Compare commits

...
Author SHA1 Message Date
jschoubben 9796444201 Merge remote-tracking branch 'origin/main' into feat/1126-nodes-compose
mesh/delivery merged: source.main moved to novox/mesh-sdk@823d52051f03
mesh/delivery-group group feat/1126-nodes-compose merged: every member is merged
mesh/check checking 979644420117 on 7187beaf042b | novox/mesh-sdk#98 in group feat/1126-nodes-compose
# Conflicts:
#	test/standing.test.ts
2026-10-26 22:03:26 +01:00
mesh-tools 7187beaf04 Merge pull request 'The provider loop's tests wait on what they mean, not on the clock (issue 1901 note 27)' (#101) from fix/standing-test-races into main 2026-10-26 20:58:44 +00:00
jschoubben 5f45175ed0 Merge remote-tracking branch 'origin/main' into feat/1126-nodes-compose 2026-10-26 21:56:47 +01:00
jschoubben 18e541c1ba The standing test allows 10 ms between the loop's clock and the test's stamp of an announcement
mesh/delivery merged: source.main moved to novox/mesh-sdk@7187beaf042b
mesh/check pass 18e541c1badc on 7187beaf042b | carried from 2cbe21937153 | a compiled program is not refused: mesh-build-node reported no programs | u…
2026-10-26 21:55:46 +01:00
jschoubben cf930388f9 The standing test's runs last 0.6 of failingAfterMs of clock as well as two attempts, so a success that does not reset a run is caught; the addition test holds a thousand stable passes, so 'first pass' is proved (review of the standing test races) 2026-10-26 21:49:03 +01:00
jschoubben 36ae0c691b The standing and retirement tests wait on what they mean, not on a slept fraction of a bound: announcements are timed from the clock at the moment they are said, short runs are counted in attempts, an addition is proved by a hold it must not wait for (the second timing race of one kind in a day; issue 1126) 2026-10-26 21:44:06 +01:00
mesh-tools 2cbe219371 Merge pull request 'A build seat's capacity follows its CPU share and its build hours; asks say their phases (issues 1890, 1806)' (#100) from fix/capacity-follows-cpu-share into main 2026-10-26 20:33:28 +00:00
jschoubben 3a095a66be The standing test keeps c failing alone after its ask is dropped, so its only recovery is its retirement: clearing every failure at once raced the loop's read of the new list under load (issue 1126; the hidden failure of try-1793042418286545971, issue 1901) 2026-10-26 21:33:20 +01:00
jschoubben 805b4d9a35 Merge main into fix/capacity-follows-cpu-share: the build modules' binary refusal (sdk #99) beside the capacity work
mesh/delivery merged: source.main moved to novox/mesh-sdk@2cbe21937153
mesh/delivery-group group fix/capacity-follows-cpu-share merged: every member is merged
mesh/check checking 805b4d9a3536 on dae8e288d2e1 | novox/mesh-sdk#100 in group fix/capacity-follows-cpu-share
2026-10-26 21:26:58 +01:00
jschoubben d00dac140b A build module takes asks only within its node's build hours
mesh/delivery-group group fix/capacity-follows-cpu-share rejected: composed together: the checker's verdict at the members' heads (63694fe9222d): unit: ..test
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: ..test: ✔ a transient empty list for four passes retires nothing; five stable passes retire, data kept (418.253369ms) | d00da…
The operator, 2026-10-26 21:18: g14 builds only from 18:00 to 06:00. A build-hours setting names a
window per node in its local time (crossing midnight allowed, absent = any hour); outside it the
capacity judged by is 0, running steps finish, and the puller wakes on one timer at the next edge.
The change is said in the log and in capacity.<node> with the window and when it opens.
2026-10-26 21:21:37 +01:00
jschoubben 03dce12378 The standing test keeps c failing alone after its ask is dropped, so its only recovery is its retirement: clearing every failure at once raced the loop's read of the new list under load (issue 1126; the hidden failure of try-1793042418286545971, issue 1901)
mesh/delivery-group group feat/1126-nodes-compose rejected: composed together: the checker's verdict at the members' heads (0d06ff386dce): ace
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-sdk#98 does not merge into the trunk: files in conflict: test/standing.test.ts | 03dce1237803 on 7187beaf042b | uni…
2026-10-26 20:33:23 +01:00
mesh-tools dae8e288d2 Merge pull request 'A change that adds a compiled binary is refused, in any repository (issue 1577)' (#99) from fix/1577-checker-refuses-binaries into main 2026-10-26 19:32:25 +00:00
jschoubben dd9d9d5b5f Compare haves with SameHave in the round-trip tests, now that a have carries a list (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking dd9d9d5b5f01 on dae8e288d2e1 | novox/mesh-sdk#98 in group feat/1126-nodes-compose
2026-10-26 20:20:12 +01:00
jschoubben 2c0ab3a592 A have carries the values a self-composing node's running composition holds sealed to its key, so its composer reuses them after a restart (review of L2, round 2; issue 1126, ADR 0547 §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed (4 tests): unit: go.go vet: # git.novox.be/novox/mesh-sdk/go/delivery [git.novox.be/novox/mesh-sdk/go/delivery.test] | 2c0ab3a59288 …
2026-10-26 20:19:54 +01:00
jschoubben 3968849b1b Review of L2: the export is a placement file at its version with its modules (S1); a node asks the vault only when a module placed on it declares an own secret, so one with none composes whether or not the vault answers (H8's failure) (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 3968849b1b36 on a7a1fbd34e1d | novox/mesh-sdk#98 in group feat/1126-nodes-compose
2026-10-26 20:08:32 +01:00
jschoubben 02f50bd9d6 A build seat's capacity follows its node's build CPU, and a running ask says its phases
Since the build CPU cap (issue 1806) shanks' mesh-build-go kept capacity 3 inside a 12-core slice:
three full checks got about 4 cores each and ran past the 30-minute step bound (issue 1890).
The puller now judges by min(configured, floor(slice quota cores / cores-per-step)), the quota
read live from its cgroup, says it on change and writes capacity.<node> to the pauses state.
A run's phases (fetch and merge, digest, tests) are logged with the ask id and written to its
running step key; Asker.Wait hands every step value to a WithStepSeen hook.
2026-10-26 19:52:54 +01:00
jschoubben fe11db9293 MergeInto also names programs a commit of the branch added and a later one removed: the merge commit would keep the blob in history (issue 1577)
mesh/delivery merged: source.main moved to novox/mesh-sdk@dae8e288d2e1
mesh/delivery-group group fix/1577-checker-refuses-binaries merged: every member is merged
mesh/check checking fe11db929397 on a7a1fbd34e1d | novox/mesh-sdk#99 in group fix/1577-checker-refuses-binaries
2026-10-26 19:47:17 +01:00
jschoubben 9713c3023a Placement carries a settings-form module's config values, which mesh-config does not hold, and the export writes them (issue 1126, ADR 0555 §1): the media modules on one node would compose without them
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 9713c3023a18 on a7a1fbd34e1d | unit 1245…
2026-10-26 19:38:37 +01:00
jschoubben 53c2def1a5 VaultSecretChanged is a variable (its seat is), and gofmt (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/nobinaries.TestTheCatalogueHoldsNoCompiledProgram: modules/mesh-config/cmd/mesh-config…
2026-10-26 19:25:19 +01:00
jschoubben f55b34febc A node composing its own hears the vault's secret-changed, names and generations only (issue 1126, ADR 0547 §2)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed (6 tests): unit: go.gofmt: not as gofmt writes them (run gofmt -w): go/buscompose/buscompose_test.go | f55b34febc2d on a7a1fbd34e1d …
2026-10-26 19:25:09 +01:00
jschoubben 944a35d200 A node's record carries the mesh's own certificate as the machine holds it, and frombus composes with the composing node's (issue 1126, ADR 0555 §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk 91413de33516, which is neither on mesh-sdk's trunk nor the g…
2026-10-26 19:23:06 +01:00
jschoubben 7d81e8326f frombus gives the composing node's core credentials as the vault's boxes sealed to it, from the same own secrets (issue 1126, ADR 0349 §16, ADR 0550 §5) 2026-10-26 19:22:58 +01:00
jschoubben 91413de335 compose names the pair credentials a node's composition reads as consumer and provider, and frombus asks them sealed before it composes (issue 1126, ADR 0550 §5)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: g14: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk f1d7c7377dc4, which is neither on mesh-sdk's trunk nor the grou…
2026-10-26 19:18:46 +01:00
jschoubben 6447a1c6d4 placement-seed writes each node's first placement file from the controller's export, checked before anything is written, once (issue 1126, ADR 0549 §4, ADR 0555 §6)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk f1d7c7377dc4, which is neither on mesh-sdk's trunk nor the g…
2026-10-26 19:17:45 +01:00
jschoubben 4d5f430224 A want the node composes itself says so (Self) and names the build alone; every other want names its composition as before, so no caller of the old form changes (issue 1126, ADR 0547 §2)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk f1d7c7377dc4, which is neither on mesh-sdk's trunk nor the grou…
2026-10-26 19:13:56 +01:00
jschoubben dfe3ba9475 frombus exports a node's placement from a composition input, Input's inverse for what a person decides, and the controller's seat declares placement-export (issue 1126, ADR 0555 §6)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking dfe3ba94750c on a7a1fbd34e1d | novox/mesh-sdk#98 in group feat/1126-nodes-compose
2026-10-26 19:11:34 +01:00
jschoubben f1d7c7377d frombus reads the composing node's own secrets sealed to it, and node principals may ask the vault's own in their own name (issue 1126, ADR 0550 §5)
mesh/delivery-group group feat/1126-nodes-compose checking: 4 of 5 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking f1d7c7377dc4 on a7a1fbd34e1d | novox/mesh-sdk#98 in group feat/1126-nodes-compose
2026-10-26 19:03:00 +01:00
jschoubben 22bc4dc8ef MergeInto names the changed files that are compiled programs, read in the workspace, so a check can refuse build output (issue 1577) 2026-10-26 18:59:35 +01:00
jschoubben b72de2b75a A node applying what it composed refuses a composition that stands in for anything its input lacks, the bus's user list alone held out (issue 1126, ADR 0547 §2) 2026-10-26 18:59:05 +01:00
jschoubben 6df0dfe208 Node principals read what they compose their own modules from; the composer copy is held per user to the controller's grants plus exactly these (issue 1126, ADR 0547 §2, ADR 0549 §3)
mesh/delivery-group group feat/1126-nodes-compose rejected: composed together: the checker's verdict at the members' heads (922da6d6ae76): forge
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 6df0dfe208c6 on a7a1fbd34e1d | novox/mesh-sdk#98 in group feat/1126-nodes-compose
2026-10-26 18:44:06 +01:00
jschoubben 9532b78724 A node's own want names no composition from the second milestone: the node composes its own (issue 1126, ADR 0547 §2) 2026-10-26 18:41:33 +01:00
jschoubben e49309d61c A want names the build alone from the second milestone, and the yes a person gave; a have waits for that yes with its judgement (issue 1126, ADR 0547 §2, §3) 2026-10-26 18:39:46 +01:00
jschoubben 9789c8379a Node principals read every node's own facts, where its record rides; the composer copy is checked to grant all the controller's grants and only that and the issue 1790 rows more (issue 1126, ADR 0555 §3)
mesh/delivery-group group feat/1126-nodes-compose checking: 1 of 3 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 9789c8379a74 on a7a1fbd34e1d | novox/mesh-sdk#98 in group feat/1126-nodes-compose
2026-10-26 18:33:36 +01:00
jschoubben 2b0689f654 A node's record rides in its own have key, <node>.node: no bucket, writer or grant of its own, and its key's change is what readers hear (issue 1126, ADR 0555 §3) 2026-10-26 18:29:45 +01:00
jschoubben 5b1e990eb4 The tunnel found at adoption and the names of the hub's peers are kept in the placement; whether the hub took it over and who enrolled under a peer's key are read from placement and records, as the controller reads them (issue 1126, ADR 0555) 2026-10-26 18:26:11 +01:00
jschoubben a431746c0e What a person decides of a node lives in its placement file and what the machine observes in its record; two nodes on one address or two hubs are refused (issue 1126, K2) 2026-10-26 18:17:59 +01:00
jschoubben 5db53e738f Name the catalogue repository whose placement files are read (issue 1126, ADR 0549 §1) 2026-10-26 18:10:25 +01:00
jschoubben 9f7f50de8d A merged message of the catalogue carries its placement files, so the indexer reads placement at the merge without asking the forge again (issue 1126, ADR 0549 §2) 2026-10-26 18:08:59 +01:00
jschoubben bdd4c7b875 The judgement of a move is a pure function in mesh-sdk, so a node agent judges what it composed against what it runs (issue 1126, ADR 0547 §3, ADR 0377 §3) 2026-10-26 18:07:20 +01:00
jschoubben e982072443 frombus fills the composition's input from placement, node records, the indexer's manifests and mesh-config's slice, so one composer serves the node agent, the check and the grants part (issue 1126, ADR 0547 §2, ADR 0549 §3) 2026-10-26 18:06:04 +01:00
jschoubben 146cd858a3 Placement and the node record have one layout in mesh-sdk, so the indexer, the node agents and the composition check read them alike (issue 1126, ADR 0549 §1-§2, ADR 0547 §2) 2026-10-26 18:02:13 +01:00
mesh-tools a7a1fbd34e Merge pull request 'A module waits for a state created after it starts (issue 1790)' (#97) from fix/1790-state-appears into main 2026-10-26 14:47:31 +00:00
mesh-tools 1888d1d230 Merge pull request 'A git or ssh-keygen failure keeps its own words (forgekey.Said)' (#96) from fix/fetch-reads-all into main 2026-10-26 14:47:25 +00:00
mesh-tools 960d6c34d2 Merge pull request 'The composer gives build modules the node's share: idle, capped builds (issue 1806, ADR 0553/0554, change B)' (#95) from fix/build-cpu-yields-compose into main 2026-10-26 14:31:13 +00:00
jschoubben ff81bdf8cc Keep git's own words in a fetch's reason: a bare "error:" line no longer stands for the forge's refusal
mesh/delivery merged: source.main moved to novox/mesh-sdk@1888d1d230b3
mesh/delivery-group group fix/fetch-reads-all merged: every member is merged
mesh/check pass ff81bdf8cc8f on 960d6c34d25e | unit 5815 run, 345 kept, 18 skipped | integration 5603 run, 640 kept, 29 skipped | over bound: unit, in…
The mirror of jschoubben/car-hunter on ace said only "git -C: error:": git writes a bare
"error:" before the forge's message, and the reason kept the first non-empty line. forgekey.Said
keeps every line that says something, leaves bare prefixes out, joins them and bounds them at
600 bytes; every git and ssh-keygen reason of buildstep and forgekey goes through it, and the
mirror's commands are named by their verb, not by -C.
2026-10-26 15:26:29 +01:00
jschoubben 65ee43f075 Merge remote-tracking branch 'origin/main' into fix/1790-state-appears
mesh/delivery merged: source.main moved to novox/mesh-sdk@a7a1fbd34e1d
mesh/delivery-group group fix/1790-state-appears merged: every member is merged
mesh/check being carried to 65ee43f07583 on a7a1fbd34e1d | from 1888d1d230b3, mesh-delivery feef96ed160e
2026-10-26 15:24:34 +01:00
jschoubben c3e8fc4ba6 process: an awaited bucket says loudly once when the bus refuses hearing its creation, and hears it again and binds at each reconnect (issue 1790, review) 2026-10-26 15:24:33 +01:00
mesh-tools 75be8acbad Merge pull request 'Build seats yield: the decoder, buildcpu and NodeLimits (issue 1806, ADR 0553/0554, change A)' (#94) from fix/build-cpu-yields into main 2026-10-26 14:22:47 +00:00
mesh-tools d103ed88ab Merge pull request 'forgekey: a command starts in a directory that exists (issue 734)' (#93) from fix/daemon-dir-kept into main 2026-10-26 14:15:57 +00:00
jschoubben 3ac644baef process: gofmt the integration fixture (issue 1790) 2026-10-26 15:13:25 +01:00
jschoubben fd7d93333e Keep a failed toolchain probe a loud refusal; only the command directory is fixed (issue 734, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-sdk@d103ed88ab17
mesh/delivery-group group fix/daemon-dir-kept merged: every member is merged
mesh/check checking fd7d93333e6b on 21252194d530 | novox/mesh-sdk#93 in group fix/daemon-dir-kept
Declining an ask whose probe failed would turn a truly broken toolchain into a
wait of up to the decline bound that then fails with an empty "needs a node
with" reason. With the node-engine no longer deleting a running build and
commands started in a directory that exists, the getwd cause is gone, so the
guard's refusal stays as it was. The decline refactor goes with it.
2026-10-26 15:11:04 +01:00
jschoubben d515c71e1c process: a bucket missing at a part's start is bound when the bus says it was created, keeping what was written meanwhile; buscompose grants each principal the creation of the buckets it may bind (issue 1790)
Since the grants part creates a manifest's buckets after a delivery (ADR 0547), a module restarted by that
delivery can open a new state before it exists; OptionalBucket answered nil and the part never looked again
(mesh-delivery's checker and builder, state hearing, 2026-10-26). AwaitBucket binds on the server's
STREAM.CREATED advisory, or at the next use, and writes the latest value kept per key. The copied composer's
agreement test leaves that one row out: the controller's list is no longer loaded since the take-over.
2026-10-26 15:11:00 +01:00
jschoubben 36061b5c56 Start commands in a directory that exists, and decline an ask whose toolchain could not be read (issue 734)
A build module runs in the directory its build was unpacked in. When the
node-engine replaced that build, every command the module started inherited a
deleted working directory, and the toolchain probe's failure became a final
refusal of the check that asked it (three checks on ace on 2026-10-26).

forgekey: ExecRunnerClean starts its command in the HOME it is given, or the
temporary directory; the other runners keep the process's directory while it
exists and use the temporary directory once it is gone.

buildstep: a guard whose probe failed answers a Refusal with Again set; the
puller declines such an ask (delivery.StepDeclined) so its asker puts it again
for another node or a later pull, instead of ending it refused. A version
mismatch stays a refusal.
2026-10-26 15:01:57 +01:00
jschoubben 4ef49d0391 Merge fix/build-cpu-yields into the composer half
mesh/delivery merged: source.main moved to novox/mesh-sdk@960d6c34d25e
mesh/check checking 4ef49d039195 on 960d6c34d25e
2026-10-26 14:46:11 +01:00
jschoubben 17b7cc480d Merge main into fix/build-cpu-yields
mesh/delivery merged: source.main moved to novox/mesh-sdk@75be8acbad4d
mesh/delivery-group group fix/build-cpu-yields merged: every member is merged
mesh/check checking 17b7cc480d6d on d103ed88ab17 | novox/mesh-sdk#94 in group fix/build-cpu-yields
2026-10-26 14:46:01 +01:00
jschoubben 6ac8c5c229 The composer gives a build seat's holder its node's one share, the node-engine's build-cpu-share (ADR 0553 §3, §4)
The second half of ADR 0553, merged only once the node-engine that decodes
yields runs on every node (§5). The share is the node-engine's own config as
the node's layers give it; each step divides it by its own module's capacity.
A share out of range bounds builds at the default and never refuses the
node's composition. A change of the share recomposes the build runtimes, which
restart once.
2026-10-26 14:42:20 +01:00
mesh-tools 21252194d5 Merge pull request 'A check ask can name needs and packages, and a node that lacks a need declines it (issue 890, ADR 0552)' (#91) from feat/890-placement into main 2026-10-26 13:40:35 +00:00
mesh-tools 545a0c9cb4 Merge pull request 'An asker of the build seats hears when its ask is given up, and the seat's users may read their own dead letters (issue 734)' (#92) from fix/734-lost-ask into main 2026-10-26 13:40:30 +00:00
jschoubben 527d87613e Without the composer's share a step has no thread limit; with it, the share over the module's own capacity (ADR 0553 round 2)
Review round 2: change A must not slow builds before any slice exists, so a
module given no share sets no GOMAXPROCS. The slice's quota bounds the total;
threads only size one step, from the node's share over the module's own
capacity (shanks: 24 x 50% / 3 = 4). The share is the node-engine's own config.
2026-10-26 14:39:39 +01:00
jschoubben 36c92861d2 Merge main into fix/build-cpu-yields 2026-10-26 14:22:26 +01:00
jschoubben 852ba81c5a Build seats yield to one shared slice per node; the composer half waits for the engines (ADR 0553, issue 1806)
Review round 1: the decoder type ships before any composer can emit it, so a
routine SDK bump never sends yields to an engine that refuses them; the
composer half moves to a later change. Every yielding process of a node goes
into one top-level slice, meshyield.slice, so the three build modules share one
quota instead of taking one each; the share is one node value, node-tools'
build-cpu-share, and each step's threads divide it by every build slot of the
node (words from the composer, a conservative fallback without them).
2026-10-26 14:20:05 +01:00
jschoubben 3203358c4c buildrun: an asker keeps whether it hears DEAD_LETTERS, and its health tool answers from it (issue 734, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-sdk@545a0c9cb43b
mesh/delivery-group group fix/734-lost-ask merged: every member is merged
mesh/check being carried to 3203358c4cb5 on 2aae6f5580f2 | from mesh-delivery c8b26735c8cf | novox/mesh-sdk#92 in group fix/734-lost-ask
A deaf asker must be loud: SayHearing keeps each Hearing in the asker's state hearing, and HearingHealth answers the
module's health tool from it, unhealthy with why while DEAD_LETTERS is not heard.
2026-10-26 14:18:41 +01:00
jschoubben 9b01c94a9e buildrun: the asker hears DEAD_LETTERS again on its next reconnect or ask, in the background, and says whether it hears; buscompose grants only what a pulled ordered consumer needs (issue 734, review round 1)
An asker started before its grant was composed was refused once and stayed deaf until it started again, silently.
DeadLetterHearing tries in the background and again on an event (Again: a reconnect, an ask waited for while deaf), and
tells each change of Hearing so the module's health tool can say it. DeadLetterReads drops the flow control a pull
consumer does not use and deletes one consumer by name.
2026-10-26 14:16:58 +01:00
jschoubben 33395f68b3 Merge main (issue 1890's step records) into feat/890-placement: a decline begins its step with what it is
mesh/delivery merged: source.main moved to novox/mesh-sdk@21252194d530
mesh/delivery-group group feat/890-placement merged: every member is merged
mesh/check checking 33395f68b309 on 2aae6f5580f2 | novox/mesh-sdk#91 in group feat/890-placement
2026-10-26 14:05:30 +01:00
jschoubben 45564e69a6 A placed check ask names the packages it tests, so the first ask's other packages are not run or counted again (issue 890, ADR 0552 §4; review round 1) 2026-10-26 14:01:05 +01:00
jschoubben d96b9fd660 buildrun: an ask the build seat gave up on ends its asker's wait as failed with why, for every asker of the build seats (issue 734)
The checker's try and the builder's lanes waited for ever on an ask given up during a bus reload: no build node writes a
step for it. LostAskWaiter hands the asker a failed step on the node bus when DEAD_LETTERS keeps the ask's copy
(HearDeadLetters, granted by DeadLetterReads), and under NotTakenBound when no node took the ask within the bound.
2026-10-26 14:00:46 +01:00
mesh-tools 2aae6f5580 Merge pull request 'buildstep: a build module ends the steps a previous process left running, and a step says what it is (issue 1890)' (#90) from fix/1890-left-running-steps into main 2026-10-26 12:57:31 +00:00
jschoubben 6b84973142 Build steps: an ask key past the step bound is ended, at start and on the module's round (issue 1890 review 2)
mesh/delivery merged: source.main moved to novox/mesh-sdk@2aae6f5580f2
mesh/delivery-group group fix/1890-left-running-steps merged: every member is merged
mesh/check pass 6b8497314297 on 2aae6f5580f2 | carried from b4cd95e7c4a4, mesh-catalog a4f5df797b68 | unit 3132 run, 3022 kept, 7 skipped | integratio…
2026-10-26 13:51:54 +01:00
jschoubben e7f0dc612e buscompose: a user of a seat pulled by kind reads its seat's dead letters, so an asker hears an ask given up (issue 734)
A check ask given up during a bus reload left its try running for ever: the asker had no grant to learn that
DEAD_LETTERS holds it. DeadLetterReads grants an ordered consumer on DEAD_LETTERS filtered to the seat stream's copies,
and nothing else of the stream.
2026-10-26 13:51:35 +01:00
jschoubben 76deab6c67 gofmt the cap's test (issue 1806) 2026-10-26 13:49:09 +01:00
jschoubben e66c6fea62 Build steps: test the ask key with a real ask id (issue 1890) 2026-10-26 13:43:05 +01:00
jschoubben 6b1c7a5613 Build steps: a restart leaves pulled asks to the stream, ends steps by compare-and-set, and step_end refuses a step this process started (issue 1890 review 1) 2026-10-26 13:42:50 +01:00
jschoubben 04902ff24a Build seats yield: idle priority and a per-node processor cap on their runtimes, threads per step within it (issue 1806)
The operator's desktop slows while it builds: the Go build runtime took 1650% of
shanks' 24 processors with nothing holding it back. A build seat's holder now
runs its runtime at idle priority under its node's cpu-cap share (declaration
Yields, composed from Seat.Yields and the module's cpu-cap setting), and
buildcpu/buildstep give each step its part of that cap in threads.
2026-10-26 13:37:00 +01:00
jschoubben bb71310337 Let a check ask name what its node needs, decline it where that is lacking, and leave unmet packages to such a node (issue 890)
A check ask's packages may need something only some build nodes have (mesh-build-node's integration tests read the
Node build module's installed toolchain). An ask now names its Needs; a holder lacking one ends the ask's step refused
as declined and drops the message for its asker to put again, skipping that worker until its next pull wait; a level's
packages whose declared needs the node lacks are left untested and named unmet, for the asker to place.
2026-10-26 13:35:12 +01:00
jschoubben 18e0a01c54 Build steps: EndStaleAtStart waits for the runtime and ends the left-running steps (issue 1890) 2026-10-26 13:25:58 +01:00
jschoubben ddafe1b1aa Build steps: a step records what it is, a restart ends the steps it left running, a tool ends one (issue 1890) 2026-10-26 13:25:13 +01:00
mesh-tools b4cd95e7c4 Merge pull request 'Take-over of the bus's user list by the grants part: sealed list layout, followed-list have, handed hashes, named readers (issue 1945)' (#89) from feat/1945-user-list-switch into main 2026-10-26 12:21:39 +00:00
mesh-tools 89c5838176 Merge pull request 'Delivery layouts for ADR 0548: merge turns, carried verdicts and the siblings a verdict read (issue 568)' (#88) from feat/548-one-check into main 2026-10-26 12:01:17 +00:00
jschoubben c2a410bb63 The have's user list says why the bus node does not watch the grants part's key (review of the switch, issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-sdk@b4cd95e7c4a4
mesh/delivery-group group feat/1945-user-list-switch merged: every member is merged
mesh/check checking c2a410bb6376 on 89c5838176b2 | novox/mesh-sdk#89 in group feat/1945-user-list-switch
2026-10-26 12:25:27 +01:00
jschoubben af66adc180 Bring in the adoption handing (fix/1945-adopt-hashes 17478994): the have's user list carries Handed beside Follows and Key; a state that names readers is not Readable to a composer that knows no readers (review of the switch, issue 1945) 2026-10-26 12:23:07 +01:00
jschoubben 9c61918ac3 delivery: a turn never holds the check of heads that wait in it, so two approved merges in one turn never hold each other's carry (ADR 0548 §3; issue 568, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-sdk@89c5838176b2
mesh/delivery-group group feat/548-one-check merged: every member is merged
mesh/check checking 9c61918ac36f on 85fa8511c4b3 | novox/mesh-sdk#88 in group feat/548-one-check
2026-10-26 12:12:53 +01:00
jschoubben 83ee7769d8 Make the grants part's state in the switch's test as the part makes a declared state, by its stream (issue 1945) 2026-10-26 12:00:29 +01:00
jschoubben 175a5567b8 The bus node follows the grants part's sealed user list, and only the grants part reads the vault's hashes (ADR 0547 clauses 7, 9; issue 1945)
The grants part writes its list as an object of its own bucket, sealed to the bus node's key, and names it in
user-list.sealed; the node-engine of the node holding mesh-broker is granted that one key and an object by name, the
grants part its bucket (the controller's copy carries the same grants, byte for byte, golden regenerated there).
A state may name its readers (announce, NamedReaders): mesh-vault.bus-accounts is read by mesh-grants alone.
2026-10-26 11:53:11 +01:00
jschoubben f5c233a928 delivery: a verdict names the siblings it read and the commits it was carried from, and a composition its fingerprint set, so a trunk or sibling move is carried, not judged again (ADR 0548 §5, §6; issue 568 S2) 2026-10-26 11:25:07 +01:00
jschoubben 0abe480dba delivery: the gate's merge turns, so the checker holds new checks off a repository an approved merge waits on (ADR 0548 §3, §4; issue 568 S1) 2026-10-26 11:24:06 +01:00
87 changed files with 7070 additions and 146 deletions
+29
View File
@@ -46,6 +46,10 @@ func Key(module string) (string, error) {
type State struct { type State struct {
Name string `json:"name"` Name string `json:"name"`
Readable bool `json:"readable,omitempty"` Readable bool `json:"readable,omitempty"`
// Readers, when named, are the only modules that may read the state (novox/hq ADR 0547 clause 9, issue 1945): a
// state readable by some and not all, such as the vault's bus account hashes. Only ReadableBy grants them; Readable,
// which a composer that knows no readers asks, stays false for it, so such a composer grants nobody.
Readers []string `json:"readers,omitempty"`
} }
// States is an announcement: the module, the machine whose instance said it, and every state the module keeps, sorted // States is an announcement: the module, the machine whose instance said it, and every state the module keeps, sorted
@@ -67,6 +71,26 @@ func (s States) Keeps(state string) bool {
return false return false
} }
// ReadableBy says whether the module announced the state readable by reader: readable, and reader among the readers
// when it names any.
func (s States) ReadableBy(state, reader string) bool {
for _, k := range s.States {
if k.Name != state {
continue
}
if len(k.Readers) > 0 {
for _, r := range k.Readers {
if r == reader {
return true
}
}
return false
}
return k.Readable
}
return false
}
// Readable says whether the module announced the state readable by others. // Readable says whether the module announced the state readable by others.
func (s States) Readable(state string) bool { func (s States) Readable(state string) bool {
for _, k := range s.States { for _, k := range s.States {
@@ -93,6 +117,11 @@ func (s *States) check() error {
return fmt.Errorf("announce: %s announces the state %q twice", s.Module, k.Name) return fmt.Errorf("announce: %s announces the state %q twice", s.Module, k.Name)
} }
seen[k.Name] = true seen[k.Name] = true
for _, r := range k.Readers {
if !name.MatchString(r) {
return fmt.Errorf("announce: %s names %q as a reader of %s, and a reader is a module: lower-case letters, digits and hyphens", s.Module, r, k.Name)
}
}
} }
return nil return nil
} }
+9 -2
View File
@@ -9,7 +9,8 @@ import (
// own event subject, in a versioned value the controller keeps by module. // own event subject, in a versioned value the controller keeps by module.
func TestAnAnnouncementRoundTripsSortedAndSaysWhatIsReadable(t *testing.T) { func TestAnAnnouncementRoundTripsSortedAndSaysWhatIsReadable(t *testing.T) {
data, err := Encode(States{Module: "mesh-controller", Node: "novox", States: []State{ data, err := Encode(States{Module: "mesh-controller", Node: "novox", States: []State{
{Name: "plans-building", Readable: true}, {Name: "assignments", Readable: true}, {Name: "own"}}}) {Name: "plans-building", Readable: true}, {Name: "assignments", Readable: true}, {Name: "own"},
{Name: "hashes", Readers: []string{"mesh-grants"}}}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -24,12 +25,17 @@ func TestAnAnnouncementRoundTripsSortedAndSaysWhatIsReadable(t *testing.T) {
for _, s := range back.States { for _, s := range back.States {
names = append(names, s.Name) names = append(names, s.Name)
} }
if strings.Join(names, ",") != "assignments,own,plans-building" { if strings.Join(names, ",") != "assignments,hashes,own,plans-building" {
t.Fatalf("the states are not sorted: %v", names) t.Fatalf("the states are not sorted: %v", names)
} }
if !back.Readable("plans-building") || back.Readable("own") || back.Readable("absent") { if !back.Readable("plans-building") || back.Readable("own") || back.Readable("absent") {
t.Fatal("Readable does not say what was announced readable") t.Fatal("Readable does not say what was announced readable")
} }
// Named readers (ADR 0547 clause 9): readable by them alone, and not readable to a composer that knows no readers.
if back.Readable("hashes") || !back.ReadableBy("hashes", "mesh-grants") || back.ReadableBy("hashes", "spy") ||
!back.ReadableBy("plans-building", "spy") || back.ReadableBy("own", "spy") {
t.Fatal("ReadableBy does not say who was named")
}
if !back.Keeps("own") || back.Keeps("absent") { if !back.Keeps("own") || back.Keeps("absent") {
t.Fatal("Keeps does not say what was announced") t.Fatal("Keeps does not say what was announced")
} }
@@ -57,6 +63,7 @@ func TestAnAnnouncementRefusesWhatNoBucketCouldBe(t *testing.T) {
{States{Module: "a", States: []State{{Name: "x_y"}}}, `"x_y"`}, {States{Module: "a", States: []State{{Name: "x_y"}}}, `"x_y"`},
{States{Module: "a", States: []State{{Name: ""}}}, `""`}, {States{Module: "a", States: []State{{Name: ""}}}, `""`},
{States{Module: "a", States: []State{{Name: "x"}, {Name: "x", Readable: true}}}, "twice"}, {States{Module: "a", States: []State{{Name: "x"}, {Name: "x", Readable: true}}}, "twice"},
{States{Module: "a", States: []State{{Name: "x", Readers: []string{"b.c"}}}}, `"b.c"`},
} { } {
if _, err := Encode(c.s); err == nil || !strings.Contains(err.Error(), c.says) { if _, err := Encode(c.s); err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("%+v: %v, wanted a refusal saying %s", c.s, err, c.says) t.Errorf("%+v: %v, wanted a refusal saying %s", c.s, err, c.says)
+215
View File
@@ -0,0 +1,215 @@
// Package buildcpu is how much of a build node's processors its builds may take, and how they yield to everything else
// there (novox/hq ADR 0553, issue 1806; the operator, 2026-10-26: "limit CPU usage on our build nodes so the total cpu
// usage never impacts the normal workings of our nodes ... We have 3 build agents so they should share the cpu power").
//
// Three parts, each with one owner:
//
// - **Priority.** A build seat's holder (catalogue.Seat.Yields) runs at the lowest priority there is: the composer
// gives its runtime process declaration.Yields, and the node-engine places every yielding process of the node in
// one slice (declaration.YieldSlice) whose CPU weight is idle, with Nice=19 for every process it starts.
// Interactive and service work always comes first; builds take the cycles nothing else wants.
// - **One cap per node, shared.** The node's share of its processors is one value, the node-engine's own config
// `build-cpu-share` (ShareSetting), set per node through the node's settings layer. The node-engine writes it as the
// slice's CPUQuota: one bound the kernel holds for all the node's build modules together, never one per module.
// - **Threads.** The quota bounds the total; threads only size each step. A step gets the share divided by its own
// module's capacity (Threads), from the share the composer hands as a word (ShareWord): GOMAXPROCS sets every go
// command's -p, the compiler's concurrency, a test binary's processors and -parallel. A module given no word sets
// no thread limit at all: without the slice there is nothing to size a step to, and a limit then would only slow
// builds.
package buildcpu
import (
"os"
"runtime"
"strconv"
"strings"
)
// ShareSetting is the node-engine's config value of the node's processor share for its builds, in percent of the
// machine's processors, 1 to 100: one value per node, set through the node's settings layer (novox/hq ADR 0553 §3).
const ShareSetting = "build-cpu-share"
// DefaultPercent is the share where nothing sets one: half the machine, so a desktop keeps half its processors for
// itself whatever builds.
const DefaultPercent = 50
// ShareWord is the word the composer gives every yielding module (novox/hq ADR 0553 §4): the node's share, in percent.
const ShareWord = "MESH_BUILD_CPU_SHARE"
// Quota is a CPUQuota for a share of a machine with cores processors: systemd counts a quota in percent of one
// processor, so half of 24 is 1200%. At least 100%, one processor, so a share on a small machine never stops a step.
func Quota(percent, cores int) int {
q := percent * cores
if q < 100 {
q = 100
}
return q
}
// Threads is how many threads one step may run at once: its part of the share when its own module runs slots steps at
// once, at least one. The slice's quota bounds the steps of all the node's build modules together; this only sizes one.
func Threads(percent, cores, slots int) int {
if slots < 1 {
slots = 1
}
if percent < 1 || percent > 100 {
percent = DefaultPercent
}
t := cores * percent / 100 / slots
if t < 1 {
t = 1
}
return t
}
// Limits is what one step is held to on this node, as a status tool says it.
type Limits struct {
// Percent is the node's share in percent of its processors, and From where it was read: ShareWord, or "none" when
// the composer gave no share and a step runs without a thread limit.
Percent int `json:"share-percent,omitempty"`
From string `json:"share-from"`
// Cores is the machine's processor count.
Cores int `json:"cores"`
// Slots is this module's own capacity on the node: steps at once.
Slots int `json:"slots"`
// Threads is what each step is given (Threads); 0 is no limit.
Threads int `json:"threads-per-step"`
}
// FromWords is the limits of a step from the share the composer gave this process (ShareWord) and the module's own
// capacity. Without a usable share there is no limit: Threads is 0.
func FromWords(getenv func(string) string, own int) Limits {
if own < 1 {
own = 1
}
l := Limits{From: "none", Cores: runtime.NumCPU(), Slots: own}
if n, err := strconv.Atoi(strings.TrimSpace(getenv(ShareWord))); err == nil && n >= 1 && n <= 100 {
l.Percent, l.From = n, ShareWord
l.Threads = Threads(n, l.Cores, own)
}
return l
}
// Env is the environment a step's commands are given so they start no more threads than they may: GOMAXPROCS for every
// Go program (the compiler, a test binary, esbuild, buildah), UV_THREADPOOL_SIZE for Node's worker pool. Returned as
// KEY=value words to append to a command's environment; nothing for 0 threads, no limit.
func Env(threads int) []string {
if threads < 1 {
return nil
}
t := strconv.Itoa(threads)
return []string{"GOMAXPROCS=" + t, "UV_THREADPOOL_SIZE=" + t}
}
// InForce is what the kernel holds this process to, read rather than assumed (evidence over assertion): its nice
// level, its cgroup (the unit) and the cgroup's parent (the slice the node-engine placed it in), each one's CPU weight,
// idle mark and quota. A field the machine does not answer is left empty and Unread says which.
//
// The IO weight is read but binds only where the disk's scheduler honours it (bfq, or the io.cost controller); under
// the scheduler `none`, the usual one for NVMe, it does nothing, and the CPU bound is what holds.
type InForce struct {
Nice int `json:"nice"`
Cgroup string `json:"cgroup,omitempty"`
Unit Group `json:"unit"`
Slice Group `json:"slice"`
// Quota is the tighter of the two groups' cpu.max as a percent of one processor, 0 with none.
Quota int `json:"quota-percent,omitempty"`
Unread []string `json:"unread,omitempty"`
}
// Group is one cgroup's controls.
type Group struct {
Path string `json:"path,omitempty"`
CPUIdle string `json:"cpu-idle,omitempty"`
CPUWeight string `json:"cpu-weight,omitempty"`
// CPUMax is the cgroup's cpu.max: "<quota µs> <period µs>", or "max" with no quota.
CPUMax string `json:"cpu-max,omitempty"`
IOWeight string `json:"io-weight,omitempty"`
}
// cgroupRoot is where the unified hierarchy is mounted. A variable so a test can point it elsewhere.
var cgroupRoot = "/sys/fs/cgroup"
// procSelfCgroup is this process's cgroup file. A variable so a test can point it elsewhere.
var procSelfCgroup = "/proc/self/cgroup"
// procSelfStat is this process's stat file. A variable so a test can point it elsewhere.
var procSelfStat = "/proc/self/stat"
// ReadInForce reads what the kernel holds this process to.
func ReadInForce() InForce {
var f InForce
f.Nice = nice()
raw, err := os.ReadFile(procSelfCgroup)
if err != nil {
f.Unread = append(f.Unread, "cgroup: "+err.Error())
return f
}
for _, line := range strings.Split(strings.TrimSpace(string(raw)), "\n") {
if strings.HasPrefix(line, "0::") {
f.Cgroup = strings.TrimPrefix(line, "0::")
}
}
if f.Cgroup == "" {
f.Unread = append(f.Unread, "cgroup: not on the unified hierarchy")
return f
}
read := func(path string) Group {
g := Group{Path: path}
value := func(name string) string {
b, err := os.ReadFile(cgroupRoot + path + "/" + name)
if err != nil {
f.Unread = append(f.Unread, path+"/"+name+": "+err.Error())
return ""
}
return strings.TrimSpace(string(b))
}
g.CPUIdle, g.CPUWeight, g.CPUMax, g.IOWeight = value("cpu.idle"), value("cpu.weight"), value("cpu.max"), value("io.weight")
return g
}
f.Unit = read(f.Cgroup)
if i := strings.LastIndex(f.Cgroup, "/"); i > 0 {
f.Slice = read(f.Cgroup[:i])
}
for _, q := range []int{quotaOf(f.Unit.CPUMax), quotaOf(f.Slice.CPUMax)} {
if q > 0 && (f.Quota == 0 || q < f.Quota) {
f.Quota = q
}
}
return f
}
// quotaOf is a cpu.max value as a percent of one processor: "1200000 100000" is 1200; "max ..." or nothing is 0.
func quotaOf(cpuMax string) int {
fields := strings.Fields(cpuMax)
if len(fields) != 2 || fields[0] == "max" {
return 0
}
q, err1 := strconv.Atoi(fields[0])
p, err2 := strconv.Atoi(fields[1])
if err1 != nil || err2 != nil || p <= 0 {
return 0
}
return q * 100 / p
}
// nice is this process's nice level from its stat file (the 19th field, counted after the command's closing
// parenthesis, which may itself hold spaces); 0 when it cannot be read.
func nice() int {
raw, err := os.ReadFile(procSelfStat)
if err != nil {
return 0
}
s := string(raw)
i := strings.LastIndex(s, ")")
if i < 0 {
return 0
}
fields := strings.Fields(s[i+1:])
// After the command: state is field 3, so nice (field 19) is index 16.
if len(fields) < 17 {
return 0
}
n, _ := strconv.Atoi(fields[16])
return n
}
+90
View File
@@ -0,0 +1,90 @@
package buildcpu
import (
"os"
"path/filepath"
"reflect"
"runtime"
"testing"
)
// The node's share as a quota, the threads one step is given for its own module's capacity, and the word the composer
// hands, in one table (ADR 0553, issue 1806).
func TestShareQuotaAndThreads(t *testing.T) {
for _, c := range []struct {
name string
percent, cores, slots int
quota, threads int
}{
{"shanks: half of 24, three slots", 50, 24, 3, 1200, 4},
{"ace: three quarters of 24, two slots", 75, 24, 2, 1800, 9},
{"g14: half of 16, one slot", 50, 16, 1, 800, 8},
{"one processor at least", 10, 2, 4, 100, 1},
{"no slots counts as one", 100, 4, 0, 400, 4},
{"a share out of range is the default", 0, 8, 1, 0, 4},
} {
if c.quota != 0 {
if q := Quota(c.percent, c.cores); q != c.quota {
t.Errorf("%s: Quota = %d, want %d", c.name, q, c.quota)
}
}
if th := Threads(c.percent, c.cores, c.slots); th != c.threads {
t.Errorf("%s: Threads = %d, want %d", c.name, th, c.threads)
}
}
l := FromWords(func(k string) string { return map[string]string{ShareWord: "75"}[k] }, 2)
if l.Percent != 75 || l.From != ShareWord || l.Slots != 2 || l.Threads != Threads(75, runtime.NumCPU(), 2) {
t.Errorf("from the word: %+v", l)
}
// Told nothing usable, a step has no thread limit: nothing in its environment.
for _, word := range []string{"", "250", "lots"} {
l := FromWords(func(string) string { return word }, 3)
if l.Threads != 0 || l.From != "none" || Env(l.Threads) != nil {
t.Errorf("share %q: %+v, env %v", word, l, Env(l.Threads))
}
}
if got := Env(3); !reflect.DeepEqual(got, []string{"GOMAXPROCS=3", "UV_THREADPOOL_SIZE=3"}) {
t.Errorf("Env = %q", got)
}
}
// What the kernel holds a process to is read from its unit's and its slice's cgroup and its stat file, not assumed.
func TestReadInForce(t *testing.T) {
dir := t.TempDir()
cg := filepath.Join(dir, "cg")
slice := filepath.Join(cg, "meshyield.slice")
unit := filepath.Join(slice, "mesh-build-go-runtime.service")
if err := os.MkdirAll(unit, 0o755); err != nil {
t.Fatal(err)
}
for at, files := range map[string]map[string]string{
slice: {"cpu.idle": "1\n", "cpu.weight": "0\n", "cpu.max": "1200000 100000\n", "io.weight": "default 1\n"},
unit: {"cpu.idle": "0\n", "cpu.weight": "100\n", "cpu.max": "max 100000\n", "io.weight": "default 100\n"},
} {
for name, v := range files {
if err := os.WriteFile(filepath.Join(at, name), []byte(v), 0o644); err != nil {
t.Fatal(err)
}
}
}
self := filepath.Join(dir, "cgroup")
stat := filepath.Join(dir, "stat")
_ = os.WriteFile(self, []byte("0::/meshyield.slice/mesh-build-go-runtime.service\n"), 0o644)
// A command with a space and a parenthesis in it, then nice 19 in field 19.
_ = os.WriteFile(stat, []byte("42 (mesh (build) go) S 1 42 42 0 -1 4194560 1 0 0 0 0 0 0 0 39 19 8 0 1 2 3\n"), 0o644)
cgroupRoot, procSelfCgroup, procSelfStat = cg, self, stat
t.Cleanup(func() {
cgroupRoot, procSelfCgroup, procSelfStat = "/sys/fs/cgroup", "/proc/self/cgroup", "/proc/self/stat"
})
got := ReadInForce()
want := InForce{Nice: 19, Cgroup: "/meshyield.slice/mesh-build-go-runtime.service",
Unit: Group{Path: "/meshyield.slice/mesh-build-go-runtime.service", CPUIdle: "0", CPUWeight: "100",
CPUMax: "max 100000", IOWeight: "default 100"},
Slice: Group{Path: "/meshyield.slice", CPUIdle: "1", CPUWeight: "0", CPUMax: "1200000 100000",
IOWeight: "default 1"},
Quota: 1200}
if !reflect.DeepEqual(got, want) {
t.Fatalf("ReadInForce = %+v\nwant %+v", got, want)
}
}
+37
View File
@@ -34,6 +34,9 @@ type AskSteps struct {
// the module's sweep. // the module's sweep.
Release func(workspace string) error Release func(workspace string) error
Logf func(format string, args ...any) Logf func(format string, args ...any)
// Has says whether this node has a need a package's tests declare (delivery.AskNeeds, issue 890); a package needing
// what it lacks is not tested here but named in the level's Unmet, for the asker to place. Nil has nothing.
Has func(need string) bool
} }
// RunAsk runs ask's job with steps and answers a delivery.BuildAnswer or a delivery.CheckAnswer. // RunAsk runs ask's job with steps and answers a delivery.BuildAnswer or a delivery.CheckAnswer.
@@ -67,6 +70,11 @@ func (s AskSteps) fetch(ctx context.Context, args map[string]any) (json.RawMessa
if s.Fetch == nil { if s.Fetch == nil {
return nil, fetched{}, Refuse("this build module fetches nothing") return nil, fetched{}, Refuse("this build module fetches nothing")
} }
if _, merges := args["base"]; merges {
Phase(ctx, "fetch and merge")
} else {
Phase(ctx, "fetch")
}
out, err := s.Fetch(ctx, args) out, err := s.Fetch(ctx, args)
if err != nil { if err != nil {
return nil, fetched{}, err return nil, fetched{}, err
@@ -123,6 +131,7 @@ func (s AskSteps) build(ctx context.Context, job delivery.BuildJob) (any, error)
if s.Copy == nil { if s.Copy == nil {
return nil, Refuse("this build module copies no artifact (%s)", a.Artifact) return nil, Refuse("this build module copies no artifact (%s)", a.Artifact)
} }
Phase(ctx, "copy "+a.Artifact)
out, err = s.Copy(ctx, map[string]any{"module": job.Module, "artifact": a.Artifact, "from": a.From}) out, err = s.Copy(ctx, map[string]any{"module": job.Module, "artifact": a.Artifact, "from": a.From})
} else { } else {
if s.Build == nil { if s.Build == nil {
@@ -132,6 +141,7 @@ func (s AskSteps) build(ctx context.Context, job delivery.BuildJob) (any, error)
if job.Publish { if job.Publish {
args["publish"] = true args["publish"] = true
} }
Phase(ctx, "build "+a.Artifact)
out, err = s.Build(ctx, args) out, err = s.Build(ctx, args)
} }
if err != nil { if err != nil {
@@ -193,6 +203,13 @@ func (s AskSteps) level(ctx context.Context, job delivery.CheckJob, level string
return args return args
} }
test := func(args map[string]any) (bool, error) { test := func(args map[string]any) (bool, error) {
what := "the whole level"
if dir, _ := args["module"].(string); dir != "" {
what = dir
} else if pkgs, _ := args["packages"].([]string); len(pkgs) > 0 {
what = fmt.Sprintf("%d packages", len(pkgs))
}
Phase(ctx, fmt.Sprintf("%s: tests of %s", level, what))
out, err := s.Test(ctx, args) out, err := s.Test(ctx, args)
if err != nil { if err != nil {
return false, err return false, err
@@ -231,6 +248,9 @@ func (s AskSteps) level(ctx context.Context, job delivery.CheckJob, level string
if !keepRunner { if !keepRunner {
runner = "" // an installed module that ignores the runner judges the level as it judges any other (ADR 0535 §2) runner = "" // an installed module that ignores the runner judges the level as it judges any other (ADR 0535 §2)
} }
if whole && len(job.Packages) > 0 {
todo, whole = job.Packages, false // a placed ask tests its packages alone, whatever the digest said (ADR 0552 §4)
}
for _, priority := range []bool{true, false} { for _, priority := range []bool{true, false} {
args := withRunner(map[string]any{"workspace": f.Workspace, "level": level}) args := withRunner(map[string]any{"workspace": f.Workspace, "level": level})
if job.Run != "" { if job.Run != "" {
@@ -281,6 +301,7 @@ func (s AskSteps) plan(ctx context.Context, job delivery.CheckJob, level string,
if job.Run != "" { if job.Run != "" {
return nil, true, true return nil, true, true
} }
Phase(ctx, level+": digest")
out, err := s.Digest(ctx, withRunner(map[string]any{"workspace": f.Workspace, "level": level})) out, err := s.Digest(ctx, withRunner(map[string]any{"workspace": f.Workspace, "level": level}))
if err != nil { if err != nil {
if s.Logf != nil { if s.Logf != nil {
@@ -307,9 +328,16 @@ func (s AskSteps) plan(ctx context.Context, job delivery.CheckJob, level string,
return nil, true, runner != "" // a digest that names nothing or not the siblings keys no verdict: the level runs whole return nil, true, runner != "" // a digest that names nothing or not the siblings keys no verdict: the level runs whole
} }
la.Digest = &d la.Digest = &d
only := map[string]bool{}
for _, p := range job.Packages {
only[p] = true
}
var todo []string var todo []string
for _, pd := range d.Packages { for _, pd := range d.Packages {
p := pd.Package p := pd.Package
if len(only) > 0 && !only[p] {
continue // a placed ask's other packages were judged by the first ask (ADR 0552 §4)
}
if job.Kept && s.Kept != nil { if job.Kept && s.Kept != nil {
tests, found, err := s.Kept(ctx, level, pd.Digest) tests, found, err := s.Kept(ctx, level, pd.Digest)
if err != nil && s.Logf != nil { if err != nil && s.Logf != nil {
@@ -320,6 +348,15 @@ func (s AskSteps) plan(ctx context.Context, job delivery.CheckJob, level string,
continue continue
} }
} }
if needs := d.NeedsOf(p); len(delivery.Lacks(needs, s.Has)) > 0 {
// Its tests would only skip here (issue 890): it is left for a node that has what they need.
la.Unmet = append(la.Unmet, delivery.PackageNeeds{Package: p, Needs: needs})
if s.Logf != nil {
s.Logf("the %s level of %s is left to a node with %s: this node lacks %s", level, p,
strings.Join(needs, ", "), strings.Join(delivery.Lacks(needs, s.Has), ", "))
}
continue
}
todo = append(todo, p) todo = append(todo, p)
} }
return todo, false, false return todo, false, false
+42 -2
View File
@@ -20,6 +20,7 @@ type recorder struct {
fetch map[string]any fetch map[string]any
tests map[string]bool // package or module directory -> passed tests map[string]bool // package or module directory -> passed
digests map[string]string digests map[string]string
needs []delivery.PackageNeeds
} }
func (r *recorder) steps() AskSteps { func (r *recorder) steps() AskSteps {
@@ -35,8 +36,10 @@ func (r *recorder) steps() AskSteps {
Build: call("build", func(args map[string]any) any { Build: call("build", func(args map[string]any) any {
return map[string]any{"digest": "sha256:" + args["artifact"].(string), "reference": "artifact-store://" + args["artifact"].(string)} return map[string]any{"digest": "sha256:" + args["artifact"].(string), "reference": "artifact-store://" + args["artifact"].(string)}
}), }),
Copy: call("copy", func(args map[string]any) any { return map[string]any{"digest": "sha256:copied"} }), Copy: call("copy", func(args map[string]any) any { return map[string]any{"digest": "sha256:copied"} }),
Digest: call("digest", func(map[string]any) any { return map[string]any{"level": "unit", "packages": r.digests} }), Digest: call("digest", func(map[string]any) any {
return map[string]any{"level": "unit", "packages": r.digests, "needs": r.needs}
}),
Test: call("test", func(args map[string]any) any { Test: call("test", func(args map[string]any) any {
passed := true passed := true
if pkgs, ok := args["packages"].([]string); ok { if pkgs, ok := args["packages"].([]string); ok {
@@ -139,6 +142,43 @@ test {"level":"unit","packages":["x/pkg/a"],"workspace":"/w"}`
if u.Level != delivery.LevelUnit || len(u.Kept) != 1 || u.Kept[0].Package != "x/pkg/k" || u.Kept[0].Tests != 7 || len(u.Tests) != 2 || u.Digest == nil { if u.Level != delivery.LevelUnit || len(u.Kept) != 1 || u.Kept[0].Package != "x/pkg/k" || u.Kept[0].Tests != 7 || len(u.Tests) != 2 || u.Digest == nil {
t.Fatalf("the unit level %+v", u) t.Fatalf("the unit level %+v", u)
} }
// Issue 890/AC2: a package declaring node-build-node is not tested on a node that lacks it but named unmet, and is
// tested on a node that has it.
for _, has := range []bool{false, true} {
r.calls, r.tests["x/pkg/b"] = nil, true
r.needs = []delivery.PackageNeeds{{Package: "x/pkg/a", Needs: []string{delivery.NeedNodeBuildNode}}}
steps := r.steps()
if has {
steps.Has = func(n string) bool { return n == delivery.NeedNodeBuildNode }
}
out, err := RunAsk(context.Background(), steps, ask(delivery.CheckJob{Levels: []string{delivery.LevelUnit}, Kept: true,
Fetch: map[string]any{"repository": "novox/x", "commit": runCommit}}))
if err != nil {
t.Fatal(err)
}
u := out.(delivery.CheckAnswer).Levels[0]
tested := strings.Contains(strings.Join(r.calls, "\n"), `"x/pkg/a"`)
if tested != has || (len(u.Unmet) == 0) == !has || (!has && (u.Unmet[0].Package != "x/pkg/a" || delivery.UnmetNeeds([]delivery.LevelAnswer{u})[0] != delivery.NeedNodeBuildNode)) {
t.Fatalf("has %v: tested x/pkg/a %v, unmet %+v, calls %v", has, tested, u.Unmet, r.calls)
}
}
// A placed ask naming its packages tests those alone and takes no other's kept verdict, also when the level would
// run whole (ADR 0552 §4).
for _, digests := range []map[string]string{r.digests, nil} {
r.calls, r.digests = nil, digests
steps := r.steps()
steps.Has = func(string) bool { return true }
out, err := RunAsk(context.Background(), steps, ask(delivery.CheckJob{Levels: []string{delivery.LevelUnit}, Kept: true, Packages: []string{"x/pkg/a"},
Fetch: map[string]any{"repository": "novox/x", "commit": runCommit}}))
if err != nil {
t.Fatal(err)
}
u := out.(delivery.CheckAnswer).Levels[0]
if last := r.calls[len(r.calls)-1]; !strings.HasSuffix(last, `"packages":["x/pkg/a"],"workspace":"/w"}`) || len(u.Kept) != 0 || len(u.Tests) != 1 {
t.Fatalf("digests %v: calls %v, kept %+v: want x/pkg/a alone tested, nothing taken kept", digests != nil, r.calls, u.Kept)
}
}
} }
// ADR 0535 §2, §4: a merge that changes the build module's own code is judged by the candidate's runner: its digest says // ADR 0535 §2, §4: a merge that changes the build module's own code is judged by the candidate's runner: its digest says
+135
View File
@@ -0,0 +1,135 @@
package buildstep
import (
"encoding/json"
"fmt"
"os"
"strings"
"time"
)
// **A node may build only within its build hours** (the operator, 2026-10-26 21:18: "we should only use g14 as a builder
// outside office hours", "from 18 o'clock until 6 in the morning"). A build module's `build-hours` setting names, per
// node, the window in the node's local time in which it takes asks; outside it the capacity judged by is 0: steps running
// finish, no new one is taken. A node it does not name, or an empty setting, builds at any hour. The puller judges it at
// every judgement and wakes on one timer to the window's next edge (a clock edge, not a poll of another part).
// BuildHoursSetting is the build modules' setting of the nodes' build windows: `<node>=HH:MM-HH:MM` pairs separated by
// commas, in the node's local time; an end before its start crosses midnight (g14=18:00-06:00).
const BuildHoursSetting = "build-hours"
// BuildWindow is one node's build hours: minutes after local midnight, Start inclusive, End exclusive.
type BuildWindow struct {
Start, End int
}
// String is the window as the setting writes it.
func (w BuildWindow) String() string {
return fmt.Sprintf("%02d:%02d-%02d:%02d", w.Start/60, w.Start%60, w.End/60, w.End%60)
}
// Open says whether t, in its own location, falls in the window.
func (w BuildWindow) Open(t time.Time) bool {
m := t.Hour()*60 + t.Minute()
if w.Start < w.End {
return m >= w.Start && m < w.End
}
return m >= w.Start || m < w.End
}
// NextEdge is the first time after t, in t's location, at which the window opens or closes.
func (w BuildWindow) NextEdge(t time.Time) time.Time {
var next time.Time
for day := 0; day <= 1; day++ {
for _, m := range []int{w.Start, w.End} {
e := time.Date(t.Year(), t.Month(), t.Day()+day, m/60, m%60, 0, 0, t.Location())
if e.After(t) && (next.IsZero() || e.Before(next)) {
next = e
}
}
}
return next
}
// ParseBuildHours is node's window in a build-hours setting: nil when the setting is empty or does not name the node.
func ParseBuildHours(setting, node string) (*BuildWindow, error) {
var found *BuildWindow
for _, pair := range strings.Split(setting, ",") {
pair = strings.TrimSpace(pair)
if pair == "" {
continue
}
name, span, ok := strings.Cut(pair, "=")
from, to, ok2 := strings.Cut(span, "-")
if !ok || !ok2 || strings.TrimSpace(name) == "" {
return nil, fmt.Errorf("the build hours %q: %q is not <node>=HH:MM-HH:MM", setting, pair)
}
start, err := minuteOf(from)
if err != nil {
return nil, fmt.Errorf("the build hours %q: %v", setting, err)
}
end, err := minuteOf(to)
if err != nil {
return nil, fmt.Errorf("the build hours %q: %v", setting, err)
}
if start == end {
return nil, fmt.Errorf("the build hours %q: %s opens and closes at the same minute", setting, pair)
}
if strings.TrimSpace(name) == node {
found = &BuildWindow{Start: start, End: end}
}
}
return found, nil
}
// minuteOf reads HH:MM as minutes after midnight.
func minuteOf(s string) (int, error) {
var h, m int
s = strings.TrimSpace(s)
if n, err := fmt.Sscanf(s, "%d:%d", &h, &m); err != nil || n != 2 || h < 0 || h > 23 || m < 0 || m > 59 || len(s) != 5 {
return 0, fmt.Errorf("%q is not a time HH:MM", s)
}
return h*60 + m, nil
}
// NodeBuildHours is node's build window, read from the module's settings file at path at every call, and why it is
// none when the value cannot be read. No path, no file value or an empty one is no window and no fault. A value that
// cannot be read is no window too, so the node builds at any hour; the setting is a preference of hours, not a safety
// bound (a node that must never build has capacity 0). It is loud: the fault is said once per distinct fault through
// logf, and the puller writes it in the capacity record that build_usage shows.
func NodeBuildHours(path, node string, logf func(string, ...any)) func() (*BuildWindow, string) {
said := ""
say := func(why string) {
if logf != nil && why != said {
said = why
logf("%s, so %s builds at any hour", why, node)
}
}
fault := func(why string) (*BuildWindow, string) {
say(why)
return nil, why
}
return func() (*BuildWindow, string) {
if path == "" {
return nil, ""
}
raw, err := os.ReadFile(path)
if err != nil {
return fault(fmt.Sprintf("the setting %s cannot be read: %v", BuildHoursSetting, err))
}
var s map[string]json.RawMessage
if err := json.Unmarshal(raw, &s); err != nil {
return fault(fmt.Sprintf("the settings file %s is not JSON: %v", path, err))
}
setting := ""
if v, ok := s[BuildHoursSetting]; ok && json.Unmarshal(v, &setting) != nil {
return fault(fmt.Sprintf("the setting %s %s is not a string", BuildHoursSetting, v))
}
w, err := ParseBuildHours(setting, node)
if err != nil {
return fault(err.Error())
}
said = ""
return w, ""
}
}
+141
View File
@@ -0,0 +1,141 @@
package buildstep
import (
"encoding/json"
"fmt"
"math"
"os"
"runtime"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/buildcpu"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A build seat's capacity follows the CPU its node gives builds** (issue 1806's follow-up; issue 1890, 2026-10-26
// 19:41: under the 50 % cap shanks' mesh-build-go kept capacity 3, three full mesh-catalog checks shared 12 cores, and
// each step ran past the 30-minute step bound with no test run). The configured capacity is an upper bound; the node's
// build slice bounds it again by the cores a step needs: floor(the slice's quota in cores / cores-per-step), one at
// least. The quota is read live from the cgroup at every judgement, so a changed cap is followed at the next take
// without a restart, and the result is said (the seat's current verb, the status tools, `capacity.<node>` in the pauses
// state that the delivery view's build_usage reads).
// CoresPerStepSetting is a build module's setting of the processors one of its steps needs: a number above zero, the
// same on every node (a step's need is the module's, not the node's). Each build module declares it with its own default
// (DefaultCoresPerStep).
const CoresPerStepSetting = "cores-per-step"
// DefaultCoresPerStep are the build modules' defaults by language, from measured steps (2026-10-26): a full mesh-catalog
// check on mesh-build-go takes what it is given (three at once took 965 % of shanks' 1200 % slice, about 4 cores each,
// and ran past the 30-minute step bound), so a Go step is given 6, half again what starved; a Node step (tsc, esbuild, a
// test runner: median 15 s) and an image build (buildah, its RUN lines mostly fetches) need 2.
var DefaultCoresPerStep = map[string]float64{delivery.AskGo: 6, delivery.AskNode: 2, delivery.AskImage: 2}
// Quota is the build slice's CPU quota as this process's cgroup holds it, in percent of one processor (1200 is 12
// cores), and where it was read: the cgroup's cpu.max (buildcpu.ReadInForce, the tighter of its unit's and its slice's),
// else the share the composer gave (buildcpu.ShareWord) over the machine's processors, else 0 and why none is known.
func Quota() (percent int, from string) {
f := buildcpu.ReadInForce()
if f.Quota > 0 {
path := f.Slice.Path
if path == "" {
path = f.Cgroup
}
return f.Quota, "the cpu.max of " + path
}
if n, err := strconv.Atoi(strings.TrimSpace(os.Getenv(buildcpu.ShareWord))); err == nil && n >= 1 && n <= 100 {
return buildcpu.Quota(n, runtime.NumCPU()), fmt.Sprintf("the share %d%% the composer gave (%s), of %d processors; no cpu.max is read", n, buildcpu.ShareWord, runtime.NumCPU())
}
return 0, "no CPU quota: neither a cpu.max on its cgroup nor a share from the composer"
}
// EffectiveCapacity is the capacity a seat's holder judges by, from its configured capacity, its node's quota (percent of
// one processor, 0 when none is known) and the cores one of its steps needs. Without a quota it is the configured one;
// with one, min(configured, floor(quota cores / coresPerStep)), and one at least while the configured one is above
// zero. A coresPerStep of zero or less is taken as 1.
func EffectiveCapacity(configured, quotaPercent int, quotaFrom string, coresPerStep float64) delivery.Capacity {
if coresPerStep <= 0 || math.IsNaN(coresPerStep) || math.IsInf(coresPerStep, 0) {
coresPerStep = 1
}
c := delivery.Capacity{Configured: configured, Effective: configured, QuotaFrom: quotaFrom, CoresPerStep: coresPerStep}
switch {
case configured <= 0:
c.Effective = 0
c.Why = fmt.Sprintf("its configured capacity is %d, so it pulls nothing", configured)
return c
case quotaPercent <= 0:
c.Why = fmt.Sprintf("its configured capacity %d; %s, so the CPU bounds nothing", configured, quotaFrom)
return c
}
c.QuotaCores = float64(quotaPercent) / 100
c.ByCPU = int(math.Floor(c.QuotaCores/coresPerStep + 1e-9))
if c.ByCPU < 1 {
c.ByCPU = 1
}
byCPU := fmt.Sprintf("%g cores of the build slice (%s) / %g cores a step = %d", c.QuotaCores, quotaFrom, coresPerStep, c.ByCPU)
if c.ByCPU < configured {
c.Effective = c.ByCPU
c.Why = fmt.Sprintf("%d, bounded by the CPU: %s, under its configured %d", c.Effective, byCPU, configured)
} else {
c.Why = fmt.Sprintf("%d, its configured capacity; the CPU allows %s", configured, byCPU)
}
return c
}
// NodeCoresPerStep is the module's cores-per-step, read from its settings file at path at every call so a change is
// followed at the next judgement; fallback when there is no path, no value or one that is not a number above zero, said
// once per distinct fault through logf.
func NodeCoresPerStep(path string, fallback float64, logf func(string, ...any)) func() float64 {
said := ""
say := func(why string) {
if logf != nil && why != said {
said = why
logf("%s, so a step is taken to need %g cores", why, fallback)
}
}
return func() float64 {
if path == "" {
return fallback
}
raw, err := os.ReadFile(path)
if err != nil {
say(fmt.Sprintf("the setting %s cannot be read: %v", CoresPerStepSetting, err))
return fallback
}
var s map[string]json.RawMessage
if err := json.Unmarshal(raw, &s); err != nil {
say(fmt.Sprintf("the settings file %s is not JSON: %v", path, err))
return fallback
}
v, ok := s[CoresPerStepSetting]
if !ok {
return fallback // not declared yet (a module before its manifest names it): the default, nothing to say
}
n, err := coresOf(v)
if err != nil {
say(fmt.Sprintf("the setting %s %s: %v", CoresPerStepSetting, v, err))
return fallback
}
said = ""
return n
}
}
// coresOf reads a cores-per-step value: a JSON number or a string holding one, above zero.
func coresOf(v json.RawMessage) (float64, error) {
var n float64
if err := json.Unmarshal(v, &n); err != nil {
var s string
if err := json.Unmarshal(v, &s); err != nil {
return 0, fmt.Errorf("is neither a number nor a string")
}
if n, err = strconv.ParseFloat(strings.TrimSpace(s), 64); err != nil {
return 0, fmt.Errorf("is not a number")
}
}
if n <= 0 || math.IsNaN(n) || math.IsInf(n, 0) {
return 0, fmt.Errorf("is not above zero")
}
return n, nil
}
+99
View File
@@ -3,7 +3,9 @@ package buildstep
import ( import (
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"time"
) )
// 1887/A7: the node's build capacity is the build module's setting, read at every pull: the control-node at 0 pulls // 1887/A7: the node's build capacity is the build module's setting, read at every pull: the control-node at 0 pulls
@@ -36,3 +38,100 @@ func TestNodeCapacity(t *testing.T) {
t.Errorf("a missing file = %d, want 0", got) t.Errorf("a missing file = %d, want 0", got)
} }
} }
// ADR 0553 §4: a step's threads are the composer's share over the module's own capacity; without the word, none. And
// issue 1806's follow-up: the capacity judged by is the configured one bounded by the build slice's cores over the cores
// a step needs (the three build nodes as they stand on 2026-10-26), and cores-per-step is read from the settings file.
func TestNodeLimitsAndTheCPUBound(t *testing.T) {
t.Setenv("MESH_BUILD_CPU_SHARE", "")
if l := NodeLimits(3); l.Threads != 0 || l.Slots != 3 {
t.Errorf("without the word: %+v", l)
}
t.Setenv("MESH_BUILD_CPU_SHARE", "75")
if l := NodeLimits(3); l.Percent != 75 || l.Slots != 3 || l.Threads < 1 {
t.Errorf("from the word: %+v", l)
}
for _, c := range []struct {
name string
configured, quota int
cores float64
effective, byCPU int
}{
{"shanks: 12 cores, configured 3", 3, 1200, 6, 2, 2},
{"ace: 18 cores, configured 2", 2, 1800, 6, 2, 3},
{"g14: 8 cores, configured 2", 2, 800, 6, 1, 1},
{"node on shanks: 12 cores at 2", 3, 1200, 2, 3, 6},
{"no quota: the configured one", 3, 0, 6, 3, 0},
{"configured 0 pulls nothing", 0, 1200, 6, 0, 0},
{"a quota under one step still builds one", 2, 100, 6, 1, 1},
{"no cores a step is one", 2, 150, 0, 1, 1},
} {
got := EffectiveCapacity(c.configured, c.quota, "test", c.cores)
if got.Effective != c.effective || got.ByCPU != c.byCPU || got.Why == "" {
t.Errorf("%s: %+v, want effective %d, by CPU %d", c.name, got, c.effective, c.byCPU)
}
}
file := filepath.Join(t.TempDir(), "settings.json")
var said []string
cores := NodeCoresPerStep(file, 6, func(f string, a ...any) { said = append(said, f) })
for _, c := range []struct {
file string
want float64
}{{`{"capacity":"shanks=2"}`, 6}, {`{"cores-per-step":4}`, 4}, {`{"cores-per-step":"3.5"}`, 3.5}, {`{"cores-per-step":0}`, 6}, {`{"cores-per-step":0}`, 6}} {
os.WriteFile(file, []byte(c.file), 0o600)
if got := cores(); got != c.want {
t.Errorf("cores-per-step from %s = %g, want %g", c.file, got, c.want)
}
}
if len(said) != 1 {
t.Errorf("a value not above zero is said once, said %d times", len(said))
}
// The operator, 2026-10-26 21:18: g14 builds only from 18:00 to 06:00, its local time. Outside its build hours the
// capacity judged by is 0, said with the window and when it opens; a node not named, or a bad value, builds at any hour.
cet := time.FixedZone("CET", 3600)
day := func(hh, mm int) time.Time { return time.Date(2026, 10, 26, hh, mm, 0, 0, cet) }
was := now
defer func() { now = was }()
for _, c := range []struct {
name, setting, node string
at time.Time
effective int
edge string // the window's next edge, HH:MM; "" for no window
bad bool
}{
{"g14 in the office", "g14=18:00-06:00", "g14", day(10, 0), 0, "18:00", false},
{"g14 in the evening", "g14=18:00-06:00", "g14", day(18, 0), 2, "06:00", false},
{"g14 past midnight", "g14=18:00-06:00", "g14", day(5, 59), 2, "06:00", false},
{"g14 at the close", "g14=18:00-06:00", "g14", day(6, 0), 0, "18:00", false},
{"a day window", "shanks=09:00-17:30,g14=18:00-06:00", "shanks", day(17, 29), 2, "17:30", false},
{"a node not named", "g14=18:00-06:00", "ace", day(10, 0), 2, "", false},
{"a bad time", "g14=18:00-6", "g14", day(10, 0), 2, "", true},
{"no span", "g14=18:00-18:00", "g14", day(10, 0), 2, "", true},
} {
w, err := ParseBuildHours(c.setting, c.node)
if (err != nil) != c.bad {
t.Errorf("%s: parse %v", c.name, err)
}
if edge := ""; w != nil {
edge = w.NextEdge(c.at).Format("15:04")
if edge != c.edge {
t.Errorf("%s: next edge %s, want %s", c.name, edge, c.edge)
}
} else if c.edge != "" {
t.Errorf("%s: no window, want one", c.name)
}
file := filepath.Join(t.TempDir(), "settings.json")
os.WriteFile(file, []byte(`{"build-hours":"`+c.setting+`"}`), 0o600)
now = func() time.Time { return c.at }
p := &Puller{Language: "go", Node: c.node, Capacity: func() int { return 2 }, BuildHours: NodeBuildHours(file, c.node, nil)}
got := p.CapacityDetail()
if got.Effective != c.effective || got.OutsideHours != (c.effective == 0) || (c.effective == 0 && !strings.Contains(got.Why, "outside its build hours 18:00-06:00 CET until 18:00")) {
t.Errorf("%s: %+v, want capacity %d", c.name, got, c.effective)
}
// A value that cannot be read builds at any hour, loudly: the record names the value and says why.
if c.bad && (!strings.Contains(got.BuildHoursFault, c.setting) || !strings.Contains(got.Why, "at any hour, because its build hours cannot be read")) {
t.Errorf("%s: a bad value is not said in the record: %+v", c.name, got)
}
}
}
+14
View File
@@ -0,0 +1,14 @@
package buildstep
import (
"os"
"git.novox.be/novox/mesh-sdk/go/buildcpu"
)
// NodeLimits is what one step of a build module is held to (novox/hq ADR 0553 §4): the node's processor share from the
// word the composer gives the module (buildcpu.ShareWord), divided by the module's own capacity, own, over the machine's
// processors. Without the word, no thread limit: Threads is 0.
func NodeLimits(own int) buildcpu.Limits {
return buildcpu.FromWords(os.Getenv, own)
}
+6 -11
View File
@@ -71,7 +71,7 @@ func FetchCommit(ctx context.Context, run forgekey.Runner, dir, remote, commit,
full := append([]string{"-c", "credential.helper=", "-C", dir}, args...) full := append([]string{"-c", "credential.helper=", "-C", dir}, args...)
r := run(ctx, env, "git", full...) r := run(ctx, env, "git", full...)
if r.Status != 0 || r.Err != "" { if r.Status != 0 || r.Err != "" {
return "", fmt.Errorf("git %s: %s", args[0], firstLine(r.Err+" "+r.Stderr)) return "", fmt.Errorf("git %s: %s", args[0], forgekey.Said(r.Err+"\n"+r.Stderr))
} }
return r.Stdout, nil return r.Stdout, nil
} }
@@ -128,7 +128,11 @@ func UpdateMirror(ctx context.Context, run forgekey.Runner, mirror, remote, comm
git := func(args ...string) error { git := func(args ...string) error {
r := run(ctx, env, "git", append([]string{"-c", "credential.helper="}, args...)...) r := run(ctx, env, "git", append([]string{"-c", "credential.helper="}, args...)...)
if r.Status != 0 || r.Err != "" { if r.Status != 0 || r.Err != "" {
return fmt.Errorf("git %s: %s", args[0], firstLine(r.Err+" "+r.Stderr)) verb := args[0]
if verb == "-C" && len(args) > 2 { // the mirror's commands are named by what they do, not by -C
verb = args[2]
}
return fmt.Errorf("git %s: %s", verb, forgekey.Said(r.Err+"\n"+r.Stderr))
} }
return nil return nil
} }
@@ -190,12 +194,3 @@ func checkFetch(o fetchOptions, dir, remote, commit, ref string) error {
} }
return nil return nil
} }
func firstLine(s string) string {
for _, l := range strings.Split(s, "\n") {
if l = strings.TrimSpace(l); l != "" {
return l
}
}
return ""
}
+14 -1
View File
@@ -15,6 +15,9 @@ type MergeResult struct {
Base, Head, Merge string Base, Head, Merge string
Conflicts []string Conflicts []string
Changed []string Changed []string
// Programs are the changed files that are compiled programs (ExecutableKind of their first bytes, read in the workspace
// at the merge); never nil for a clean merge, so a caller tells none from a module that reports none (issue 1577).
Programs []Program
} }
// MergeInto makes, in the workspace dir that FetchCommit left at commit, the merge of commit into the trunk commit // MergeInto makes, in the workspace dir that FetchCommit left at commit, the merge of commit into the trunk commit
@@ -52,7 +55,7 @@ func MergeInto(ctx context.Context, run forgekey.Runner, dir, remote, base, comm
full := append([]string{"-c", "credential.helper=", "-C", dir}, args...) full := append([]string{"-c", "credential.helper=", "-C", dir}, args...)
r := run(ctx, env, "git", full...) r := run(ctx, env, "git", full...)
if r.Status != 0 || r.Err != "" { if r.Status != 0 || r.Err != "" {
return "", fmt.Errorf("git %s: %s", args[0], firstLine(r.Err+" "+r.Stderr)) return "", fmt.Errorf("git %s: %s", args[0], forgekey.Said(r.Err+"\n"+r.Stderr))
} }
return r.Stdout, nil return r.Stdout, nil
} }
@@ -98,6 +101,16 @@ func MergeInto(ctx context.Context, run forgekey.Runner, dir, remote, base, comm
return res, fmt.Errorf("the paths the merge changes cannot be read: %w", err) return res, fmt.Errorf("the paths the merge changes cannot be read: %w", err)
} }
res.Changed = strings.Fields(changed) res.Changed = strings.Fields(changed)
raw, err := git("diff", "--raw", "-z", "--no-renames", "--diff-filter=AM", base, res.Merge)
if err != nil {
return res, fmt.Errorf("the files the merge adds or modifies cannot be read: %w", err)
}
res.Programs = programsIn(dir, raw)
history, err := historyPrograms(git, base, commit, res.Programs)
if err != nil {
return res, fmt.Errorf("the commits of the branch cannot be read for compiled programs: %w", err)
}
res.Programs = append(res.Programs, history...)
return res, nil return res, nil
} }
+30 -3
View File
@@ -79,6 +79,30 @@ func (r *mergeRig) head(files map[string]string) string {
return h return h
} }
// headThatAddsAndRemoves is head with an earlier commit on the branch that adds a program and a later one that removes it: the
// branch's tree is clean, its history is not. It answers the head and the earlier commit.
func (r *mergeRig) headThatAddsAndRemoves(files map[string]string) (head, added string) {
realGit(r.t, r.work, "checkout", "-q", "-B", "pr", r.fork)
writeFile(r.t, r.work, "gone/prog", "\x7fELF\x02\x01")
realGit(r.t, r.work, "add", ".")
realGit(r.t, r.work, "commit", "-q", "-m", "oops")
added = realGit(r.t, r.work, "rev-parse", "HEAD")
realGit(r.t, r.work, "rm", "-q", "gone/prog")
return r.head2(files), added
}
// head2 commits files on the current branch and pushes it as the pull request's head.
func (r *mergeRig) head2(files map[string]string) string {
for n, c := range files {
writeFile(r.t, r.work, n, c)
}
realGit(r.t, r.work, "add", ".")
realGit(r.t, r.work, "commit", "-q", "-m", "head")
h := realGit(r.t, r.work, "rev-parse", "HEAD")
realGit(r.t, r.work, "push", "-q", r.bare, "HEAD:refs/pull/1/head")
return h
}
// advance moves the trunk past the fork point with a commit changing files, pushes it and answers its sha. // advance moves the trunk past the fork point with a commit changing files, pushes it and answers its sha.
func (r *mergeRig) advance(files map[string]string) string { func (r *mergeRig) advance(files map[string]string) string {
realGit(r.t, r.work, "checkout", "-q", "-B", "main", r.fork) realGit(r.t, r.work, "checkout", "-q", "-B", "main", r.fork)
@@ -102,7 +126,7 @@ func (r *mergeRig) fetch(head string) {
// holds both sides' changes. // holds both sides' changes.
func TestMergeIntoACleanMergeAnswersTheMergeCommitAndTheChangedFiles(t *testing.T) { func TestMergeIntoACleanMergeAnswersTheMergeCommitAndTheChangedFiles(t *testing.T) {
r := newMergeRig(t) r := newMergeRig(t)
head := r.head(map[string]string{"b.txt": "head\n"}) head, added := r.headThatAddsAndRemoves(map[string]string{"b.txt": "head\n", "prog": "\x7fELF\x02\x01"})
base := r.advance(map[string]string{"a.txt": "one\ntwo\nthree\nfour\n"}) base := r.advance(map[string]string{"a.txt": "one\ntwo\nthree\nfour\n"})
r.fetch(head) r.fetch(head)
res, err := MergeInto(context.Background(), forgekey.ExecRunner, r.ws, r.remote, base, head, AllowFileRemote()) res, err := MergeInto(context.Background(), forgekey.ExecRunner, r.ws, r.remote, base, head, AllowFileRemote())
@@ -112,8 +136,11 @@ func TestMergeIntoACleanMergeAnswersTheMergeCommitAndTheChangedFiles(t *testing.
if res.Base != base || res.Head != head || len(res.Merge) != 40 || res.Merge == base || res.Merge == head || len(res.Conflicts) != 0 { if res.Base != base || res.Head != head || len(res.Merge) != 40 || res.Merge == base || res.Merge == head || len(res.Conflicts) != 0 {
t.Fatalf("result = %+v", res) t.Fatalf("result = %+v", res)
} }
if !reflect.DeepEqual(res.Changed, []string{"b.txt"}) { if !reflect.DeepEqual(res.Changed, []string{"b.txt", "prog"}) {
t.Errorf("changed = %v, want only what the head adds to the trunk: b.txt", res.Changed) t.Errorf("changed = %v, want only what the head adds to the trunk: b.txt, prog", res.Changed)
}
if !reflect.DeepEqual(res.Programs, []Program{{"prog", "ELF", ""}, {"gone/prog", "ELF", added}}) { // issue 1577
t.Errorf("programs = %v, want the ELF the head adds and the one an earlier commit added and a later one removed", res.Programs)
} }
if at := realGit(t, r.ws, "rev-parse", "HEAD"); at != res.Merge { if at := realGit(t, r.ws, "rev-parse", "HEAD"); at != res.Merge {
t.Errorf("the workspace is at %s, not at the merge %s", at, res.Merge) t.Errorf("the workspace is at %s, not at the merge %s", at, res.Merge)
+18
View File
@@ -104,3 +104,21 @@ func TestUpdateMirrorRefusesBeforeRunning(t *testing.T) {
t.Errorf("ran %v", f.calls) t.Errorf("ran %v", f.calls)
} }
} }
// The forge's refusal reaches the reason in git's own words: the mirror of a repository its account cannot read said
// only "git -C: error:" (2026-10-26), since git writes a bare "error:" line before the forge's message.
func TestUpdateMirrorSaysTheForgesRefusal(t *testing.T) {
f := &fakeGit{}
f.rule = func(args string) (r forgekey.Ran) {
if strings.Contains(args, "fetch") {
r.Status = 128
r.Stderr = "error: \nerror: User: 22:nox-builder with Key: 30:mesh:ace.mesh-build-go is not authorized to read o/r.\nerror: \nfatal: Could not read from remote repository.\n"
}
return
}
err := UpdateMirror(context.Background(), f.run, filepath.Join(t.TempDir(), "m.git"), "ssh://mesh-forge/o/r.git", sha, "")
if err == nil || !strings.Contains(err.Error(), "nox-builder with Key: 30:mesh:ace.mesh-build-go is not authorized to read o/r.") ||
!strings.Contains(err.Error(), "Could not read from remote repository") || !strings.HasPrefix(err.Error(), "git fetch: ") {
t.Errorf("err = %v", err)
}
}
+67
View File
@@ -0,0 +1,67 @@
package buildstep
import (
"context"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A running ask says its phases** (issue 1890, 2026-10-26 19:09: a mesh-catalog try's step on shanks ran 30 minutes
// and its module's journal held no line naming its ask, so nobody could tell whether it fetched, linted or tested). The
// puller hands every ask's run a phase reporter in its context; RunAsk and a module's own step bodies call Phase as they
// move (fetch and merge, a level's digest, lint, a level's tests), and each phase is logged with the ask's id and its
// time since the start, and written to the ask's running step key (delivery.Step.Phase), so its asker and the delivery
// view read where it stands from the bus.
type phaseKey struct{}
// Phase says that the ask whose run ctx belongs to entered phase; nothing outside an ask's run (a tool called by hand).
func Phase(ctx context.Context, phase string) {
if r, ok := ctx.Value(phaseKey{}).(*phases); ok && r != nil {
r.enter(strings.TrimSpace(phase))
}
}
// phases is one running ask's reporter: it writes the step key again with each new phase until the step ends.
type phases struct {
mu sync.Mutex
put Put
key string
step delivery.Step
ended bool
logf func(format string, args ...any)
prefix string
}
func (r *phases) enter(phase string) {
if phase == "" {
return
}
r.mu.Lock()
defer r.mu.Unlock()
if r.ended || phase == r.step.Phase {
return
}
at := now().UTC()
r.step.Phase, r.step.PhaseAt = phase, at
if r.logf != nil {
r.logf("%s: %s, %s after its start", r.prefix, phase, at.Sub(r.step.Started).Round(time.Second))
}
if r.put == nil {
return
}
if err := write(r.put, r.key, r.step); err != nil && r.logf != nil {
r.logf("%s: the phase %s could not be written to %s: %v", r.prefix, phase, r.key, err)
}
}
// end stops the writes (the step's end is written after it) and answers the step with its last phase, for the end.
func (r *phases) end() delivery.Step {
r.mu.Lock()
defer r.mu.Unlock()
r.ended = true
return r.step
}
+115
View File
@@ -0,0 +1,115 @@
package buildstep
import (
"bytes"
"io"
"os"
"path/filepath"
"strings"
)
// Program is a changed file whose first bytes are an executable binary: a compiled program, which is build output and not
// source (novox/hq issue 1577: `go build` in a module's folder writes one beside go.mod, and `git add -A` takes it).
type Program struct {
Path string `json:"path"`
Kind string `json:"kind"` // ELF, Mach-O or PE
// Commit is set for a program a commit of the branch added and a later one removed (or replaced): the merge's tree is clean,
// but the merge commit carries the blob into the trunk's history for good. Empty for a program the merge's tree holds.
Commit string `json:"commit,omitempty"`
}
// ExecutableKind says what executable format the first bytes of a file are, or "" when they are none. A PE needs its header
// ("PE\0\0" at the offset in bytes 0x3c) so a text starting with "MZ" is not one; a fat Mach-O (0xCAFEBABE) is told from a
// Java class by its architecture count (bytes 4 to 7, fewer than 0x20; a class has its version there).
func ExecutableKind(head []byte) string {
switch {
case bytes.HasPrefix(head, []byte("\x7fELF")):
return "ELF"
case bytes.HasPrefix(head, []byte("MZ")):
if len(head) >= 0x40 {
off := int(head[0x3c]) | int(head[0x3d])<<8 | int(head[0x3e])<<16 | int(head[0x3f])<<24
if off >= 0 && off+4 <= len(head) && bytes.Equal(head[off:off+4], []byte("PE\x00\x00")) {
return "PE"
}
}
case bytes.HasPrefix(head, []byte{0xcf, 0xfa, 0xed, 0xfe}), bytes.HasPrefix(head, []byte{0xce, 0xfa, 0xed, 0xfe}),
bytes.HasPrefix(head, []byte{0xfe, 0xed, 0xfa, 0xcf}), bytes.HasPrefix(head, []byte{0xfe, 0xed, 0xfa, 0xce}),
bytes.HasPrefix(head, []byte{0xca, 0xfe, 0xba, 0xbe}) && len(head) >= 8 && head[4] == 0 && head[5] == 0 && head[6] == 0 && head[7] < 0x20:
return "Mach-O"
}
return ""
}
// programHeadSize is how many bytes of a file are read to tell its format: enough for a PE header at its usual offsets.
const programHeadSize = 0x100
// historyPrograms names the compiled programs that a commit of base..head added or modified and the merge's tree no longer
// holds at that path (those it holds are in final already): each with the commit that added it. git is the merge's git runner.
func historyPrograms(git func(args ...string) (string, error), base, head string, final []Program) ([]Program, error) {
revs, err := git("rev-list", "--no-merges", "--reverse", base+".."+head)
if err != nil {
return nil, err
}
held := map[string]bool{}
for _, p := range final {
held[p.Path] = true
}
out, seen := []Program{}, map[string]bool{}
for _, c := range strings.Fields(revs) {
raw, err := git("diff-tree", "-r", "-z", "--no-commit-id", "--no-renames", "--root", "--diff-filter=AM", c)
if err != nil {
return nil, err
}
f := strings.Split(raw, "\x00")
for i := 0; i+1 < len(f); i += 2 {
meta := strings.Fields(strings.TrimPrefix(f[i], ":"))
if len(meta) < 4 || (meta[1] != "100644" && meta[1] != "100755") || held[f[i+1]] || seen[meta[3]] {
continue
}
seen[meta[3]] = true
blob, err := git("cat-file", "blob", meta[3])
if err != nil {
return nil, err
}
head := []byte(blob)
if len(head) > programHeadSize {
head = head[:programHeadSize]
}
if k := ExecutableKind(head); k != "" {
out = append(out, Program{Path: f[i+1], Kind: k, Commit: c})
}
}
}
return out, nil
}
// programsIn reads the output of `git diff --raw -z --no-renames --diff-filter=AM` (":<old mode> <new mode> <old sha> <new
// sha> <status>\0<path>\0" per file) and names each added or modified regular file of the tree dir that is a compiled
// program, from its first bytes read locally. A symlink (mode 120000) and a submodule (160000) are not read. It never
// answers nil, so a caller can tell no programs from none reported.
func programsIn(dir, raw string) []Program {
out := []Program{}
f := strings.Split(raw, "\x00")
for i := 0; i+1 < len(f); i += 2 {
meta := strings.Fields(strings.TrimPrefix(f[i], ":"))
if len(meta) < 2 || (meta[1] != "100644" && meta[1] != "100755") {
continue
}
p := f[i+1]
full := filepath.Join(dir, filepath.FromSlash(p))
if st, err := os.Lstat(full); err != nil || !st.Mode().IsRegular() {
continue
}
fh, err := os.Open(full)
if err != nil {
continue
}
head := make([]byte, programHeadSize)
n, _ := io.ReadFull(fh, head)
fh.Close()
if k := ExecutableKind(head[:n]); k != "" {
out = append(out, Program{Path: p, Kind: k})
}
}
return out
}
+72
View File
@@ -0,0 +1,72 @@
package buildstep
import (
"os"
"path/filepath"
"reflect"
"testing"
)
func pe(offset int, magic string) []byte {
b := make([]byte, 0x100)
copy(b, "MZ")
b[0x3c] = byte(offset)
copy(b[offset:], magic)
return b
}
func TestExecutableKindTellsProgramsFromText(t *testing.T) { // issue 1577
for _, c := range []struct {
name string
head []byte
want string
}{
{"elf", []byte("\x7fELF\x02\x01\x01"), "ELF"},
{"mach-o 64", []byte{0xcf, 0xfa, 0xed, 0xfe, 7, 0, 0, 1}, "Mach-O"},
{"mach-o 32 big", []byte{0xfe, 0xed, 0xfa, 0xce, 0, 0, 0, 7}, "Mach-O"},
{"fat mach-o", []byte{0xca, 0xfe, 0xba, 0xbe, 0, 0, 0, 2}, "Mach-O"},
{"java class", []byte{0xca, 0xfe, 0xba, 0xbe, 0, 0, 0, 0x34}, ""},
{"pe", pe(0x80, "PE\x00\x00"), "PE"},
{"mz text", []byte("MZ is a text file starting with two letters"), ""},
{"mz without a pe header", pe(0x80, "XX\x00\x00"), ""},
{"text", []byte("package main"), ""},
{"empty", nil, ""},
} {
if got := ExecutableKind(c.head); got != c.want {
t.Errorf("%s: kind = %q, want %q", c.name, got, c.want)
}
}
}
func TestProgramsInNamesTheAddedAndModifiedRegularFilesThatArePrograms(t *testing.T) { // issue 1577
dir := t.TempDir()
write := func(name string, b []byte) {
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, name), b, 0o755); err != nil {
t.Fatal(err)
}
}
write("m/prog", []byte("\x7fELF\x02\x01"))
write("m/notes", []byte("plain text"))
write("m/mac", []byte{0xcf, 0xfa, 0xed, 0xfe, 7, 0, 0, 1})
if err := os.Symlink("prog", filepath.Join(dir, "m/link")); err != nil {
t.Fatal(err)
}
line := func(mode, path string) string { return ":000000 " + mode + " 0000 1111 A\x00" + path + "\x00" }
for _, c := range []struct {
name, raw string
want []Program
}{
{"none", "", []Program{}},
{"a program and a text", line("100755", "m/prog") + line("100644", "m/notes"), []Program{{"m/prog", "ELF", ""}}},
{"two programs", line("100644", "m/prog") + line("100755", "m/mac"), []Program{{"m/prog", "ELF", ""}, {"m/mac", "Mach-O", ""}}},
{"a symlink and a submodule are not read", line("120000", "m/link") + line("160000", "m/sub"), []Program{}},
{"a path the tree lacks", line("100644", "m/gone"), []Program{}},
} {
if got := programsIn(dir, c.raw); !reflect.DeepEqual(got, c.want) {
t.Errorf("%s: programs = %v, want %v", c.name, got, c.want)
}
}
}
+257 -14
View File
@@ -6,6 +6,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"sort" "sort"
"strings"
"sync" "sync"
"time" "time"
@@ -76,6 +77,20 @@ type Puller struct {
// for this node, and wakes the puller on each change (watchCapacity): WatchConfigServed in a module, a test's own // for this node, and wakes the puller on each change (watchCapacity): WatchConfigServed in a module, a test's own
// double. Until it hands a value the puller judges by Capacity. // double. Until it hands a value the puller judges by Capacity.
WatchConfig func(key string, handle func(stdio.ProvisionChange) error) error WatchConfig func(key string, handle func(stdio.ProvisionChange) error) error
// CoresPerStep is what one ask of this module needs, read at every judgement (NodeCoresPerStep); with it the capacity
// judged by is bounded by the node's build slice: min(the configured capacity, floor(the quota's cores / CoresPerStep))
// (EffectiveCapacity). Nil bounds nothing by the CPU. Quota reads the quota (Quota when nil; a test's own).
CoresPerStep func() float64
Quota func() (percent int, from string)
// BuildHours is the node's build window, read at every judgement (NodeBuildHours); with WatchConfig set the value
// mesh-config resolves for `build-hours` takes its place once handed. Outside the window the capacity judged by is 0:
// steps running finish, no new one is taken; the puller wakes at the window's next edge. Nil builds at any hour.
BuildHours func() (*BuildWindow, string)
// Has says whether this node has a need an ask names (delivery.AskNeeds, issue 890). An ask needing what it lacks is
// declined: its step ends refused as declined (delivery.StepDeclined), the message is dropped for its asker to put
// again, and the worker it came from is not looked at again until the next pull's wait, so a node lacking the need
// does not take the asker's next copy at once. Nil has nothing.
Has func(need string) bool
mu sync.Mutex mu sync.Mutex
inHand int inHand int
@@ -86,6 +101,19 @@ type Puller struct {
// watched is the capacity the config watch handed last, used in place of Capacity once hasWatched. // watched is the capacity the config watch handed last, used in place of Capacity once hasWatched.
watched int watched int
hasWatched bool hasWatched bool
// hours is the build window the config watch handed last, and hoursFault why its value cannot be read; used in place
// of BuildHours once hasHours.
hours *BuildWindow
hoursFault string
hasHours bool
// judged is the capacity said last (log and capacity.<node>), so a change is said once.
judged delivery.Capacity
hasJudged bool
capWritten bool
capFault string
// declined are the kinds whose worker handed an ask this node declined, left out of the pulls without a wait until
// the next pull's wait ends.
declined map[string]bool
} }
func (p *Puller) init() { func (p *Puller) init() {
@@ -207,23 +235,128 @@ func (p *Puller) room() bool {
// roomNow is room, with the capacity and the asks in hand it was judged on. // roomNow is room, with the capacity and the asks in hand it was judged on.
func (p *Puller) roomNow() (ok bool, capacity, inHand int) { func (p *Puller) roomNow() (ok bool, capacity, inHand int) {
p.mu.Lock() capacity = p.CapacityDetail().Effective
watched, has := p.watched, p.hasWatched
p.mu.Unlock()
if has {
capacity = watched
} else if p.Capacity != nil {
capacity = p.Capacity()
}
p.mu.Lock() p.mu.Lock()
defer p.mu.Unlock() defer p.mu.Unlock()
return !p.paused && p.inHand < capacity, capacity, p.inHand return !p.paused && p.inHand < capacity, capacity, p.inHand
} }
// CapacityNow is the capacity the puller judges by: the config watch's last value, else Capacity. // CapacityNow is the capacity the puller judges by (CapacityDetail's Effective).
func (p *Puller) CapacityNow() int { func (p *Puller) CapacityNow() int {
_, capacity, _ := p.roomNow() return p.CapacityDetail().Effective
return capacity }
// CapacityDetail is the capacity the puller judges by now and why: the configured one (the config watch's last value,
// else Capacity), bounded by the node's build slice when CoresPerStep is set (EffectiveCapacity), its quota read now. A
// change from what it said last is logged and written to `capacity.<node>` of the pauses state (PausePut), so the
// delivery view says it too.
func (p *Puller) CapacityDetail() delivery.Capacity {
p.mu.Lock()
watched, has := p.watched, p.hasWatched
p.mu.Unlock()
configured, from := 0, "the settings file"
switch {
case has:
configured, from = watched, "mesh-config, as the config watch handed it"
case p.Capacity != nil:
configured = p.Capacity()
}
var c delivery.Capacity
if p.CoresPerStep == nil {
c = delivery.Capacity{Configured: configured, Effective: configured, QuotaFrom: "not judged: the module names no cores a step needs",
Why: fmt.Sprintf("its configured capacity %d", configured)}
} else {
quota := p.Quota
if quota == nil {
quota = Quota
}
percent, qfrom := quota()
c = EffectiveCapacity(configured, percent, qfrom, p.CoresPerStep())
}
c.ConfiguredFrom = from
w, fault := p.window()
if fault != "" {
c.BuildHoursFault = fault
c.Why = fmt.Sprintf("%s; at any hour, because its build hours cannot be read: %s", c.Why, fault)
}
if w != nil {
at := now()
zone, _ := at.Zone()
c.BuildHours = w.String() + " " + zone
if !w.Open(at) && c.Effective > 0 {
c.OutsideHours = true
c.Why = fmt.Sprintf("0, outside its build hours %s until %s (otherwise %s)", c.BuildHours, w.NextEdge(at).Format("15:04"), c.Why)
c.Effective = 0
} else if c.Effective > 0 {
c.Why = fmt.Sprintf("%s; within its build hours %s until %s", c.Why, c.BuildHours, w.NextEdge(at).Format("15:04"))
}
}
p.sayCapacity(c)
return c
}
// window is the node's build window now (the config watch's last value, else BuildHours, else none), and why its value
// cannot be read when it cannot.
func (p *Puller) window() (*BuildWindow, string) {
p.mu.Lock()
w, fault, has := p.hours, p.hoursFault, p.hasHours
p.mu.Unlock()
if has {
return w, fault
}
if p.BuildHours != nil {
return p.BuildHours()
}
return nil, ""
}
// untilEdge is how long until the build window's next edge, and false when the node has no window.
func (p *Puller) untilEdge() (time.Duration, bool) {
w, _ := p.window()
if w == nil {
return 0, false
}
at := now()
return w.NextEdge(at).Sub(at), true
}
// sayCapacity logs c when it differs from what was said last, and writes it to the node's capacity key until a write
// lands (a failed write is tried again at the next judgement, its error said once).
func (p *Puller) sayCapacity(c delivery.Capacity) {
p.mu.Lock()
same := p.hasJudged && p.judged.Effective == c.Effective && p.judged.Configured == c.Configured && p.judged.ByCPU == c.ByCPU &&
p.judged.CoresPerStep == c.CoresPerStep && p.judged.QuotaCores == c.QuotaCores && p.judged.OutsideHours == c.OutsideHours &&
p.judged.BuildHours == c.BuildHours && p.judged.BuildHoursFault == c.BuildHoursFault
if !same {
p.judged, p.hasJudged, p.capWritten = c, true, false
}
write := !p.capWritten && p.PausePut != nil
p.mu.Unlock()
if !same {
p.logf("the %s build capacity of %s is %s (configured from %s)", p.Language, p.Node, c.Why, c.ConfiguredFrom)
}
if !write {
return
}
key, err := delivery.CapacityKey(p.Node)
if err != nil {
return
}
c.At = now()
raw, err := delivery.EncodeCapacity(c)
if err == nil {
_, err = p.PausePut(key, json.RawMessage(raw))
}
p.mu.Lock()
defer p.mu.Unlock()
if err == nil {
p.capWritten, p.capFault = true, ""
return
}
if err.Error() != p.capFault {
p.capFault = err.Error()
p.logf("the %s build capacity of %s could not be written to %s (tried again at the next judgement): %v", p.Language, p.Node, key, err)
}
} }
// Pull takes and runs asks until ctx ends. An ask still running then is neither ended nor acknowledged: the stream // Pull takes and runs asks until ctx ends. An ask still running then is neither ended nor acknowledged: the stream
@@ -239,6 +372,7 @@ func (p *Puller) Pull(ctx context.Context) error {
p.logf("the pause of %s cannot be read: %v; pulling as not paused", p.Node, err) p.logf("the pause of %s cannot be read: %v; pulling as not paused", p.Node, err)
} }
go p.watchCapacity() // WatchConfigServed waits for Serve, which a module starts after its pull go p.watchCapacity() // WatchConfigServed waits for Serve, which a module starts after its pull
go p.watchHours()
wait := p.PullWait wait := p.PullWait
if wait <= 0 { if wait <= 0 {
wait = DefaultPullWait wait = DefaultPullWait
@@ -251,16 +385,29 @@ func (p *Puller) Pull(ctx context.Context) error {
// Parked until something wakes it: a capacity change (the config watch), a resume, or a step's end. Every // Parked until something wakes it: a capacity change (the config watch), a resume, or a step's end. Every
// park logs once, so a puller that sits idle says why (issue 1887). // park logs once, so a puller that sits idle says why (issue 1887).
p.logf("the %s puller on %s parks: %d of capacity %d in hand, paused %v", p.Language, p.Node, inHand, capacity, p.Paused()) p.logf("the %s puller on %s parks: %d of capacity %d in hand, paused %v", p.Language, p.Node, inHand, capacity, p.Paused())
var edge <-chan time.Time // the build window's next edge, when the node has one: one timer, not a poll
stop := func() bool { return false }
if d, ok := p.untilEdge(); ok {
t := time.NewTimer(d + time.Second)
edge, stop = t.C, t.Stop
}
select { select {
case <-ctx.Done(): case <-ctx.Done():
stop()
return ctx.Err() return ctx.Err()
case <-p.wake: case <-p.wake:
case <-edge:
} }
stop()
_, capacity, inHand = p.roomNow() _, capacity, inHand = p.roomNow()
p.logf("the %s puller on %s wakes: %d of capacity %d in hand, paused %v", p.Language, p.Node, inHand, capacity, p.Paused()) p.logf("the %s puller on %s wakes: %d of capacity %d in hand, paused %v", p.Language, p.Node, inHand, capacity, p.Paused())
continue continue
} }
msg, err := p.next(ctx, wait) pullWait := wait
if d, ok := p.untilEdge(); ok && d < pullWait {
pullWait = d // a pull does not wait past the window's close
}
msg, err := p.next(ctx, pullWait)
if err != nil { if err != nil {
if ctx.Err() != nil { if ctx.Err() != nil {
return ctx.Err() return ctx.Err()
@@ -283,17 +430,59 @@ func (p *Puller) Pull(ctx context.Context) error {
// next is one ask from the workers in their order: one already waiting on delivery, check or recheck, and when none // next is one ask from the workers in their order: one already waiting on delivery, check or recheck, and when none
// waits, a pull of the delivery worker that waits for one. // waits, a pull of the delivery worker that waits for one.
//
// A kind whose worker handed an ask this node declined is left out until that wait ended (issue 890): the pull's own
// wait, no timer of its own.
func (p *Puller) next(ctx context.Context, wait time.Duration) (*Pulled, error) { func (p *Puller) next(ctx context.Context, wait time.Duration) (*Pulled, error) {
p.mu.Lock()
declined := p.declined
p.mu.Unlock()
for _, kind := range delivery.AskKinds { for _, kind := range delivery.AskKinds {
if declined[kind] {
continue
}
msg, err := p.Source.Next(ctx, kind, 0) msg, err := p.Source.Next(ctx, kind, 0)
if err != nil || msg != nil { if err != nil || msg != nil {
return withKind(msg, kind), err return withKind(msg, kind), err
} }
} }
msg, err := p.Source.Next(ctx, delivery.AskDelivery, wait) msg, err := p.Source.Next(ctx, delivery.AskDelivery, wait)
p.mu.Lock()
p.declined = nil
p.mu.Unlock()
return withKind(msg, delivery.AskDelivery), err return withKind(msg, delivery.AskDelivery), err
} }
// decline ends the step of an ask this node lacks a need of, refused as declined with what it lacks, and drops the
// message: the asker puts the ask again under a new id (issue 890). A message whose decline cannot be written is handed
// back, as one whose start cannot be.
func (p *Puller) decline(msg *Pulled, ask delivery.StepAsk, lacks []string) {
step, key, err := begin(p.Put, p.Node, ask.Kind+" ask", ask.ID, AskWhat(ask))
if err == nil {
why := Refuse("the ask %s needs %s, and %s lacks %s: it is left to a node that has it", ask.ID,
strings.Join(ask.Needs, ", "), p.Node, strings.Join(lacks, ", "))
step.Ended = now().UTC()
step.State, step.Reason = delivery.StepRefused, why.Error()
step.Result, err = json.Marshal(delivery.StepDeclined{Declined: true, Needs: ask.Needs, Lacks: lacks})
if err == nil {
err = writeEnd(p.Put, key, step)
}
}
if err != nil {
p.logf("the ask %s cannot be declined on %s, so it is given back: %v", ask.ID, p.Node, err)
_ = msg.Nak()
return
}
p.logf("the ask %s is declined on %s: it needs %s, and this node lacks %s", ask.ID, p.Node, strings.Join(ask.Needs, ", "), strings.Join(lacks, ", "))
_ = msg.Term()
p.mu.Lock()
if p.declined == nil {
p.declined = map[string]bool{}
}
p.declined[msg.Kind] = true
p.mu.Unlock()
}
func withKind(msg *Pulled, kind string) *Pulled { func withKind(msg *Pulled, kind string) *Pulled {
if msg != nil { if msg != nil {
msg.Kind = kind msg.Kind = kind
@@ -314,7 +503,11 @@ func (p *Puller) take(ctx context.Context, msg *Pulled) {
_ = msg.Term() _ = msg.Term()
return return
} }
step, key, err := begin(p.Put, p.Node, ask.Kind+" ask", ask.ID) if lacks := delivery.Lacks(ask.Needs, p.Has); len(lacks) > 0 {
p.decline(msg, ask, lacks)
return
}
step, key, err := begin(p.Put, p.Node, ask.Kind+" ask", ask.ID, AskWhat(ask))
if err != nil { if err != nil {
p.logf("the ask %s cannot start on %s: %v", ask.ID, p.Node, err) p.logf("the ask %s cannot start on %s: %v", ask.ID, p.Node, err)
_ = msg.Nak() _ = msg.Nak()
@@ -340,6 +533,7 @@ func (p *Puller) take(ctx context.Context, msg *Pulled) {
} }
p.running[ask.ID] = RunningAsk{ID: ask.ID, Kind: ask.Kind, Module: module, Started: step.Started} p.running[ask.ID] = RunningAsk{ID: ask.ID, Kind: ask.Kind, Module: module, Started: step.Started}
p.mu.Unlock() p.mu.Unlock()
p.logf("the %s ask %s starts on %s: %s", ask.Kind, ask.ID, p.Node, step.What)
go p.run(ctx, msg, ask, key, step) go p.run(ctx, msg, ask, key, step)
} }
@@ -370,7 +564,11 @@ func (p *Puller) run(ctx context.Context, msg *Pulled, ask delivery.StepAsk, key
} }
} }
}() }()
result, err := protect(ctx, func(ctx context.Context) (any, error) { return p.Run(ctx, ask) }) ph := &phases{put: p.Put, key: key, step: step, logf: p.Logf, prefix: fmt.Sprintf("the %s ask %s on %s", ask.Kind, ask.ID, p.Node)}
result, err := protect(ctx, func(ctx context.Context) (any, error) {
return p.Run(context.WithValue(ctx, phaseKey{}, ph), ask)
})
step = ph.end() // its end keeps the phase it ended in
close(done) close(done)
if ctx.Err() != nil { if ctx.Err() != nil {
// The process stops: the ask is neither ended nor acknowledged here, so the stream gives it to a node that runs. // The process stops: the ask is neither ended nor acknowledged here, so the stream gives it to a node that runs.
@@ -383,6 +581,15 @@ func (p *Puller) run(ctx context.Context, msg *Pulled, ask delivery.StepAsk, key
_ = msg.Nak() _ = msg.Nak()
return return
} }
outcome := "passed"
if err != nil {
outcome = "did not pass: " + err.Error()
if len(outcome) > 300 {
outcome = outcome[:300] + "..."
}
}
p.logf("the %s ask %s ended on %s after %s, in its phase %q; %s", ask.Kind, ask.ID, p.Node,
now().Sub(step.Started).Round(time.Second), step.Phase, outcome)
if aerr := msg.Ack(); aerr != nil { if aerr := msg.Ack(); aerr != nil {
p.logf("the ask %s ended on %s and its acknowledgement failed: %v", ask.ID, p.Node, aerr) p.logf("the ask %s ended on %s and its acknowledgement failed: %v", ask.ID, p.Node, aerr)
} }
@@ -419,6 +626,42 @@ func (p *Puller) watchCapacity() {
} }
} }
// watchHours follows the module's config value `build-hours` when WatchConfig and BuildHours are set, as watchCapacity
// follows `capacity`: each value handed is kept, used in place of BuildHours, and wakes the puller. A value that cannot
// be read is no window, said; a watch that cannot start leaves the puller on BuildHours, said once.
func (p *Puller) watchHours() {
if p.WatchConfig == nil || p.BuildHours == nil {
return
}
err := p.WatchConfig(BuildHoursSetting, func(c stdio.ProvisionChange) error {
setting := ""
var w *BuildWindow
var why string
switch {
case c.Refused != "":
why = "its build hours are no longer answered: " + c.Refused
case len(c.Value) > 0 && string(c.Value) != "null" && json.Unmarshal(c.Value, &setting) != nil:
why = fmt.Sprintf("the setting %s %s is not a string", BuildHoursSetting, c.Value)
default:
var err error
if w, err = ParseBuildHours(setting, p.Node); err != nil {
why = err.Error()
}
}
p.mu.Lock()
p.hours, p.hoursFault, p.hasHours = w, why, true
p.mu.Unlock()
if why != "" {
p.logf("the %s puller on %s builds at any hour: %s", p.Language, p.Node, why)
}
p.Wake()
return nil
})
if err != nil {
p.logf("the %s build hours of %s cannot be followed on the bus, so they are read from the settings file: %v", p.Language, p.Node, err)
}
}
// WatchConfigServed is stdio.WatchConfig once Serve has the runtime's channel (stdio.Served): a build module starts its // WatchConfigServed is stdio.WatchConfig once Serve has the runtime's channel (stdio.Served): a build module starts its
// pull before it serves, and a watch asked before that would be refused. // pull before it serves, and a watch asked before that would be refused.
func WatchConfigServed(key string, handle func(stdio.ProvisionChange) error) error { func WatchConfigServed(key string, handle func(stdio.ProvisionChange) error) error {
+81 -7
View File
@@ -104,20 +104,38 @@ func TestPullTakesDeliveryBeforeCheck(t *testing.T) {
} }
// 1887/A2a, 1887/AC-pull: given capacity 2 and three asks waiting, two run and the third waits, and it starts when // 1887/A2a, 1887/AC-pull: given capacity 2 and three asks waiting, two run and the third waits, and it starts when
// one ends, with no timer. // one ends, with no timer. Issue 1806's follow-up: the 2 is the CPU's, a configured 3 bounded by a 12-core slice at 6
// cores a step, said once in capacity.<node>; and issue 1890: a run's phase is written to its step key and logged with
// its ask id, and its end keeps the last phase.
func TestPullHoldsAtMostItsCapacity(t *testing.T) { func TestPullHoldsAtMostItsCapacity(t *testing.T) {
src, st := NewFakeAsks(), newSteps() src, st := NewFakeAsks(), newSteps()
asks := []delivery.StepAsk{}
for i := 0; i < 3; i++ { for i := 0; i < 3; i++ {
_ = src.PublishAsk(testAsk(delivery.AskDelivery)) a := testAsk(delivery.AskDelivery)
asks = append(asks, a)
_ = src.PublishAsk(a)
} }
release := make(chan struct{}) release := make(chan struct{})
var mu sync.Mutex var mu sync.Mutex
started := 0 started := 0
p := &Puller{Source: src, Language: delivery.AskGo, Node: "shanks", Put: st.put, Capacity: func() int { return 2 }, var said []string
capacities := map[string][]byte{}
quota := 1200
p := &Puller{Source: src, Language: delivery.AskGo, Node: "shanks", Put: st.put, Capacity: func() int { return 3 },
CoresPerStep: func() float64 { return 6 },
Quota: func() (int, string) { mu.Lock(); defer mu.Unlock(); return quota, "a test's slice" },
PausePut: func(key string, v any) (uint64, error) {
mu.Lock()
capacities[key] = v.(json.RawMessage)
mu.Unlock()
return 1, nil
},
Logf: func(f string, a ...any) { mu.Lock(); said = append(said, fmt.Sprintf(f, a...)); mu.Unlock() },
PullWait: 10 * time.Millisecond, Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { PullWait: 10 * time.Millisecond, Run: func(ctx context.Context, a delivery.StepAsk) (any, error) {
mu.Lock() mu.Lock()
started++ started++
mu.Unlock() mu.Unlock()
Phase(ctx, "unit: tests of 3 packages")
<-release <-release
return nil, nil return nil, nil
}} }}
@@ -129,31 +147,87 @@ func TestPullHoldsAtMostItsCapacity(t *testing.T) {
if p.InHand() != 2 || src.Waiting(delivery.AskDelivery) != 1 { if p.InHand() != 2 || src.Waiting(delivery.AskDelivery) != 1 {
t.Fatalf("in hand %d, waiting %d; want 2 and 1", p.InHand(), src.Waiting(delivery.AskDelivery)) t.Fatalf("in hand %d, waiting %d; want 2 and 1", p.InHand(), src.Waiting(delivery.AskDelivery))
} }
c := p.CapacityDetail()
if c.Effective != 2 || c.Configured != 3 || c.ByCPU != 2 || c.QuotaCores != 12 || !strings.Contains(c.Why, "bounded by the CPU") {
t.Fatalf("the capacity judged by = %+v, want 2 of a configured 3 bounded by 12 cores / 6", c)
}
mu.Lock()
raw := capacities["capacity.shanks"]
capLines := 0
for _, l := range said {
if strings.Contains(l, "build capacity of shanks is 2, bounded by the CPU") {
capLines++
}
}
mu.Unlock()
if got, err := delivery.DecodeCapacity(raw); err != nil || got.Effective != 2 || capLines != 1 {
t.Fatalf("capacity.shanks = %s (%v), said %d times; want effective 2, said once", raw, err, capLines)
}
waitFor(t, "a phase on the step key", func() bool {
key, _ := delivery.AskStepKey("shanks", p.Running()[0].ID)
s, _ := st.get(key)
return s.State == delivery.StepRunning && s.Phase == "unit: tests of 3 packages" && !s.PhaseAt.IsZero()
})
release <- struct{}{} release <- struct{}{}
waitFor(t, "the third taken", func() bool { return src.Waiting(delivery.AskDelivery) == 0 }) waitFor(t, "the third taken", func() bool { return src.Waiting(delivery.AskDelivery) == 0 })
// Taken from the source comes a moment before its run starts: wait for the start, not only the take. // Taken from the source comes a moment before its run starts: wait for the start, not only the take.
waitFor(t, "the third started", func() bool { mu.Lock(); defer mu.Unlock(); return started == 3 }) waitFor(t, "the third started", func() bool { mu.Lock(); defer mu.Unlock(); return started == 3 })
close(release) close(release)
waitFor(t, "all acked", func() bool { a, _, _ := src.Fates(); return len(a) == 3 }) waitFor(t, "all acked", func() bool { a, _, _ := src.Fates(); return len(a) == 3 })
for _, a := range asks {
key, _ := delivery.AskStepKey("shanks", a.ID)
if s, _ := st.get(key); s.State != delivery.StepPassed || s.Phase != "unit: tests of 3 packages" {
t.Errorf("the end of %s = %+v, want passed in its last phase", a.ID, s)
}
}
mu.Lock()
logged := strings.Join(said, "\n")
quota = 600 // the cap lowered: one step at 6 cores, followed at the next judgement
mu.Unlock()
if !strings.Contains(logged, "the delivery ask "+asks[0].ID+" on shanks: unit: tests of 3 packages") {
t.Errorf("no journal line names the ask's phase: %s", logged)
}
if c := p.CapacityDetail(); c.Effective != 1 {
t.Errorf("after the quota fell to 6 cores, capacity %+v, want 1", c)
}
} }
// An ask that is not one, or not this seat's, is dropped and runs nowhere. // An ask that is not one, or not this seat's, is dropped and runs nowhere. And issue 890/AC1: a check ask needing
// node-build-node is declined on a node that lacks it (its step refused as declined, naming what it lacks, the message
// dropped for the asker to put again), and runs on a node that has it.
func TestPullDropsWhatIsNoAsk(t *testing.T) { func TestPullDropsWhatIsNoAsk(t *testing.T) {
src, st := NewFakeAsks(), newSteps() src, st := NewFakeAsks(), newSteps()
src.Publish(delivery.AskDelivery, []byte(`{"v":1}`)) src.Publish(delivery.AskDelivery, []byte(`{"v":1}`))
other := testAsk(delivery.AskDelivery) other := testAsk(delivery.AskDelivery)
other.Language = delivery.AskNode other.Language = delivery.AskNode
_ = src.PublishAsk(other) _ = src.PublishAsk(other)
placed := testAsk(delivery.AskCheck)
placed.Needs = []string{delivery.NeedNodeBuildNode}
if err := src.PublishAsk(placed); err != nil {
t.Fatal(err)
}
ran := false ran := false
p := &Puller{Source: src, Language: delivery.AskGo, Node: "shanks", Put: st.put, Capacity: func() int { return 1 }, p := &Puller{Source: src, Language: delivery.AskGo, Node: "ace", Put: st.put, Capacity: func() int { return 1 },
PullWait: 10 * time.Millisecond, Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { ran = true; return nil, nil }} PullWait: 10 * time.Millisecond, Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { ran = true; return nil, nil }}
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
defer cancel() defer cancel()
go p.Pull(ctx) go p.Pull(ctx)
waitFor(t, "two termed", func() bool { _, tm, _ := src.Fates(); return len(tm) == 2 }) waitFor(t, "three termed", func() bool { _, tm, _ := src.Fates(); return len(tm) == 3 })
if ran { if ran {
t.Fatal("a wrong ask ran") t.Fatal("a wrong or a declined ask ran")
} }
key, _ := delivery.AskStepKey("ace", placed.ID)
s, _ := st.get(key)
if d, ok := delivery.DeclinedOf(s); !ok || len(d.Lacks) != 1 || d.Lacks[0] != delivery.NeedNodeBuildNode || !strings.Contains(s.Reason, "lacks node-build-node") {
t.Fatalf("the declined step = %+v", s)
}
has, hasSteps := NewFakeAsks(), newSteps()
_ = has.PublishAsk(placed)
q := &Puller{Source: has, Language: delivery.AskGo, Node: "g14", Put: hasSteps.put, Capacity: func() int { return 1 },
PullWait: 10 * time.Millisecond, Has: func(n string) bool { return n == delivery.NeedNodeBuildNode },
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return nil, nil }}
go q.Pull(ctx)
waitFor(t, "the placed ask acked where its need is met", func() bool { a, _, _ := has.Fates(); return len(a) == 1 })
} }
// load reads the recorded steps state's raw values: the pause key is not a step, so it is kept apart. // load reads the recorded steps state's raw values: the pause key is not a step, so it is kept apart.
+5 -2
View File
@@ -14,10 +14,13 @@ func SeatTools(seat, what string, p *Puller) []stdio.Tool {
Run: func(map[string]any) (any, error) { return run() }} Run: func(map[string]any) (any, error) { return run() }}
} }
current := func() (any, error) { current := func() (any, error) {
return map[string]any{"node": p.Node, "capacity": p.CapacityNow(), "running": p.Running(), "paused": p.Paused()}, nil c := p.CapacityDetail()
return map[string]any{"node": p.Node, "capacity": c.Effective, "capacity_why": c.Why, "capacity_detail": c,
"running": p.Running(), "paused": p.Paused()}, nil
} }
return []stdio.Tool{ return []stdio.Tool{
verb("current", "The "+what+" asks this node runs now (each one's id, kind and when it started), its build capacity, "+ verb("current", "The "+what+" asks this node runs now (each one's id, kind and when it started), its build capacity (the configured one, "+
"bounded by the cores its node's build slice gives over the cores a step needs) and why, "+
"and whether it is paused.", current), "and whether it is paused.", current),
verb("pause", "Take no new "+what+" ask on this node until resumed; an ask running here ends in its place. Kept "+ verb("pause", "Take no new "+what+" ask on this node until resumed; an ask running here ends in its place. Kept "+
"across a restart (the node's pause key).", func() (any, error) { "across a restart (the node's pause key).", func() (any, error) {
+272
View File
@@ -0,0 +1,272 @@
package buildstep
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A step left running** (novox/hq issue 1890, note 5). A build module that stops while a step runs leaves its key in
// state running for good: nothing else writes that key's end, and the stream's redelivery only rewrites it when the same
// ask comes back. At its start a module ends the steps its node left running before the process began (EndStale), and
// its step-end tool ends one by hand (EndOne).
// RestartReason is why a step a previous process left running is ended.
const RestartReason = "the build module restarted while it ran"
// Boot is when this process began. A step started at or after it belongs to this process, which may still run it, so
// EndOne refuses it.
var Boot = time.Now()
// Scan reads and writes a steps state with its revisions: its keys, a key's stored value and revision (nil and 0 for
// none), and a write that lands only while the key is still at the revision read (KeptState.Update), failing with
// stdio.ErrChanged when another writer was first.
type Scan struct {
Keys func() ([]string, error)
Get func(key string) (json.RawMessage, uint64, error)
Update func(key string, value any, revision uint64) (uint64, error)
}
// ScanOf is the Scan of a state the process keeps.
func ScanOf(s stdio.KeptState) Scan {
return Scan{Keys: s.Keys, Update: s.Update, Get: func(key string) (json.RawMessage, uint64, error) {
e, err := s.Get(key)
if err != nil || e == nil {
return nil, 0, err
}
return e.Value, e.Revision, nil
}}
}
// isAsk says a step is a pulled ask's: its key is <node>.<ask id> and its tool is "<kind> ask" (a hand-started step
// has the tool's name). An ask's key is the stream's to settle, never a restart's: the puller leaves an ask it was
// running unacknowledged so the stream redelivers it, and the asker's wait takes the first end it reads on the key.
// Only once the asker's bound has passed (delivery.StepBound) is the key nobody's, and then it is ended.
func isAsk(key string, s delivery.Step) bool {
_, _, err := delivery.ParseAskStepKey(key)
return err == nil && strings.HasSuffix(s.Tool, " ask")
}
// AskWhat says what an ask is, in one line: its kind, and its repository, head and levels, or its module and
// fingerprint.
func AskWhat(a delivery.StepAsk) string {
short := func(s string) string {
if len(s) > 12 {
return s[:12]
}
return s
}
str := func(m map[string]any, k string) string { v, _ := m[k].(string); return v }
parts := []string{a.Kind + " ask"}
switch {
case a.Check != nil:
parts = append(parts, str(a.Check.Fetch, "repository")+"@"+short(str(a.Check.Fetch, "commit")), "levels "+strings.Join(a.Check.Levels, ","))
if a.Check.Run != "" {
parts = append(parts, "run "+a.Check.Run)
}
case a.Build != nil:
parts = append(parts, "module "+a.Build.Module)
if r := str(a.Build.Fetch, "repository"); r != "" {
parts = append(parts, r+"@"+short(str(a.Build.Fetch, "commit")))
}
if a.Build.FP != "" {
parts = append(parts, "fp "+short(a.Build.FP))
}
case a.Compose != nil:
parts = append(parts, "compose", str(a.Compose.Fetch, "repository")+"@"+short(str(a.Compose.Fetch, "commit")))
}
return strings.Join(parts, ", ")
}
// endRunning writes step ended failed with why, only while its key is still at revision.
func endRunning(scan Scan, key string, step delivery.Step, revision uint64, why string) error {
step.State, step.Reason, step.Ended, step.Result = delivery.StepFailed, why, now().UTC(), nil
raw, err := delivery.EncodeStep(step)
if err != nil {
return err
}
_, err = scan.Update(key, json.RawMessage(raw), revision)
return err
}
// EndStale ends this node's steps that are running and started before `before` (the start of this process) as failed
// with RestartReason, and answers their ids. Only keys of node are read; a value that is no step is left alone. A step
// of a pulled ask is left to the stream's redelivery (isAsk) until its start is older than delivery.StepBound: its asker
// stopped waiting then, whether the stream gave it to another node or gave it up, and it is ended as failed, "past the
// step bound: its asker stopped waiting at <time>". A key written by another writer since it was read
// (a redelivered ask's start, a real end) is skipped.
func EndStale(scan Scan, node string, before time.Time, logf func(string, ...any)) ([]string, error) {
keys, err := scan.Keys()
if err != nil {
return nil, fmt.Errorf("buildstep: listing the steps of %s: %w", node, err)
}
var ended []string
for _, k := range keys {
n, _, perr := delivery.ParseStepKey(k)
if perr != nil || n != node {
continue
}
raw, rev, err := scan.Get(k)
if err != nil {
return ended, fmt.Errorf("buildstep: reading the step %s: %w", k, err)
}
if len(raw) == 0 {
continue
}
step, err := delivery.DecodeStep(raw)
if err != nil || step.State != delivery.StepRunning {
continue
}
why := RestartReason
if isAsk(k, step) {
if !step.Started.Add(delivery.StepBound).Before(now()) {
continue
}
why = "past the step bound: its asker stopped waiting at " + step.Started.Add(delivery.StepBound).UTC().Format("2006-01-02T15:04:05Z")
} else if !step.Started.Before(before) {
continue
}
if err := endRunning(scan, k, step, rev, why); err != nil {
if errors.Is(err, stdio.ErrChanged) {
continue
}
return ended, fmt.Errorf("buildstep: ending the step %s: %w", k, err)
}
if logf != nil {
logf("the step %s (%s) was left running and is ended: %s", k, step.Tool, why)
}
ended = append(ended, step.ID)
}
return ended, nil
}
// EndOne ends this node's running step id (or the key <node>.<id>) as failed with why. A step that is not running is
// refused, and so is one started since this process began (Boot: it may still run, whether a pulled ask or a step a
// tool started), one live reports as held, and one whose key changed since it was read.
func EndOne(scan Scan, node, id, why string, live func(id string) bool) (delivery.Step, error) {
id = strings.TrimSpace(id)
if n, rest, ok := strings.Cut(id, "."); ok {
if n != node {
return delivery.Step{}, fmt.Errorf("the step %q is of node %q, this module runs on %q", id, n, node)
}
id = rest
}
if strings.TrimSpace(why) == "" {
return delivery.Step{}, fmt.Errorf("say why the step is ended (why)")
}
key, err := delivery.StepKey(node, id)
if err != nil {
return delivery.Step{}, err
}
raw, rev, err := scan.Get(key)
if err != nil {
return delivery.Step{}, err
}
if len(raw) == 0 {
return delivery.Step{}, fmt.Errorf("no step %s in the steps state", key)
}
step, err := delivery.DecodeStep(raw)
if err != nil {
return delivery.Step{}, err
}
if step.State != delivery.StepRunning {
return delivery.Step{}, fmt.Errorf("the step %s is %s, not running", key, step.State)
}
if !step.Started.Before(Boot) || (live != nil && live(id)) {
return delivery.Step{}, fmt.Errorf("the step %s was started by this process and may run now; only a step a previous process left is ended", key)
}
if err := endRunning(scan, key, step, rev, why); err != nil {
if errors.Is(err, stdio.ErrChanged) {
return delivery.Step{}, fmt.Errorf("the step %s changed while it was being ended; read it again", key)
}
return delivery.Step{}, err
}
step.State, step.Reason = delivery.StepFailed, why
return step, nil
}
// StepEndTool is the module's step-end tool, named name (build_go_step_end): it ends one step left running, with a
// reason, refusing one this process started. p may be nil.
func StepEndTool(name string, node string, scan Scan, p *Puller) stdio.Tool {
return stdio.Tool{Name: name,
Description: "End one step of this machine that is stuck in state running (for instance left by a restart of the module) " +
"as failed, with the reason given; its asker then sees it over, so ending an ask still in its stream fails its asker. " +
"A step not running, or one this process started, is refused. " +
"id is the step id or the key <node>.<id>.",
Input: map[string]any{"type": "object", "required": []string{"id", "why"}, "properties": map[string]any{
"id": map[string]any{"type": "string", "description": "the step id, or the key <node>.<id>"},
"why": map[string]any{"type": "string", "description": "why it is ended; it becomes the step's reason"}}},
Run: func(args map[string]any) (any, error) {
id, _ := args["id"].(string)
why, _ := args["why"].(string)
var live func(string) bool
if p != nil {
live = func(id string) bool {
for _, r := range p.Running() {
if r.ID == id {
return true
}
}
return false
}
}
return EndOne(scan, node, id, why, live)
}}
}
// StaleAttempts and StaleRetry bound EndStaleAtStart's tries: the steps state is readable once the runtime serves.
var (
StaleAttempts = 6
StaleRetry = 5 * time.Second
)
// EndStaleAtStart is what a build module runs once at its start, in a goroutine: it waits for the runtime to serve,
// then ends this node's steps left running before `before` (EndStale), trying again when the state cannot be read yet.
// A step this process starts is never one of them: it starts after `before`.
func EndStaleAtStart(ctx context.Context, scan Scan, node string, before time.Time, logf func(string, ...any)) {
select {
case <-ctx.Done():
return
case <-stdio.Served():
}
for attempt := 1; ; attempt++ {
ended, err := EndStale(scan, node, before, logf)
if err == nil {
if logf != nil {
logf("%d steps of %s left running by a previous process ended", len(ended), node)
}
return
}
if logf != nil {
logf("ending the steps left running (try %d of %d): %v", attempt, StaleAttempts, err)
}
if attempt >= StaleAttempts {
return
}
select {
case <-ctx.Done():
return
case <-time.After(StaleRetry):
}
}
}
// EndStaleRound is EndStale as the module's periodic round runs it: at once, and only when the runtime serves (a round
// before that is skipped, the start's EndStaleAtStart and the next round cover it). It ends the steps a previous
// process left and the asks past delivery.StepBound.
func EndStaleRound(scan Scan, node string, logf func(string, ...any)) {
select {
case <-stdio.Served():
default:
return
}
if _, err := EndStale(scan, node, Boot, logf); err != nil && logf != nil {
logf("ending the steps left running: %v", err)
}
}
+131
View File
@@ -0,0 +1,131 @@
package buildstep
import (
"encoding/json"
stdio "git.novox.be/novox/mesh-sdk/go"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// scanOf is the steps a keys double holds, last value per key, its revision the count of its puts; a write through
// Update lands only at the revision read, as the state's does. race, when set, runs once before the next Update's check.
func scanOf(k *keys, race func(key string)) Scan {
return Scan{
Keys: func() ([]string, error) {
k.mu.Lock()
defer k.mu.Unlock()
var out []string
for key := range k.puts {
out = append(out, key)
}
return out, nil
},
Get: func(key string) (json.RawMessage, uint64, error) {
k.mu.Lock()
defer k.mu.Unlock()
s, ok := k.puts[key]
if !ok {
return nil, 0, nil
}
raw, err := delivery.EncodeStep(s[len(s)-1])
return raw, uint64(len(s)), err
},
Update: func(key string, value any, revision uint64) (uint64, error) {
if race != nil {
race(key)
}
k.mu.Lock()
n := uint64(len(k.puts[key]))
k.mu.Unlock()
if n != revision {
return 0, stdio.ErrChanged
}
return k.put(key, value)
},
}
}
// 1890: EndStale ends only this node's hand-started steps left running before the process began and the asks past the
// step bound (a younger ask is the stream's to redeliver), and never a key written meanwhile; EndOne ends one by hand, never one this process started.
func TestEndStaleAndEndOne(t *testing.T) {
k := newKeys()
boot := time.Now()
oldBoot := Boot
Boot = boot
defer func() { Boot = oldBoot }()
put := func(node, id, tool, state string, started time.Time) {
key, _ := delivery.StepKey(node, id)
_, _ = k.put(key, json.RawMessage(mustStep(t, delivery.Step{ID: id, Tool: tool, Node: node, State: state, Started: started})))
}
put("ace", "old1", "build_go_fetch", delivery.StepRunning, boot.Add(-time.Hour))
put("ace", "mjk3l9x0abcdef", "check ask", delivery.StepRunning, boot.Add(-time.Hour))
put("ace", "mjk3l9x0abcdee", "check ask", delivery.StepRunning, time.Now().Add(-5*time.Minute))
put("ace", "raced1", "build_go_test", delivery.StepRunning, boot.Add(-time.Hour))
put("ace", "new1", "build_go_fetch", delivery.StepRunning, boot.Add(time.Second))
put("ace", "done1", "build_go_fetch", delivery.StepPassed, boot.Add(-time.Hour))
put("g14", "other1", "build_go_fetch", delivery.StepRunning, boot.Add(-time.Hour))
// raced1 is ended by its own run between the read and the write.
raced := false
scan := scanOf(k, func(key string) {
if key == "ace.raced1" && !raced {
raced = true
put("ace", "raced1", "build_go_test", delivery.StepPassed, boot.Add(-time.Hour))
}
})
ended, err := EndStale(scan, "ace", boot, nil)
if err != nil || len(ended) != 2 || ended[0] == ended[1] {
t.Fatalf("ended %v, %v", ended, err)
}
if s := k.last("ace.old1"); s.State != delivery.StepFailed || s.Reason != RestartReason || s.Ended.IsZero() {
t.Errorf("old1 = %+v", s)
}
if s := k.last("ace.mjk3l9x0abcdef"); s.State != delivery.StepFailed || !strings.HasPrefix(s.Reason, "past the step bound: its asker stopped waiting at ") {
t.Errorf("the old ask = %+v", s)
}
for key, state := range map[string]string{"ace.mjk3l9x0abcdee": delivery.StepRunning, "ace.raced1": delivery.StepPassed, "ace.new1": delivery.StepRunning, "ace.done1": delivery.StepPassed, "g14.other1": delivery.StepRunning} {
if s := k.last(key); s.State != state {
t.Errorf("%s = %s, want %s", key, s.State, state)
}
}
// EndOne: refuses without why, a held step, one started by this process (a pulled ask or a step tool's), a step not
// running and another node's; ends a step a previous process left.
for _, c := range []struct{ id, why string }{{"old1", ""}, {"new1", "stuck"}, {"done1", "x"}, {"g14.other1", "x"}} {
if _, err := EndOne(scan, "ace", c.id, c.why, nil); err == nil {
t.Errorf("ended %s with why %q", c.id, c.why)
}
}
put("ace", "old2", "build_go_fetch", delivery.StepRunning, boot.Add(-time.Hour))
if _, err := EndOne(scan, "ace", "old2", "x", func(string) bool { return true }); err == nil {
t.Error("ended a held step")
}
if _, err := EndOne(scan, "ace", "ace.old2", "stuck for good", nil); err != nil {
t.Fatal(err)
}
if s := k.last("ace.old2"); s.State != delivery.StepFailed || s.Reason != "stuck for good" {
t.Errorf("old2 = %+v", s)
}
}
func mustStep(t *testing.T, s delivery.Step) []byte {
t.Helper()
raw, err := delivery.EncodeStep(s)
if err != nil {
t.Fatal(err)
}
return raw
}
// 1890: AskWhat names the repository, head and levels of a check ask and the module of a build ask.
func TestAskWhat(t *testing.T) {
check := delivery.StepAsk{Kind: "check", Check: &delivery.CheckJob{Fetch: map[string]any{"repository": "novox/mesh-sdk", "commit": "0123456789abcdef0123"}, Levels: []string{"unit", "integration"}}}
if got, want := AskWhat(check), "check ask, novox/mesh-sdk@0123456789ab, levels unit,integration"; got != want {
t.Errorf("check = %q", got)
}
build := delivery.StepAsk{Kind: "delivery", Build: &delivery.BuildJob{Module: "mesh-x", FP: "abcdef0123456789"}}
if got, want := AskWhat(build), "delivery ask, module mesh-x, fp abcdef012345"; got != want {
t.Errorf("build = %q", got)
}
}
+3 -3
View File
@@ -43,7 +43,7 @@ func NewID() string {
// the error's text, or refused when the error is a *Refusal; a panic in run ends the step failed, naming it, and // the error's text, or refused when the error is a *Refusal; a panic in run ends the step failed, naming it, and
// the process goes on. If the running write fails, nothing was started and Start answers the error. // the process goes on. If the running write fails, nothing was started and Start answers the error.
func Start(ctx context.Context, put Put, node, tool string, run func(ctx context.Context) (result any, err error)) (string, error) { func Start(ctx context.Context, put Put, node, tool string, run func(ctx context.Context) (result any, err error)) (string, error) {
step, key, err := begin(put, node, tool, NewID()) step, key, err := begin(put, node, tool, NewID(), "")
if err != nil { if err != nil {
return "", err return "", err
} }
@@ -52,8 +52,8 @@ func Start(ctx context.Context, put Put, node, tool string, run func(ctx context
} }
// begin writes the step running under id. // begin writes the step running under id.
func begin(put Put, node, tool, id string) (delivery.Step, string, error) { func begin(put Put, node, tool, id, what string) (delivery.Step, string, error) {
step := delivery.Step{ID: id, Tool: tool, Node: node, State: delivery.StepRunning, Started: now().UTC()} step := delivery.Step{ID: id, Tool: tool, Node: node, State: delivery.StepRunning, Started: now().UTC(), What: what}
key, err := delivery.StepKey(node, step.ID) key, err := delivery.StepKey(node, step.ID)
if err != nil { if err != nil {
return delivery.Step{}, "", err return delivery.Step{}, "", err
+3 -3
View File
@@ -36,7 +36,7 @@ var lockFiles = []string{"go.sum", "package-lock.json"}
func ModuleTrees(ctx context.Context, run forgekey.Runner, env []string, dir, commit string) (map[string]ModuleTree, error) { func ModuleTrees(ctx context.Context, run forgekey.Runner, env []string, dir, commit string) (map[string]ModuleTree, error) {
r := run(ctx, env, "git", "-C", dir, "ls-files", "--", "module.json", "*/module.json") r := run(ctx, env, "git", "-C", dir, "ls-files", "--", "module.json", "*/module.json")
if r.Status != 0 || r.Err != "" { if r.Status != 0 || r.Err != "" {
return nil, fmt.Errorf("git ls-files: %s", firstLine(r.Err+" "+r.Stderr)) return nil, fmt.Errorf("git ls-files: %s", forgekey.Said(r.Err+"\n"+r.Stderr))
} }
trees := map[string]ModuleTree{} trees := map[string]ModuleTree{}
for _, l := range strings.Split(r.Stdout, "\n") { for _, l := range strings.Split(r.Stdout, "\n") {
@@ -50,7 +50,7 @@ func ModuleTrees(ctx context.Context, run forgekey.Runner, env []string, dir, co
} }
t := run(ctx, env, "git", "-C", dir, "rev-parse", spec) t := run(ctx, env, "git", "-C", dir, "rev-parse", spec)
if t.Status != 0 || t.Err != "" { if t.Status != 0 || t.Err != "" {
return nil, fmt.Errorf("git rev-parse %s: %s", spec, firstLine(t.Err+" "+t.Stderr)) return nil, fmt.Errorf("git rev-parse %s: %s", spec, forgekey.Said(t.Err+"\n"+t.Stderr))
} }
mt := ModuleTree{Tree: strings.TrimSpace(t.Stdout)} mt := ModuleTree{Tree: strings.TrimSpace(t.Stdout)}
raw, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(l))) raw, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(l)))
@@ -93,7 +93,7 @@ func narrow(ctx context.Context, run forgekey.Runner, env []string, dir, commit,
} }
r := run(ctx, env, "git", append([]string{"-C", dir, "ls-tree", "-z", commit, "--"}, full...)...) r := run(ctx, env, "git", append([]string{"-C", dir, "ls-tree", "-z", commit, "--"}, full...)...)
if r.Status != 0 || r.Err != "" { if r.Status != 0 || r.Err != "" {
return fmt.Errorf("git ls-tree %s of the module %s: %s", commit, mdir, firstLine(r.Err+" "+r.Stderr)) return fmt.Errorf("git ls-tree %s of the module %s: %s", commit, mdir, forgekey.Said(r.Err+"\n"+r.Stderr))
} }
objects := map[string]string{} objects := map[string]string{}
for _, l := range strings.Split(r.Stdout, "\x00") { for _, l := range strings.Split(r.Stdout, "\x00") {
+15
View File
@@ -45,10 +45,25 @@ func ControllerAnnouncement(node string) announce.States {
// announced the state readable, or announced nothing yet (the transition); refused when the owner announced and did // announced the state readable, or announced nothing yet (the transition); refused when the owner announced and did
// not announce that state readable. // not announce that state readable.
func ReadsAnnounced(reads []string, announced map[string]announce.States) (granted, refused []string) { func ReadsAnnounced(reads []string, announced map[string]announce.States) (granted, refused []string) {
return ReadsAnnouncedBy("", reads, announced)
}
// ReadsAnnouncedBy is ReadsAnnounced for one reader module, which also honours the readers an owner named for a state
// (announce.State.Readers) and the composer's named readers (NamedReaders, ADR 0547 clause 9): a read named for other
// modules is refused. An empty reader is judged by the readable mark alone.
func ReadsAnnouncedBy(reader string, reads []string, announced map[string]announce.States) (granted, refused []string) {
for _, r := range reads { for _, r := range reads {
owner, name, ok := strings.Cut(r, ".") owner, name, ok := strings.Cut(r, ".")
s, said := announced[owner] s, said := announced[owner]
switch { switch {
case reader != "" && !ReaderMay(reader, r):
refused = append(refused, r)
case reader != "" && ok && said && s.Keeps(name):
if s.ReadableBy(name, reader) {
granted = append(granted, r)
} else {
refused = append(refused, r)
}
case !ok || !said || s.Readable(name): case !ok || !said || s.Readable(name):
granted = append(granted, r) granted = append(granted, r)
default: default:
+160 -7
View File
@@ -5,6 +5,8 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"reflect" "reflect"
"regexp"
"slices"
"strings" "strings"
"testing" "testing"
) )
@@ -13,7 +15,108 @@ import (
// byte what the controller's composes. testdata/controller-user-lists.json was written by the controller's own // byte what the controller's composes. testdata/controller-user-lists.json was written by the controller's own
// internal/broker (mesh-controller branch fix/1945-mesh-grants-publishes-memberships, and fix/1790-grants-create-buckets for the grants part creating declared streams), lists and memberships, from fixtures_test.go compiled there unchanged; composed.conf is the // internal/broker (mesh-controller branch fix/1945-mesh-grants-publishes-memberships, and fix/1790-grants-create-buckets for the grants part creating declared streams), lists and memberships, from fixtures_test.go compiled there unchanged; composed.conf is the
// controller's golden of the whole server configuration. A row that differs is the copy drifting from the controller, // controller's golden of the whole server configuration. A row that differs is the copy drifting from the controller,
// which is a fault while both compose (until the controller loses its lease, ADR 0547 §13). // which is a fault while both compose (until the controller loses its lease, ADR 0547 §13), but for the grants the copy
// gives beyond it on purpose (beyondTheControllers): the copy grants every user everything the controller's grants, and
// only those more.
// createdAdvisory is one subscription to a bucket's creation in a composed list, as stateCreatedHeard composes it.
var createdAdvisory = regexp.MustCompile(`"\$JS\.EVENT\.ADVISORY\.STREAM\.CREATED\.KV_[^"]*", `)
// beyondTheControllers is what the copy grants beyond the controller's on purpose (novox/hq ADR 0547 §2, ADR 0555 §3,
// issue 1126), by user: a node's account reads every node's own facts, `<node>.node` of the have bucket where the node's
// record rides, and what it composes its own modules from (NodeComposeReads). After the take-over (ADR 0547 §8) the
// grants part's list is the one the bus loads; a hand-back loads the controller's, which lacks these reads, and the nodes
// keep applying without composing their own until the next take-over. Each principal's hearing of its buckets' creation
// (issue 1790) is the other row beyond, matched by createdAdvisory.
func beyondTheControllers(user string) map[string]bool {
out := map[string]bool{}
node, ok := strings.CutPrefix(user, "node.")
if !ok {
return out
}
for _, s := range append(readGrants("node-engine_have", "*.node"), NodeComposeReads(node)...) {
out[s] = true
}
return out
}
// mayBeyondTheControllers is what the copy may grant beyond the controller's where the records allow it, and need not
// where they do not: a node's ask of the vault's `own` while the vault serves it (issue 1126, ADR 0550 §5).
func mayBeyondTheControllers(user string) map[string]bool {
if _, ok := strings.CutPrefix(user, "node."); !ok {
return nil
}
return map[string]bool{VaultOwnCall(Principal{Kind: KindNode, Node: strings.TrimPrefix(user, "node.")}): true,
VaultSecretChanged: true}
}
// quotedToken is a quoted token of a composed list.
var quotedToken = regexp.MustCompile(`"[^"]*"`)
// grantsByUser splits a composed list into the text outside its users and each user's quoted tokens (its name, its
// password's hash, every subject it may publish or subscribe to), by user.
func grantsByUser(list string) (string, map[string]map[string]bool) {
var outside strings.Builder
users := map[string]map[string]bool{}
user := ""
for _, line := range strings.SplitAfter(list, "\n") {
trimmed := strings.TrimSpace(line)
if name, ok := strings.CutPrefix(trimmed, `{ user: "`); ok {
user, _, _ = strings.Cut(name, `"`)
users[user] = map[string]bool{}
} else if user != "" && trimmed == "]" {
user = ""
}
if user == "" {
outside.WriteString(line)
continue
}
for _, q := range quotedToken.FindAllString(line, -1) {
users[user][strings.Trim(q, `"`)] = true
}
}
return outside.String(), users
}
// grantsAsTheControllers fails t unless the copy's list is the controller's, user for user, but for the grants it
// names on purpose (beyondTheControllers, the creation advisories): every grant of the controller's is the copy's, and
// the copy's beyond them are exactly those, every one of them there.
func grantsAsTheControllers(t *testing.T, got, want string) {
t.Helper()
gotOutside, gotUsers := grantsByUser(got)
wantOutside, wantUsers := grantsByUser(want)
if gotOutside != wantOutside {
t.Errorf("the list differs from the controller's outside its users:\n%s", gotOutside)
}
for user, was := range wantUsers {
now, ok := gotUsers[user]
if !ok {
t.Errorf("the copy has no user %s", user)
continue
}
intended := beyondTheControllers(user)
for g := range was {
if !now[g] {
t.Errorf("%s: the copy does not grant %s, which the controller's does", user, g)
}
}
for g := range now {
if !was[g] && !intended[g] && !mayBeyondTheControllers(user)[g] && !createdAdvisory.MatchString(`"`+g+`", `) {
t.Errorf("%s: the copy grants %s beyond the controller's, and no decision names it", user, g)
}
}
for g := range intended {
if !now[g] {
t.Errorf("%s: the copy does not grant %s, which a node composing its own modules reads", user, g)
}
}
}
for user := range gotUsers {
if _, ok := wantUsers[user]; !ok {
t.Errorf("the copy has a user %s the controller's has not", user)
}
}
}
func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) { func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("testdata", "controller-user-lists.json")) raw, err := os.ReadFile(filepath.Join("testdata", "controller-user-lists.json"))
if err != nil { if err != nil {
@@ -48,9 +151,11 @@ func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
if strings.Join(missing, ",") != strings.Join(want[i].Missing, ",") { if strings.Join(missing, ",") != strings.Join(want[i].Missing, ",") {
t.Errorf("left out %v, the controller %v", missing, want[i].Missing) t.Errorf("left out %v, the controller %v", missing, want[i].Missing)
} }
if list != want[i].List { // The one row the copy composes beyond the controller's, on purpose (novox/hq issue 1790, 2026-10-26, after
t.Errorf("the list differs from the controller's:\n%s", list) // the take-over of ADR 0547 milestone 2): each principal hears its own and read buckets created, so a state
} // missing at its start is bound when it appears. The controller's composer is frozen and its list no longer
// loaded; while it is not, this row is left out of the comparison and nothing else.
grantsAsTheControllers(t, list, want[i].List)
// And every membership the records issue, as the controller's MembershipFor makes it (ADR 0547 §5): the // And every membership the records issue, as the controller's MembershipFor makes it (ADR 0547 §5): the
// grants part publishes them. // grants part publishes them.
raw, err := json.Marshal(MembershipsFor(f)) raw, err := json.Marshal(MembershipsFor(f))
@@ -69,6 +174,56 @@ func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
} }
}) })
} }
// An asker on a seat pulled by kind hears what that seat's workers gave up on, and no other stream's dead letters
// (novox/hq issue 734: a check ask given up during a bus reload left its try running for ever).
t.Run("an asker reads its seat's dead letters", func(t *testing.T) {
build := Seat{Name: "node-build-go", Scope: "node", Accepts: []string{"delivery", "check", "recheck"}, PulledByKind: true}
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "mesh-delivery-checker", Uses: []Seat{build}})
if err != nil {
t.Fatal(err)
}
for subject, want := range map[string]bool{
"$JS.API.CONSUMER.CREATE.DEAD_LETTERS.x1.mesh.events.dead.SEAT_NODE_BUILD_GO.>": true,
"$JS.API.CONSUMER.CREATE.DEAD_LETTERS.x1.mesh.events.dead.SEAT_NODE_BUILD_GO.w.next": true,
"$JS.API.CONSUMER.DELETE.DEAD_LETTERS.x1": true,
"$JS.API.CONSUMER.MSG.NEXT.DEAD_LETTERS.x1": true,
"$JS.API.CONSUMER.CREATE.DEAD_LETTERS.x1.mesh.events.dead.EVENTS.>": false,
"$JS.API.CONSUMER.CREATE.DEAD_LETTERS.x1": false,
"$JS.API.STREAM.PURGE.DEAD_LETTERS": false,
"$JS.FC.DEAD_LETTERS.x1": false,
} {
if got := MayPublish(perms, subject); got != want {
t.Errorf("may publish %s: %v, want %v", subject, got, want)
}
}
plain, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "shop",
Uses: []Seat{{Name: "telegram-sender", Accepts: []string{"send"}}}})
if err != nil {
t.Fatal(err)
}
if MayPublish(plain, "$JS.API.CONSUMER.DELETE.DEAD_LETTERS.x1") {
t.Error("a user of a seat not pulled by kind reads DEAD_LETTERS")
}
})
// Issue 1126 (ADR 0550 §5): a node's engine asks the vault's `own` in its own name while the vault serves it, and no
// other node's.
t.Run("a node asks the vault its own secrets", func(t *testing.T) {
one := Principal{Kind: KindNode, Node: "one", AsksItsOwnSecrets: true}
perms, err := PermissionsFor(one)
if err != nil {
t.Fatal(err)
}
if !MayPublish(perms, VaultOwnCall(one)) || MayPublish(perms, VaultOwnCall(Principal{Kind: KindNode, Node: "two"})) {
t.Errorf("node one asks own as %v, and as two %v", MayPublish(perms, VaultOwnCall(one)),
MayPublish(perms, VaultOwnCall(Principal{Kind: KindNode, Node: "two"})))
}
if plain, _ := PermissionsFor(Principal{Kind: KindNode, Node: "one"}); MayPublish(plain, VaultOwnCall(one)) {
t.Error("a node asks own while the vault does not serve it")
}
if !slices.Contains(perms.Subscribe, VaultSecretChanged) {
t.Errorf("node one does not hear the vault's secret-changed: %v", perms.Subscribe)
}
})
t.Run("the controller's golden server configuration", func(t *testing.T) { t.Run("the controller's golden server configuration", func(t *testing.T) {
seat := Seat{Name: "telegram-sender", Accepts: []string{"send"}, Emits: []string{"delivered", "failed"}} seat := Seat{Name: "telegram-sender", Accepts: []string{"send"}, Emits: []string{"delivered", "failed"}}
got, err := Compose( got, err := Compose(
@@ -95,8 +250,6 @@ func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if got != string(want) { grantsAsTheControllers(t, got, string(want))
t.Errorf("the configuration differs from the controller's golden:\n%s", got)
}
}) })
} }
+57 -4
View File
@@ -4,16 +4,24 @@ import "testing"
// Defends issue 1790 and novox/hq ADR 0547 §5 (the operator, 2026-10-26 08:54): the grants part may create the stream // Defends issue 1790 and novox/hq ADR 0547 §5 (the operator, 2026-10-26 08:54): the grants part may create the stream
// of each bucket, log and stream a placed module's manifest declares, by its own name, and read it back; nothing // of each bucket, log and stream a placed module's manifest declares, by its own name, and read it back; nothing
// wider, nothing updated or deleted, and no other principal so. Held byte-equal in the controller's copy. // wider, nothing updated or deleted, and no other principal so. Held byte-equal in the controller's copy. And ADR 0547
// clause 7 (issue 1945, the operator's second exception): the grants part writes its own object bucket of sealed user
// lists, and the node-engine of the node holding mesh-broker alone reads its one key and an object of it by name.
func TestTheGrantsPartCreatesOnlyDeclaredStreams(t *testing.T) { func TestTheGrantsPartCreatesOnlyDeclaredStreams(t *testing.T) {
records := Records{ records := Records{
Nodes: []string{"novox"}, Nodes: []string{"novox", "ace"},
Assigned: map[string][]Declared{"novox": { Assigned: map[string][]Declared{"ace": {{Module: "plex"}}, "novox": {
{Module: "nats", SnapshotsTheBus: true},
// mesh-sdk's copy alone (ADR 0547 clause 9, named readers): the vault's hashes are read by the grants part
// alone, whatever another module's reads say; the controller's copy grants any module that declares it.
{Module: "mesh-vault", State: []Bucket{{Module: "mesh-vault", Name: "bus-accounts"}}},
{Module: "spy", Serves: []string{"spy_x"}, Reads: []string{VaultBusAccounts}},
{Module: "mesh-delivery-builder", {Module: "mesh-delivery-builder",
State: []Bucket{{Module: "mesh-delivery-builder", Name: "steps", History: 5}}, State: []Bucket{{Module: "mesh-delivery-builder", Name: "steps", History: 5}},
Logs: []Log{{Module: "mesh-delivery-builder", Name: "builds"}}, Logs: []Log{{Module: "mesh-delivery-builder", Name: "builds"}},
Streams: []ModuleStream{{Module: "mesh-delivery-builder", Name: "BUILDASKS", Events: []string{"ask"}}}}, Streams: []ModuleStream{{Module: "mesh-delivery-builder", Name: "BUILDASKS", Events: []string{"ask"}}}},
{Module: GrantsModule, State: []Bucket{{Module: GrantsModule, Name: "assertions"}}}, {Module: GrantsModule, State: []Bucket{{Module: GrantsModule, Name: "assertions"}}, Reads: []string{VaultBusAccounts}},
{Module: RuntimeModule},
}}, }},
} }
users, err := Users(records) users, err := Users(records)
@@ -22,6 +30,14 @@ func TestTheGrantsPartCreatesOnlyDeclaredStreams(t *testing.T) {
} }
perms := map[string]Permissions{} perms := map[string]Permissions{}
for _, u := range users { for _, u := range users {
if u.Kind == KindNode || u.Kind == KindNodeTools {
p, err := PermissionsFor(u)
if err != nil {
t.Fatalf("%s: %v", u.Username(), err)
}
perms[u.Username()] = p
continue
}
if u.Kind != KindModule { if u.Kind != KindModule {
continue continue
} }
@@ -48,11 +64,48 @@ func TestTheGrantsPartCreatesOnlyDeclaredStreams(t *testing.T) {
{GrantsModule, "$JS.API.STREAM.CREATE.ASSIGNMENTS", false}, {GrantsModule, "$JS.API.STREAM.CREATE.ASSIGNMENTS", false},
{GrantsModule, "$JS.API.STREAM.INFO.EVENTS", false}, {GrantsModule, "$JS.API.STREAM.INFO.EVENTS", false},
{"mesh-delivery-builder", "$JS.API.STREAM.CREATE.KV_mesh-delivery-builder_steps", false}, {"mesh-delivery-builder", "$JS.API.STREAM.CREATE.KV_mesh-delivery-builder_steps", false},
// The sealed user list (ADR 0547 clause 7): the grants part writes its own bucket and no other.
{GrantsModule, "$JS.API.STREAM.CREATE.OBJ_mesh-grants-user-list", true},
{GrantsModule, "$O.mesh-grants-user-list.C.abc", true},
{GrantsModule, "$JS.API.STREAM.PURGE.OBJ_mesh-grants-user-list", true},
{GrantsModule, "$O.declaration-files-novox.C.abc", false},
{GrantsModule, "$JS.API.STREAM.DELETE.OBJ_mesh-grants-user-list", false},
{"mesh-delivery-builder", "$O.mesh-grants-user-list.C.abc", false},
// The bus node's engine reads the one key and an object by name; nothing of the bucket's list, no write.
{"node.novox", "$JS.API.DIRECT.GET.KV_mesh-grants_user-list.$KV.mesh-grants_user-list.sealed", true},
{"node.novox", "$JS.API.CONSUMER.CREATE.KV_mesh-grants_user-list.abc.$KV.mesh-grants_user-list.sealed", true},
{"node.novox", "$JS.API.DIRECT.GET.OBJ_mesh-grants-user-list.$O.mesh-grants-user-list.M.abc", true},
{"node.novox", "$JS.API.CONSUMER.CREATE.OBJ_mesh-grants-user-list.abc.$O.mesh-grants-user-list.C.abc", true},
{"node.novox", "$JS.API.DIRECT.GET.KV_mesh-grants_user-list.$KV.mesh-grants_user-list.list", false},
{"node.novox", "$JS.API.STREAM.INFO.OBJ_mesh-grants-user-list", false},
{"node.novox", "$JS.API.STREAM.MSG.GET.OBJ_mesh-grants-user-list", false},
{"node.novox", "$O.mesh-grants-user-list.C.abc", false},
{"node.ace", "$JS.API.DIRECT.GET.KV_mesh-grants_user-list.$KV.mesh-grants_user-list.sealed", false},
{"node.ace", "$JS.API.DIRECT.GET.OBJ_mesh-grants-user-list.$O.mesh-grants-user-list.M.abc", false},
{GrantsModule, "$JS.API.DIRECT.GET.KV_mesh-vault_bus-accounts.$KV.mesh-vault_bus-accounts.x", true},
{"spy", "$JS.API.DIRECT.GET.KV_mesh-vault_bus-accounts.$KV.mesh-vault_bus-accounts.x", false},
{"novox.node-tools", "$JS.API.DIRECT.GET.KV_mesh-vault_bus-accounts.$KV.mesh-vault_bus-accounts.x", false},
} { } {
if got := MayPublish(perms[row.who], row.subject); got != row.may { if got := MayPublish(perms[row.who], row.subject); got != row.may {
t.Errorf("%s may publish %s: %v, want %v", row.who, row.subject, got, row.may) t.Errorf("%s may publish %s: %v, want %v", row.who, row.subject, got, row.may)
} }
} }
// Issue 1790 (2026-10-26): a module hears the bus create each bucket it may bind, its own and those it reads, so a
// state missing at its start is bound when it appears; no other bucket's, and no other advisory.
for _, row := range []struct {
who, subject string
may bool
}{
{"mesh-delivery-builder", "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_mesh-delivery-builder_steps", true},
{GrantsModule, "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_mesh-vault_bus-accounts", true},
{"mesh-delivery-builder", "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_mesh-grants_assertions", false},
{"mesh-delivery-builder", "$JS.EVENT.ADVISORY.STREAM.CREATED.LOG_mesh-delivery-builder_builds", false},
{"mesh-delivery-builder", "$JS.EVENT.ADVISORY.STREAM.DELETED.KV_mesh-delivery-builder_steps", false},
} {
if got := MaySubscribe(perms[row.who], row.subject); got != row.may {
t.Errorf("%s may hear %s: %v, want %v", row.who, row.subject, got, row.may)
}
}
// A name that is no plain token is never granted, and a module other than the grants part is never granted one. // A name that is no plain token is never granted, and a module other than the grants part is never granted one.
if got := CreateStreamGrants([]string{"*", ">", "KV_a.b", "KV_a_b"}); len(got) != 2 { if got := CreateStreamGrants([]string{"*", ">", "KV_a.b", "KV_a_b"}); len(got) != 2 {
t.Errorf("grants for wildcard names: %v", got) t.Errorf("grants for wildcard names: %v", got)
+49 -2
View File
@@ -213,6 +213,13 @@ type Principal struct {
// StatesItsKeyToTheVault is a machine principal whose node-engine may ask the vault's `node-key` in its own name // StatesItsKeyToTheVault is a machine principal whose node-engine may ask the vault's `node-key` in its own name
// (issue 786, novox/hq ADR 0372): set while the vault's holder serves it. That one verb, and nothing else new. // (issue 786, novox/hq ADR 0372): set while the vault's holder serves it. That one verb, and nothing else new.
StatesItsKeyToTheVault bool StatesItsKeyToTheVault bool
// AsksItsOwnSecrets is a machine principal whose node-engine may ask the vault's `own` in its own name (novox/hq
// ADR 0547 §2, ADR 0550 §5, issue 1126): its modules' own secrets sealed to its key, for the compositions it makes
// itself. Set while the vault's holder serves it.
AsksItsOwnSecrets bool
// FollowsGrantsUserList is the machine principal of the node holding mesh-broker: its node-engine reads the grants
// part's sealed user list, one key and an object by name (novox/hq ADR 0547 clause 7, GrantsUserListFollows).
FollowsGrantsUserList bool
// BoundBuckets are the providers' buckets of entries, `<module>_bound`, a provisioner client reads its own // BoundBuckets are the providers' buckets of entries, `<module>_bound`, a provisioner client reads its own
// machine's keys of (mesh-sdk go/provider, issue 679): every bucket named bound of a module the mesh places. // machine's keys of (mesh-sdk go/provider, issue 679): every bucket named bound of a module the mesh places.
BoundBuckets []string BoundBuckets []string
@@ -341,6 +348,10 @@ var VaultMoveIn = SeatVerb{Seat: "mesh-vault", Verb: "move-in"}
// novox/hq ADR 0372); granted as Principal.StatesItsKeyToTheVault says. // novox/hq ADR 0372); granted as Principal.StatesItsKeyToTheVault says.
var VaultNodeKey = SeatVerb{Seat: "mesh-vault", Verb: "node-key"} var VaultNodeKey = SeatVerb{Seat: "mesh-vault", Verb: "node-key"}
// VaultOwn is the vault's verb a machine's node-engine asks in its own name for its modules' own secrets sealed to its
// key (issue 1126, ADR 0550 §5); granted as Principal.AsksItsOwnSecrets says.
var VaultOwn = SeatVerb{Seat: "mesh-vault", Verb: "own"}
// VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant // VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant
// chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the // chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the
// controller's own `plans`, a machine's held builds sent by its own `push` and the builds held after a failed release // controller's own `plans`, a machine's held builds sent by its own `push` and the builds held after a failed release
@@ -879,6 +890,14 @@ func PermissionsFor(p Principal) (Permissions, error) {
return Permissions{}, err return Permissions{}, err
} }
pub = append(pub, haveKeys...) pub = append(pub, haveKeys...)
// And every node's own facts, `<node>.node`, its record among them (issue 1126, ADR 0555 §3): a node agent composing
// its own modules reads the other nodes' public records, which its firewall and resolver need (ADR 0547 §2). Read
// only: each node still writes its own keys alone.
pub = append(pub, readGrants(delivery.HaveBucket, "*."+delivery.HaveNodeWord)...)
// And what it composes its own modules from (ADR 0547 §2, ADR 0549 §3): the indexer's placement, modules and
// manifests, the builder's builds (the manifest of a build its want names), and mesh-config's resolved values of
// its own node alone. Read only.
pub = append(pub, NodeComposeReads(p.Node)...)
// And an object of the declaration bucket by its name, and nothing of the bucket's list or writing // And an object of the declaration bucket by its name, and nothing of the bucket's list or writing
// (novox/hq ADR 0486 §2, issue 1697). // (novox/hq ADR 0486 §2, issue 1697).
fileReads, err := DeclarationFilesReads(p.Node) fileReads, err := DeclarationFilesReads(p.Node)
@@ -912,6 +931,19 @@ func PermissionsFor(p Principal) (Permissions, error) {
if p.StatesItsKeyToTheVault { if p.StatesItsKeyToTheVault {
pub = append(pub, "mesh.seat."+VaultNodeKey.Seat+"."+CallKind+"."+VaultNodeKey.Verb+"."+CallerToken(p.Username())) pub = append(pub, "mesh.seat."+VaultNodeKey.Seat+"."+CallKind+"."+VaultNodeKey.Verb+"."+CallerToken(p.Username()))
} }
// And, while the vault serves it, its modules' own secrets in its own name (issue 1126): sealed to its key, for the
// compositions it makes itself.
if p.AsksItsOwnSecrets {
pub = append(pub, VaultOwnCall(p))
// And the vault's word that a secret changed (names and generations, never a value), so the node composes again
// on it rather than at the next unrelated change (ADR 0547 §2).
sub = append(sub, VaultSecretChanged)
}
// And, on the node holding mesh-broker, the grants part's sealed user list it follows from take-over (novox/hq ADR
// 0547 clause 7): one key, and an object by its name.
if p.FollowsGrantsUserList {
pub = append(pub, GrantsUserListFollows()...)
}
case KindModule: case KindModule:
// 0. The watchman alone hears who connects and disconnects, and asks who is connected (ADR 0431 §2). // 0. The watchman alone hears who connects and disconnects, and asks who is connected (ADR 0431 §2).
@@ -927,6 +959,8 @@ func PermissionsFor(p Principal) (Permissions, error) {
// And creating each stream a manifest declares, by its own name (the operator, 2026-10-26 08:54; issue // And creating each stream a manifest declares, by its own name (the operator, 2026-10-26 08:54; issue
// 1790): the create and the read of that one stream, never an update, a delete or a stream not declared. // 1790): the create and the read of that one stream, never an update, a delete or a stream not declared.
pub = append(pub, CreateStreamGrants(p.CreatesStreams)...) pub = append(pub, CreateStreamGrants(p.CreatesStreams)...)
// And its own object bucket, where it writes the user list sealed to the bus node (ADR 0547 clause 7).
pub = append(pub, GrantsUserListWrites()...)
} }
// The view alone reads the build seats' workers for the backlog of asks (issue 1887 A8a). // The view alone reads the build seats' workers for the backlog of asks (issue 1887 A8a).
if p.Module == ViewModule { if p.Module == ViewModule {
@@ -1075,8 +1109,12 @@ func PermissionsFor(p Principal) (Permissions, error) {
// 5. Its state, and the state of others it reads (novox/hq ADR 0201): every one read and // 5. Its state, and the state of others it reads (novox/hq ADR 0201): every one read and
// watched, its own written too. // watched, its own written too.
pub = append(pub, stateGrants(stateAccess{Module: p.Module, Node: p.Node, Keeps: p.State, kept := stateAccess{Module: p.Module, Node: p.Node, Keeps: p.State, PerMachine: p.PerMachine, Reads: p.Reads,
PerMachine: p.PerMachine, Reads: p.Reads, KeyedReads: p.KeyedReads})...) KeyedReads: p.KeyedReads}
pub = append(pub, stateGrants(kept)...)
// And it hears each of those buckets created, so one not on the bus at its start is bound when it appears
// (novox/hq issue 1790; mesh-sdk process.AwaitBucket).
sub = append(sub, stateCreatedHeard(kept)...)
// And its logs (novox/hq ADR 0297): appended to and read directly, its own alone. // And its logs (novox/hq ADR 0297): appended to and read directly, its own alone.
pub = append(pub, logGrants(p.Module, p.Logs)...) pub = append(pub, logGrants(p.Module, p.Logs)...)
@@ -1770,3 +1808,12 @@ const (
nodeToolsService = "node-tools" nodeToolsService = "node-tools"
leaseKey = "holder" leaseKey = "holder"
) )
// VaultOwnCall is the subject a machine principal's node-engine asks the vault's `own` on, in its own name.
func VaultOwnCall(p Principal) string {
return "mesh.seat." + VaultOwn.Seat + "." + CallKind + "." + VaultOwn.Verb + "." + CallerToken(p.Username())
}
// VaultSecretChanged is the vault's event that a secret changed (a name and its generation, never a value), which a
// node composing its own hears to compose again (issue 1126, ADR 0547 §2).
var VaultSecretChanged = "mesh.mod." + VaultOwn.Seat + ".event.secret-changed"
+5 -1
View File
@@ -30,7 +30,11 @@ func processGrants(p Principal) (pub, sub []string, err error) {
// 3. Its module's own state, by the local names its bus names alone — never the whole of what the // 3. Its module's own state, by the local names its bus names alone — never the whole of what the
// module keeps (novox/hq ADR 0436 §1), read and written as stateGrants gives any module's own bucket. // module keeps (novox/hq ADR 0436 §1), read and written as stateGrants gives any module's own bucket.
pub = append(pub, stateGrants(stateAccess{Module: p.Module, Node: p.Node, Keeps: p.State})...) kept := stateAccess{Module: p.Module, Node: p.Node, Keeps: p.State}
pub = append(pub, stateGrants(kept)...)
// And it hears each of them created, so one not on the bus at its start is bound when it appears (novox/hq issue
// 1790; process.AwaitBucket).
sub = append(sub, stateCreatedHeard(kept)...)
// 4. The router's records of its module's own asks (novox/hq ADR 0436 §1, ADR 0259 §3): the same // 4. The router's records of its module's own asks (novox/hq ADR 0436 §1, ADR 0259 §3): the same
// direct-get subjects a module's own principal reads for the seats its Uses names that keep them — // direct-get subjects a module's own principal reads for the seats its Uses names that keep them —
+20 -2
View File
@@ -24,8 +24,9 @@ import (
// subjects cover it, so what travels there — a code typed by the operator — is never persisted. A kinded // subjects cover it, so what travels there — a code typed by the operator — is never persisted. A kinded
// holder asks with its own kind; the modules that watch the seat answer. // holder asks with its own kind; the modules that watch the seat answer.
// //
// And one read: **a holder's records**, a bucket the seat names, read by each user under its own name only // And two reads: **a holder's records**, a bucket the seat names, read by each user under its own name only
// (`$KV.<bucket>.<module>.>`), so an asker reads the state of its own asks and no other asker's. // (`$KV.<bucket>.<module>.>`), so an asker reads the state of its own asks and no other asker's; and on a seat pulled by
// kind, **what its workers gave up on**, the copies DEAD_LETTERS keeps of the seat's stream (DeadLetterReads, issue 734).
// BuildAskMaxDeliver bounds the deliveries of a build seat's ask (issue 1907): mesh-sdk delivery.AskMaxDeliver, written // BuildAskMaxDeliver bounds the deliveries of a build seat's ask (issue 1907): mesh-sdk delivery.AskMaxDeliver, written
// here as the broker mirrors the catalogue rather than importing it. // here as the broker mirrors the catalogue rather than importing it.
@@ -129,6 +130,9 @@ func SeatTrafficOf(module string, holds, uses, watches []Seat) SeatTraffic {
} }
t.Records = append(t.Records, "$JS.API.DIRECT.GET.KV_"+b+".$KV."+b+"."+module+".>") t.Records = append(t.Records, "$JS.API.DIRECT.GET.KV_"+b+".$KV."+b+"."+module+".>")
} }
if s.PulledByKind && len(s.Accepts) > 0 && safeSubject.MatchString(s.Name) {
t.Records = append(t.Records, DeadLetterReads(seatStreamName(s.Name))...)
}
} }
for _, w := range watches { for _, w := range watches {
if w.Kinded { if w.Kinded {
@@ -151,6 +155,20 @@ func SeatTrafficOf(module string, holds, uses, watches []Seat) SeatTraffic {
return t return t
} }
// DeadLetterReads is what a user of a seat pulled by kind is granted to hear which of the asks it put the seat's workers
// gave up on (novox/hq issue 734): an ordered consumer on DEAD_LETTERS filtered to the copies the controller keeps of that
// seat's stream (DeadLetterSubject(stream, *)), pulled, and its deletion. A copy carries the ask's own message id
// (Mesh-Dead-Msg-Id), so the asker ends the step it waits for instead of waiting for ever; it reads the seat's dead
// letters and no other stream's, and changes none.
func DeadLetterReads(stream string) []string {
return []string{
"$JS.API.STREAM.INFO." + DeadLettersStream,
"$JS.API.CONSUMER.CREATE." + DeadLettersStream + ".*." + DeadLetterSubject(stream, ">"),
"$JS.API.CONSUMER.MSG.NEXT." + DeadLettersStream + ".*",
"$JS.API.CONSUMER.DELETE." + DeadLettersStream + ".*",
}
}
// seatTrafficGrants is the bus permissions seat traffic needs: the subjects themselves, and the JetStream API a // seatTrafficGrants is the bus permissions seat traffic needs: the subjects themselves, and the JetStream API a
// worker is pulled and acknowledged through and a record is read through. // worker is pulled and acknowledged through and a record is read through.
func seatTrafficGrants(t SeatTraffic) (pub, sub []string) { func seatTrafficGrants(t SeatTraffic) (pub, sub []string) {
+59
View File
@@ -3,6 +3,7 @@ package buscompose
import ( import (
"fmt" "fmt"
"git.novox.be/novox/mesh-sdk/go/announce" "git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/delivery"
"sort" "sort"
"strings" "strings"
"time" "time"
@@ -166,6 +167,49 @@ func stateGrants(a stateAccess) []string {
return out return out
} }
// StreamCreatedAdvisory is the subject the bus says a stream was created on (the server's own advisory), here for a
// bucket's stream KV_<bucket>.
func StreamCreatedAdvisory(bucket string) string {
return "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_" + bucket
}
// stateCreatedHeard is what a principal subscribes to so that a bucket it may bind and that is not on the bus yet is
// bound when it appears (novox/hq issue 1790, 2026-10-26: since the grants part creates a manifest's buckets, ADR 0547,
// a module's delivery can start it before its new state exists): the bus's advisory that the bucket's stream was
// created, for each bucket stateGrants lets it bind, and no other. Read-only: the server publishes it in the mesh's own
// account, and hearing it changes nothing.
func stateCreatedHeard(a stateAccess) []string {
var out []string
seen := map[string]bool{}
add := func(bucket string) {
if !seen[bucket] {
seen[bucket] = true
out = append(out, StreamCreatedAdvisory(bucket))
}
}
perMachine := map[string]bool{}
for _, n := range a.PerMachine {
perMachine[n] = true
}
for _, name := range a.Keeps {
if !safeSubject.MatchString(name) || (perMachine[name] && !safeSubject.MatchString(a.Node)) {
continue
}
add(BucketName(a.Module, name))
}
for _, r := range a.Reads {
if bucket, ok := bucketOfRead(r); ok {
add(bucket)
}
}
for _, k := range a.KeyedReads {
if bucket, ok := bucketOfRead(k.Read); ok && safeSubject.MatchString(k.Key) {
add(bucket)
}
}
return out
}
// KeyedRead is one key of another module's state a principal reads, and no other key of it. // KeyedRead is one key of another module's state a principal reads, and no other key of it.
type KeyedRead struct { type KeyedRead struct {
Read string `json:"read"` Read string `json:"read"`
@@ -263,3 +307,18 @@ func RaiseBuckets(a BucketAsserter, buckets []Bucket) (undeclared []string, err
sort.Strings(undeclared) sort.Strings(undeclared)
return undeclared, nil return undeclared, nil
} }
// ResolvedBucket is mesh-config's state of resolved values, keyed `<node>.<module>.<key>`.
const ResolvedBucket = "mesh-config_resolved"
// NodeComposeReads is what a node's agent reads to compose its own modules (novox/hq ADR 0547 §2, ADR 0549 §3, issue
// 1126): every key of the indexer's placement, modules and manifests and of the builder's builds, and of mesh-config's
// resolved values only its own node's, `<node>.>`. Reads only; the node is checked as every subject token is by the
// caller's principal.
func NodeComposeReads(node string) []string {
var out []string
for _, bucket := range []string{delivery.PlacementBucket, delivery.ModulesBucket, delivery.ManifestsBucket, delivery.BuilderBuildsBucket} {
out = append(out, readGrants(bucket, "")...)
}
return append(out, readGrants(ResolvedBucket, node+".>")...)
}
File diff suppressed because one or more lines are too long
+111
View File
@@ -0,0 +1,111 @@
package buscompose
import (
"slices"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **The grants part's user list, sealed to the bus node** (novox/hq ADR 0547 clause 7, issue 1945). The grants part writes
// the list it composed as one object of its own object bucket, sealed to the key of the node holding mesh-broker, and
// names it in one key of its own state `user-list`. The node-engine of that node follows the key when the switch says
// so (clause 8), opens the object and writes the bus's user-list file.
//
// Who may do what, and nothing more:
//
// - the grants part writes its object bucket (create, put, purge of an object's old chunks, delete) and its own state
// as before;
// - the node-engine of the node holding mesh-broker reads that one key of the state, watched, and an object of the
// bucket by its name: never the bucket's list of names, never a write, never another key of the grants part's state;
// - no other principal touches either.
// GrantsUserListBucket, GrantsUserListKey, GrantsUserListObjects and GrantsUserListObjectStream are delivery's names,
// here where the grants are composed.
const (
GrantsUserListBucket = delivery.GrantsUserListBucket
GrantsUserListKey = delivery.GrantsUserListKey
GrantsUserListObjects = delivery.GrantsUserListObjects
GrantsUserListObjectStream = delivery.GrantsUserListObjectStream
)
// GrantsUserListFollows is what the node-engine of the node holding mesh-broker may publish to follow the grants part's
// list: the one key `user-list.sealed` read and watched (its own ordered consumer, filtered on that key alone), and an
// object of the bucket by its name, as a node reads its declaration's files (ADR 0486 §2: never STREAM.INFO of the object
// stream, which lists every name, and never STREAM.MSG.GET).
func GrantsUserListFollows() []string {
kv := "KV_" + GrantsUserListBucket
key := "$KV." + GrantsUserListBucket + "." + GrantsUserListKey
obj, s := GrantsUserListObjects, GrantsUserListObjectStream
return []string{
"$JS.API.DIRECT.GET." + kv + "." + key,
"$JS.API.CONSUMER.CREATE." + kv + ".*." + key,
"$JS.API.CONSUMER.DELETE." + kv + ".*",
"$JS.API.STREAM.INFO." + kv,
"$JS.FC." + kv + ".>",
"$JS.API.DIRECT.GET." + s + ".$O." + obj + ".M.*",
"$JS.API.CONSUMER.CREATE." + s + ".*.$O." + obj + ".C.*",
"$JS.API.CONSUMER.DELETE." + s + ".*",
"$JS.FC." + s + ".>",
}
}
// GrantsUserListWrites is what the grants part may publish to keep its object bucket: create it and read its info back,
// put an object (its chunks and its metadata, the old chunks of a name purged), read an object's metadata, and delete
// one. Its own state `user-list` it writes already, as every module writes its own.
func GrantsUserListWrites() []string {
obj, s := GrantsUserListObjects, GrantsUserListObjectStream
return []string{
"$JS.API.STREAM.CREATE." + s,
"$JS.API.STREAM.INFO." + s,
"$JS.API.STREAM.PURGE." + s,
"$JS.API.DIRECT.GET." + s + ".>",
"$JS.API.STREAM.MSG.GET." + s,
"$O." + obj + ".>",
}
}
// grantsOwnObjects is the grants part writing its own object bucket, and nothing of another.
func grantsOwnObjects(p Principal, pattern string) bool {
return p.Kind == KindModule && p.Module == GrantsModule &&
(pattern == "$O."+GrantsUserListObjects+".>" || pattern == "$JS.API.STREAM.CREATE."+GrantsUserListObjectStream)
}
// **Named readers** (novox/hq ADR 0547 clause 9, progressive insight of 2026-10-26; issue 1945). A state named here is
// read by the modules named and by no other, whatever another module's `reads` says: the vault's `bus-accounts` holds
// every bus account's bcrypt hash, and from take-over the grants part alone may read it. A module's announcement may name
// readers too (announce.State.Readers); both are honoured.
const VaultBusAccounts = "mesh-vault.bus-accounts"
// NamedReaders are the states only the modules named may read, by `<owner>.<name>`.
var NamedReaders = map[string][]string{VaultBusAccounts: {GrantsModule}}
// ReaderMay says whether module may read the state read (`<owner>.<name>`) by the named readers: always, for a state
// that names none.
func ReaderMay(module, read string) bool {
readers, named := NamedReaders[read]
return !named || slices.Contains(readers, module)
}
// readsNamed splits a module's reads by the named readers: those it may read, and those named for others.
func readsNamed(module string, reads []string) (granted, refused []string) {
for _, r := range reads {
if ReaderMay(module, strings.TrimSpace(r)) {
granted = append(granted, r)
} else {
refused = append(refused, r)
}
}
return granted, refused
}
// withNamedReaders is d with the reads named for other modules moved to its refused reads.
func withNamedReaders(d Declared) Declared {
granted, refused := readsNamed(d.Module, d.Reads)
if len(refused) == 0 {
return d
}
d.Reads = granted
d.ReadsRefused = append(append([]string(nil), d.ReadsRefused...), refused...)
return d
}
@@ -0,0 +1,213 @@
//go:build integration
package buscompose
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"os"
"path/filepath"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/seal"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// Defends novox/hq ADR 0547 clause 7 on a real server (issue 1945): with the list this composer writes, the grants part
// creates its object bucket and its state, puts the sealed list and names it in `user-list.sealed`; the node-engine of
// the node holding mesh-broker watches that one key, reads the object by its name and opens it with its key, and may
// neither list the bucket, nor read another key of the state, nor write; another node reads nothing of either.
func TestTheBusNodeFollowsTheGrantsPartsSealedListUnderTheComposedGrants(t *testing.T) {
records := Records{
Nodes: []string{"novox", "ace"},
Assigned: map[string][]Declared{
"novox": {{Module: "nats", SnapshotsTheBus: true},
{Module: GrantsModule, State: []Bucket{{Module: GrantsModule, Name: "user-list"}}}},
"ace": {{Module: "plex"}},
},
}
users, err := Users(records)
if err != nil {
t.Fatal(err)
}
for i := range users {
users[i].PasswordHash = "pw-" + users[i].Username()
}
accounts, err := ComposeAccounts(users)
if err != nil {
t.Fatal(err)
}
dir := t.TempDir()
conf := filepath.Join(dir, "server.conf")
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\njetstream { store_dir: \""+dir+"\" }\n"+accounts), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the composed list is not the server's: %v", err)
}
opts.NoLog, opts.NoSigs = true, true
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(30 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(s.Shutdown)
dial := func(user string) *nats.Conn {
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw-"+user), nats.CustomInboxPrefix("_INBOX."+user))
if err != nil {
t.Fatalf("%s: %v", user, err)
}
t.Cleanup(nc.Close)
return nc
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
// The grants part: its state and its object bucket, the sealed list put and named.
grants, err := jetstream.New(dial("novox." + GrantsModule))
if err != nil {
t.Fatal(err)
}
// Its state is made as the grants part makes a declared state, by its stream's name (CreateKeyValue would ask the
// account's info, which no module is granted).
if _, err := grants.CreateStream(ctx, jetstream.StreamConfig{Name: "KV_" + GrantsUserListBucket,
Subjects: []string{"$KV." + GrantsUserListBucket + ".>"}, MaxMsgsPerSubject: 1, AllowDirect: true,
AllowRollup: true, DenyDelete: true, Discard: jetstream.DiscardNew}); err != nil {
t.Fatalf("the grants part's state: %v", err)
}
kv, err := grants.KeyValue(ctx, GrantsUserListBucket)
if err != nil {
t.Fatalf("the grants part binds its state: %v", err)
}
objects, err := grants.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: GrantsUserListObjects})
if err != nil {
t.Fatalf("the grants part's object bucket: %v", err)
}
node, err := seal.GenerateKeyPair()
if err != nil {
t.Fatal(err)
}
list := []byte(accounts)
digest := delivery.UserListDigest(list)
object, err := delivery.SealUserList(list, node.PublicKey)
if err != nil {
t.Fatal(err)
}
name := delivery.UserListObjectName(digest)
for range 2 { // put twice: a second put of a name purges its old chunks
if _, err := objects.PutBytes(ctx, name, object); err != nil {
t.Fatalf("the grants part put its object: %v", err)
}
}
value, err := delivery.EncodeGrantsUserList(delivery.GrantsUserList{Follow: delivery.FollowGrants, Digest: digest,
Object: name, SealedTo: delivery.SealingKeyID(node.PublicKey), Node: "novox", Revision: 1, Users: len(users)})
if err != nil {
t.Fatal(err)
}
if _, err := kv.Put(ctx, GrantsUserListKey, value); err != nil {
t.Fatal(err)
}
if _, err := kv.Put(ctx, "list", []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
// The bus node's engine: the key watched, the object by name (as mesh-host's link reads a declaration's file).
busNode := dial("node.novox")
js, err := busNode.JetStream(nats.MaxWait(5 * time.Second))
if err != nil {
t.Fatal(err)
}
keys, err := js.KeyValue(GrantsUserListBucket)
if err != nil {
t.Fatalf("the bus node binds the grants part's state: %v", err)
}
w, err := keys.Watch(GrantsUserListKey)
if err != nil {
t.Fatalf("the bus node watches the key: %v", err)
}
defer w.Stop()
var got delivery.GrantsUserList
select {
case e := <-w.Updates():
if e == nil {
t.Fatal("the watch found no value")
}
if got, err = delivery.DecodeGrantsUserList(e.Value()); err != nil {
t.Fatal(err)
}
case <-ctx.Done():
t.Fatal("the watch said nothing")
}
read := func(js nats.JetStreamContext, name string) ([]byte, error) {
meta, err := js.GetLastMsg(GrantsUserListObjectStream, "$O."+GrantsUserListObjects+".M."+
base64.URLEncoding.EncodeToString([]byte(name)), nats.DirectGet())
if err != nil {
return nil, err
}
var info struct {
NUID string `json:"nuid"`
Size int `json:"size"`
}
if err := json.Unmarshal(meta.Data, &info); err != nil {
return nil, err
}
sub, err := js.SubscribeSync("$O."+GrantsUserListObjects+".C."+info.NUID, nats.OrderedConsumer(),
nats.BindStream(GrantsUserListObjectStream))
if err != nil {
return nil, err
}
defer sub.Unsubscribe()
var body []byte
for len(body) < info.Size {
m, err := sub.NextMsg(5 * time.Second)
if err != nil {
return nil, err
}
body = append(body, m.Data...)
}
return body, nil
}
body, err := read(js, got.Object)
if err != nil {
t.Fatalf("the bus node reads the object by its name: %v", err)
}
opened, err := delivery.OpenUserList(body, got.Digest, func(b seal.Box) ([]byte, error) {
s, err := seal.Open(b, node.PrivateKey)
return []byte(s), err
})
if err != nil || !bytes.Equal(opened, list) {
t.Fatalf("the bus node opened %d bytes, %v", len(opened), err)
}
// And nothing beyond: no other key, no list of the bucket, no write.
if _, err := keys.Get("list"); err == nil {
t.Error("the bus node read another key of the grants part's state")
}
if _, err := js.StreamInfo(GrantsUserListObjectStream); err == nil {
t.Error("the bus node read the object stream's info, which lists every name")
}
if _, err := js.Publish("$O."+GrantsUserListObjects+".M.x", []byte("x")); err == nil {
t.Error("the bus node wrote the grants part's bucket")
}
// Another node reads nothing of it.
other, err := dial("node.ace").JetStream(nats.MaxWait(2 * time.Second))
if err != nil {
t.Fatal(err)
}
if _, err := read(other, got.Object); err == nil {
t.Error("another node read the object")
}
if _, err := other.GetLastMsg("KV_"+GrantsUserListBucket, "$KV."+GrantsUserListBucket+"."+GrantsUserListKey, nats.DirectGet()); err == nil {
t.Errorf("another node read the key: %v", err)
}
}
+11 -1
View File
@@ -147,14 +147,20 @@ func Users(r Records) ([]Principal, error) {
for _, node := range sortedCopy(r.Nodes) { for _, node := range sortedCopy(r.Nodes) {
witness := false witness := false
var checks []string var checks []string
bus := false
for _, d := range r.Assigned[node] { for _, d := range r.Assigned[node] {
if d.Module == controllerModule { if d.Module == controllerModule {
witness = true witness = true
} }
if d.SnapshotsTheBus {
bus = true
}
checks = append(checks, d.Checks...) checks = append(checks, d.Checks...)
} }
out = append(out, Principal{Kind: KindNode, Node: node, WitnessesController: witness, Checks: checks, out = append(out, Principal{Kind: KindNode, Node: node, WitnessesController: witness, Checks: checks,
MovesIntoTheVault: servesMoveIn(r), StatesItsKeyToTheVault: servesVerb(r, VaultNodeKey)}) MovesIntoTheVault: servesMoveIn(r), StatesItsKeyToTheVault: servesVerb(r, VaultNodeKey),
AsksItsOwnSecrets: servesVerb(r, VaultOwn),
FollowsGrantsUserList: bus})
// **And the machine's provisioner client, on an account of its own** (novox/hq ADR 0349 clause 11): one // **And the machine's provisioner client, on an account of its own** (novox/hq ADR 0349 clause 11): one
// per machine that runs it, delivered to the module running it. // per machine that runs it, delivered to the module running it.
for _, d := range r.Assigned[node] { for _, d := range r.Assigned[node] {
@@ -176,6 +182,9 @@ func Users(r Records) ([]Principal, error) {
} }
} }
for _, d := range r.Assigned[node] { for _, d := range r.Assigned[node] {
// A state with named readers is read by them alone (ADR 0547 clause 9): a read named for another module is
// granted nothing, here and in the runtime carrying the module.
d = withNamedReaders(d)
// **A process account is its own, delivered in its own secret** (broker-<process id>, novox/hq // **A process account is its own, delivered in its own secret** (broker-<process id>, novox/hq
// ADR 0436 §1, §2), never the module's `broker` — so it is composed whether or not the module // ADR 0436 §1, §2), never the module's `broker` — so it is composed whether or not the module
// itself has an account, and is never carried by the machine's runtime, which runs as the // itself has an account, and is never carried by the machine's runtime, which runs as the
@@ -223,6 +232,7 @@ func Users(r Records) ([]Principal, error) {
var carried []Declared var carried []Declared
for _, d := range r.Assigned[node] { for _, d := range r.Assigned[node] {
if d.RunsAs == "" && runnerServes(d) { if d.RunsAs == "" && runnerServes(d) {
d = withNamedReaders(d)
// Never a restricted stream for what the tool runner carries (novox/hq ADR 0376 §1). // Never a restricted stream for what the tool runner carries (novox/hq ADR 0376 §1).
d.Hears = ComposedHears(d) d.Hears = ComposedHears(d)
carried = append(carried, d) carried = append(carried, d)
+7 -1
View File
@@ -44,6 +44,9 @@ func createsDeclared(p Principal, pattern string) bool {
return true return true
} }
name, ok := strings.CutPrefix(pattern, "$JS.API.STREAM.CREATE.") name, ok := strings.CutPrefix(pattern, "$JS.API.STREAM.CREATE.")
if ok && p.Kind == KindModule && p.Module == GrantsModule && name == GrantsUserListObjectStream {
return true
}
return ok && p.Kind == KindModule && p.Module == GrantsModule && slices.Contains(p.CreatesStreams, name) && return ok && p.Kind == KindModule && p.Module == GrantsModule && slices.Contains(p.CreatesStreams, name) &&
safeSubject.MatchString(name) safeSubject.MatchString(name)
} }
@@ -249,7 +252,10 @@ var WritersTable = []WriterRow{
KeptIn: "the object bucket " + DeclarationFilesBucket + "-<node> of each node, an object removed a day after the " + KeptIn: "the object bucket " + DeclarationFilesBucket + "-<node> of each node, an object removed a day after the " +
"node's latest declaration no longer names it", "node's latest declaration no longer names it",
Others: "a node-engine reads an object of its own bucket by its name and may neither list the bucket nor write it", Others: "a node-engine reads an object of its own bucket by its name and may neither list the bucket nor write it",
Subjects: []string{"$O.*.>"}, Writes: isControllerOrTheHandPath}, Subjects: []string{"$O.*.>"}, Writes: func(p Principal, pattern string) bool {
return isControllerOrTheHandPath(p, pattern) || grantsOwnObjects(p, pattern)
},
Shared: "the grants part writes its own object bucket " + GrantsUserListObjects + ", the user list sealed to the bus node, and no other (novox/hq ADR 0547 clause 7)"},
{State: "the declared-config mirror", Writer: "controller (lease holder), from its store, only on a change", {State: "the declared-config mirror", Writer: "controller (lease holder), from its store, only on a change",
KeptIn: "key-value " + DeclaredConfigBucket, Others: "read through `reads` as mesh-controller." + MirrorDeclaredConfig, KeptIn: "key-value " + DeclaredConfigBucket, Others: "read through `reads` as mesh-controller." + MirrorDeclaredConfig,
Subjects: kvOf(DeclaredConfigBucket), Writes: isController}, Subjects: kvOf(DeclaredConfigBucket), Writes: isController},
+1 -1
View File
@@ -172,7 +172,7 @@ func TestAStoreRowWithoutAProtocolKeepsTheCompiledOne(t *testing.T) {
t.Cleanup(func() { UseSeats(was) }) t.Cleanup(func() { UseSeats(was) })
UseSeats([]Seat{{Name: "node-build-go", Scope: ScopeNode, Decision: "row"}}) UseSeats([]Seat{{Name: "node-build-go", Scope: ScopeNode, Decision: "row"}})
got, ok := SeatNamed("node-build-go") got, ok := SeatNamed("node-build-go")
if !ok || len(got.Accepts) == 0 || !got.PulledByKind { if !ok || len(got.Accepts) == 0 || !got.PulledByKind || !got.Yields {
t.Fatalf("the Go build seat lost what it accepts when loaded from the store: %+v", got) t.Fatalf("the Go build seat lost what it accepts when loaded from the store: %+v", got)
} }
if got.Decision != "row" { if got.Decision != "row" {
+14
View File
@@ -311,6 +311,17 @@ func (r Resolution) ownRuntimes(with Rendering) ([]map[string]any, error) {
if words, err = r.readsWords(m, words, with); err != nil { if words, err = r.readsWords(m, words, with); err != nil {
return nil, err return nil, err
} }
// A build seat's holder gives way on its node, and its steps are told their part of the node's share
// (novox/hq ADR 0553).
yielding, yieldWords := r.yieldsOf(m, with)
if len(yieldWords) > 0 {
if words == nil {
words = map[string]string{}
}
for k, v := range yieldWords {
words[k] = v
}
}
if len(words) > 0 { if len(words) > 0 {
body, err := json.Marshal(map[string]map[string]string{m.Module: words}) body, err := json.Marshal(map[string]map[string]string{m.Module: words})
if err != nil { if err != nil {
@@ -329,6 +340,9 @@ func (r Resolution) ownRuntimes(with Rendering) ([]map[string]any, error) {
process[HealthField] = m.RuntimeHealth process[HealthField] = m.RuntimeHealth
healthInto(process, m, with) healthInto(process, m, with)
} }
if yielding != nil {
process["yields"] = yielding
}
if err := artifactsInto(process, RuntimeModule, with); err != nil { if err := artifactsInto(process, RuntimeModule, with); err != nil {
return nil, err return nil, err
} }
+48
View File
@@ -498,6 +498,54 @@ func TestAModuleOfItsOwnAccountIsServedByARuntimeOfItsOwn(t *testing.T) {
if _, told := env[RuntimeOperatorAccount]; told { if _, told := env[RuntimeOperatorAccount]; told {
t.Error("a runtime of a module's own account is told the operator's account") t.Error("a runtime of a module's own account is told the operator's account")
} }
if _, yields := own["yields"]; yields {
t.Errorf("telegram holds no build seat and yields: %v", own["yields"])
}
// The build seats' holders yield (ADR 0553): idle, under the node's one share from the node-engine's build-cpu-share,
// and their steps are told it.
builder := func(name, seat string) Manifest {
b := aToolsModule(t, name, "tools/index.js")
b.RunsAs, b.SecretsOwner = name, name
b.OwnSecrets = OwnSecrets{"broker": {Path: "/var/lib/" + name + "/broker"}}
b.Claims = []Claim{{Name: seat}}
return b
}
gob, img := builder("gob", "node-build-go"), builder("img", "node-build-image")
with.Needed["gob"], with.Needed["img"] = map[string]string{"broker": "own"}, map[string]string{"broker": "own"}
for _, c := range []struct {
settings SettingsBy
share string
}{
{nil, "50"},
{SettingsBy{"mesh-host": {{From: "anchor", Values: map[string]any{"build-cpu-share": float64(75)}}}}, "75"},
{SettingsBy{"mesh-host": {{From: "anchor", Values: map[string]any{"build-cpu-share": float64(0)}}}}, "50"},
} {
with.Settings = c.settings
runtimes, err := Resolution{Node: "anchor", Modules: []Manifest{gob, img, telegram, goRuntime}}.ownRuntimes(with)
if err != nil {
t.Fatal(err)
}
if len(runtimes) != 3 {
t.Fatalf("%d runtimes of their own, want gob's, img's and telegram's", len(runtimes))
}
for _, p := range runtimes {
name := fmt.Sprint(p["name"])
words := map[string]map[string]string{}
_ = json.Unmarshal([]byte(p["env"].(map[string]string)[RuntimeToolEnv]), &words)
mod := strings.TrimSuffix(name, "-runtime")
if mod == "telegram" {
if p["yields"] != nil || words[mod]["MESH_BUILD_CPU_SHARE"] != "" {
t.Errorf("telegram yields under %v: %v %v", c.settings, p["yields"], words)
}
continue
}
if got := fmt.Sprint(p["yields"]); got != "map[cpu-share:"+c.share+" priority:idle]" ||
words[mod]["MESH_BUILD_CPU_SHARE"] != c.share {
t.Errorf("%s under %v yields %s with words %v; want share %s", name, c.settings, got, words[mod], c.share)
}
}
}
with.Settings = nil
// Without the machine's runtime to run it from, it is refused in words. // Without the machine's runtime to run it from, it is refused in words.
if _, err := (Resolution{Node: "anchor", Modules: []Manifest{telegram}}).ownRuntimes(with); err == nil { if _, err := (Resolution{Node: "anchor", Modules: []Manifest{telegram}}).ownRuntimes(with); err == nil {
t.Error("a module of its own account composed without a runtime program") t.Error("a module of its own account composed without a runtime program")
+9 -3
View File
@@ -49,6 +49,11 @@ type Seat struct {
// order Accepts lists them (novox/hq ADR 0525 §2, ADR 0536 §4): the build seats'. Compiled, never stored, like // order Accepts lists them (novox/hq ADR 0525 §2, ADR 0536 §4): the build seats'. Compiled, never stored, like
// Receives. // Receives.
PulledByKind bool PulledByKind bool
// Yields says the seat's holder gives way to everything else on its node (novox/hq ADR 0553): its runtime runs at
// idle priority in the node's one yielding slice, under the node's one processor share (buildcpu.ShareSetting) that
// every yielding holder of the node shares. The build seats': a build must never slow the operator's desktop.
// Compiled, never stored, like Receives.
Yields bool
// Serves carries each verb in full — name, description, schema — because a role's tools are the // Serves carries each verb in full — name, description, schema — because a role's tools are the
// mesh's to define and an agent's to call (novox/hq ADR 0132, design 33 §2). // mesh's to define and an agent's to call (novox/hq ADR 0132, design 33 §2).
Serves []Verb Serves []Verb
@@ -256,11 +261,11 @@ var defaultSeats = append([]Seat{
// its holder taking nothing, kept on the module's own pause key (issue 1887's design limit 2). They replace // its holder taking nothing, kept on the module's own pause key (issue 1887's design limit 2). They replace
// node-build-agent, retired with build-agent (ADR 0524 §5). // node-build-agent, retired with build-agent (ADR 0524 §5).
{Name: "node-build-go", Scope: ScopeNode, Accepts: buildAskKinds(), PulledByKind: true, Serves: buildSeatVerbs("Go"), {Name: "node-build-go", Scope: ScopeNode, Accepts: buildAskKinds(), PulledByKind: true, Serves: buildSeatVerbs("Go"),
Decision: "novox/hq ADR 0525, ADR 0536"}, Yields: true, Decision: "novox/hq ADR 0525, ADR 0536, ADR 0553"},
{Name: "node-build-node", Scope: ScopeNode, Accepts: buildAskKinds(), PulledByKind: true, Serves: buildSeatVerbs("Node"), {Name: "node-build-node", Scope: ScopeNode, Accepts: buildAskKinds(), PulledByKind: true, Serves: buildSeatVerbs("Node"),
Decision: "novox/hq ADR 0525, ADR 0536"}, Yields: true, Decision: "novox/hq ADR 0525, ADR 0536, ADR 0553"},
{Name: "node-build-image", Scope: ScopeNode, Accepts: buildAskKinds(), PulledByKind: true, Serves: buildSeatVerbs("image"), {Name: "node-build-image", Scope: ScopeNode, Accepts: buildAskKinds(), PulledByKind: true, Serves: buildSeatVerbs("image"),
Decision: "novox/hq ADR 0525, ADR 0536"}, Yields: true, Decision: "novox/hq ADR 0525, ADR 0536, ADR 0553"},
// **The mesh's resolvers** (novox/hq ADR 0194, 0196, 0223): every node's internal domain, and // **The mesh's resolvers** (novox/hq ADR 0194, 0196, 0223): every node's internal domain, and
// every node and container asks them and nothing else. Replicated since ADR 0223: held on more // every node and container asks them and nothing else. Replicated since ADR 0223: held on more
// than one machine, each answering the same names from the same roster, and every machine's // than one machine, each answering the same names from the same roster, and every machine's
@@ -497,6 +502,7 @@ func UseSeats(s []Seat) {
row.Paths = d.Paths row.Paths = d.Paths
row.Replicated = d.Replicated row.Replicated = d.Replicated
row.PulledByKind = d.PulledByKind row.PulledByKind = d.PulledByKind
row.Yields = d.Yields // never stored (ADR 0553)
row.Serves = optionalAsCompiled(row.Serves, d.Serves) row.Serves = optionalAsCompiled(row.Serves, d.Serves)
} }
merged = append(merged, row) merged = append(merged, row)
+13
View File
@@ -324,6 +324,9 @@ func TestNodeBuildAgentIsRetiredForTheBuildSeats(t *testing.T) {
if !ok || s.Scope != ScopeNode || !s.PulledByKind || strings.Join(s.Accepts, ",") != "delivery,check,recheck" { if !ok || s.Scope != ScopeNode || !s.PulledByKind || strings.Join(s.Accepts, ",") != "delivery,check,recheck" {
t.Fatalf("%s = %+v", name, s) t.Fatalf("%s = %+v", name, s)
} }
if !s.Yields {
t.Errorf("%s does not yield (ADR 0553)", name)
}
var verbs []string var verbs []string
for _, v := range s.Serves { for _, v := range s.Serves {
verbs = append(verbs, v.Name) verbs = append(verbs, v.Name)
@@ -332,4 +335,14 @@ func TestNodeBuildAgentIsRetiredForTheBuildSeats(t *testing.T) {
t.Errorf("%s serves %v", name, verbs) t.Errorf("%s serves %v", name, verbs)
} }
} }
// The build seats are the seats that yield, and no other (ADR 0553 §1).
var yielding []string
for _, s := range Seats() {
if s.Yields {
yielding = append(yielding, s.Name)
}
}
if strings.Join(yielding, ",") != "node-build-go,node-build-node,node-build-image" {
t.Errorf("the seats that yield are %v", yielding)
}
} }
+7
View File
@@ -126,6 +126,13 @@ var ControllerVerbs = []Verb{
{Name: "node", Description: "What one machine reported it can do, what it is assigned, and why.", {Name: "node", Description: "What one machine reported it can do, what it is assigned, and why.",
Input: schema(map[string]string{"node": "the machine's name"}, []string{"node"}), Input: schema(map[string]string{"node": "the machine's name"}, []string{"node"}),
Replaces: []string{"hostnamectl", "uptime"}}, Replaces: []string{"hostnamectl", "uptime"}},
{Name: "placement-export", Description: "Each machine's placement as the catalogue's placement file says it (novox/hq ADR 0555 " +
"§6, issue 1126): what a person decided of the machine (endpoint, site, hub, address on the private network, public " +
"domain, accounts, adoption, the tunnel found at adoption) and where its modules run (in order, with host ports, " +
"bindings, config placement layers, pins and holdings), read by the controller's own code from its own store, as JSON. " +
"Read-only; the one verb the controller gains for its exit (the operator, 2026-10-26 19:06:40), read once by the seed " +
"of the placement files.",
Input: schema(map[string]string{"node": "one machine's name; every machine when absent"}, nil)},
{Name: "modules", Description: "Every module the mesh holds: version, the commit it was built from, " + {Name: "modules", Description: "Every module the mesh holds: version, the commit it was built from, " +
"and which machines run it.", "and which machines run it.",
Input: schema(nil, nil)}, Input: schema(nil, nil)},
+69
View File
@@ -0,0 +1,69 @@
package catalogue
import (
"strconv"
"git.novox.be/novox/mesh-sdk/go/buildcpu"
)
// yieldsOf is how the runtime of its own a module is served by gives way on this node (novox/hq ADR 0553), or nil when
// it does not, and the word its steps size their threads from:
//
// - a module holding a seat that yields (Seat.Yields, the build seats) runs at idle priority under the node's one share
// of its processors: the node-engine's own config build-cpu-share (buildcpu.ShareSetting) as this node's layers give
// it, its default where none does. Every yielding module of the node carries the same share, so the node-engine's
// one yielding slice has one quota (ADR 0553 §2, §3).
// - its word: that share (buildcpu.ShareWord), which a step divides by its own module's capacity (ADR 0553 §4).
//
// Never a refusal: a share out of range is the default, so a mistyped value bounds the builds at the default and never
// stops the node's whole composition.
func (r Resolution) yieldsOf(m Manifest, with Rendering) (map[string]any, map[string]string) {
if !yields(m) {
return nil, nil
}
engine := Manifest{Module: engineModule}
for _, e := range r.Modules {
if e.Module == engineModule {
engine = e
}
}
share := buildcpu.DefaultPercent
for _, s := range Effective(engine, with.Settings[engineModule]) {
if s.Key == buildcpu.ShareSetting {
if n, ok := percentOf(s.Value); ok {
share = n
}
}
}
return map[string]any{"priority": "idle", "cpu-share": share},
map[string]string{buildcpu.ShareWord: strconv.Itoa(share)}
}
// yields is whether a module claims a seat that yields.
func yields(m Manifest) bool {
for _, c := range m.Claims {
if s, ok := SeatNamed(c.Name); ok && s.Yields {
return true
}
}
return false
}
// percentOf is a setting's value as a percent from 1 to 100: a number, or a string holding one.
func percentOf(v any) (int, bool) {
var n int
switch x := v.(type) {
case float64:
n = int(x)
case int:
n = x
case string:
var err error
if n, err = strconv.Atoi(x); err != nil {
return 0, false
}
default:
return 0, false
}
return n, n >= 1 && n <= 100
}
+139
View File
@@ -0,0 +1,139 @@
// placement-seed writes the catalogue's first placement files (novox/hq ADR 0549 §4, ADR 0555 §6, issue 1126) from the
// answer of the controller's one read-only verb `placement-export`: one `placement/<node>.json` per node, each the
// node's placement as the controller's own code read it from its own store, checked as the catalogue's check checks it
// (delivery.CheckPlacements) before anything is written. The pull request it makes merges only when its composition check
// composes every node equal to what it runs.
//
// placement-seed -catalogue <checkout of mesh-catalog> < export.json
// placement-seed -catalogue <dir> -export export.json
//
// export.json is the verb's answer as it prints it ({"<node>": <placement>, ...}, or the verb's wrapping with it under
// "answer"). It opens no network connection, bus or store; it writes only <dir>/placement/<node>.json, and refuses to
// write over a placement file already there. Exit codes: 0 written; 1 the export cannot be read, is refused by the
// check, or a file is there; 2 the flags are wrong.
package main
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func main() {
catalogue := flag.String("catalogue", "", "the checkout of the catalogue to write placement/<node>.json into")
export := flag.String("export", "", "the verb's answer; standard input when absent")
flag.Parse()
if *catalogue == "" || flag.NArg() > 0 {
fmt.Fprintln(os.Stderr, "usage: placement-seed -catalogue <dir> [-export <file>]")
os.Exit(2)
}
in := io.Reader(os.Stdin)
if *export != "" {
f, err := os.Open(*export)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
defer f.Close()
in = f
}
written, err := Seed(in, *catalogue)
for _, path := range written {
fmt.Println(path)
}
if err != nil {
fmt.Fprintln(os.Stderr, "placement-seed:", err)
os.Exit(1)
}
}
// Seed reads the export and writes each node's placement file under root, answering the paths written. Nothing is
// written when the export does not read, the check refuses it, or a placement file is already there.
func Seed(export io.Reader, root string) ([]string, error) {
raw, err := io.ReadAll(export)
if err != nil {
return nil, err
}
placements, err := read(raw)
if err != nil {
return nil, err
}
all := func(string) bool { return true }
if refused := delivery.CheckPlacements(placements, all, all); len(refused) > 0 {
return nil, fmt.Errorf("the export is refused by the catalogue's check: %v", refused)
}
nodes := make([]string, 0, len(placements))
for node := range placements {
nodes = append(nodes, node)
}
sort.Strings(nodes)
files := map[string][]byte{}
for _, node := range nodes {
p := placements[node]
p.Revision = "" // only the indexer writes a revision, into the key
if _, err := delivery.EncodePlacement(p); err != nil {
return nil, fmt.Errorf("%s: %w", node, err)
}
p.V = delivery.Version
body, err := json.MarshalIndent(p, "", " ")
if err != nil {
return nil, err
}
path := filepath.Join(root, filepath.FromSlash(delivery.PlacementFile(node)))
if _, err := os.Stat(path); err == nil {
return nil, fmt.Errorf("%s is already there; the seed is written once (ADR 0549 §4)", path)
} else if !errors.Is(err, os.ErrNotExist) {
return nil, err
}
files[path] = append(body, '\n')
}
var written []string
for _, node := range nodes {
path := filepath.Join(root, filepath.FromSlash(delivery.PlacementFile(node)))
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return written, err
}
if err := os.WriteFile(path, files[path], 0o644); err != nil {
return written, err
}
written = append(written, path)
}
return written, nil
}
// read is the export as the verb prints it, or wrapped as a verb's answer (the controller's verbAnswer: its JSON under
// "answer").
func read(raw []byte) (map[string]delivery.Placement, error) {
var wrapped struct {
Answer json.RawMessage `json:"answer"`
}
if json.Unmarshal(raw, &wrapped) == nil && len(wrapped.Answer) > 0 {
raw = wrapped.Answer
}
var byNode map[string]json.RawMessage
if err := json.Unmarshal(raw, &byNode); err != nil {
return nil, fmt.Errorf("the export is not {<node>: <placement>}: %w", err)
}
if len(byNode) == 0 {
return nil, errors.New("the export names no node")
}
out := map[string]delivery.Placement{}
for node, body := range byNode {
p, err := delivery.DecodePlacement(body)
if err != nil {
return nil, fmt.Errorf("%s: %w", node, err)
}
if p.Node != node {
return nil, fmt.Errorf("the export's placement of %s names the node %q", node, p.Node)
}
out[node] = p
}
return out, nil
}
+61
View File
@@ -0,0 +1,61 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Issue 1126 (ADR 0549 §4, ADR 0555 §6): the seed writes one placement file per node of the export, as the catalogue's
// check and the indexer read it, from the verb's answer bare or wrapped; it writes nothing for an export the check
// refuses, and never over a file already there.
func TestTheSeedWritesEachNodesPlacementFileOnce(t *testing.T) {
export := `{"answer":{"ace":{"v":1,"node":"ace","revision":"x","decided":{"address":"10.77.0.2"},"modules":[{"module":"plex","ports":{"32400":32400}}]},` +
`"novox":{"v":1,"node":"novox","decided":{"hub":true,"address":"10.77.0.1"},"modules":[]}}}`
root := t.TempDir()
written, err := Seed(strings.NewReader(export), root)
if err != nil || len(written) != 2 {
t.Fatalf("written %v, %v", written, err)
}
raw, err := os.ReadFile(filepath.Join(root, "placement", "ace.json"))
if err != nil {
t.Fatal(err)
}
p, err := delivery.DecodePlacement(raw)
if err != nil || p.Node != "ace" || p.Revision != "" || p.HostPorts()["plex"][32400] != 32400 {
t.Errorf("ace's file reads %+v, %v", p, err)
}
if _, err := Seed(strings.NewReader(export), root); err == nil || !strings.Contains(err.Error(), "already there") {
t.Errorf("a second seed over the files: %v", err)
}
clash := `{"ace":{"v":1,"node":"ace","decided":{"address":"10.77.0.1"},"modules":[]},"novox":{"v":1,"node":"novox","decided":{"address":"10.77.0.1"},"modules":[]}}`
other := t.TempDir()
if _, err := Seed(strings.NewReader(clash), other); err == nil || !strings.Contains(err.Error(), "both have the address") {
t.Errorf("an export the check refuses: %v", err)
}
if entries, _ := os.ReadDir(filepath.Join(other, "placement")); len(entries) != 0 {
t.Errorf("a refused export wrote %v", entries)
}
// What the controller's placement-export prints, frombus.Export per node, is what the seed takes: a node with no
// module placed included.
in := compose.Input{Nodes: []compose.Node{{Name: "solo", Place: compose.Place{Hub: true, Address: "10.77.0.9"}}}}
exported, err := frombus.Export(in, "solo")
if err != nil {
t.Fatal(err)
}
answer, _ := json.Marshal(map[string]any{"answer": map[string]delivery.Placement{"solo": exported}})
if written, err := Seed(strings.NewReader(string(answer)), t.TempDir()); err != nil || len(written) != 1 {
t.Errorf("Export's answer %s was not seeded: %v, %v", answer, written, err)
}
for _, bad := range []string{`[]`, `{}`, `{"ace":{"v":1,"node":"novox","modules":[]}}`} {
if _, err := Seed(strings.NewReader(bad), t.TempDir()); err == nil {
t.Errorf("%s was seeded", bad)
}
}
}
+40
View File
@@ -2,8 +2,10 @@ package compose
import ( import (
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"sort" "sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/declaration" "git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -16,6 +18,26 @@ import (
// (mesh-hand-compose) gets that one. Each module's Data says whether its manifest declares data. The compositions are not // (mesh-hand-compose) gets that one. Each module's Data says whether its manifest declares data. The compositions are not
// signed and not yet by digest: the writer does both (delivery.SignComposed, Composed.ByDigest). // signed and not yet by digest: the writer does both (delivery.SignComposed, Composed.ByDigest).
func Compositions(in Input, node string, fps map[string]string, mode Mode) (out map[string]delivery.Composed, skipped []string, err error) { func Compositions(in Input, node string, fps map[string]string, mode Mode) (out map[string]delivery.Composed, skipped []string, err error) {
return compositions(in, node, fps, mode, false)
}
// ErrStoodIn is a composition that holds a stand-in for a value the input does not carry (an own secret, a grant's
// sealed copy, a vault box, the certificate, the bus's user list) or a port it does not give: what a send would make,
// never what may be applied.
var ErrStoodIn = errors.New("compose: the composition stands in for what its input does not carry")
// BusUsersStandIn is what a composition says it stood in for when its input carries no bus user list.
const BusUsersStandIn = "the bus's user list"
// CompositionsApplied is Compositions for a node that applies what it composed itself (novox/hq ADR 0547 §2, issue
// 1126): a declaration that stands in for anything (Declared.StoodIn, Foreseen, ForeseenPorts) is refused with
// ErrStoodIn, naming each, so a stand-in is never applied as if it were the value; but for the bus's user list
// (BusUsersStandIn), which the bus node holds out of every apply of a composition it made itself.
func CompositionsApplied(in Input, node string, fps map[string]string) (map[string]delivery.Composed, []string, error) {
return compositions(in, node, fps, Reading, true)
}
func compositions(in Input, node string, fps map[string]string, mode Mode, applied bool) (out map[string]delivery.Composed, skipped []string, err error) {
reading := in reading := in
reading.Nodes = append([]Node(nil), in.Nodes...) reading.Nodes = append([]Node(nil), in.Nodes...)
found := false found := false
@@ -35,6 +57,24 @@ func Compositions(in Input, node string, fps map[string]string, mode Mode) (out
if len(problems) > 0 { if len(problems) > 0 {
return nil, nil, fmt.Errorf("compose: %s's declaration is refused by the node-engine's validator: %v", node, problems) return nil, nil, fmt.Errorf("compose: %s's declaration is refused by the node-engine's validator: %v", node, problems)
} }
if applied {
var stood []string
for _, x := range d.StoodIn {
// The bus's user list is the bus node's own to follow, written alone and never from a composition the node
// made itself (ADR 0547 §7, §8): its stand-in is held out of the apply, not applied.
if x != BusUsersStandIn {
stood = append(stood, x)
}
}
stood = append(stood, d.Foreseen...)
for _, p := range d.ForeseenPorts {
stood = append(stood, "port "+p)
}
if len(stood) > 0 {
sort.Strings(stood)
return nil, nil, fmt.Errorf("%w on %s: %s", ErrStoodIn, node, strings.Join(stood, ", "))
}
}
body, err := d.Body() body, err := d.Body()
if err != nil { if err != nil {
return nil, nil, err return nil, nil, err
+64
View File
@@ -0,0 +1,64 @@
package compose
import (
"fmt"
"sort"
)
// CredentialKey names one pair credential by the ends it joins: the provision, the consumer's node and module, the
// provider's node, and the local name where a module keeps several (Credential's and Issued's key).
type CredentialKey struct {
Provision string `json:"provision"`
Consumer string `json:"consumer"`
ConsumerModule string `json:"consumer_module"`
Provider string `json:"provider"`
Local string `json:"local,omitempty"`
}
// CredentialsNeeded is every pair credential node's composition reads (novox/hq ADR 0547 §2, ADR 0550 §5, issue 1126):
// as a consumer, each need of its resolution a provider's credential answers (a need answered by a record is a licence's,
// not a pair's); as a provider, each credential it must carry a grant for (rowsFor). A node that composes its own asks the
// vault for exactly these, sealed to its key, and composes with them; one it does not get is stood in for, which the
// node's own apply refuses (CompositionsApplied).
func CredentialsNeeded(in Input, node string) (asConsumer, asProvider []CredentialKey, err error) {
c := newComposer(in)
resolved, err := c.resolution(node)
if err != nil {
return nil, nil, err
}
for _, n := range resolved.Needs {
if n.ByRecord || n.From == "" {
continue
}
asConsumer = append(asConsumer, CredentialKey{Provision: n.Name, Consumer: node, ConsumerModule: n.For,
Provider: n.From, Local: n.Local})
}
rows, err := c.rowsFor(node)
if err != nil {
return nil, nil, fmt.Errorf("compose: the credentials %s provides: %w", node, err)
}
for _, r := range rows {
asProvider = append(asProvider, CredentialKey{Provision: r.Provision, Consumer: r.Consumer,
ConsumerModule: r.ConsumerModule, Provider: node, Local: r.Local})
}
sortKeys(asConsumer)
sortKeys(asProvider)
return asConsumer, asProvider, nil
}
func sortKeys(ks []CredentialKey) {
sort.Slice(ks, func(i, j int) bool {
a, b := ks[i], ks[j]
switch {
case a.Provision != b.Provision:
return a.Provision < b.Provision
case a.Provider != b.Provider:
return a.Provider < b.Provider
case a.Consumer != b.Consumer:
return a.Consumer < b.Consumer
case a.ConsumerModule != b.ConsumerModule:
return a.ConsumerModule < b.ConsumerModule
}
return a.Local < b.Local
})
}
+20
View File
@@ -2,6 +2,7 @@ package compose
import ( import (
"encoding/json" "encoding/json"
"errors"
"strings" "strings"
"testing" "testing"
@@ -78,6 +79,16 @@ func TestAnOwnSecretHeldIsCarriedAndOneNotHeldIsForeseenOrLeftOut(t *testing.T)
t.Errorf("a held secret with no value is not named: %v", got.Rendering.Foreseen) t.Errorf("a held secret with no value is not named: %v", got.Rendering.Foreseen)
} }
unchanged(t, mk, func(in Input) { rendered(t, in, "laptop", Foreseeing) }) unchanged(t, mk, func(in Input) { rendered(t, in, "laptop", Foreseeing) })
// 1126 (ADR 0547 §2): a node applying what it composed itself never applies a stand-in: a held secret with no value in the
// Input is refused, naming it; held with its value, it composes.
fps := map[string]string{"assistant": "src2:" + strings.Repeat("a", 64), overlay.Name: "src2:" + strings.Repeat("b", 64)}
if _, _, err := CompositionsApplied(in, "laptop", fps); !errors.Is(err, ErrStoodIn) || !strings.Contains(err.Error(), "assistant/password") {
t.Errorf("a stand-in was not refused for a node's own apply: %v", err)
}
if out, _, err := CompositionsApplied(mk(), "laptop", fps); err != nil || len(out) == 0 {
t.Errorf("a held secret with its value is refused for a node's own apply: %v", err)
}
} }
// An own secret issued outside the mesh that nobody gave is never made (novox/hq issue 401, ADR 0228): nothing is composed or // An own secret issued outside the mesh that nobody gave is never made (novox/hq issue 401, ADR 0228): nothing is composed or
@@ -161,6 +172,15 @@ func aGrantMesh(t *testing.T) Input {
// the sealed copy a stand-in is composed and named (novox/hq plan.go grantsFor). // the sealed copy a stand-in is composed and named (novox/hq plan.go grantsFor).
func TestAGrantIsCreatedForEachCredentialMadeForAConsumer(t *testing.T) { func TestAGrantIsCreatedForEachCredentialMadeForAConsumer(t *testing.T) {
in := aGrantMesh(t) in := aGrantMesh(t)
// 1126 (ADR 0550 §5): the credentials each node's composition reads, by their ends: laptop's files consumes anchor's
// s3-bucket, and anchor grants it; the node composing its own asks the vault for exactly these.
pair := CredentialKey{Provision: "s3-bucket", Consumer: "laptop", ConsumerModule: "files", Provider: "anchor"}
if consumer, provider, err := CredentialsNeeded(in, "laptop"); err != nil || len(provider) != 0 || len(consumer) != 1 || consumer[0] != pair {
t.Errorf("laptop needs %v as consumer and %v as provider (%v), want %v as consumer", consumer, provider, err, pair)
}
if consumer, provider, err := CredentialsNeeded(in, "anchor"); err != nil || len(consumer) != 0 || len(provider) != 1 || provider[0] != pair {
t.Errorf("anchor needs %v as consumer and %v as provider (%v), want %v as provider", consumer, provider, err, pair)
}
got := rendered(t, in, "anchor", Foreseeing) got := rendered(t, in, "anchor", Foreseeing)
if len(got.Rendering.Grants) != 1 { if len(got.Rendering.Grants) != 1 {
t.Fatalf("grants are %+v", got.Rendering.Grants) t.Fatalf("grants are %+v", got.Rendering.Grants)
+111
View File
@@ -0,0 +1,111 @@
package frombus
import (
"fmt"
"sort"
"strconv"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Export is one node's placement file as the composition input the controller fills says it (novox/hq ADR 0555 §6,
// issue 1126): the inverse of Input for what a person decides, so the controller's one read-only verb
// `placement-export` answers, from its own store by its own code, exactly what the seed writes into the catalogue.
// What the machine observes (its record) is not in it.
func Export(in compose.Input, node string) (delivery.Placement, error) {
var n *compose.Node
for i := range in.Nodes {
if in.Nodes[i].Name == node {
n = &in.Nodes[i]
}
}
if n == nil {
return delivery.Placement{}, fmt.Errorf("frombus: the input has no node %q", node)
}
// At the version the placement file is read at, with a list of modules even when none is placed: the export is the
// file the seed writes (cmd/placement-seed), and the reader refuses a value of version 0 or without its modules.
p := delivery.Placement{V: delivery.Version, Node: node, Modules: []delivery.PlacedModule{}, Decided: delivery.NodeDecided{
Endpoint: n.Place.Endpoint, Site: n.Place.Site, Hub: n.Place.Hub, Address: n.Place.Address,
PublicDomain: n.PublicDomain, Account: n.Account, AccountHome: n.AccountHome, AgentAccount: n.AgentAccount,
AgentHome: n.AgentAccountHome, Adopted: n.Adopted, Taken: n.Taken}}
// The tunnel found at the node's adoption: the one its private network took over, or, on the hub, the mesh's adopted
// one whether or not the hub took it; and the names a person gave the hub's carried peers.
if t, ok := in.Tunnels[node]; ok {
rt := recordTunnelOf(t.Tunnel)
p.Decided.Tunnel = &rt
} else if in.Adopted != nil && in.Adopted.Hub == node {
rt := recordTunnelOf(in.Adopted.Tunnel)
p.Decided.Tunnel = &rt
}
if n.Place.Hub {
for _, c := range in.CarriedPeers {
if c.Named != "" {
if p.Decided.PeerNames == nil {
p.Decided.PeerNames = map[string]string{}
}
p.Decided.PeerNames[c.PublicKey] = c.Named
}
}
}
for _, module := range n.Assigned {
m := delivery.PlacedModule{Module: module}
for port, at := range n.Ports[module] {
if m.Ports == nil {
m.Ports = map[string]int{}
}
m.Ports[strconv.Itoa(port)] = at
}
for provision, to := range n.Bound[module] {
if m.Bindings == nil {
m.Bindings = map[string]delivery.Provider{}
}
m.Bindings[provision] = delivery.Provider{Node: to.Node, Module: to.Module}
}
// A module in the settings form keeps its config values in its layers, not in mesh-config: they are exported with
// the placement. A module in the config form has mesh-config's values laid into its layer, which are not.
settingsForm := !in.Catalogue[module].DeclaresConfig()
for _, layer := range n.Settings[module] {
if layer.From == catalogue.MeshWideLayer {
m.HasMesh, m.MeshWide = true, layer.Placement
if settingsForm && len(layer.Values) > 0 {
m.MeshValues = layer.Values
}
} else {
m.HasLayer, m.Layer = true, layer.Placement
if settingsForm && len(layer.Values) > 0 {
m.Values = layer.Values
}
}
}
p.Modules = append(p.Modules, m)
}
for provision, to := range n.Pins {
if p.Pins == nil {
p.Pins = map[string]delivery.Provider{}
}
p.Pins[provision] = delivery.Provider{Node: to.Node, Module: to.Module}
}
for _, h := range in.Holdings {
if h.Node == node {
p.Holds = append(p.Holds, delivery.Holding{Claim: h.Claim, Scope: h.Scope, Module: h.Module, Site: h.Site, Kind: h.Kind})
}
}
sort.SliceStable(p.Holds, func(i, j int) bool {
if p.Holds[i].Claim != p.Holds[j].Claim {
return p.Holds[i].Claim < p.Holds[j].Claim
}
return p.Holds[i].Module < p.Holds[j].Module
})
return p, nil
}
func recordTunnelOf(t compose.Tunnel) delivery.RecordTunnel {
out := delivery.RecordTunnel{Interface: t.Interface, Unit: t.Unit, Config: t.Config, Port: t.Port, MTU: t.MTU,
Address: t.Address, Range: t.Range, PublicKey: t.PublicKey}
for _, peer := range t.Peers {
out.Peers = append(out.Peers, delivery.RecordTunnelPeer{PublicKey: peer.PublicKey, Address: peer.Address})
}
return out
}
+358
View File
@@ -0,0 +1,358 @@
// Package frombus fills mesh-sdk's composition input from what is on the bus, so a node agent, the composition check
// and the grants part compose with one composer over the same inputs (novox/hq ADR 0547 §2, ADR 0549 §3; issue 1126).
//
// **One composer, three callers.** compose.Compositions is the composer; Input is how each caller fills its
// compose.Input from the bus: the node agent for its own node (ADR 0547 §2), the assigner's composition check for every
// node (ADR 0539 §6's check, ADR 0547 §4), and the grants part (ADR 0547 §5). Each reads the bus with its own credential
// through a Reader, so this package holds no bus client, no store and no timer: what it is given is all it reads.
//
// **What it reads, and from whom.**
// - which modules run on each node, with their pins, holdings, host ports, bindings and settings placement: the
// indexer's state `placement`, one key per node (ADR 0549 §2);
// - what a person decided of each node (its endpoint, site, hub, address on the private network, public domain and
// accounts): the same placement key, Placement.Decided;
// - what each node observes of itself (its tunnel key, capabilities, keys, certificate, links and networks): the
// node's own have key, `<node>.node` (HaveNode.Record); the tunnel found at a node's adoption and the names of the hub's peers are decided,
// in its placement, and whether the hub took the tunnel over and who enrolled under a peer's key are read from both;
// - every module's manifest: the indexer's `modules.<module>` and its manifests by digest, or the digest the
// composing node's want names where it runs another build (Scope.Kept);
// - each module's config values on a node: mesh-config's resolved state, the composing node's slice alone unless the
// caller composes every node (ADR 0547 §2, the prepare step's L4).
//
// **Own secrets and pair credentials** come from the vault sealed to the composing node's key (OwnSecrets, Credentials),
// for that node alone: as a consumer and as a provider, exactly the keys compose.CredentialsNeeded names. **Not yet
// read** (each a later subtask of issue 1126): licences (none is placed today) and core boxes. Without one
// the composition stands in for it, and Compose refuses a composition that stands in for anything (compose.ErrStoodIn):
// a node never applies a stand-in. No node is joining before ADR 0547's third milestone (ADR 0550 §6), so Joining is
// always empty.
package frombus
import (
"context"
"encoding/json"
"fmt"
"net/netip"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
)
// Reader is the bus as a caller reads it with its own credential. Each method reads one part's state; an error is a
// read that failed, never an absent key (an absent key is an empty answer).
type Reader interface {
// Placements is every node's placement, by node (delivery.PlacementBucket).
Placements(ctx context.Context) (map[string]delivery.Placement, error)
// Records is every node's record, by node: HaveNode.Record of each node's own have key, `<node>.node` of
// delivery.HaveBucket (ADR 0555 §3).
Records(ctx context.Context) (map[string]delivery.NodeRecord, error)
// Modules is every module the indexer holds, by module (delivery.ModulesBucket).
Modules(ctx context.Context) (map[string]delivery.Module, error)
// Manifest is the raw manifest of a digest (delivery.ManifestsBucket).
Manifest(ctx context.Context, digest string) ([]byte, error)
// Values is one module's config values on one node as mesh-config resolved them (mesh-config's state `resolved`,
// keys `<node>.<module>.<key>`), decoded; nil when it has none.
Values(ctx context.Context, node, module string) (map[string]any, error)
// OwnSecrets is the own secrets of the composing node's modules, each sealed to that node's key, as the vault answers
// the node's engine (the vault's verb `own`, ADR 0550 §5); asked only for the node composing itself.
OwnSecrets(ctx context.Context, node string) ([]compose.OwnSecretRow, error)
// Credentials is, for the composing node, each pair credential it reads as a consumer, held and sealed to its key,
// and each it grants as a provider, with the copy sealed to it (the vault's `own` with the keys, ADR 0550 §5). A key
// the vault holds no value for is left out; the composition then stands in for it, and Compose refuses it.
Credentials(ctx context.Context, node string, asConsumer, asProvider []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error)
}
// Mesh is what is true of the whole mesh and is no part's state on the bus yet: the private network's range and the
// internal names' suffix as genesis was told them, and the mesh's authority certificate (public). Empty fields take
// compose's defaults.
type Mesh struct {
OverlayCIDR string
InternalSuffix string
Authority string
// Predefined is the list of predefined keys mesh-config published, which a manifest's values falling back to a
// predefined key are judged against; nil reads each manifest as the indexer does (catalogue.ParseManifestForInputs).
Predefined catalogue.PredefinedList
}
// Scope says who composes. Node is the node composing; only its config values are read unless Every is set, as the
// composition check sets it to compose every node. Kept is, by module, the manifest digest the composing node's wants
// name where it differs from the indexer's trunk: that node composes the module as it runs it.
type Scope struct {
Node string
Every bool
Kept map[string]string
}
// Input is the composition input for s, read through r.
func Input(ctx context.Context, r Reader, mesh Mesh, s Scope) (compose.Input, error) {
placements, err := r.Placements(ctx)
if err != nil {
return compose.Input{}, fmt.Errorf("frombus: placement: %w", err)
}
records, err := r.Records(ctx)
if err != nil {
return compose.Input{}, fmt.Errorf("frombus: node records: %w", err)
}
modules, err := r.Modules(ctx)
if err != nil {
return compose.Input{}, fmt.Errorf("frombus: the indexer's modules: %w", err)
}
if _, ok := placements[s.Node]; !ok && !s.Every {
return compose.Input{}, fmt.Errorf("frombus: %s has no placement in the catalogue", s.Node)
}
in := compose.Input{OverlayCIDR: mesh.OverlayCIDR, InternalSuffix: mesh.InternalSuffix, Authority: mesh.Authority,
Catalogue: map[string]catalogue.Manifest{}}
if in.Authority == "" {
// The mesh's own certificate, public, as the composing node holds it; for the check of every node, as the first
// node holding one does.
if rec, ok := records[s.Node]; ok && rec.Authority != "" {
in.Authority = rec.Authority
} else {
for _, node := range sortedKeys(records) {
if a := records[node].Authority; a != "" {
in.Authority = a
break
}
}
}
}
read := func(module, digest string) (catalogue.Manifest, error) {
raw, err := r.Manifest(ctx, digest)
if err != nil {
return catalogue.Manifest{}, fmt.Errorf("frombus: the manifest of %s (digest %s): %w", module, digest, err)
}
if mesh.Predefined != nil {
return catalogue.ParseManifestWith(raw, mesh.Predefined)
}
return catalogue.ParseManifestForInputs(raw)
}
for _, name := range sortedKeys(modules) {
m := modules[name]
if m.Manifest == "" {
continue // a registered module the indexer has not read: nothing to compose it from
}
man, err := read(name, m.Manifest)
if err != nil {
return compose.Input{}, err
}
in.Catalogue[name] = man
}
// The module the mesh provides itself (the private network) has no repository the indexer reads: its manifest comes
// with mesh-sdk and always wins, as the controller's providedModules and the grants part have it.
provided, err := json.Marshal(overlay.Manifest())
if err != nil {
return compose.Input{}, err
}
var private catalogue.Manifest
if err := json.Unmarshal(provided, &private); err != nil {
return compose.Input{}, fmt.Errorf("frombus: the private network's manifest mesh-sdk provides: %w", err)
}
in.Catalogue[overlay.Name] = private
var missing []string
for _, node := range nodesOf(placements, records) {
rec, hasRecord := records[node]
p := placements[node]
if !hasRecord {
if len(p.Modules) > 0 {
missing = append(missing, node)
}
continue
}
n := nodeOf(rec, p)
withValues := s.Every || node == s.Node
for _, m := range p.Modules {
var values map[string]any
if withValues {
if values, err = r.Values(ctx, node, m.Module); err != nil {
return compose.Input{}, fmt.Errorf("frombus: %s's config values on %s: %w", m.Module, node, err)
}
}
if layers := m.Layers(node, values); layers != nil {
if n.Settings == nil {
n.Settings = map[string][]catalogue.Layer{}
}
n.Settings[m.Module] = layers
}
}
if node == s.Node {
for _, module := range sortedKeys(s.Kept) {
man, err := read(module, s.Kept[module])
if err != nil {
return compose.Input{}, err
}
if n.Kept == nil {
n.Kept = map[string]catalogue.Manifest{}
}
n.Kept[module] = man
}
}
in.Nodes = append(in.Nodes, n)
in.Holdings = append(in.Holdings, p.Held()...)
// A tunnel found at adoption is the node's private network's when the node's tunnel key is the tunnel's (the
// controller's inv.Tunnels); the hub's found tunnel is the mesh's adopted one whether or not the hub took it over
// (inv.AdoptedTunnel), and its peers are carried once it did (inv.CarriedPeers).
if t := p.Decided.Tunnel; t != nil {
if rec.TunnelKey == t.PublicKey {
if in.Tunnels == nil {
in.Tunnels = map[string]compose.FoundTunnel{}
}
in.Tunnels[node] = compose.FoundTunnel{Tunnel: tunnelOf(*t), NodeAdopted: p.Decided.Adopted}
}
if p.Decided.Hub {
in.Adopted = &compose.AdoptedTunnel{Tunnel: tunnelOf(*t), Hub: node, Adopted: rec.TunnelKey == t.PublicKey}
}
}
}
if in.Adopted != nil && in.Adopted.Adopted {
in.CarriedPeers = carriedPeers(in.Adopted.Tunnel, placements[in.Adopted.Hub].Decided.PeerNames, records)
}
if len(missing) > 0 {
return compose.Input{}, fmt.Errorf("frombus: modules are placed on %s, whose agent has written no record yet",
strings.Join(missing, ", "))
}
if !s.Every {
if _, ok := records[s.Node]; !ok {
return compose.Input{}, fmt.Errorf("frombus: %s has written no record of itself yet", s.Node)
}
// Its own modules' secrets, sealed to its own key by the vault: a composition carries them sealed (ADR 0550 §5).
// Asked only when a module placed on it declares one: a node none of whose modules has a secret composes
// whether or not the vault answers.
if declaresOwnSecrets(in, s.Node) {
if in.OwnSecrets, err = r.OwnSecrets(ctx, s.Node); err != nil {
return compose.Input{}, fmt.Errorf("frombus: the own secrets of %s's modules: %w", s.Node, err)
}
}
// The core's credentials travel as the vault's box sealed to the machine (ADR 0349 §16): the same values, which
// the composition reads for a core module as its kept boxes.
for _, o := range in.OwnSecrets {
if json.Valid([]byte(o.Sealed)) {
in.KeptCoreBoxes = append(in.KeptCoreBoxes, compose.CoreBox{Node: o.Node, Module: o.Module, Name: o.Name,
SealedTo: o.SealedTo, Box: json.RawMessage(o.Sealed)})
}
}
}
return in, nil
}
// Compose is the node's compositions over the bus (ADR 0547 §2): Input for s, then compose.CompositionsApplied at the
// fingerprints fps gives each module: composed in the Reading mode as the controller composes them, and refused when it
// stands in for anything the input does not carry (compose.ErrStoodIn), since the node applies what it composed.
func Compose(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]string) (map[string]delivery.Composed, []string, error) {
in, err := Input(ctx, r, mesh, s)
if err != nil {
return nil, nil, err
}
// The pair credentials its composition reads, asked of the vault sealed to it (ADR 0550 §5).
asConsumer, asProvider, err := compose.CredentialsNeeded(in, s.Node)
if err != nil {
return nil, nil, err
}
if len(asConsumer)+len(asProvider) > 0 {
if in.Credentials, in.Issued, err = r.Credentials(ctx, s.Node, asConsumer, asProvider); err != nil {
return nil, nil, fmt.Errorf("frombus: the credentials of %s: %w", s.Node, err)
}
}
return compose.CompositionsApplied(in, s.Node, fps)
}
// declaresOwnSecrets says a module assigned to node declares an own secret, in the manifest the node composes it from.
func declaresOwnSecrets(in compose.Input, node string) bool {
for _, n := range in.Nodes {
if n.Name != node {
continue
}
for _, module := range n.Assigned {
m, ok := n.Kept[module]
if !ok {
m = in.Catalogue[module]
}
if len(m.OwnSecrets) > 0 {
return true
}
}
}
return false
}
// nodeOf is the composition's node from its record (what the machine observes) and its placement (what a person
// decided of it, and where its modules run).
func nodeOf(rec delivery.NodeRecord, p delivery.Placement) compose.Node {
d := p.Decided
n := compose.Node{
Name: rec.Node,
Place: compose.Place{Key: rec.TunnelKey, Endpoint: d.Endpoint, Site: d.Site, Hub: d.Hub, Address: d.Address},
Assigned: p.Assigned(), PublicDomain: d.PublicDomain,
Account: d.Account, AccountHome: d.AccountHome, AgentAccount: d.AgentAccount, AgentAccountHome: d.AgentHome,
Adopted: d.Adopted, Taken: d.Taken, Pins: p.ChosenPins(), Bound: p.Bound(), Ports: p.HostPorts(), SealingKey: rec.SealingKey,
OutwardLinks: rec.OutwardLinks, Networks: rec.Networks, NetworksSaid: rec.NetworksSaid,
ServingKey: rec.ServingKey, Certificate: rec.Certificate, Contract: rec.Contract, Stated: rec.Contract > 0,
}
if len(rec.Capabilities) > 0 {
n.Capabilities = make(map[string]bool, len(rec.Capabilities))
for _, c := range rec.Capabilities {
n.Capabilities[c] = true
}
}
return n
}
func tunnelOf(t delivery.RecordTunnel) compose.Tunnel {
out := compose.Tunnel{Interface: t.Interface, Unit: t.Unit, Config: t.Config, Port: t.Port, MTU: t.MTU,
Address: t.Address, Range: t.Range, PublicKey: t.PublicKey}
for _, p := range t.Peers {
out.Peers = append(out.Peers, compose.TunnelPeer{PublicKey: p.PublicKey, Address: p.Address})
}
return out
}
// carriedPeers is every peer of the hub's adopted tunnel, by address, each with the node that enrolled under its key
// where one has (a node whose tunnel key is the peer's) and the name a person gave it (inv.CarriedPeers).
func carriedPeers(t compose.Tunnel, names map[string]string, records map[string]delivery.NodeRecord) []compose.CarriedPeer {
byKey := map[string]string{}
for node, rec := range records {
if rec.TunnelKey != "" {
byKey[rec.TunnelKey] = node
}
}
var out []compose.CarriedPeer
for _, p := range t.Peers {
address, _, _ := strings.Cut(p.Address, "/")
out = append(out, compose.CarriedPeer{PublicKey: p.PublicKey, Address: address, EnrolledAs: byKey[p.PublicKey], Named: names[p.PublicKey]})
}
sort.SliceStable(out, func(i, j int) bool {
a, errA := netip.ParseAddr(out[i].Address)
b, errB := netip.ParseAddr(out[j].Address)
if errA != nil || errB != nil {
return out[i].Address < out[j].Address
}
return a.Less(b)
})
return out
}
// nodesOf is every node named by a placement or a record, by name: the order the controller reads them in.
func nodesOf(placements map[string]delivery.Placement, records map[string]delivery.NodeRecord) []string {
seen := map[string]bool{}
for n := range placements {
seen[n] = true
}
for n := range records {
seen[n] = true
}
return sortedKeys(seen)
}
func sortedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
+241
View File
@@ -0,0 +1,241 @@
package frombus
import (
"context"
"crypto/ecdh"
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"os"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
)
// bus is a Reader over values held in memory, counting whose config values were read.
type bus struct {
placements map[string]delivery.Placement
records map[string]delivery.NodeRecord
modules map[string]delivery.Module
manifests map[string][]byte
values map[string]map[string]any // by node.module
valuesRead []string
own []compose.OwnSecretRow
ownErr error
}
func (b *bus) Placements(context.Context) (map[string]delivery.Placement, error) {
return b.placements, nil
}
func (b *bus) Records(context.Context) (map[string]delivery.NodeRecord, error) { return b.records, nil }
func (b *bus) Modules(context.Context) (map[string]delivery.Module, error) { return b.modules, nil }
func (b *bus) Manifest(_ context.Context, digest string) ([]byte, error) {
raw, ok := b.manifests[digest]
if !ok {
return nil, errors.New("no such manifest")
}
return raw, nil
}
func (b *bus) OwnSecrets(context.Context, string) ([]compose.OwnSecretRow, error) {
return b.own, b.ownErr
}
func (b *bus) Credentials(context.Context, string, []compose.CredentialKey, []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) {
return nil, nil, nil
}
func (b *bus) Values(_ context.Context, node, module string) (map[string]any, error) {
b.valuesRead = append(b.valuesRead, node+"."+module)
return b.values[node+"."+module], nil
}
func publicKey(t *testing.T) string {
t.Helper()
k, err := ecdh.X25519().GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
return base64.StdEncoding.EncodeToString(k.PublicKey().Bytes())
}
// Issue 1126 (S3, S4, S5): the bus's placement, node records, manifests and config values fill the same compose.Input the
// controller's store fills for compose's own two-machine mesh (compose's aMesh: anchor the hub, laptop), and that Input
// composes; a node reads only its own config values; what cannot be composed is refused, naming it.
func TestInput(t *testing.T) {
ctx := context.Background()
xterm, err := os.ReadFile("../../catalogue/testdata/catalogue-835bf22b-xterm.module.json")
if err != nil {
t.Fatal(err)
}
digest := delivery.ManifestDigest(xterm)
// A module in the settings form (plex and the media modules on one node, 2026-10-26): its config values are in its
// layers, not mesh-config, so the placement carries them (MeshValues, Values) and mesh-config's are not read for it.
media := []byte(`{"module":"media","version":"1","settings":{"lines":{"kind":"preference","default":0,"why":"kept"},` +
`"name":{"kind":"preference","default":"x","why":"shown"}},"own-secrets":{"token":"/srv/media/token"},"resources":[{"id":"conf","type":"file",` +
`"path":"/srv/media/conf","mode":"0644","content":"uid=${setting:lines} name=${setting:name}\n"}]}`)
mediaDigest := delivery.ManifestDigest(media)
keys := map[string]string{"anchor": publicKey(t), "laptop": publicKey(t)}
caps := []string{"container-runtime", "systemd", "wireguard"}
tunnel := &delivery.RecordTunnel{Interface: "wg0", PublicKey: keys["anchor"], Address: "10.77.0.1/24",
Peers: []delivery.RecordTunnelPeer{{PublicKey: "peer", Address: "10.77.0.9/32"}, {PublicKey: keys["laptop"], Address: "10.77.0.2/32"}}}
mk := func() *bus {
return &bus{
placements: map[string]delivery.Placement{
"anchor": {Node: "anchor", Decided: delivery.NodeDecided{Endpoint: "anchor.example:51820", Site: "here", Hub: true, Address: "10.77.0.1",
Tunnel: tunnel, PeerNames: map[string]string{"peer": "phone"}},
Modules: []delivery.PlacedModule{{Module: overlay.Name}},
Holds: []delivery.Holding{{Claim: "mesh-private-network", Scope: "mesh", Module: overlay.Name}}},
"laptop": {Node: "laptop", Decided: delivery.NodeDecided{Endpoint: "laptop.example:51820", Site: "here", Address: "10.77.0.2"},
Modules: []delivery.PlacedModule{{Module: overlay.Name},
{Module: "xterm", HasLayer: true, Ports: map[string]int{"80": 8080}},
{Module: "media", HasMesh: true, MeshValues: map[string]any{"name": "mesh"}, HasLayer: true,
Layer: map[string]any{"places": map[string]any{"data": "/srv/data"}}, Values: map[string]any{"lines": float64(1000)}}}},
},
records: map[string]delivery.NodeRecord{
"anchor": {Node: "anchor", TunnelKey: keys["anchor"], Capabilities: caps, Contract: 2},
"laptop": {Node: "laptop", TunnelKey: keys["laptop"], Capabilities: caps, Authority: "the mesh's certificate"},
},
modules: map[string]delivery.Module{"xterm": {Manifest: digest}, "media": {Manifest: mediaDigest}, "unread": {}},
manifests: map[string][]byte{digest: xterm, mediaDigest: media},
values: map[string]map[string]any{"laptop.xterm": {"font": "mono"}, "anchor.xterm": {"font": "serif"},
"laptop.media": {"lines": "not mesh-config's"}},
own: []compose.OwnSecretRow{{Node: "laptop", Module: "media", Name: "token", Origin: "made", SealedTo: "sk", Sealed: "{}"}},
}
}
// What the controller's store gives for the same mesh, as compose's aMesh and SettingsFor build it.
var private catalogue.Manifest
raw, _ := json.Marshal(overlay.Manifest())
if err := json.Unmarshal(raw, &private); err != nil {
t.Fatal(err)
}
parsed, err := catalogue.ParseManifestForInputs(xterm)
if err != nil {
t.Fatal(err)
}
parsedMedia, err := catalogue.ParseManifestForInputs(media)
if err != nil {
t.Fatal(err)
}
capsSet := map[string]bool{"container-runtime": true, "systemd": true, "wireguard": true}
tun := compose.Tunnel{Interface: "wg0", PublicKey: keys["anchor"], Address: "10.77.0.1/24",
Peers: []compose.TunnelPeer{{PublicKey: "peer", Address: "10.77.0.9/32"}, {PublicKey: keys["laptop"], Address: "10.77.0.2/32"}}}
want := compose.Input{
Catalogue: map[string]catalogue.Manifest{overlay.Name: private, "xterm": parsed, "media": parsedMedia},
Nodes: []compose.Node{
{Name: "anchor", Place: compose.Place{Key: keys["anchor"], Endpoint: "anchor.example:51820", Site: "here", Hub: true,
Address: "10.77.0.1"}, Assigned: []string{overlay.Name}, Capabilities: capsSet, Contract: 2, Stated: true},
{Name: "laptop", Place: compose.Place{Key: keys["laptop"], Endpoint: "laptop.example:51820", Site: "here",
Address: "10.77.0.2"}, Assigned: []string{overlay.Name, "xterm", "media"}, Capabilities: capsSet,
Ports: map[string]map[int]int{"xterm": {80: 8080}},
Settings: map[string][]catalogue.Layer{"xterm": {{From: "laptop", Values: map[string]any{"font": "mono"}}},
"media": {{From: catalogue.MeshWideLayer, Values: map[string]any{"name": "mesh"}},
{From: "laptop", Values: map[string]any{"lines": float64(1000)}, Placement: map[string]any{"places": map[string]any{"data": "/srv/data"}}}}}},
},
Holdings: []catalogue.Held{{Claim: "mesh-private-network", Scope: "mesh", Node: "anchor", Module: overlay.Name}},
OwnSecrets: []compose.OwnSecretRow{{Node: "laptop", Module: "media", Name: "token", Origin: "made", SealedTo: "sk", Sealed: "{}"}},
// The same value as a core credential's box, for a core module (ADR 0349 §16).
KeptCoreBoxes: []compose.CoreBox{{Node: "laptop", Module: "media", Name: "token", SealedTo: "sk", Box: json.RawMessage("{}")}},
Tunnels: map[string]compose.FoundTunnel{"anchor": {Tunnel: tun}},
Adopted: &compose.AdoptedTunnel{Tunnel: tun, Hub: "anchor", Adopted: true},
// The controller's order is by address; laptop enrolled under its peer's key, and a person named the other.
Authority: "the mesh's certificate",
CarriedPeers: []compose.CarriedPeer{{PublicKey: keys["laptop"], Address: "10.77.0.2", EnrolledAs: "laptop"},
{PublicKey: "peer", Address: "10.77.0.9", Named: "phone"}},
}
b := mk()
got, err := Input(ctx, b, Mesh{}, Scope{Node: "laptop"})
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, want) {
gj, _ := json.MarshalIndent(got, "", " ")
wj, _ := json.MarshalIndent(want, "", " ")
t.Fatalf("Input from the bus differs from the store's:\n got %s\nwant %s", gj, wj)
}
if !reflect.DeepEqual(b.valuesRead, []string{"laptop." + overlay.Name, "laptop.xterm", "laptop.media"}) {
t.Errorf("laptop read config values %v; a node reads its own slice alone", b.valuesRead)
}
// The same composer over both: the compositions from the bus are the store's, byte for byte.
fps := map[string]string{overlay.Name: "src2:" + strings.Repeat("a", 64), "xterm": "src2:" + strings.Repeat("b", 64),
"media": "src2:" + strings.Repeat("c", 64)}
fromBus, _, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop"}, fps)
if err != nil {
t.Fatal(err)
}
fromStore, _, err := compose.Compositions(want, "laptop", fps, compose.Reading)
if err != nil {
t.Fatal(err)
}
if len(fromBus) == 0 || !reflect.DeepEqual(fromBus, fromStore) {
t.Fatalf("compositions differ or are empty: %d from the bus, %d from the store", len(fromBus), len(fromStore))
}
// ADR 0555 §6: Export is Input's inverse for what a person decides: the store's input exported, node by node, is the
// placement the bus holds, and read back it is the store's input again.
for node, placed := range mk().placements {
got, err := Export(want, node)
if err != nil {
t.Fatal(err)
}
if gj, _ := delivery.EncodePlacement(got); !delivery.SamePlacement(got, placed) {
pj, _ := delivery.EncodePlacement(placed)
t.Errorf("%s exported as %s, want %s", node, gj, pj)
}
}
if _, err := Export(want, "stranger"); err == nil {
t.Error("a node the input lacks was exported")
}
b = mk()
if every, err := Input(ctx, b, Mesh{}, Scope{Every: true}); err != nil || every.OwnSecrets != nil {
t.Fatalf("composing every node: %v, own secrets %v (the vault answers a node about itself alone)", err, every.OwnSecrets)
}
if !reflect.DeepEqual(b.valuesRead, []string{"anchor." + overlay.Name, "laptop." + overlay.Name, "laptop.xterm", "laptop.media"}) {
// Every node's placed modules, and nothing placed nowhere: anchor places no xterm.
t.Errorf("the check read %v", b.valuesRead)
}
// A node none of whose modules declares an own secret composes without asking the vault (the review of L2: a node
// with no secret composed nothing while the vault did not answer); one whose module does, refuses without it.
quiet := mk()
quiet.ownErr = errors.New("no responders")
if _, err := Input(ctx, quiet, Mesh{}, Scope{Node: "anchor"}); err != nil {
t.Errorf("anchor, with no own secret, needed the vault: %v", err)
}
if _, err := Input(ctx, quiet, Mesh{}, Scope{Node: "laptop"}); err == nil {
t.Error("laptop's media, which declares an own secret, composed without the vault")
}
kept := mk()
kept.manifests["old"] = xterm
in, err := Input(ctx, kept, Mesh{}, Scope{Node: "laptop", Kept: map[string]string{"xterm": "old"}})
if err != nil || !reflect.DeepEqual(in.Nodes[1].Kept, map[string]catalogue.Manifest{"xterm": parsed}) || in.Nodes[0].Kept != nil {
t.Errorf("the want's manifest is kept on the composing node alone: %v, %v", in.Nodes, err)
}
for name, c := range map[string]struct {
change func(*bus)
scope Scope
says string
}{
"no record": {func(b *bus) { delete(b.records, "laptop") }, Scope{Node: "anchor"}, "laptop, whose agent has written no record"},
"no placement": {func(b *bus) {}, Scope{Node: "stranger"}, "stranger has no placement"},
"a manifest unread": {func(b *bus) { delete(b.manifests, digest) }, Scope{Node: "laptop"}, "the manifest of xterm"},
"a kept one unread": {func(b *bus) {}, Scope{Node: "laptop", Kept: map[string]string{"xterm": "gone"}}, "digest gone"},
} {
b := mk()
c.change(b)
if _, err := Input(ctx, b, Mesh{}, c.scope); err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: %v, want it to say %q", name, err, c.says)
}
}
}
+1 -1
View File
@@ -113,7 +113,7 @@ func (c *composer) render(nodeName string, mode Mode) (Rendered, error) {
r.BusUsers = c.in.BusUsers r.BusUsers = c.in.BusUsers
if r.BusUsers == "" { if r.BusUsers == "" {
r.BusUsers = ForeseenBusUsers r.BusUsers = ForeseenBusUsers
foreseen("the bus's user list") foreseen(BusUsersStandIn)
} }
if len(c.in.BusMissing) > 0 { if len(c.in.BusMissing) > 0 {
out.Said = append(out.Said, fmt.Sprintf("the bus's user list leaves out %d user(s) the mesh has minted no credential "+ out.Said = append(out.Said, fmt.Sprintf("the bus's user list leaves out %d user(s) the mesh has minted no credential "+
+51
View File
@@ -916,6 +916,56 @@ type Process struct {
// Health is how this resource is ready (novox/hq ADR 0240 rule 2, Phase B): one kind and its // Health is how this resource is ready (novox/hq ADR 0240 rule 2, Phase B): one kind and its
// timing, judged by the node-engine beside liveness. Absent: judged alive or not, and nothing more. // timing, judged by the node-engine beside liveness. Absent: judged alive or not, and nothing more.
Health *Health `json:"health,omitempty"` Health *Health `json:"health,omitempty"`
// Yields says this process gives way to everything else on its machine (novox/hq ADR 0553, issue 1806): the
// runtime of a build seat's holder, whose builds must never slow the operator's desktop. Absent: the machine's
// defaults. For a process that stays up.
Yields *Yields `json:"yields,omitempty"`
}
// YieldIdle is the one priority a process yields at: the lowest there is.
const YieldIdle = "idle"
// YieldSlice is the one slice the node-engine places every yielding process of its node in (novox/hq ADR 0553 §2): a
// slice at the top of the hierarchy, beside the operator's user slice and the system's, so its idle weight and its
// quota bind against everything else on the machine. No dash in its name: a dash makes a slice a child of the slice
// named before it, where its weight would count only against its siblings.
const YieldSlice = "meshyield.slice"
// Yields is how a process gives way, written by the node-engine (novox/hq ADR 0553).
//
// **Priority** "idle": the process runs in YieldSlice, whose CPUWeight is idle (it runs only on cycles nothing else
// wants) and whose IOWeight is 1, with Nice=19 and the lowest best-effort IO level for every process it starts. The IO
// controls bind only where the disk's scheduler honours them (bfq, or io.cost for the weight); under `none`, the
// usual one for NVMe, they do nothing, and the CPU bound is what holds. **CPUShare** is the most of the machine's
// processors the slice may take, in percent, shared by every yielding process of the node together: the node-engine
// writes it as the slice's CPUQuota for the processors the machine has, so the bound is the kernel's. Every yielding
// process of one node carries the same share: the composer gives the node's one value (ADR 0553 §3). 0 is no cap.
type Yields struct {
Priority string `json:"priority,omitempty"`
CPUShare int `json:"cpu-share,omitempty"`
}
func (y *Yields) problems(where string, staysUp bool) []string {
if y == nil {
return nil
}
var problems []string
if !staysUp {
problems = append(problems, where+": yields is for a process that stays up")
}
switch y.Priority {
case "", YieldIdle:
default:
problems = append(problems, fmt.Sprintf("%s: yields priority %q is not one this host knows: idle", where, y.Priority))
}
if y.CPUShare < 0 || y.CPUShare > 100 {
problems = append(problems, fmt.Sprintf("%s: yields cpu-share %d is not a percent of the machine, 0 to 100", where, y.CPUShare))
}
if y.Priority == "" && y.CPUShare == 0 {
problems = append(problems, where+": yields says neither a priority nor a cpu-share")
}
return problems
} }
func (d *Process) Identity() string { return d.ID } func (d *Process) Identity() string { return d.ID }
@@ -948,6 +998,7 @@ func ProcessNameProblem(name string) string {
func (d *Process) validate(where string, _ bool) []string { func (d *Process) validate(where string, _ bool) []string {
problems := d.Health.problems(where, false, !d.RunOnce && d.Schedule == "") problems := d.Health.problems(where, false, !d.RunOnce && d.Schedule == "")
problems = append(problems, d.Yields.problems(where, !d.RunOnce && d.Schedule == "")...)
if problem := ProcessNameProblem(d.Name); problem != "" { if problem := ProcessNameProblem(d.Name); problem != "" {
problems = append(problems, where+": "+problem) problems = append(problems, where+": "+problem)
} }
+19
View File
@@ -92,6 +92,20 @@ func TestTheModesAreExclusive(t *testing.T) {
if problems := watching.validate("a process", false); len(problems) == 0 { if problems := watching.validate("a process", false); len(problems) == 0 {
t.Fatal("a scheduled process was given something to restart on, and it is never running") t.Fatal("a scheduled process was given something to restart on, and it is never running")
} }
// A process yields only while it stays up, at a priority the host knows, within the machine (issue 1806).
for name, bad := range map[string]func(*Process){
"a step that yields": func(p *Process) { p.RunOnce = true; p.Yields = &Yields{Priority: YieldIdle} },
"an unknown priority": func(p *Process) { p.Yields = &Yields{Priority: "low"} },
"a share over the whole": func(p *Process) { p.Yields = &Yields{CPUShare: 150} },
"a yield saying nothing": func(p *Process) { p.Yields = &Yields{} },
} {
p := aProcess()
bad(p)
if problems := p.validate("a process", false); len(problems) == 0 {
t.Errorf("%s was accepted", name)
}
}
} }
// A cadence that is not a cadence is refused near its author, rather than by a machine at the far // A cadence that is not a cadence is refused near its author, rather than by a machine at the far
@@ -118,6 +132,11 @@ func TestEachModeOnItsOwnIsAccepted(t *testing.T) {
if problems := every.validate("a process", false); len(problems) != 0 { if problems := every.validate("a process", false); len(problems) != 0 {
t.Fatalf("a scheduled process was refused: %v", problems) t.Fatalf("a scheduled process was refused: %v", problems)
} }
yielding := aProcess()
yielding.Yields = &Yields{Priority: YieldIdle, CPUShare: 50}
if problems := yielding.validate("a process", false); len(problems) != 0 {
t.Fatalf("a daemon yielding at idle under half the machine was refused: %v", problems)
}
} }
// **The one that cannot be escaped, only refused.** // **The one that cannot be escaped, only refused.**
+43 -10
View File
@@ -166,8 +166,9 @@ func checkFingerprint(fp, of string) error {
return nil return nil
} }
// Want is a node's want for a module: the fingerprint FP the node should run and the digest Composed of the one // Want is a node's want for a module: the fingerprint FP the node should run and, before ADR 0547's second milestone,
// composition of it the node may apply (ADR 0539 clause 1), or Removed for a module that left the node's modules (then // the digest Composed of the one composition of it the node may apply (ADR 0539 clause 1; from that milestone the
// node composes its own, ADR 0547 §2, and a want names none), or Removed for a module that left the node's modules (then
// both are empty). When the module's newest fingerprint is not yet allowed on this node, Newest names it and Rule the // both are empty). When the module's newest fingerprint is not yet allowed on this node, Newest names it and Rule the
// rule that holds the node back; both are empty when FP is the newest. Backup is the answer to a failed backup before // rule that holds the node back; both are empty when FP is the newest. Backup is the answer to a failed backup before
// the apply, BackupAgain or BackupSkip (clause 16), empty when none was asked. Words say why in a sentence. The // the apply, BackupAgain or BackupSkip (clause 16), empty when none was asked. Words say why in a sentence. The
@@ -185,11 +186,18 @@ type Want struct {
// is released. A put's want names no newest fingerprint or rule. // is released. A put's want names no newest fingerprint or rule.
Put bool `json:"put,omitempty"` Put bool `json:"put,omitempty"`
Hold bool `json:"hold,omitempty"` Hold bool `json:"hold,omitempty"`
// Yes is the digest of a composition a person allowed the node to move to (novox/hq ADR 0547 §3): a move of a module
// of policy record that the node agent judged to recreate a container or move its data is applied only when its
// want names the composition so allowed. Empty when none was asked.
Yes string `json:"yes,omitempty"`
// Self says the node composes this module itself (ADR 0547 §2): the want names the build alone, and no composition.
// A want that is not Self names the one composition it allows (ADR 0539 clause 1), as before.
Self bool `json:"self,omitempty"`
} }
func (w Want) check() error { func (w Want) check() error {
if w.Removed { if w.Removed {
if w.FP != "" || w.Composed != "" || w.Newest != "" || w.Rule != 0 || w.Backup != "" || w.Put || w.Hold { if w.FP != "" || w.Composed != "" || w.Newest != "" || w.Rule != 0 || w.Backup != "" || w.Put || w.Hold || w.Self || w.Yes != "" {
return fmt.Errorf("delivery: a removed want names no fingerprint, composition, rule, backup answer or put") return fmt.Errorf("delivery: a removed want names no fingerprint, composition, rule, backup answer or put")
} }
return nil return nil
@@ -203,11 +211,22 @@ func (w Want) check() error {
if err := checkFingerprint(w.FP, "a want's fingerprint"); err != nil { if err := checkFingerprint(w.FP, "a want's fingerprint"); err != nil {
return err return err
} }
if w.Composed == "" { // From ADR 0547's second milestone a node the catalogue places composes its own modules (§2): its want is Self and
return fmt.Errorf("delivery: the want of %s names no composition: a want names the digest of the one composition it allows (ADR 0539 clause 1)", w.FP) // names the build alone. Any other want names the one composition it allows (ADR 0539 clause 1).
switch {
case w.Self && w.Composed != "":
return fmt.Errorf("delivery: the want of %s says the node composes it, and names a composition", w.FP)
case !w.Self && w.Composed == "":
return fmt.Errorf("delivery: the want of %s names no composition: a want names the digest of the one composition it allows (ADR 0539 clause 1), unless the node composes it itself", w.FP)
case w.Composed != "":
if err := checkDigest(w.Composed, "a want's composition digest"); err != nil {
return err
}
} }
if err := checkDigest(w.Composed, "a want's composition digest"); err != nil { if w.Yes != "" {
return err if err := checkDigest(w.Yes, "the digest a want's yes allows"); err != nil {
return err
}
} }
switch w.Backup { switch w.Backup {
case "", BackupAgain, BackupSkip: case "", BackupAgain, BackupSkip:
@@ -309,11 +328,14 @@ type NodeWant struct {
V int `json:"v"` V int `json:"v"`
Composed string `json:"composed"` Composed string `json:"composed"`
Words string `json:"words,omitempty"` Words string `json:"words,omitempty"`
// Self says the node composes its own composition (ADR 0547 §2) and the want names none.
Self bool `json:"self,omitempty"`
} }
// EncodeNodeWant is w as JSON at Version; it refuses a digest not of the sha256 form. // EncodeNodeWant is w as JSON at Version; it refuses a digest not of the sha256 form, and a want that names no
// composition unless it says the node composes its own (Self, ADR 0547 §2).
func EncodeNodeWant(w NodeWant) ([]byte, error) { func EncodeNodeWant(w NodeWant) ([]byte, error) {
if err := checkDigest(w.Composed, "the node's own want's composition digest"); err != nil { if err := w.check(); err != nil {
return nil, err return nil, err
} }
w.V = Version w.V = Version
@@ -332,12 +354,23 @@ func DecodeNodeWant(data []byte) (NodeWant, error) {
if err := d.Decode(&w); err != nil { if err := d.Decode(&w); err != nil {
return NodeWant{}, fmt.Errorf("delivery: a node's own want does not fit its shape: %w", err) return NodeWant{}, fmt.Errorf("delivery: a node's own want does not fit its shape: %w", err)
} }
if err := checkDigest(w.Composed, "the node's own want's composition digest"); err != nil { if err := w.check(); err != nil {
return NodeWant{}, err return NodeWant{}, err
} }
return w, nil return w, nil
} }
// check refuses a node's own want that names no composition and does not say the node composes its own, or does both.
func (w NodeWant) check() error {
switch {
case w.Self && w.Composed != "":
return fmt.Errorf("delivery: the node's own want says the node composes it, and names a composition")
case w.Self:
return nil
}
return checkDigest(w.Composed, "the node's own want's composition digest")
}
// EncodeHeld is h as JSON at Version; it refuses a node name a key refuses. // EncodeHeld is h as JSON at Version; it refuses a node name a key refuses.
func EncodeHeld(h Held) ([]byte, error) { func EncodeHeld(h Held) ([]byte, error) {
if err := h.check(); err != nil { if err := h.check(); err != nil {
+17
View File
@@ -86,6 +86,9 @@ func (k Asker) Wait(ctx context.Context, a delivery.StepAsk) (delivery.Step, err
if !ok { if !ok {
return delivery.Step{}, errors.New("the watch of the ask's step ended") return delivery.Step{}, errors.New("the watch of the ask's step ended")
} }
if seen := stepSeenOf(ctx); seen != nil {
seen(step)
}
if step.State != delivery.StepRunning { if step.State != delivery.StepRunning {
return step, nil return step, nil
} }
@@ -108,6 +111,20 @@ func (k Asker) Wait(ctx context.Context, a delivery.StepAsk) (delivery.Step, err
} }
} }
type stepSeenKey struct{}
// WithStepSeen is ctx carrying seen, which Asker.Wait hands every value the ask's step key takes while it waits: the node
// that took it, each phase it says while running (delivery.Step.Phase), and its end. A try names its asks' progress so
// (issue 1890). seen runs in the waiting goroutine and must not block.
func WithStepSeen(ctx context.Context, seen func(delivery.Step)) context.Context {
return context.WithValue(ctx, stepSeenKey{}, seen)
}
func stepSeenOf(ctx context.Context) func(delivery.Step) {
seen, _ := ctx.Value(stepSeenKey{}).(func(delivery.Step))
return seen
}
// Ask puts a and waits for its step; a step that did not pass is an error naming its node, its state and its reason. // Ask puts a and waits for its step; a step that did not pass is an error naming its node, its state and its reason.
func (k Asker) Ask(ctx context.Context, a delivery.StepAsk) (delivery.Step, error) { func (k Asker) Ask(ctx context.Context, a delivery.StepAsk) (delivery.Step, error) {
if err := k.Put(ctx, a); err != nil { if err := k.Put(ctx, a); err != nil {
+99 -3
View File
@@ -2,11 +2,15 @@ package buildrun
import ( import (
"context" "context"
"encoding/json"
"errors"
"strings" "strings"
"sync"
"testing" "testing"
"time" "time"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
) )
func goAsk(id string) delivery.StepAsk { func goAsk(id string) delivery.StepAsk {
@@ -22,9 +26,10 @@ func TestAnAskerPutsOnceAndReadsTheStep(t *testing.T) {
return delivery.Step{ID: a.ID, Node: "ace", State: delivery.StepPassed, Result: []byte(`{"artifacts":[]}`)} return delivery.Step{ID: a.ID, Node: "ace", State: delivery.StepPassed, Result: []byte(`{"artifacts":[]}`)}
}} }}
k := Asker{Publisher: f, Waiter: f} k := Asker{Publisher: f, Waiter: f}
step, err := k.Ask(context.Background(), goAsk("mvl0frv18d1957")) var seen []delivery.Step // issue 1890: a waiter that asks to see each value of the step sees it
if err != nil || step.Node != "ace" || len(f.Asks()) != 1 { step, err := k.Ask(WithStepSeen(context.Background(), func(s delivery.Step) { seen = append(seen, s) }), goAsk("mvl0frv18d1957"))
t.Fatalf("step %+v, %v, asks %d", step, err, len(f.Asks())) if err != nil || step.Node != "ace" || len(f.Asks()) != 1 || len(seen) != 1 || seen[0].Node != "ace" {
t.Fatalf("step %+v, %v, asks %d, seen %+v", step, err, len(f.Asks()), seen)
} }
f.Answer = func(a delivery.StepAsk) delivery.Step { f.Answer = func(a delivery.StepAsk) delivery.Step {
return delivery.Step{ID: a.ID, Node: "shanks", State: delivery.StepRefused, Reason: "go 1.27 installed"} return delivery.Step{ID: a.ID, Node: "shanks", State: delivery.StepRefused, Reason: "go 1.27 installed"}
@@ -48,6 +53,74 @@ func TestAnAskerPutsOnceAndReadsTheStep(t *testing.T) {
if len(g.Asks()) != 0 { if len(g.Asks()) != 0 {
t.Fatal("a wait put an ask") t.Fatal("a wait put an ask")
} }
// Issue 734: an ask the build seat gave up on (DEAD_LETTERS keeps its copy with the ask's id) ends its wait as failed
// with the words, though a node had taken it; under NotTakenBound one no node takes ends at the bound; a step written
// first is answered as the node wrote it.
dead := &DeadLetters{}
lw := LostAskWaiter{Inner: g, Dead: dead}
k = Asker{Publisher: g, Waiter: lw, Bound: time.Minute}
g.Write(delivery.Step{ID: "mvl0frv18d1970", Node: "ace", State: delivery.StepRunning, Started: time.Now()})
given, ok := GivenUpOf(map[string][]string{"Mesh-Dead-Msg-Id": {"mvl0frv18d1970"}, "Mesh-Dead-Deliveries": {"5"},
"Mesh-Dead-Stream": {"SEAT_NODE_BUILD_GO"}, "Mesh-Dead-Gave-Up": {"2026-10-26T12:26:40Z"}})
if !ok {
t.Fatal("a dead letter's headers were not read")
}
go dead.Heard(given)
step, err = k.Wait(context.Background(), goAsk("mvl0frv18d1970"))
if err != nil || step.State != delivery.StepFailed || !IsLost(step) ||
!strings.Contains(step.Reason, LostWords+" after 5 deliveries at 2026-10-26T12:26:40Z") {
t.Fatalf("a given-up ask: %+v, %v", step, err)
}
if _, err := k.Passed(step, nil); err == nil || !strings.Contains(err.Error(), "on "+LostNode+" failed: "+LostWords) {
t.Fatalf("a given-up ask's error: %v", err)
}
step, err = k.Wait(NotTakenBound(context.Background(), 20*time.Millisecond), goAsk("mvl0frv18d1971"))
if err != nil || !IsNotTaken(step) || IsLost(step) {
t.Fatalf("an ask no node took: %+v, %v", step, err)
}
g.Write(delivery.Step{ID: "mvl0frv18d1972", Node: "g14", State: delivery.StepPassed})
if step, err = k.Wait(NotTakenBound(context.Background(), time.Minute), goAsk("mvl0frv18d1972")); err != nil || step.Node != "g14" {
t.Fatalf("a step written: %+v, %v", step, err)
}
// Review round 1 of issue 734: a hearing the bus refuses at start is deaf and says so, never holds the start up, and
// tries again on the next ask waited for; heard, it says so and hands the given-up asks on.
fol := &refusingFollower{refuse: 1}
var mu sync.Mutex
var told []Hearing
hearing := NewDeadLetterHearing(context.Background(), fol, []string{delivery.AskGo}, dead, nil,
func(h Hearing) { mu.Lock(); told = append(told, h); mu.Unlock() })
hearing.Wait()
if why := hearing.Deaf(); !strings.Contains(why, "SEAT_NODE_BUILD_GO: permissions violation") {
t.Fatalf("a refused hearing says %q", why)
}
k = Asker{Publisher: g, Waiter: LostAskWaiter{Inner: g, Dead: dead, Hearing: hearing}, Bound: time.Minute}
g.Write(delivery.Step{ID: "mvl0frv18d1973", Node: "g14", State: delivery.StepPassed})
if _, err := k.Wait(context.Background(), goAsk("mvl0frv18d1973")); err != nil {
t.Fatal(err)
}
hearing.Wait()
mu.Lock()
if hearing.Deaf() != "" || len(told) != 2 || told[0].Heard || !told[1].Heard || fol.calls() != 2 {
t.Fatalf("after an ask: deaf %q, told %+v, followed %d times", hearing.Deaf(), told, fol.calls())
}
mu.Unlock()
fol.heard(GivenUp{AskID: "mvl0frv18d1974", Deliveries: 5})
step, err = k.Wait(context.Background(), goAsk("mvl0frv18d1974"))
if err != nil || !IsLost(step) {
t.Fatalf("an ask heard given up after the hearing came back: %+v, %v", step, err)
}
for _, row := range []struct {
h *Hearing
want string
}{{nil, health.StateUnhealthy}, {&told[0], health.StateUnhealthy}, {&told[1], health.StateHealthy}} {
var raw []byte
if row.h != nil {
raw, _ = json.Marshal(row.h)
}
if a := HearingHealth(raw); a.State != row.want || a.Why == "" {
t.Errorf("health of %+v = %+v, want %s", row.h, a, row.want)
}
}
} }
// Review (issue 734 path): an ask taken again by the same node (its build module replaced mid-run) is bounded from the // Review (issue 734 path): an ask taken again by the same node (its build module replaced mid-run) is bounded from the
@@ -69,3 +142,26 @@ func TestARedeliveryToTheSameNodeRestartsTheBound(t *testing.T) {
t.Fatalf("step %+v, %v, want the redelivered run's pass", step, err) t.Fatalf("step %+v, %v, want the redelivered run's pass", step, err)
} }
} }
// refusingFollower refuses the first `refuse` follows, as the bus does before the asker's grant is composed.
type refusingFollower struct {
mu sync.Mutex
refuse int
n int
hand func(GivenUp)
}
func (f *refusingFollower) Follow(_ context.Context, _ string, _ time.Time, heard func(GivenUp)) (func(), error) {
f.mu.Lock()
defer f.mu.Unlock()
f.n++
if f.n <= f.refuse {
return nil, errors.New("permissions violation")
}
f.hand = heard
return func() {}, nil
}
func (f *refusingFollower) calls() int { f.mu.Lock(); defer f.mu.Unlock(); return f.n }
func (f *refusingFollower) heard(g GivenUp) { f.mu.Lock(); h := f.hand; f.mu.Unlock(); h(g) }
+503
View File
@@ -0,0 +1,503 @@
package buildrun
import (
"context"
"encoding/json"
"errors"
"fmt"
"strconv"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
)
// **An ask the build seat gave up on ends its step** (novox/hq issue 734). A build seat's worker offers an ask
// delivery.AskMaxDeliver times; one handed out that often without an answer (a bus reload that drops every hand-out, a node
// that dies taking it each time) is given up by the server, and the controller keeps a copy of it in DEAD_LETTERS under
// mesh.events.dead.<seat stream>.<worker>, with the ask's message id (its ask id), how often it was handed out and when it
// was given up. No build node writes a step for it, so an asker that waits for the step waits for ever: a try stays
// running and holds a try place, a check's verdict stays running and its pull request waits.
//
// An asker of the build seats (the checker, the builder) hears those copies (DeadLetterHearing: an ordered consumer on
// DEAD_LETTERS filtered to the build seats' streams, granted to a user of a seat pulled by kind, buscompose
// DeadLetterReads) into DeadLetters, and LostAskWaiter ends the waiting step at once as failed on the node LostNode, its
// reason saying LostWords; the asker decides what follows (the checker ends a try as error and asks a check again, the
// builder asks its part of a build again).
//
// Belt and braces, where the asker says so in the context (NotTakenBound; the checker's tries): an ask no build node took
// within a bound ends the same way, saying NotTakenWords. An ask waits in the stream as long as the backlog is otherwise
// (ADR 0525 §9): a backlog is no fault.
// LostNode is the node a step ended for a lost ask names: no build node wrote it.
const LostNode = "bus"
// The words of a step ended for its ask.
const (
LostWords = "its ask was given up by the build seat"
NotTakenWords = "no build node took its ask"
)
// The headers the controller keeps a given-up message's copy with in DEAD_LETTERS (mesh-controller internal/link
// deadletters.go): the original message id (an ask's is its ask id), how often it was handed out, when it was given up.
const (
DeadHeaderMsgID = "Mesh-Dead-Msg-Id"
DeadHeaderDeliveries = "Mesh-Dead-Deliveries"
DeadHeaderGaveUp = "Mesh-Dead-Gave-Up"
DeadHeaderStream = "Mesh-Dead-Stream"
)
// DeadLettersStream is the stream the controller keeps given-up messages in, and DeadLetterFilter the subjects of one seat
// stream's copies there (buscompose DeadLetterSubject).
const DeadLettersStream = "DEAD_LETTERS"
// DeadLetterFilter is the subjects DEAD_LETTERS keeps a seat stream's given-up messages under.
func DeadLetterFilter(stream string) string { return "mesh.events.dead." + stream + ".>" }
// SeatStream is the stream of a build seat's language (buscompose seatStreamName of node-build-<language>).
func SeatStream(language string) string { return "SEAT_NODE_BUILD_" + strings.ToUpper(language) }
// GivenUp is one ask a build seat gave up on, as DEAD_LETTERS keeps it.
type GivenUp struct {
AskID string
Stream string
Deliveries int
At time.Time
}
// GivenUpOf reads a DEAD_LETTERS copy's headers; false when it names no message id.
func GivenUpOf(header map[string][]string) (GivenUp, bool) {
get := func(k string) string {
if v := header[k]; len(v) > 0 {
return strings.TrimSpace(v[0])
}
return ""
}
g := GivenUp{AskID: get(DeadHeaderMsgID), Stream: get(DeadHeaderStream)}
if g.AskID == "" {
return GivenUp{}, false
}
g.Deliveries, _ = strconv.Atoi(get(DeadHeaderDeliveries))
g.At, _ = time.Parse(time.RFC3339Nano, get(DeadHeaderGaveUp))
return g, true
}
// Words are what a step ended for the given-up ask says.
func (g GivenUp) Words() string {
at := "at an unknown time"
if !g.At.IsZero() {
at = "at " + g.At.UTC().Format(time.RFC3339)
}
return fmt.Sprintf("%s after %d deliveries %s (kept in %s); ask again", LostWords, g.Deliveries, at, DeadLettersStream)
}
// keptGivenUp is how many given-up asks DeadLetters remembers for a waiter that starts after the copy was heard.
const keptGivenUp = 1024
// DeadLetters holds the given-up asks heard, and hands each to the waiter of its ask. Safe for concurrent use.
type DeadLetters struct {
mu sync.Mutex
heard map[string]GivenUp
order []string
watchers map[string][]chan GivenUp
}
// Heard takes one given-up ask in.
func (d *DeadLetters) Heard(g GivenUp) {
d.mu.Lock()
if d.heard == nil {
d.heard, d.watchers = map[string]GivenUp{}, map[string][]chan GivenUp{}
}
if _, ok := d.heard[g.AskID]; !ok {
d.order = append(d.order, g.AskID)
if len(d.order) > keptGivenUp {
delete(d.heard, d.order[0])
d.order = d.order[1:]
}
}
d.heard[g.AskID] = g
ws := d.watchers[g.AskID]
delete(d.watchers, g.AskID)
d.mu.Unlock()
for _, w := range ws {
w <- g // buffered, one each
}
}
// watch answers a channel that gets the ask's given-up copy once (at once when it was heard already), and its cancel.
func (d *DeadLetters) watch(askID string) (<-chan GivenUp, func()) {
ch := make(chan GivenUp, 1)
d.mu.Lock()
defer d.mu.Unlock()
if d.heard == nil {
d.heard, d.watchers = map[string]GivenUp{}, map[string][]chan GivenUp{}
}
if g, ok := d.heard[askID]; ok {
ch <- g
return ch, func() {}
}
d.watchers[askID] = append(d.watchers[askID], ch)
return ch, func() {
d.mu.Lock()
defer d.mu.Unlock()
ws := d.watchers[askID]
for i, w := range ws {
if w == ch {
d.watchers[askID] = append(ws[:i], ws[i+1:]...)
break
}
}
if len(d.watchers[askID]) == 0 {
delete(d.watchers, askID)
}
}
}
type notTakenKey struct{}
// NotTakenBound says, in ctx, that an ask no build node took within bound ends its step (a try's, never a check's).
func NotTakenBound(ctx context.Context, bound time.Duration) context.Context {
return context.WithValue(ctx, notTakenKey{}, bound)
}
func notTakenBoundOf(ctx context.Context) time.Duration {
d, _ := ctx.Value(notTakenKey{}).(time.Duration)
return d
}
// LostAskWaiter is an AskWaiter that ends an ask's step when the build seat gave the ask up (DeadLetters), or, under
// NotTakenBound, when no build node took it within the bound: it hands the asker a failed step whose reason says why, on
// the node "bus". Every step the build module writes is handed on as it is.
type LostAskWaiter struct {
Inner AskWaiter
Dead *DeadLetters
// Hearing, when set, is asked to hear DEAD_LETTERS again each time an ask is waited for while it cannot: an ask put is
// the event that makes hearing matter.
Hearing interface {
Deaf() string
Again()
}
Now func() time.Time
}
func (w LostAskWaiter) now() time.Time {
if w.Now != nil {
return w.Now()
}
return time.Now()
}
// WatchAsk is AskWaiter's.
func (w LostAskWaiter) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
if w.Hearing != nil && w.Hearing.Deaf() != "" {
w.Hearing.Again()
}
inner, err := w.Inner.WatchAsk(ctx, module, askID)
if err != nil {
return nil, err
}
var dead <-chan GivenUp
cancel := func() {}
if w.Dead != nil {
dead, cancel = w.Dead.watch(askID)
}
// The not-taken bound is a context's (ADR 0437 §8: no timer), ended when a node takes the ask.
var notTaken <-chan struct{}
bound := notTakenBoundOf(ctx)
stopBound := func() {}
if bound > 0 {
var nt context.Context
nt, stopBound = context.WithTimeout(context.Background(), bound)
notTaken = nt.Done()
}
out := make(chan delivery.Step)
go func() {
defer close(out)
defer cancel()
defer func() { stopBound() }()
send := func(s delivery.Step) bool {
select {
case out <- s:
return true
case <-ctx.Done():
return false
}
}
ended := func(reason string) delivery.Step {
at := w.now().UTC()
return delivery.Step{ID: askID, Node: LostNode, State: delivery.StepFailed, Reason: reason, Ended: at}
}
for {
select {
case <-ctx.Done():
return
case s, ok := <-inner:
if !ok {
return
}
if notTaken != nil {
stopBound() // a node took it: its step is bounded by the asker from its start
notTaken = nil
}
if !send(s) {
return
}
case g := <-dead:
dead = nil
send(ended(g.Words()))
return
case <-notTaken:
send(ended(fmt.Sprintf("%s within %s of its put; it was lost or waits behind a backlog; ask again", NotTakenWords, bound)))
return
}
}
}()
return out, nil
}
// IsLost says whether a step was ended by LostAskWaiter because the build seat gave its ask up (LostWords).
func IsLost(s delivery.Step) bool { return s.Node == LostNode && strings.Contains(s.Reason, LostWords) }
// IsNotTaken says whether a step was ended by LostAskWaiter because no build node took its ask within the bound.
func IsNotTaken(s delivery.Step) bool {
return s.Node == LostNode && strings.Contains(s.Reason, NotTakenWords)
}
// **The asker hears DEAD_LETTERS, tries again when it cannot, and says whether it hears** (issue 734). The read is
// granted with the asker's seat traffic (buscompose DeadLetterReads); an asker started before its grant was composed is
// refused, and an asker that only tried once would stay deaf until it starts again, silently. DeadLetterHearing tries in
// the background, never holding the start up, and again on an event (Again): the asker's reconnect to the bus, and every
// ask waited for while it is deaf (LostAskWaiter.Hearing); no timer. Each change of whether it hears is said (Said), so
// the module's health tool says it: a deaf asker is not healthy.
// Hearing is what a DeadLetterHearing says: whether it hears every build seat stream it follows, and why not.
type Hearing struct {
Heard bool `json:"heard"`
Streams []string `json:"streams"`
Why string `json:"why,omitempty"`
At time.Time `json:"at"`
}
// HearingKey is the key a module keeps its Hearing under, in its own state (the checker's and the builder's hearing).
const HearingKey = "dead-letters"
// DeadLetterFollower starts following one seat stream's copies in DEAD_LETTERS from since, handing each given-up ask to
// heard; it answers the stop, or why it cannot.
type DeadLetterFollower interface {
Follow(ctx context.Context, stream string, since time.Time, heard func(GivenUp)) (func(), error)
}
// JetStreamDeadLetters is the bus side of a DeadLetterFollower: an ordered consumer on DEAD_LETTERS filtered to the
// stream's copies, which resumes where it was after a reconnect and, reset before it heard any, starts at since again.
type JetStreamDeadLetters struct{ JS jetstream.JetStream }
// Follow is DeadLetterFollower's.
func (j JetStreamDeadLetters) Follow(ctx context.Context, stream string, since time.Time, heard func(GivenUp)) (func(), error) {
cons, err := j.JS.OrderedConsumer(ctx, DeadLettersStream, jetstream.OrderedConsumerConfig{
FilterSubjects: []string{DeadLetterFilter(stream)}, DeliverPolicy: jetstream.DeliverByStartTimePolicy, OptStartTime: &since})
if err != nil {
return nil, err
}
cc, err := cons.Consume(func(m jetstream.Msg) {
if g, ok := GivenUpOf(m.Headers()); ok {
heard(g)
}
})
if err != nil {
return nil, err
}
return cc.Stop, nil
}
// DeadLetterHearing follows the build seats' given-up asks into Dead. Make it with NewDeadLetterHearing.
type DeadLetterHearing struct {
follower DeadLetterFollower
streams []string
dead *DeadLetters
logf func(string, ...any)
said func(Hearing)
ctx context.Context
since time.Time
mu sync.Mutex
stops map[string]func()
why map[string]string
trying bool
again bool
told *Hearing
wg sync.WaitGroup
}
// NewDeadLetterHearing starts hearing the given-up asks of each language's build seat stream, in the background. said
// is told each change of whether it hears (nil tells nothing); ctx ends it.
func NewDeadLetterHearing(ctx context.Context, follower DeadLetterFollower, languages []string, dead *DeadLetters,
logf func(string, ...any), said func(Hearing)) *DeadLetterHearing {
h := &DeadLetterHearing{follower: follower, dead: dead, logf: logf, said: said, ctx: ctx, since: time.Now(),
stops: map[string]func(){}, why: map[string]string{}}
for _, l := range languages {
s := SeatStream(l)
h.streams = append(h.streams, s)
h.why[s] = "not tried yet"
}
if h.logf == nil {
h.logf = func(string, ...any) {}
}
h.Again()
return h
}
// Deaf says why a stream is not heard, or "" when every one is.
func (h *DeadLetterHearing) Deaf() string {
h.mu.Lock()
defer h.mu.Unlock()
return h.deafLocked()
}
func (h *DeadLetterHearing) deafLocked() string {
var out []string
for _, s := range h.streams {
if why, deaf := h.why[s]; deaf {
out = append(out, s+": "+why)
}
}
if len(out) == 0 {
return ""
}
return DeadLettersStream + " is not heard, so an ask a build seat gives up on is waited for until its bound: " +
strings.Join(out, "; ")
}
// Again tries, in the background, every stream not heard yet; a call while it tries makes it try once more after.
func (h *DeadLetterHearing) Again() {
h.mu.Lock()
if h.trying {
h.again = true
h.mu.Unlock()
return
}
h.trying = true
h.mu.Unlock()
h.wg.Add(1)
go func() {
defer h.wg.Done()
for {
h.try()
h.mu.Lock()
if !h.again || h.ctx.Err() != nil {
h.trying = false
h.mu.Unlock()
return
}
h.again = false
h.mu.Unlock()
}
}()
}
// Wait returns when no try runs.
func (h *DeadLetterHearing) Wait() { h.wg.Wait() }
// Stop ends every stream's following.
func (h *DeadLetterHearing) Stop() {
h.mu.Lock()
defer h.mu.Unlock()
for s, stop := range h.stops {
stop()
delete(h.stops, s)
}
}
func (h *DeadLetterHearing) try() {
for _, s := range h.streams {
h.mu.Lock()
_, deaf := h.why[s]
h.mu.Unlock()
if !deaf || h.ctx.Err() != nil {
continue
}
stream := s
stop, err := h.follower.Follow(h.ctx, stream, h.since, func(g GivenUp) {
h.logf("the ask %s on %s was given up after %d deliveries at %s: the wait for its step ends", g.AskID, g.Stream,
g.Deliveries, g.At.Format(time.RFC3339))
h.dead.Heard(g)
})
h.mu.Lock()
if err != nil {
h.why[stream] = err.Error()
h.logf("the asks %s gives up on are not heard (%s of %s); tried again at the next reconnect or ask: %v",
stream, DeadLetterFilter(stream), DeadLettersStream, err)
} else {
delete(h.why, stream)
h.stops[stream] = stop
h.logf("the asks %s gives up on are heard from %s", stream, DeadLettersStream)
}
h.mu.Unlock()
}
h.tell()
}
// tell says the hearing when it changed.
func (h *DeadLetterHearing) tell() {
h.mu.Lock()
why := h.deafLocked()
now := Hearing{Heard: why == "", Streams: append([]string(nil), h.streams...), Why: why, At: time.Now().UTC()}
same := h.told != nil && h.told.Heard == now.Heard && h.told.Why == now.Why
if !same {
h.told = &now
}
h.mu.Unlock()
if !same && h.said != nil {
h.said(now)
}
}
// HearingState is the name of the state an asker keeps its Hearing in (`<module>_hearing`, key HearingKey), which its
// health tool reads.
const HearingState = "hearing"
// HearingKeeper is where a Hearing is kept: a jetstream.KeyValue, or a process.AwaitedBucket for a state that may not be
// on the bus at the part's start (issue 1790), which keeps the latest until it appears and answers process.ErrNotYet.
type HearingKeeper interface {
Put(ctx context.Context, key string, value []byte) (uint64, error)
}
// SayHearing is a DeadLetterHearing's said that keeps each Hearing in kv under HearingKey; nil kv keeps nothing and
// says so once. A write kv keeps for later (process.ErrNotYet) is not an error: the awaited bucket said it waits.
func SayHearing(ctx context.Context, kv HearingKeeper, logf func(string, ...any)) func(Hearing) {
if kv == nil {
if logf != nil {
logf("whether %s is heard is not kept: the state %s is not there, so the health tool cannot say it", DeadLettersStream, HearingState)
}
return nil
}
return func(h Hearing) {
data, err := json.Marshal(h)
if err == nil {
_, err = kv.Put(ctx, HearingKey, data)
}
if err != nil && !errors.Is(err, process.ErrNotYet) && logf != nil {
logf("whether %s is heard could not be kept: %v", DeadLettersStream, err)
}
}
}
// HearingHealth is the health answer of an asker's kept Hearing (raw, empty when none is kept): healthy while it hears
// DEAD_LETTERS, unhealthy saying why while it does not or has not said.
func HearingHealth(raw []byte) health.Answer {
if len(raw) == 0 {
return health.Unhealthy("it has not said whether it hears " + DeadLettersStream + " (issue 734): it is starting, or its state " + HearingState + " is not there")
}
var h Hearing
if err := json.Unmarshal(raw, &h); err != nil {
return health.Unhealthy("what it keeps of hearing " + DeadLettersStream + " is not a hearing: " + err.Error())
}
if !h.Heard {
return health.Unhealthy(h.Why + " (since " + h.At.Format(time.RFC3339) + "; tried again at its next reconnect or ask)")
}
return health.Healthy("it hears " + DeadLettersStream + " for " + strings.Join(h.Streams, ", "))
}
+39
View File
@@ -115,6 +115,10 @@ type Composition struct {
Words string `json:"words,omitempty"` Words string `json:"words,omitempty"`
SDK string `json:"sdk,omitempty"` SDK string `json:"sdk,omitempty"`
Machines []string `json:"machines,omitempty"` Machines []string `json:"machines,omitempty"`
// FPSet is the candidate's fingerprint set (Candidate.FPSet) the composition judged: a carry keeps a composition that
// passed while the candidate's set is the same, and composes again when a fingerprint changed (novox/hq ADR 0548 §5).
// Absent in a value written before it, which a carry composes again.
FPSet string `json:"fpset,omitempty"`
} }
// Took is the time of each part of a check in integer milliseconds (ADR 0447 clause 3): waiting in the queue, cloning, // Took is the time of each part of a check in integer milliseconds (ADR 0447 clause 3): waiting in the queue, cloning,
@@ -166,6 +170,14 @@ type Verdict struct {
Composition *Composition `json:"composition,omitempty"` Composition *Composition `json:"composition,omitempty"`
// OverBound names the levels whose running went over their bound; it does not change the State (ADR 0453 clause 9). // OverBound names the levels whose running went over their bound; it does not change the State (ADR 0453 clause 9).
OverBound []string `json:"over-bound,omitempty"` OverBound []string `json:"over-bound,omitempty"`
// Beside is the commit of each sibling repository the check read beside the repositories it merged (ADR 0484 §4), by
// repository: a move of one puts the verdict out of date as a trunk move does, and it is carried (novox/hq ADR 0548 §6).
// A repository is in Trunk or here, never both. Absent in a value written before it.
Beside map[string]string `json:"beside,omitempty"`
// CarriedFrom names, of a carried verdict, the trunk and sibling commits it was judged on before the moves it was
// carried over, by repository (ADR 0548 §5): the verdict was rewritten for Trunk and Beside by running only the packages
// whose digest the moves changed. Absent from a verdict judged whole, and from one still running.
CarriedFrom map[string]string `json:"carried-from,omitempty"`
} }
func (c *Composition) check() error { func (c *Composition) check() error {
@@ -203,6 +215,33 @@ func (v Verdict) check() error {
return fmt.Errorf("delivery: the verdict's trunk head of %s is empty", repository) return fmt.Errorf("delivery: the verdict's trunk head of %s is empty", repository)
} }
} }
for repository, commit := range v.Beside {
if commit == "" {
return fmt.Errorf("delivery: the verdict's sibling commit of %s is empty", repository)
}
if _, ok := v.Trunk[repository]; ok {
return fmt.Errorf("delivery: the verdict names %s as a trunk it merged and as a sibling it read", repository)
}
}
if len(v.CarriedFrom) > 0 {
if v.State == VerdictRunning {
return fmt.Errorf("delivery: a running verdict is carried from nothing: carried-from is written with its end")
}
moved := false
for repository, commit := range v.CarriedFrom {
now, inTrunk := v.Trunk[repository]
if !inTrunk {
now = v.Beside[repository]
}
if commit == "" || now == "" {
return fmt.Errorf("delivery: the verdict's carried-from names %s, which it judged on no commit of", repository)
}
moved = moved || commit != now
}
if !moved {
return fmt.Errorf("delivery: the verdict's carried-from names the commits it is judged on: nothing moved, so nothing was carried")
}
}
for level, r := range v.Levels { for level, r := range v.Levels {
if err := checkLevel(level); err != nil { if err := checkLevel(level); err != nil {
return err return err
+25 -3
View File
@@ -29,7 +29,12 @@ func verdictSamples() map[string]Verdict {
failTest.Levels = map[string]LevelResult{LevelUnit: {Run: 1}} failTest.Levels = map[string]LevelResult{LevelUnit: {Run: 1}}
failMachine := passVerdict() failMachine := passVerdict()
failMachine.State, failMachine.Failed = VerdictFail, FailedFirst{Machine: "ace", Words: "path /etc/x is owned by two modules"} failMachine.State, failMachine.Failed = VerdictFail, FailedFirst{Machine: "ace", Words: "path /etc/x is owned by two modules"}
return map[string]Verdict{"running": running, "pass": passVerdict(), "fail by a test": failTest, "fail by a machine": failMachine} carried := passVerdict()
carried.Beside = map[string]string{"novox/mesh-catalog": "cccc"}
carried.CarriedFrom = map[string]string{"novox/mesh-sdk": "9999", "novox/mesh-control": "bbbb", "novox/mesh-catalog": "cccc"}
carried.Composition = &Composition{State: CompositionPass, SDK: carried.SDK, Machines: []string{"ace"}, FPSet: "fps1:00"}
return map[string]Verdict{"running": running, "pass": passVerdict(), "fail by a test": failTest, "fail by a machine": failMachine,
"carried": carried}
} }
func keptSample() Kept { func keptSample() Kept {
@@ -248,7 +253,14 @@ func TestVerdictCarriesSkippedCompositionAndOverBound(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
for _, field := range []string{`"skipped":4`, `"composition":{"state":"pass"`, `"over-bound":["integration"]`} { v.Beside, v.CarriedFrom = map[string]string{"novox/mesh-catalog": "cccc"}, map[string]string{"novox/mesh-catalog": "cbcb"}
v.Composition.FPSet = "fps1:ab"
data, err = EncodeVerdict(v)
if err != nil {
t.Fatal(err)
}
for _, field := range []string{`"skipped":4`, `"composition":{"state":"pass"`, `"over-bound":["integration"]`, `"beside":{"novox/mesh-catalog":"cccc"}`,
`"carried-from":{"novox/mesh-catalog":"cbcb"}`, `"fpset":"fps1:ab"`} {
if !strings.Contains(string(data), field) { if !strings.Contains(string(data), field) {
t.Errorf("the value lacks %s: %s", field, data) t.Errorf("the value lacks %s: %s", field, data)
} }
@@ -295,6 +307,13 @@ func TestVerdictNewFieldsChecked(t *testing.T) {
"a not-run composition naming machines": {func(v *Verdict) { v.Composition = &Composition{State: CompositionNotRun, Machines: []string{"ace"}} }, []string{"composition", "not run"}}, "a not-run composition naming machines": {func(v *Verdict) { v.Composition = &Composition{State: CompositionNotRun, Machines: []string{"ace"}} }, []string{"composition", "not run"}},
"an unknown over-bound level": {func(v *Verdict) { v.OverBound = []string{"e2e"} }, []string{"e2e"}}, "an unknown over-bound level": {func(v *Verdict) { v.OverBound = []string{"e2e"} }, []string{"e2e"}},
"an over-bound level named twice": {func(v *Verdict) { v.OverBound = []string{LevelUnit, LevelUnit} }, []string{"over-bound", "unit"}}, "an over-bound level named twice": {func(v *Verdict) { v.OverBound = []string{LevelUnit, LevelUnit} }, []string{"over-bound", "unit"}},
// ADR 0548 §5, §6: the siblings read and the commits a verdict was carried from.
"a sibling with no commit": {func(v *Verdict) { v.Beside = map[string]string{"novox/mesh-catalog": ""} }, []string{"sibling", "mesh-catalog"}},
"a trunk named as a sibling": {func(v *Verdict) { v.Beside = map[string]string{"novox/mesh-sdk": "aaaa"} }, []string{"trunk", "sibling", "mesh-sdk"}},
"carried from an unjudged repo": {func(v *Verdict) { v.CarriedFrom = map[string]string{"novox/other": "1111"} }, []string{"carried-from", "novox/other"}},
"carried from an empty commit": {func(v *Verdict) { v.CarriedFrom = map[string]string{"novox/mesh-sdk": ""} }, []string{"carried-from", "mesh-sdk"}},
"carried over no move": {func(v *Verdict) { v.CarriedFrom = map[string]string{"novox/mesh-sdk": "aaaa"} }, []string{"nothing moved"}},
"a running verdict carried": {func(v *Verdict) { v.State, v.CarriedFrom = VerdictRunning, map[string]string{"novox/mesh-sdk": "9999"} }, []string{"running", "carried"}},
} }
for name, c := range cases { for name, c := range cases {
v := passVerdict() v := passVerdict()
@@ -319,7 +338,7 @@ func TestVerdictOldAndNewReadersAgree(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("the new reader of an old value: %v", err) t.Fatalf("the new reader of an old value: %v", err)
} }
if v.Composition != nil || v.OverBound != nil || v.Levels[LevelUnit].Skipped != 0 || v.Levels[LevelUnit].Run != 12 { if v.Composition != nil || v.OverBound != nil || v.Beside != nil || v.CarriedFrom != nil || v.Levels[LevelUnit].Skipped != 0 || v.Levels[LevelUnit].Run != 12 {
t.Errorf("an old value grew fields: %+v", v) t.Errorf("an old value grew fields: %+v", v)
} }
again, err := EncodeVerdict(v) again, err := EncodeVerdict(v)
@@ -330,6 +349,9 @@ func TestVerdictOldAndNewReadersAgree(t *testing.T) {
n.Levels[LevelUnit] = LevelResult{Run: 12, Kept: 301, Skipped: 4} n.Levels[LevelUnit] = LevelResult{Run: 12, Kept: 301, Skipped: 4}
n.Composition = &Composition{State: CompositionNotRun} n.Composition = &Composition{State: CompositionNotRun}
n.OverBound = []string{LevelUnit} n.OverBound = []string{LevelUnit}
n.Beside = map[string]string{"novox/mesh-catalog": "cccc"}
n.CarriedFrom = map[string]string{"novox/mesh-sdk": "9999"}
n.Composition.FPSet = "fps1:00"
data, err := EncodeVerdict(n) data, err := EncodeVerdict(n)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
+49 -4
View File
@@ -48,7 +48,9 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"reflect"
"regexp" "regexp"
"slices"
"strings" "strings"
"time" "time"
) )
@@ -97,6 +99,10 @@ const (
// composition is applied, and the node waits for the answer in its want (novox/hq ADR 0539 clause 16). Detail // composition is applied, and the node waits for the answer in its want (novox/hq ADR 0539 clause 16). Detail
// carries the backup's own words. // carries the backup's own words.
PhaseHeld Phase = "held" PhaseHeld Phase = "held"
// PhaseWaitingYes is a module of policy record whose composed move the node agent judged to recreate a container or
// move its data (novox/hq ADR 0547 §3): nothing of it is applied, Judgement says the judgement and Waits the digest
// of the composition waiting, and the assigner asks a person; the node applies it once its want's Yes names Waits.
PhaseWaitingYes Phase = "waiting-yes"
) )
// Health is what the module's own check says. // Health is what the module's own check says.
@@ -213,12 +219,15 @@ func parseModuleKeyPart(part string) (string, error) {
return module, nil return module, nil
} }
// HaveNodeWord is the last token of a node's own facts' key in HaveBucket.
const HaveNodeWord = nodeWord
// HaveNodeKey is the key, in HaveBucket, of a node's own facts: `<node>.node`. // HaveNodeKey is the key, in HaveBucket, of a node's own facts: `<node>.node`.
func HaveNodeKey(node string) (string, error) { func HaveNodeKey(node string) (string, error) {
if err := checkName("node", node); err != nil { if err := checkName("node", node); err != nil {
return "", err return "", err
} }
return node + ".node", nil return node + "." + HaveNodeWord, nil
} }
// ParseHaveKey is the inverse of HaveKey and HaveNodeKey: isNode is true for `<node>.node`, with module empty. // ParseHaveKey is the inverse of HaveKey and HaveNodeKey: isNode is true for `<node>.node`, with module empty.
@@ -337,6 +346,15 @@ type Have struct {
// UserList is what the node-engine of the node holding mesh-broker wrote of the bus's user list, in the have of the // UserList is what the node-engine of the node holding mesh-broker wrote of the bus's user list, in the have of the
// bus module (novox/hq ADR 0547 §7, §8). Nil on every other have. // bus module (novox/hq ADR 0547 §7, §8). Nil on every other have.
UserList *UserListWritten `json:"user-list,omitempty"` UserList *UserListWritten `json:"user-list,omitempty"`
// Judgement and Waits are, in phase waiting-yes, the node agent's judgement of the move and the digest of the
// composition that waits for a person's yes (ADR 0547 §3); empty otherwise.
Judgement Judgement `json:"judgement,omitempty"`
Waits string `json:"waits,omitempty"`
// Sealed is, for a module the node composes itself, each value its running composition carries sealed to the node's
// key (a secret, a credential, a vault box), as applied (ADR 0547 §3, issue 1126): what the node's composer reuses
// after a restart while it opens to the vault's value, so a value sealed again moves no composition. Openable by the
// node alone, as the vault hands them over the bus. Empty on every other have.
Sealed []string `json:"sealed,omitempty"`
} }
// UserListWritten is the bus's user list as the node-engine of the bus node wrote it to the bus module's file, applied // UserListWritten is the bus's user list as the node-engine of the bus node wrote it to the bus module's file, applied
@@ -355,6 +373,15 @@ type UserListWritten struct {
Users int `json:"users"` Users int `json:"users"`
At time.Time `json:"at"` At time.Time `json:"at"`
Error string `json:"error,omitempty"` Error string `json:"error,omitempty"`
// Follows is which list the node follows now (FollowController or FollowGrants, novox/hq ADR 0547 clause 8), empty
// from a node-engine before the switch.
Follows string `json:"follows,omitempty"`
// Key is the bus node's sealing key, its public half as SPKI PEM: the key the grants part seals its list to
// (clause 7). Public, as the vault holds it.
Key string `json:"key,omitempty"`
// Unwatched says why the bus node does not watch the grants part's key now (its watch refused or ended, and tried
// again), empty while it does or where it need not (ADR 0547 clause 8): it then keeps the last list it followed.
Unwatched string `json:"unwatched,omitempty"`
// Handed is the hashes of the users of the list on the machine, sealed to the vault, for its adoption of the // Handed is the hashes of the users of the list on the machine, sealed to the vault, for its adoption of the
// accounts it holds no value for (ADR 0547 clause 9, handed.go); nil while the vault's key is not known. // accounts it holds no value for (ADR 0547 clause 9, handed.go); nil while the vault's key is not known.
Handed *AccountsHanded `json:"handed,omitempty"` Handed *AccountsHanded `json:"handed,omitempty"`
@@ -372,6 +399,9 @@ type HaveNode struct {
Strays json.RawMessage `json:"strays,omitempty"` Strays json.RawMessage `json:"strays,omitempty"`
Filters json.RawMessage `json:"filters,omitempty"` Filters json.RawMessage `json:"filters,omitempty"`
Profile json.RawMessage `json:"profile,omitempty"` Profile json.RawMessage `json:"profile,omitempty"`
// Record is what the machine observes of itself, typed, for every node and composing part to read (issue 1126,
// ADR 0555 §3): the node's record lives in its own have key, so it needs no bucket, writer or grant of its own.
Record *NodeRecord `json:"record,omitempty"`
} }
// Presence is whether a node or a part's instance is online; the entry's own time says since when. The watchman // Presence is whether a node or a part's instance is online; the entry's own time says since when. The watchman
@@ -420,8 +450,18 @@ func decode(data []byte, into any) error {
func (h Have) check() error { func (h Have) check() error {
switch h.Phase { switch h.Phase {
case PhaseBackingUp, PhaseApplying, PhaseChecking, PhaseRunning, PhaseFailed, PhaseRemoved, PhaseHeld: case PhaseBackingUp, PhaseApplying, PhaseChecking, PhaseRunning, PhaseFailed, PhaseRemoved, PhaseHeld:
if h.Judgement != "" || h.Waits != "" {
return fmt.Errorf("delivery: only a have waiting for a yes names a judgement and the composition waiting")
}
case PhaseWaitingYes:
if h.Judgement != JudgedRecreates && h.Judgement != JudgedMovesData && h.Judgement != JudgedCannot {
return fmt.Errorf("delivery: a have waiting for a yes names the judgement %q, not recreates, moves-data or cannot-judge", h.Judgement)
}
if err := checkDigest(h.Waits, "the composition a have waits a yes for"); err != nil {
return err
}
default: default:
return fmt.Errorf("delivery: phase %q is not backing-up, applying, checking, running, failed, removed or held", h.Phase) return fmt.Errorf("delivery: phase %q is not backing-up, applying, checking, running, failed, removed, held or waiting-yes", h.Phase)
} }
if h.Composed != "" { if h.Composed != "" {
if err := checkDigest(h.Composed, "the have's composition digest"); err != nil { if err := checkDigest(h.Composed, "the have's composition digest"); err != nil {
@@ -500,7 +540,11 @@ func DecodePulse(data []byte) (Pulse, error) {
func SameHave(a, b Have) bool { func SameHave(a, b Have) bool {
ua, ub := a.UserList, b.UserList ua, ub := a.UserList, b.UserList
a.UserList, b.UserList = nil, nil a.UserList, b.UserList = nil, nil
if a != b || (ua == nil) != (ub == nil) { if !slices.Equal(a.Sealed, b.Sealed) {
return false
}
a.Sealed, b.Sealed = nil, nil
if !reflect.DeepEqual(a, b) || (ua == nil) != (ub == nil) {
return false return false
} }
if ua == nil { if ua == nil {
@@ -515,7 +559,8 @@ func SameHave(a, b Have) bool {
// SameHaveNode says whether two node facts are the same value, comparing each carried JSON without its spacing. // SameHaveNode says whether two node facts are the same value, comparing each carried JSON without its spacing.
func SameHaveNode(a, b HaveNode) bool { func SameHaveNode(a, b HaveNode) bool {
return a.V == b.V && a.Composed == b.Composed && sameJSON(a.Windows, b.Windows) && sameJSON(a.Busy, b.Busy) && return a.V == b.V && a.Composed == b.Composed && sameJSON(a.Windows, b.Windows) && sameJSON(a.Busy, b.Busy) &&
sameJSON(a.Strays, b.Strays) && sameJSON(a.Filters, b.Filters) && sameJSON(a.Profile, b.Profile) sameJSON(a.Strays, b.Strays) && sameJSON(a.Filters, b.Filters) && sameJSON(a.Profile, b.Profile) &&
(a.Record == nil) == (b.Record == nil) && (a.Record == nil || SameNodeRecord(*a.Record, *b.Record))
} }
// SamePresence says whether two presences are the same value. // SamePresence says whether two presences are the same value.
+102 -7
View File
@@ -1,6 +1,7 @@
package delivery package delivery
import ( import (
"bytes"
"encoding/json" "encoding/json"
"errors" "errors"
"reflect" "reflect"
@@ -13,6 +14,9 @@ import (
// 1131/AC1: the have key forms and parses back; bad names are refused, naming them. // 1131/AC1: the have key forms and parses back; bad names are refused, naming them.
func TestHaveKey(t *testing.T) { func TestHaveKey(t *testing.T) {
t.Run("presence", testPresenceKey)
t.Run("pulse", testPulseKey)
t.Run("pulse ttl", testPulseTTL)
if HaveBucket != "node-engine_have" { if HaveBucket != "node-engine_have" {
t.Fatalf("HaveBucket = %q", HaveBucket) t.Fatalf("HaveBucket = %q", HaveBucket)
} }
@@ -57,7 +61,7 @@ func TestHaveKey(t *testing.T) {
} }
// 1131/AC2: the presence keys form and parse back; a node named part is refused. // 1131/AC2: the presence keys form and parse back; a node named part is refused.
func TestPresenceKey(t *testing.T) { func testPresenceKey(t *testing.T) {
if PresenceBucket != "mesh-delivery-watchman_presence" { if PresenceBucket != "mesh-delivery-watchman_presence" {
t.Fatalf("PresenceBucket = %q", PresenceBucket) t.Fatalf("PresenceBucket = %q", PresenceBucket)
} }
@@ -97,7 +101,7 @@ func TestPresenceKey(t *testing.T) {
} }
// 1131/AC3: the pulse key splits at the first dot, the key holding dots. // 1131/AC3: the pulse key splits at the first dot, the key holding dots.
func TestPulseKey(t *testing.T) { func testPulseKey(t *testing.T) {
if PulseBucket != "mesh-delivery-watchman_pulse" { if PulseBucket != "mesh-delivery-watchman_pulse" {
t.Fatalf("PulseBucket = %q", PulseBucket) t.Fatalf("PulseBucket = %q", PulseBucket)
} }
@@ -123,6 +127,7 @@ func TestPulseKey(t *testing.T) {
// 1131/AC4: each value encodes, decodes equal, and carries "v":1. // 1131/AC4: each value encodes, decodes equal, and carries "v":1.
func TestRoundTrip(t *testing.T) { func TestRoundTrip(t *testing.T) {
t.Run("phase and health refused", testPhaseHealthRefused)
hasV1 := func(t *testing.T, b []byte) { hasV1 := func(t *testing.T, b []byte) {
t.Helper() t.Helper()
var m map[string]any var m map[string]any
@@ -148,7 +153,7 @@ func TestRoundTrip(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
in.V = Version in.V = Version
if out != in { if !SameHave(out, in) {
t.Fatalf("got %+v want %+v", out, in) t.Fatalf("got %+v want %+v", out, in)
} }
}) })
@@ -211,6 +216,67 @@ func TestRoundTrip(t *testing.T) {
t.Fatal("handed with no vault key") t.Fatal("handed with no vault key")
} }
}) })
// The grants part's list sealed to the bus node (novox/hq ADR 0547 clause 7): the key's value round-trips and refuses
// what the bus node could not follow, and the object opens with the node's key alone, to the digest the key names.
t.Run("the grants part's sealed user list", func(t *testing.T) {
in := Have{FP: "f1", Phase: PhaseRunning, Health: HealthOK, UserList: &UserListWritten{Source: GrantsUserListSource,
Revision: 3, Wanted: "sha256:ab", Digest: "sha256:ab", Users: 2, Follows: FollowGrants, Key: "PEM"}}
b, err := EncodeHave(in)
if err != nil {
t.Fatal(err)
}
if out, err := DecodeHave(b); err != nil || out.UserList.Follows != FollowGrants || out.UserList.Key != "PEM" {
t.Fatalf("got %+v, %v", out.UserList, err)
}
list := []byte("accounts { MESH { users = [ { user: \"controller\" } ] } }\n")
digest := UserListDigest(list)
node, err := seal.GenerateKeyPair()
if err != nil {
t.Fatal(err)
}
other, _ := seal.GenerateKeyPair()
g := GrantsUserList{Follow: FollowGrants, Digest: digest, Object: UserListObjectName(digest),
SealedTo: SealingKeyID(node.PublicKey), Node: "novox", Revision: 1, Users: 1}
raw, err := EncodeGrantsUserList(g)
if err != nil {
t.Fatal(err)
}
back, err := DecodeGrantsUserList(raw)
if err != nil || back.Digest != digest || back.LeftOut == nil || back.Follow != FollowGrants {
t.Fatalf("decoded %+v, %v", back, err)
}
for _, bad := range []GrantsUserList{{Follow: "both", Digest: digest, Object: g.Object},
{Follow: FollowGrants, Digest: digest, Object: "other"}, {Follow: FollowGrants, Digest: "md5:x", Object: "md5:x"}} {
if _, err := EncodeGrantsUserList(bad); err == nil {
t.Errorf("%+v was written", bad)
}
}
if _, err := EncodeGrantsUserList(GrantsUserList{Follow: FollowController}); err != nil {
t.Errorf("a switch with no list yet refused: %v", err)
}
object, err := SealUserList(list, node.PublicKey)
if err != nil {
t.Fatal(err)
}
if bytes.Contains(object, []byte("controller")) {
t.Fatal("the object carries the list in the clear")
}
opener := func(priv string) func(seal.Box) ([]byte, error) {
return func(b seal.Box) ([]byte, error) { s, err := seal.Open(b, priv); return []byte(s), err }
}
if got, err := OpenUserList(object, digest, opener(node.PrivateKey)); err != nil || !bytes.Equal(got, list) {
t.Fatalf("opened %q, %v", got, err)
}
if _, err := OpenUserList(object, digest, opener(other.PrivateKey)); err == nil {
t.Error("another key opened the list")
}
if _, err := OpenUserList(object, "sha256:00", opener(node.PrivateKey)); err == nil {
t.Error("a list of another digest was taken")
}
if _, err := SealUserList(list, "not a key"); err == nil {
t.Error("a list was sealed to no key")
}
})
t.Run("have node", func(t *testing.T) { t.Run("have node", func(t *testing.T) {
in := HaveNode{Windows: json.RawMessage(`{"a":1}`), Busy: json.RawMessage(`true`)} in := HaveNode{Windows: json.RawMessage(`{"a":1}`), Busy: json.RawMessage(`true`)}
b, err := EncodeHaveNode(in) b, err := EncodeHaveNode(in)
@@ -278,7 +344,7 @@ func TestUnknownVersion(t *testing.T) {
} }
// 1131/AC6: a phase or health outside the lists is refused on encode and decode. // 1131/AC6: a phase or health outside the lists is refused on encode and decode.
func TestPhaseHealthRefused(t *testing.T) { func testPhaseHealthRefused(t *testing.T) {
for _, p := range []Phase{PhaseBackingUp, PhaseApplying, PhaseChecking, PhaseRunning, PhaseFailed} { for _, p := range []Phase{PhaseBackingUp, PhaseApplying, PhaseChecking, PhaseRunning, PhaseFailed} {
for _, h := range []Health{HealthOK, HealthFailed, HealthWaits, HealthUnknown} { for _, h := range []Health{HealthOK, HealthFailed, HealthWaits, HealthUnknown} {
if _, err := EncodeHave(Have{Phase: p, Health: h}); err != nil { if _, err := EncodeHave(Have{Phase: p, Health: h}); err != nil {
@@ -301,10 +367,27 @@ func TestPhaseHealthRefused(t *testing.T) {
if _, err := DecodeHave([]byte(`{"v":1,"phase":"running","health":"fine"}`)); err == nil { if _, err := DecodeHave([]byte(`{"v":1,"phase":"running","health":"fine"}`)); err == nil {
t.Error("bad health decoded") t.Error("bad health decoded")
} }
// ADR 0547 §3 (issue 1126): a have waits for a yes with its judgement and the composition waiting, and no other have
// names either.
waits := Have{Phase: PhaseWaitingYes, Health: HealthOK, Judgement: JudgedRecreates, Waits: "sha256:" + strings.Repeat("a", 64)}
if raw, err := EncodeHave(waits); err != nil {
t.Errorf("a have waiting for a yes: %v", err)
} else if back, err := DecodeHave(raw); err != nil || back.Judgement != JudgedRecreates || back.Waits != waits.Waits {
t.Errorf("a have waiting for a yes came back %+v, %v", back, err)
}
for name, h := range map[string]Have{
"neither judged": {Phase: PhaseWaitingYes, Health: HealthOK, Judgement: JudgedNeither, Waits: waits.Waits},
"no waiting digest": {Phase: PhaseWaitingYes, Health: HealthOK, Judgement: JudgedMovesData},
"running judged": {Phase: PhaseRunning, Health: HealthOK, Judgement: JudgedRecreates},
} {
if _, err := EncodeHave(h); err == nil {
t.Errorf("%s: encoded", name)
}
}
} }
// 1131/AC7: the pulse's time to live is 120 seconds minus the move's age; at the bound the move is stalled. // 1131/AC7: the pulse's time to live is 120 seconds minus the move's age; at the bound the move is stalled.
func TestPulseTTL(t *testing.T) { func testPulseTTL(t *testing.T) {
for _, c := range []struct { for _, c := range []struct {
age time.Duration age time.Duration
ttl time.Duration ttl time.Duration
@@ -328,6 +411,7 @@ func TestPulseTTL(t *testing.T) {
// 1131/AC8: equal values say same, different ones do not. // 1131/AC8: equal values say same, different ones do not.
func TestSame(t *testing.T) { func TestSame(t *testing.T) {
t.Run("have removed", testHaveRemoved)
a := Have{V: 1, Build: "b", Phase: PhaseRunning, Health: HealthOK} a := Have{V: 1, Build: "b", Phase: PhaseRunning, Health: HealthOK}
b := a b := a
if !SameHave(a, b) { if !SameHave(a, b) {
@@ -337,6 +421,17 @@ func TestSame(t *testing.T) {
if SameHave(a, b) { if SameHave(a, b) {
t.Error("different haves same") t.Error("different haves same")
} }
// The sealed values a self-composing node runs are part of its have (issue 1126).
sa, sb := a, a
sa.Sealed, sb.Sealed = []string{`{"box":1}`}, []string{`{"box":1}`}
if !SameHave(sa, sb) || SameHave(sa, a) {
t.Error("equal sealed values differ, or one said and one not are the same")
}
if raw, err := EncodeHave(sa); err != nil {
t.Error(err)
} else if back, err := DecodeHave(raw); err != nil || !SameHave(back, sa) {
t.Errorf("sealed values do not travel: %+v, %v", back, err)
}
// The user list is compared by what it says, never by where it is held (ADR 0547 §8). // The user list is compared by what it says, never by where it is held (ADR 0547 §8).
at := time.Date(2026, 10, 25, 21, 0, 0, 0, time.UTC) at := time.Date(2026, 10, 25, 21, 0, 0, 0, time.UTC)
ua, ub := a, a ua, ub := a, a
@@ -381,7 +476,7 @@ func TestSame(t *testing.T) {
} }
} }
func TestHaveRemoved(t *testing.T) { // 1370/AC3 func testHaveRemoved(t *testing.T) { // 1370/AC3
in := Have{Build: "b1", Phase: PhaseRemoved, Health: HealthUnknown, Detail: "the declaration no longer holds it"} in := Have{Build: "b1", Phase: PhaseRemoved, Health: HealthUnknown, Detail: "the declaration no longer holds it"}
b, err := EncodeHave(in) b, err := EncodeHave(in)
if err != nil { if err != nil {
@@ -392,7 +487,7 @@ func TestHaveRemoved(t *testing.T) { // 1370/AC3
t.Fatal(err) t.Fatal(err)
} }
in.V = Version in.V = Version
if out != in || out.Phase != "removed" { if !SameHave(out, in) || out.Phase != "removed" {
t.Errorf("round trip: %+v", out) t.Errorf("round trip: %+v", out)
} }
} }
+8 -4
View File
@@ -9,9 +9,10 @@ const oddManifest = "{\n\t\"module\": \"a\",\n \"z\": 1, \"a\": [ \"x\",\"y\"
func mergedWithInputs() Forge { func mergedWithInputs() Forge {
return Forge{Kind: ForgeMerged, Repository: "novox/mesh-sdk", Number: 36, Head: "aaaa", Merge: "cccc", Root: "t0", return Forge{Kind: ForgeMerged, Repository: "novox/mesh-sdk", Number: 36, Head: "aaaa", Merge: "cccc", Root: "t0",
Trees: map[string]string{"a": "t1", "b": "t2"}, Trees: map[string]string{"a": "t1", "b": "t2"},
Manifests: map[string]string{"a": oddManifest}, Manifests: map[string]string{"a": oddManifest},
Locks: map[string]map[string]string{"a": {"go.sum": strings.Repeat("ab", 32)}}} Locks: map[string]map[string]string{"a": {"go.sum": strings.Repeat("ab", 32)}},
Placements: map[string]string{"ace": `{"v":1,"node":"ace","modules":[]}`}, PlacementsRead: true}
} }
// 1442/AC1 // 1442/AC1
@@ -21,7 +22,7 @@ func TestForgeMergedCarriesManifestsAndLocks(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
for _, frag := range []string{`"manifests":{"a":`, `"locks":{"a":{"go.sum":`} { for _, frag := range []string{`"manifests":{"a":`, `"locks":{"a":{"go.sum":`, `"placements":{"ace":`, `"placements-read":true`} {
if !strings.Contains(string(data), frag) { if !strings.Contains(string(data), frag) {
t.Errorf("the encoding lacks %s: %s", frag, data) t.Errorf("the encoding lacks %s: %s", frag, data)
} }
@@ -47,6 +48,9 @@ func TestForgeMergedRefusesBadInputs(t *testing.T) {
"locks outside trees": {func(f *Forge) { f.Locks["yy"] = map[string]string{"go.sum": "h"} }, `"yy"`}, "locks outside trees": {func(f *Forge) { f.Locks["yy"] = map[string]string{"go.sum": "h"} }, `"yy"`},
"empty manifest": {func(f *Forge) { f.Manifests["b"] = "" }, `"b"`}, "empty manifest": {func(f *Forge) { f.Manifests["b"] = "" }, `"b"`},
"empty lock hash": {func(f *Forge) { f.Locks["a"]["go.sum"] = "" }, `"a"`}, "empty lock hash": {func(f *Forge) { f.Locks["a"]["go.sum"] = "" }, `"a"`},
"placements unsaid": {func(f *Forge) { f.PlacementsRead = false }, "placements-read"},
"placement of no node": {func(f *Forge) { f.Placements["a.b"] = "{}" }, "names no node"},
"placements on a pull": {func(f *Forge) { f.Kind, f.Action, f.Merge, f.Root, f.Trees = ForgePull, PullOpened, "", "", nil }, "only a merged message"},
} { } {
f := mergedWithInputs() f := mergedWithInputs()
tc.mut(&f) tc.mut(&f)
+64 -1
View File
@@ -123,6 +123,15 @@ func sampleMerge() Merge {
}} }}
} }
func sampleTurn() Turn {
at := time.Date(2026, 10, 26, 9, 0, 0, 0, time.UTC)
return Turn{Repository: "novox/mesh-sdk", Waiting: []TurnWait{
{Group: "feat/x", Heads: mustHash(gateHeads()), Ask: "merge.feat/x." + mustHash(gateHeads()), Approved: at, Since: at,
WaitsFor: "feat/y.0123456789ab", WaitsSince: at.Add(-time.Minute), Step: "the check ask of novox/mesh-sdk#40"},
{Group: "fix/1.2", Heads: "0123456789ac", Approved: at.Add(time.Minute), Since: at.Add(time.Minute)},
}}
}
func sampleAsk() Ask { func sampleAsk() Ask {
key, _ := MergeAskKey("feat/x", gateHeads()) key, _ := MergeAskKey("feat/x", gateHeads())
return Ask{Group: "feat/x", Heads: gateHeads(), Ask: asks.Ask{ return Ask{Group: "feat/x", Heads: gateHeads(), Ask: asks.Ask{
@@ -181,7 +190,24 @@ func TestGateValuesRoundTrip(t *testing.T) { // 1201/AC4
t.Errorf("owed %s round trip: %+v %v", o.Act, got, err) t.Errorf("owed %s round trip: %+v %v", o.Act, got, err)
} }
} }
tu := sampleTurn()
data, err = EncodeTurn(tu)
hasV("turn", data, err)
tu.V = Version
if got, err := DecodeTurn(data); err != nil || !SameTurn(got, tu) {
t.Errorf("turn round trip: %+v %v", got, err)
}
for _, field := range []string{`"waits-for":"feat/y.0123456789ab"`, `"waits-since":`, `"approved":`, `"step":`} {
if !strings.Contains(string(data), field) {
t.Errorf("the turn lacks %s: %s", field, data)
}
}
// ADR 0548 §3: a turn holds off every check but those of the heads that wait in it, whoever else waits there.
if !TurnHolds(tu, "fedcba987654") || TurnHolds(tu, tu.Waiting[0].Heads) || TurnHolds(tu, tu.Waiting[1].Heads) || TurnHolds(Turn{}, "fedcba987654") {
t.Error("TurnHolds: a turn holds other heads' checks and never the check of heads waiting in it")
}
for name, dec := range map[string]func([]byte) error{ for name, dec := range map[string]func([]byte) error{
"turn": func(b []byte) error { _, err := DecodeTurn(b); return err },
"merge": func(b []byte) error { _, err := DecodeMerge(b); return err }, "merge": func(b []byte) error { _, err := DecodeMerge(b); return err },
"ask": func(b []byte) error { _, err := DecodeAsk(b); return err }, "ask": func(b []byte) error { _, err := DecodeAsk(b); return err },
"answer": func(b []byte) error { _, err := DecodeAnswer(b); return err }, "answer": func(b []byte) error { _, err := DecodeAnswer(b); return err },
@@ -265,6 +291,27 @@ func TestGateValuesRefused(t *testing.T) { // 1201/AC5
} }
_, err = EncodeOwed(Owed{Act: OwedNote, Text: "line"}) _, err = EncodeOwed(Owed{Act: OwedNote, Text: "line"})
wantErr("note without ref", err, "ref") wantErr("note without ref", err, "ref")
for name, c := range map[string]struct {
edit func(*Turn)
words []string
}{
"no delivery": {func(t *Turn) { t.Waiting = nil }, []string{"no delivery", "deleted"}},
"a repository in capitals": {func(t *Turn) { t.Repository = "novox/Mesh-SDK" }, []string{"lowercase"}},
"no repository": {func(t *Turn) { t.Repository = "" }, []string{"repository"}},
"no group": {func(t *Turn) { t.Waiting[0].Group = "" }, []string{"group"}},
"a bad heads hash": {func(t *Turn) { t.Waiting[0].Heads = "xyz" }, []string{"xyz"}},
"the same heads twice": {func(t *Turn) { t.Waiting[1].Heads = t.Waiting[0].Heads }, []string{"twice"}},
"no Approve time": {func(t *Turn) { t.Waiting[1].Approved = time.Time{} }, []string{"Approve"}},
"out of the Approve's order": {func(t *Turn) { t.Waiting[1].Approved = t.Waiting[0].Approved.Add(-time.Hour) }, []string{"order"}},
"waits for no verdict key": {func(t *Turn) { t.Waiting[0].WaitsFor = "feat/y" }, []string{"verdict key"}},
"a step and no check": {func(t *Turn) { t.Waiting[1].Step = "x" }, []string{"names no check"}},
} {
tu := sampleTurn()
c.edit(&tu)
_, err = EncodeTurn(tu)
wantErr("turn with "+name, err, c.words...)
}
_, err = EncodeOwed(Owed{Act: OwedStatus, State: "success"}) _, err = EncodeOwed(Owed{Act: OwedStatus, State: "success"})
wantErr("status without context", err, "context") wantErr("status without context", err, "context")
} }
@@ -297,7 +344,8 @@ func TestGateSame(t *testing.T) { // 1201/AC6
func TestGateBuckets(t *testing.T) { func TestGateBuckets(t *testing.T) {
if MergesBucket != "mesh-delivery-gate_merges" || AsksBucket != "mesh-delivery-gate_asks" || if MergesBucket != "mesh-delivery-gate_merges" || AsksBucket != "mesh-delivery-gate_asks" ||
OwedBucket != "mesh-delivery-gate_owed" || AnswersBucket != "mesh-delivery-teller_answers" || OwedBucket != "mesh-delivery-gate_owed" || AnswersBucket != "mesh-delivery-teller_answers" ||
MergesHistory != 10 || AsksHistory != 5 || OwedHistory != 5 || AnswersHistory != 5 { MergesHistory != 10 || AsksHistory != 5 || OwedHistory != 5 || AnswersHistory != 5 ||
TurnsBucket != "mesh-delivery-gate_turns" || TurnsHistory != 5 {
t.Error("a bucket or history differs from the decided layout") t.Error("a bucket or history differs from the decided layout")
} }
} }
@@ -370,6 +418,21 @@ func TestKeysEscapeAndRoundTrip(t *testing.T) { // 1323/AC1
if r, n, h, a, err := ParseOwedKey(ok); err != nil || r != repo || n != 3 || h != "aaaa" || a != "view" { if r, n, h, a, err := ParseOwedKey(ok); err != nil || r != repo || n != 3 || h != "aaaa" || a != "view" {
t.Errorf("owed round trip %q = %q %d %q %q %v", repo, r, n, h, a, err) t.Errorf("owed round trip %q = %q %d %q %q %v", repo, r, n, h, a, err)
} }
// ADR 0548 §3: a repository's turn key is <owner>.<name>, lowercase, each part escaped.
tk, err := TurnsKey(repo)
if err != nil || strings.Count(tk, ".") != 1 {
t.Errorf("turns key for repository %q = %q, %v", repo, tk, err)
} else if r, err := ParseTurnsKey(tk); err != nil || r != strings.ToLower(repo) {
t.Errorf("turns round trip %q = %q, %v", repo, r, err)
}
}
if k, err := TurnsKey("novox/Mesh-SDK"); err != nil || k != "novox.mesh-sdk" {
t.Errorf("turns key = %q, %v; want novox.mesh-sdk", k, err)
}
for _, bad := range []string{"", "novox", "novox.", ".x", "a.b.c", "Novox.mesh-sdk", "novox.a=zz"} {
if _, err := ParseTurnsKey(bad); err == nil {
t.Errorf("parse turns key %q accepted", bad)
}
} }
if EscapeName("a.b") != "a=2eb" || EscapeName("a=b") != "a=3db" || EscapeName("a+b") != "a=2bb" { if EscapeName("a.b") != "a=2eb" || EscapeName("a=b") != "a=3db" || EscapeName("a+b") != "a=2bb" {
t.Errorf("escapes: %q %q %q", EscapeName("a.b"), EscapeName("a=b"), EscapeName("a+b")) t.Errorf("escapes: %q %q %q", EscapeName("a.b"), EscapeName("a=b"), EscapeName("a+b"))
+229
View File
@@ -0,0 +1,229 @@
package delivery
import (
"bytes"
"encoding/json"
"path"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// The judgement of a move (ADR 0377 §3), as a pure function over two compositions of one module, so the node agent that
// composes its own modules judges what it would apply (novox/hq ADR 0547 §3; issue 1126's S8). Moved from
// mesh-controller cmd/mesh-controller/composed_judge.go at origin/main (the controller keeps its copy, unchanged, until
// it loses the lease, ADR 0547 §1): judged, onlyTheUserList and sameResources, with their helpers, are its code.
//
// ADR 0377 §3 over two compositions of one module: **recreates** when a container resource is changed, added or
// removed; **moves data** when, with no container changed, a directory or file resource is changed, added or removed
// that a container of the module mounts or that holds or lies within one of its directories (where a module keeps its
// data), when its `data` declaration changes, or, for a module that declares data, when one of its long-running
// processes changes; **neither** otherwise, and for a module the node has never run.
// Judged is one move's judgement: what it does, its words, and whether only the bus's user list changes (ADR 0539 §7:
// a file change and a reload of the bus, no move of it, never asked).
type Judged struct {
Judgement Judgement
Words string
UserListOnly bool
}
// JudgeMove judges moving a node's module from the composition it runs to the one composed for it. Running is nil when the
// node runs no composition of the module. A composition equal to the one running (resource for resource, each
// canonical) is unchanged: no move, and no ask (ADR 0547 §3).
func JudgeMove(running *Composed, composed Composed) Judged {
if composed.Module == "" {
return Judged{Judgement: JudgedNeither, Words: "the node's own composition"}
}
if running == nil {
return Judged{Judgement: JudgedNeither, Words: "the node runs no composition of " + composed.Module}
}
if running.Data == composed.Data && SameComposedResources(*running, composed) {
return Judged{Judgement: JudgedUnchanged, Words: "equal to the composition the node runs"}
}
if onlyTheUserList(*running, composed) {
return Judged{Judgement: JudgedNeither, Words: "only the bus's user list changes", UserListOnly: true}
}
j, words := judged(*running, composed)
return Judged{Judgement: j, Words: words}
}
// busUsersSuffix ends the id of the bus module's user list as the composer writes it: `<module>.bus-users`.
const busUsersSuffix = ".bus-users"
// onlyTheUserList says two compositions of a module differ, and only in its file of the bus's user list.
func onlyTheUserList(from, to Composed) bool {
if from.Data != to.Data {
return false
}
before, after := judgedResourcesByID(from), judgedResourcesByID(to)
if len(before) != len(after) {
return false
}
changed := false
for id, now := range after {
was, ok := before[id]
if !ok {
return false
}
if bytes.Equal(was.raw, now.raw) {
continue
}
if !strings.HasSuffix(id, busUsersSuffix) || now.kind != string(declaration.TypeFile) {
return false
}
changed = true
}
return changed
}
// judged applies ADR 0377 §3 to two compositions of one module.
func judged(from, to Composed) (Judgement, string) {
before, after := judgedResourcesByID(from), judgedResourcesByID(to)
if ids := changedOf(before, after, func(kind string) bool { return kind == string(declaration.TypeContainer) }); len(ids) > 0 {
return JudgedRecreates, "it recreates " + strings.Join(ids, ", ")
}
if from.Data != to.Data {
return JudgedMovesData, "its data declaration changes"
}
places := dataPlaces(before, after)
held := func(kind string) bool {
return kind == string(declaration.TypeDirectory) || kind == string(declaration.TypeFile)
}
var moved []string
for _, id := range changedOf(before, after, held) {
r := before[id]
if r == nil {
r = after[id]
}
if withinPlaces(r.path, places) {
moved = append(moved, id)
}
}
if to.Data || from.Data {
moved = append(moved, changedOf(before, after, func(kind string) bool { return kind == string(declaration.TypeProcess) })...)
}
if len(moved) > 0 {
sort.Strings(moved)
return JudgedMovesData, "it changes where its data is kept: " + strings.Join(moved, ", ")
}
return JudgedNeither, "it recreates no container and moves no data"
}
// judgedResource is a resource as the judgement reads it: its kind, the path it acts on, what its containers mount,
// and its canonical bytes.
type judgedResource struct {
kind string
path string
mounts []string
raw []byte
}
func judgedResourcesByID(c Composed) map[string]*judgedResource {
out := map[string]*judgedResource{}
for _, raw := range c.Resources {
var head struct {
ID string `json:"id"`
Type string `json:"type"`
Path string `json:"path"`
// A container's volumes, `<path on the machine>:<path inside>[:<options>]`.
Volumes []string `json:"volumes"`
}
if err := json.Unmarshal(raw, &head); err != nil {
continue
}
r := &judgedResource{kind: head.Type, path: head.Path, raw: compactResource(raw)}
for _, v := range head.Volumes {
source, _, _ := strings.Cut(v, ":")
r.mounts = append(r.mounts, source)
}
out[head.ID] = r
}
return out
}
// compactResource is a resource's bytes canonical: its keys sorted, its spacing dropped, its numbers as written, so two
// compositions that say the same compare equal however each was written.
func compactResource(raw []byte) []byte {
var v any
d := json.NewDecoder(bytes.NewReader(raw))
d.UseNumber()
if err := d.Decode(&v); err != nil {
return raw
}
out, err := json.Marshal(v)
if err != nil {
return raw
}
return out
}
// changedOf are the ids of the resources of the kinds counted that are changed, added or removed, sorted.
func changedOf(before, after map[string]*judgedResource, counted func(kind string) bool) []string {
var out []string
for id, now := range after {
if !counted(now.kind) {
continue
}
if was, ok := before[id]; !ok || !bytes.Equal(was.raw, now.raw) {
out = append(out, id)
}
}
for id, was := range before {
if _, ok := after[id]; !ok && counted(was.kind) {
out = append(out, id)
}
}
sort.Strings(out)
return out
}
// dataPlaces are the paths where the module keeps what a move must not lose: what its containers mount, and its
// directories, before and after.
func dataPlaces(before, after map[string]*judgedResource) []string {
var out []string
for _, rs := range []map[string]*judgedResource{before, after} {
for _, r := range rs {
if r.kind == string(declaration.TypeContainer) {
out = append(out, r.mounts...)
}
if r.kind == string(declaration.TypeDirectory) && r.path != "" {
out = append(out, r.path)
}
}
}
return out
}
// withinPlaces says p is one of places, or lies within one, or holds one.
func withinPlaces(p string, places []string) bool {
if p == "" {
return false
}
p = path.Clean(p)
for _, place := range places {
if place == "" || !strings.HasPrefix(place, "/") {
continue
}
place = path.Clean(place)
if p == place || strings.HasPrefix(p, place+"/") || strings.HasPrefix(place, p+"/") {
return true
}
}
return false
}
// SameComposedResources says two compositions of a module hold the same resources in the same order, each compared
// canonical.
func SameComposedResources(a, b Composed) bool {
if len(a.Resources) != len(b.Resources) {
return false
}
for i := range a.Resources {
if !bytes.Equal(compactResource(a.Resources[i]), compactResource(b.Resources[i])) {
return false
}
}
return true
}
+68
View File
@@ -0,0 +1,68 @@
package delivery
import (
"encoding/json"
"strings"
"testing"
)
func composedOf(module string, c byte, data bool, resources ...string) Composed {
out := Composed{Node: "ace", Module: module, FP: SourcePrefix2 + strings.Repeat(string(c), 64), Data: data}
for _, r := range resources {
out.Resources = append(out.Resources, json.RawMessage(r))
}
return out
}
// The pairs mesh-controller's composed_judge_test.go captured (issue 1945 AC12), and the bus's user list, judged as the
// controller judges them (ADR 0377 §3, ADR 0539 §7), by the node agent's function (ADR 0547 §3, issue 1126 S8).
func TestJudgeMove(t *testing.T) {
const (
container = `{"id":"mesh-mail.server","type":"container","module":"mesh-mail","name":"mail","image":"mail@sha256:1111111111111111111111111111111111111111111111111111111111111111","volumes":["/var/lib/mesh-mail/spool:/spool"]}`
container2 = `{"id":"mesh-mail.server","type":"container","module":"mesh-mail","name":"mail","image":"mail@sha256:2222222222222222222222222222222222222222222222222222222222222222","volumes":["/var/lib/mesh-mail/spool:/spool"]}`
spool = `{"id":"mesh-mail.spool","type":"directory","module":"mesh-mail","path":"/var/lib/mesh-mail/spool"}`
spoolMoved = `{"id":"mesh-mail.spool","type":"directory","module":"mesh-mail","path":"/srv/mesh-mail/spool"}`
tools = `{"id":"mesh-mail.tools","type":"file","module":"mesh-mail","path":"/usr/lib/mesh-mail/tools.json","content":"a"}`
tools2 = `{"id":"mesh-mail.tools","type":"file","module":"mesh-mail","path":"/usr/lib/mesh-mail/tools.json","content":"b"}`
config = `{"id":"mesh-mail.conf","type":"file","module":"mesh-mail","path":"/var/lib/mesh-mail/spool/conf","content":"a"}`
config2 = `{"id":"mesh-mail.conf","type":"file","module":"mesh-mail","path":"/var/lib/mesh-mail/spool/conf","content":"b"}`
process = `{"id":"mesh-mail.daemon","type":"process","module":"mesh-mail","exec":"/usr/bin/a"}`
process2 = `{"id":"mesh-mail.daemon","type":"process","module":"mesh-mail","exec":"/usr/bin/b"}`
users = `{"id":"nats.bus-users","type":"file","module":"nats","path":"/etc/nats/users","content":"a"}`
users2 = `{"id":"nats.bus-users","type":"file","module":"nats","path":"/etc/nats/users","content":"b"}`
natsConf = `{"id":"nats.conf","type":"file","module":"nats","path":"/etc/nats/conf","content":"a"}`
natsConf2 = `{"id":"nats.conf","type":"file","module":"nats","path":"/etc/nats/conf","content":"b"}`
)
mail := func(c byte, data bool, rs ...string) *Composed {
m := composedOf("mesh-mail", c, data, rs...)
return &m
}
bus := func(rs ...string) *Composed { m := composedOf("nats", 'b', false, rs...); return &m }
for _, c := range []struct {
name string
from, to *Composed
want Judgement
userList bool
}{
{"a new image", mail('a', true, container, spool, tools), mail('b', true, container2, spool, tools), JudgedRecreates, false},
{"a container gone", mail('a', true, container, spool), mail('b', true, spool), JudgedRecreates, false},
{"its spool moved", mail('a', true, container, spool), mail('b', true, container, spoolMoved), JudgedMovesData, false},
{"a file in its spool", mail('a', true, container, spool, config), mail('b', true, container, spool, config2), JudgedMovesData, false},
{"its data declaration", mail('a', false, container, spool), mail('b', true, container, spool), JudgedMovesData, false},
{"a process of a module with data", mail('a', true, spool, process), mail('b', true, spool, process2), JudgedMovesData, false},
{"a process of a module without", mail('a', false, process), mail('b', false, process2), JudgedNeither, false},
{"its tools only", mail('a', true, container, spool, tools), mail('b', true, container, spool, tools2), JudgedNeither, false},
{"never ran", nil, mail('b', true, container), JudgedNeither, false},
{"equal, written otherwise", mail('a', true, `{"type":"directory","id":"mesh-mail.spool","module":"mesh-mail","path":"/var/lib/mesh-mail/spool"}`), mail('b', true, spool), JudgedUnchanged, false},
{"only the user list", bus(natsConf, users), bus(natsConf, users2), JudgedNeither, true},
{"the user list and more", bus(natsConf, users), bus(natsConf2, users2), JudgedNeither, false},
} {
got := JudgeMove(c.from, *c.to)
if got.Judgement != c.want || got.UserListOnly != c.userList || got.Words == "" {
t.Errorf("%s: %+v, want %s (user list only %v)", c.name, got, c.want, c.userList)
}
}
if got := JudgeMove(nil, Composed{Node: "ace"}); got.Judgement != JudgedNeither {
t.Errorf("the node's own composition: %+v", got)
}
}
+98
View File
@@ -0,0 +1,98 @@
package delivery
import (
"encoding/json"
"fmt"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// A node's record is what the machine observes of itself (issue 1126, ADR 0555 §3), written by its own agent in its
// own have key, `<node>.node` of the node-engine's have bucket (HaveNode.Record), whenever an observed fact changes and
// only when the value differs. The key's change is the event its readers hear: every node and the parts that compose
// watch it. Where a module runs, and every fact a person decides of the node (its endpoint, site, hub, address on the
// private network, public domain, accounts, the tunnel found at adoption), are the catalogue's placement
// (placement.go). Nothing in it is secret: its keys are public halves.
// RecordTunnel is a tunnel found on a node when it was adopted, as compose.Tunnel holds it: public keys and addresses
// only. After the node took it over the tunnel is no longer there to observe, so it is kept as what was decided at
// adoption, in the node's placement (NodeDecided.Tunnel).
type RecordTunnel struct {
Interface string `json:"interface"`
Unit string `json:"unit,omitempty"`
Config string `json:"config,omitempty"`
Port int `json:"port,omitempty"`
MTU int `json:"mtu,omitempty"`
Address string `json:"address,omitempty"`
Range string `json:"range,omitempty"`
PublicKey string `json:"public-key,omitempty"`
Peers []RecordTunnelPeer `json:"peers,omitempty"`
}
// RecordTunnelPeer is one peer of a found tunnel.
type RecordTunnelPeer struct {
PublicKey string `json:"public-key"`
Address string `json:"address,omitempty"`
}
// NodeRecord is one node's record: what the machine observes of itself (issue 1126, ADR 0555: what a person decides
// lives in the catalogue's placement file, Placement.Decided; what the machine observes lives here). TunnelKey is the
// public key of its private-network tunnel; Certificate the one the mesh issued for its internal name, public, as the
// node holds it. The tunnel found at adoption and the names given to its peers are decisions kept in the placement;
// whether the hub took its found tunnel over, and which node enrolled under a carried peer's key, are read from both.
type NodeRecord struct {
V int `json:"v"`
Node string `json:"node"`
TunnelKey string `json:"tunnel-key,omitempty"`
Capabilities []string `json:"capabilities,omitempty"`
SealingKey string `json:"sealing-key,omitempty"`
ServingKey string `json:"serving-key,omitempty"`
Certificate string `json:"certificate,omitempty"`
OutwardLinks []string `json:"outward-links,omitempty"`
Networks []catalogue.AttachedNetwork `json:"networks,omitempty"`
NetworksSaid bool `json:"networks-said,omitempty"`
Contract int `json:"contract,omitempty"`
// Authority is the mesh's own certificate as this machine holds it (public): the one its declaration last wrote where
// a module asked for it. A node composing its own composes a certificate with it (compose.Input.Authority).
Authority string `json:"authority,omitempty"`
}
func (r NodeRecord) check() error {
if err := checkName("node", r.Node); err != nil {
return err
}
for _, c := range r.Capabilities {
if c == "" {
return fmt.Errorf("delivery: the record of %s names an empty capability", r.Node)
}
}
return nil
}
// EncodeNodeRecord is r as JSON at Version; it refuses what DecodeNodeRecord refuses.
func EncodeNodeRecord(r NodeRecord) ([]byte, error) {
if err := r.check(); err != nil {
return nil, err
}
r.V = Version
return json.Marshal(r)
}
// DecodeNodeRecord reads a node record; it refuses an unknown version (ErrUnknownVersion), a node that is not a name, and
// an empty capability.
func DecodeNodeRecord(data []byte) (NodeRecord, error) {
var r NodeRecord
if err := decode(data, &r); err != nil {
return NodeRecord{}, err
}
if err := r.check(); err != nil {
return NodeRecord{}, err
}
return r, nil
}
// SameNodeRecord says whether two records are the same value, so the agent writes only when they differ.
func SameNodeRecord(a, b NodeRecord) bool {
a.V, b.V = 0, 0
return jsonEqual(a, b)
}
+73
View File
@@ -0,0 +1,73 @@
package delivery
import (
"reflect"
"regexp"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// secretNamed is facts.secretKey's pattern: a field so named would read as a secret, and the runtime's credential-field
// check (ADR 0201 §13) refuses such a field in a state every node reads.
var secretNamed = regexp.MustCompile(`(?i)(secret|passw|token|api[-_]?key|private[-_]?key|credential|bearer|cookie|salt|signing)`)
// Issue 1126 (ADR 0555): a node's record reads back, rides in its have key, refuses a node that is not a name and an empty capability, and
// neither it nor the placement file names a field like a secret.
func TestNodeRecord(t *testing.T) {
hub := NodeRecord{Node: "novox", TunnelKey: "k", Capabilities: []string{"docker"}, SealingKey: "age1x", ServingKey: "pub",
Certificate: "cert", OutwardLinks: []string{"eth0"}, Networks: []catalogue.AttachedNetwork{{Link: "eth0", Network: "192.0.2.0/24"}},
NetworksSaid: true, Contract: 3}
raw, err := EncodeNodeRecord(hub)
if err != nil {
t.Fatal(err)
}
back, err := DecodeNodeRecord(raw)
if err != nil || !SameNodeRecord(back, hub) || back.V != Version {
t.Fatalf("round trip: %+v, %v", back, err)
}
// It rides in the node's own have key: a have that differs only in its record is another value.
with, without := HaveNode{V: Version, Record: &hub}, HaveNode{V: Version}
if raw, err := EncodeHaveNode(with); err != nil {
t.Fatal(err)
} else if got, err := DecodeHaveNode(raw); err != nil || got.Record == nil || !SameHaveNode(got, with) || SameHaveNode(got, without) {
t.Fatalf("the have's record: %+v, %v", got, err)
}
changed := hub
changed.Contract = 4
if SameHaveNode(with, HaveNode{V: Version, Record: &changed}) {
t.Error("a changed record reads as the same have")
}
for name, r := range map[string]NodeRecord{
"empty capability": {Node: "ace", Capabilities: []string{""}},
"node that is not a name": {Node: "a.b"},
} {
if _, err := EncodeNodeRecord(r); err == nil {
t.Errorf("%s: taken", name)
}
}
// Every JSON field name of the record and the types it carries, walked: none reads as a secret.
var walk func(reflect.Type, string)
walk = func(ty reflect.Type, at string) {
for ty.Kind() == reflect.Pointer || ty.Kind() == reflect.Slice {
ty = ty.Elem()
}
if ty.Kind() != reflect.Struct {
return
}
for i := 0; i < ty.NumField(); i++ {
f := ty.Field(i)
name, _, _ := strings.Cut(f.Tag.Get("json"), ",")
if secretNamed.MatchString(name) || secretNamed.MatchString(f.Name) {
t.Errorf("%s.%s (%q) is named like a secret", at, f.Name, name)
}
walk(f.Type, at+"."+f.Name)
}
}
walk(reflect.TypeOf(NodeRecord{}), "NodeRecord")
// And the placement file, which holds what a person decides of a node (K2): accounts by name, never a credential.
walk(reflect.TypeOf(Placement{}), "Placement")
}
+464
View File
@@ -0,0 +1,464 @@
package delivery
import (
"encoding/json"
"fmt"
"path"
"sort"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Placement of modules on nodes lives in the catalogue (novox/hq ADR 0549): one file per node, `placement/<node>.json`
// in the catalogue repository, changed only by a pull request. At each trunk merge of the catalogue the indexer reads
// the files and writes each node's key `placement.<node>` in its state `placement`, only when it differs, and says
// `placement-changed` naming the node, the revision and the modules added, removed or changed (ADR 0549 §2). The node
// agents, the assigner, the grants part and mesh-config read it (§3); nothing in it is secret.
const (
// PlacementState is the indexer's state of placements, and PlacementBucket its bucket.
PlacementState = "placement"
PlacementBucket = IndexerModule + "_" + PlacementState
// PlacementHistory is the revisions the placement bucket keeps per key.
PlacementHistory = 10
// PlacementDir is the catalogue's directory of placement files, one `<node>.json` per node.
PlacementDir = "placement"
// PlacementChangedEvent is what the indexer says when a node's placement key changed.
PlacementChangedEvent = "placement-changed"
placementWord = "placement"
// CatalogueRepository is the repository whose placement files are the mesh's placement (ADR 0549 §1): the forge bridge
// reads them at each of its merges, and no other repository's.
CatalogueRepository = "novox/mesh-catalog"
)
// PlacementKey is the key, in PlacementBucket, of one node's placement: `placement.<node>`.
func PlacementKey(node string) (string, error) {
if err := checkName("node", node); err != nil {
return "", err
}
return placementWord + "." + node, nil
}
// ParsePlacementKey is the inverse of PlacementKey; it refuses any other key, naming it.
func ParsePlacementKey(key string) (node string, err error) {
node, ok := strings.CutPrefix(key, placementWord+".")
if !ok || checkName("node", node) != nil {
return "", fmt.Errorf("delivery: %q is not a placement key, placement.<node>", key)
}
return node, nil
}
// PlacementFile is the path, in the catalogue repository, of one node's placement file.
func PlacementFile(node string) string { return path.Join(PlacementDir, node+".json") }
// PlacementFileNode is the node a catalogue path is the placement file of, and whether it is one: `placement/<node>.json`
// directly under PlacementDir.
func PlacementFileNode(p string) (string, bool) {
dir, file := path.Split(p)
node, ok := strings.CutSuffix(file, ".json")
if path.Clean(dir) != PlacementDir || !ok || checkName("node", node) != nil {
return "", false
}
return node, true
}
// Provider is the module on a node that answers a requirement or holds a seat: a pin's or a binding's provider.
type Provider struct {
Node string `json:"node"`
Module string `json:"module"`
}
// Chosen is p as the composition names a chosen provider.
func (p Provider) Chosen() catalogue.Chosen { return catalogue.Chosen{Node: p.Node, Module: p.Module} }
// PlacedModule is one module placed on the node: the host port each port it declares is published on (by the port the
// software listens on), and, by provision, the provider each of its requirements that keeps data is bound to.
//
// MeshWide and Layer are where the module's things are, as a settings layer's placement holds them (catalogue.Layer's
// Placement: `places`, `accesses`, `networks`, `endpoints`, `expose`, `reach`, `ports`; ADR 0348 §11): the whole mesh's
// layer and this node's, which the controller keeps in its table `placement` (ADR 0549 §6 names it) and composes as two
// layers, the mesh's first. Each node's file repeats the mesh-wide layer of the modules placed on it, so a node reads
// its own key alone. HasLayer says the node has a layer of its own even when it holds no placement, since a layer
// that is there and empty still carries the module's config values (Layers).
//
// MeshValues and Values are the config values of a module in the settings form (its manifest declares no config, so
// mesh-config holds none for it): what a person set in the whole mesh's layer and this node's, beside the placement
// (issue 1126; on 2026-10-26 the media modules on one node, plex's puid and pgid among them). A module in the config
// form takes its values from mesh-config and has none here.
type PlacedModule struct {
Module string `json:"module"`
Ports map[string]int `json:"ports,omitempty"`
Bindings map[string]Provider `json:"bindings,omitempty"`
MeshWide map[string]any `json:"mesh-wide,omitempty"`
HasMesh bool `json:"has-mesh-wide,omitempty"`
Layer map[string]any `json:"layer,omitempty"`
HasLayer bool `json:"has-layer,omitempty"`
MeshValues map[string]any `json:"mesh-wide-values,omitempty"`
Values map[string]any `json:"values,omitempty"`
}
// Holding is one seat a module placed on the node holds (catalogue.Held without its node, which is the file's).
type Holding struct {
Claim string `json:"claim"`
Scope string `json:"scope,omitempty"`
Module string `json:"module"`
Site string `json:"site,omitempty"`
Kind string `json:"kind,omitempty"`
}
// Placement is one node's placement: the file the catalogue keeps and, with Revision, the value of its key. Modules are
// in the order the file names them, which is the order the composition reads the node's modules in. Pins is, by
// provision, the provider chosen for the node where more than one could answer.
type Placement struct {
V int `json:"v"`
Node string `json:"node"`
Revision string `json:"revision,omitempty"`
Decided NodeDecided `json:"decided"`
Modules []PlacedModule `json:"modules"`
Pins map[string]Provider `json:"pins,omitempty"`
Holds []Holding `json:"holds,omitempty"`
}
// NodeDecided is what a person decides of the node, kept in its placement file (issue 1126's K2: what a person decides
// lives in the catalogue, what the machine observes lives in its record, NodeRecord): where it is dialled (endpoint) and
// its site, whether it is the private network's hub, its address on the private network, the domain its routed names
// are composed under, its operator and agent logins by name and their homes, whether it is adopted rather than
// converged, the modules taken on an adopted node, the tunnel found on it at adoption and, on the hub, the names given
// to that tunnel's peers. The address is a decision written once, at enrolment, by a
// catalogue pull request: nothing allocates it at run time, and CheckPlacements refuses two nodes with one address.
// No secret and no key material: accounts are names, never credentials.
type NodeDecided struct {
Endpoint string `json:"endpoint,omitempty"`
Site string `json:"site,omitempty"`
Hub bool `json:"hub,omitempty"`
Address string `json:"address,omitempty"`
PublicDomain string `json:"public-domain,omitempty"`
Account string `json:"account,omitempty"`
AccountHome string `json:"account-home,omitempty"`
AgentAccount string `json:"agent-account,omitempty"`
AgentHome string `json:"agent-account-home,omitempty"`
Adopted bool `json:"adopted,omitempty"`
Taken []string `json:"taken,omitempty"`
// Tunnel is the tunnel found on the node when it was adopted, kept as decided then: once the node takes it over it is
// no longer there to observe. PeerNames is, on the hub, the name a person gave a peer of that tunnel, by its public key.
Tunnel *RecordTunnel `json:"tunnel,omitempty"`
PeerNames map[string]string `json:"peer-names,omitempty"`
}
func (p Placement) check() error {
if err := checkName("node", p.Node); err != nil {
return err
}
seen := map[string]bool{}
for _, m := range p.Modules {
if err := checkName("module", m.Module); err != nil {
return err
}
if seen[m.Module] {
return fmt.Errorf("delivery: the placement of %s names %s twice", p.Node, m.Module)
}
seen[m.Module] = true
for port, at := range m.Ports {
if n, err := strconv.Atoi(port); err != nil || n <= 0 || n > 65535 {
return fmt.Errorf("delivery: %s's port %q on %s is not a port number", m.Module, port, p.Node)
}
if at <= 0 || at > 65535 {
return fmt.Errorf("delivery: %s's port %s on %s is published on %d, which is not a port number", m.Module, port, p.Node, at)
}
}
for provision, to := range m.Bindings {
if provision == "" || to.Node == "" || to.Module == "" {
return fmt.Errorf("delivery: %s on %s has a binding with no provision, node or module", m.Module, p.Node)
}
}
}
for provision, to := range p.Pins {
if provision == "" || to.Node == "" || to.Module == "" {
return fmt.Errorf("delivery: the placement of %s has a pin with no provision, node or module", p.Node)
}
}
for _, h := range p.Holds {
if h.Claim == "" || !seen[h.Module] {
return fmt.Errorf("delivery: the placement of %s has %s hold %q, which is not a module placed there or names no seat", p.Node, h.Module, h.Claim)
}
}
return nil
}
// EncodePlacement is p as JSON at Version; it refuses what DecodePlacement refuses.
func EncodePlacement(p Placement) ([]byte, error) {
if err := p.check(); err != nil {
return nil, err
}
p.V = Version
return json.Marshal(p)
}
// DecodePlacement reads a placement file or key value; it refuses an unknown version (ErrUnknownVersion), a node or
// module that is not a name, a module named twice, a port that is not a number, a binding or pin naming no provider,
// and a holding by a module not placed on the node.
func DecodePlacement(data []byte) (Placement, error) {
var p Placement
if err := decode(data, &p); err != nil {
return Placement{}, err
}
if err := p.check(); err != nil {
return Placement{}, err
}
return p, nil
}
// SamePlacement says whether two placements are the same value, so the indexer writes only when they differ. The
// revision is not compared: a merge that changes no placement writes nothing (ADR 0549 §2).
func SamePlacement(a, b Placement) bool {
a.Revision, b.Revision = "", ""
ja, errA := EncodePlacement(a)
jb, errB := EncodePlacement(b)
return errA == nil && errB == nil && string(ja) == string(jb)
}
// PlacementChange is what `placement-changed` says: the node, the catalogue revision, and its modules added, removed
// or changed against the placement before.
type PlacementChange struct {
Node string `json:"node"`
Revision string `json:"revision"`
Added []string `json:"added,omitempty"`
Removed []string `json:"removed,omitempty"`
Changed []string `json:"changed,omitempty"`
}
// ChangeOf is what changed from before to after on one node; before is the zero Placement for a node new to placement.
// A change of the node's pins or holdings counts as a change of every module placed on it both times, since the
// composition of each may read them.
func ChangeOf(before, after Placement) PlacementChange {
c := PlacementChange{Node: after.Node, Revision: after.Revision}
if c.Node == "" {
c.Node = before.Node
}
was := map[string]PlacedModule{}
for _, m := range before.Modules {
was[m.Module] = m
}
nodeWide := !jsonEqual(before.Pins, after.Pins) || !jsonEqual(before.Holds, after.Holds)
now := map[string]bool{}
for _, m := range after.Modules {
now[m.Module] = true
old, ok := was[m.Module]
switch {
case !ok:
c.Added = append(c.Added, m.Module)
case nodeWide || !jsonEqual(old, m):
c.Changed = append(c.Changed, m.Module)
}
}
for _, m := range before.Modules {
if !now[m.Module] {
c.Removed = append(c.Removed, m.Module)
}
}
sort.Strings(c.Added)
sort.Strings(c.Removed)
sort.Strings(c.Changed)
return c
}
func jsonEqual(a, b any) bool {
ja, _ := json.Marshal(a)
jb, _ := json.Marshal(b)
return string(ja) == string(jb)
}
// Assigned is the modules placed on the node, in the file's order (compose.Node's Assigned).
func (p Placement) Assigned() []string {
out := make([]string, 0, len(p.Modules))
for _, m := range p.Modules {
out = append(out, m.Module)
}
return out
}
// ChosenPins is the node's pins as the composition reads them (compose.Node's Pins); nil when it has none.
func (p Placement) ChosenPins() map[string]catalogue.Chosen {
if len(p.Pins) == 0 {
return nil
}
out := make(map[string]catalogue.Chosen, len(p.Pins))
for provision, to := range p.Pins {
out[provision] = to.Chosen()
}
return out
}
// Bound is, by consumer module and provision, where each binding sends it (compose.Node's Bound); nil when none.
func (p Placement) Bound() map[string]map[string]catalogue.Chosen {
var out map[string]map[string]catalogue.Chosen
for _, m := range p.Modules {
if len(m.Bindings) == 0 {
continue
}
if out == nil {
out = map[string]map[string]catalogue.Chosen{}
}
out[m.Module] = make(map[string]catalogue.Chosen, len(m.Bindings))
for provision, to := range m.Bindings {
out[m.Module][provision] = to.Chosen()
}
}
return out
}
// HostPorts is, by module and the port it declares, the host port it is published on (compose.Node's Ports); nil when
// none. The check has refused a port that is not a number, so none is dropped here.
func (p Placement) HostPorts() map[string]map[int]int {
var out map[string]map[int]int
for _, m := range p.Modules {
if len(m.Ports) == 0 {
continue
}
if out == nil {
out = map[string]map[int]int{}
}
out[m.Module] = make(map[int]int, len(m.Ports))
for port, at := range m.Ports {
n, _ := strconv.Atoi(port)
out[m.Module][n] = at
}
}
return out
}
// Layers is one module's settings layers on the node, as the composition reads them (compose.Node's Settings): the
// whole mesh's layer when it has one, then the node's, which carries the module's config values as mesh-config resolved
// them for this node (values, nil when none) beside its placement; a module with no layer and no value has none. This is
// the controller's SettingsFor for a module that takes its config values from mesh-config (issue 664). A module in the
// settings form carries its values in the file (MeshValues, Values), which then stand for mesh-config's.
func (m PlacedModule) Layers(node string, values map[string]any) []catalogue.Layer {
var out []catalogue.Layer
if m.HasMesh || len(m.MeshWide) > 0 || len(m.MeshValues) > 0 {
mesh := map[string]any{}
for k, v := range m.MeshValues {
mesh[k] = v
}
out = append(out, catalogue.Layer{From: catalogue.MeshWideLayer, Values: mesh, Placement: m.MeshWide})
}
if len(m.Values) > 0 {
values = make(map[string]any, len(m.Values))
for k, v := range m.Values {
values[k] = v
}
}
if m.HasLayer || len(m.Layer) > 0 || len(values) > 0 {
if values == nil {
values = map[string]any{}
}
out = append(out, catalogue.Layer{From: node, Values: values, Placement: m.Layer})
}
return out
}
// Held is the node's holdings as the composition reads them (compose.Input's Holdings rows for this node).
func (p Placement) Held() []catalogue.Held {
out := make([]catalogue.Held, 0, len(p.Holds))
for _, h := range p.Holds {
out = append(out, catalogue.Held{Claim: h.Claim, Scope: h.Scope, Node: p.Node, Module: h.Module, Site: h.Site, Kind: h.Kind})
}
return out
}
// CheckPlacements is the catalogue's own check of every placement file (ADR 0549 §1), over the placements by node, the
// modules the catalogue holds and the nodes enrolled. It returns every refusal, in node order, never only the first: a
// placement naming a node not enrolled or a module the catalogue does not hold; two modules of one node on the same host
// port; a pin, binding or holding naming a provider that is not placed where it says; a seat of scope mesh held twice;
// two nodes with one address on the private network; more than one hub.
// A host port a manifest declares without a number and a requirement left unbound are the composition's to find (its
// ForeseenPorts and Unbound), which the catalogue's composition check reads.
func CheckPlacements(placements map[string]Placement, held func(module string) bool, enrolled func(node string) bool) []string {
var refusals []string
nodes := make([]string, 0, len(placements))
for n := range placements {
nodes = append(nodes, n)
}
sort.Strings(nodes)
placed := func(p Provider) bool {
other, ok := placements[p.Node]
if !ok {
return false
}
for _, m := range other.Modules {
if m.Module == p.Module {
return true
}
}
return false
}
meshSeats := map[string]string{}
addresses := map[string]string{}
var hubs []string
for _, node := range nodes {
p := placements[node]
if p.Node != node {
refusals = append(refusals, fmt.Sprintf("the placement file of %s names the node %q", node, p.Node))
}
if a := strings.TrimSpace(p.Decided.Address); a != "" {
if other, twice := addresses[a]; twice {
refusals = append(refusals, fmt.Sprintf("%s and %s both have the address %s on the private network", other, node, a))
}
addresses[a] = node
}
if p.Decided.Hub {
hubs = append(hubs, node)
}
if !enrolled(node) {
refusals = append(refusals, fmt.Sprintf("%s is not an enrolled node", node))
}
hostPorts := map[int]string{}
for _, m := range p.Modules {
if !held(m.Module) {
refusals = append(refusals, fmt.Sprintf("%s: the catalogue holds no module %s", node, m.Module))
}
for _, port := range sortedKeys(m.Ports) {
at := m.Ports[port]
if other, clash := hostPorts[at]; clash && other != m.Module {
refusals = append(refusals, fmt.Sprintf("%s: %s and %s are both published on host port %d", node, other, m.Module, at))
}
hostPorts[at] = m.Module
}
for _, provision := range sortedKeys(m.Bindings) {
if to := m.Bindings[provision]; !placed(to) {
refusals = append(refusals, fmt.Sprintf("%s: %s's %s is bound to %s/%s, which is not placed there", node, m.Module, provision, to.Node, to.Module))
}
}
}
for _, provision := range sortedKeys(p.Pins) {
if to := p.Pins[provision]; !placed(to) {
refusals = append(refusals, fmt.Sprintf("%s: the pin of %s names %s/%s, which is not placed there", node, provision, to.Node, to.Module))
}
}
for _, h := range p.Holds {
if h.Scope != "mesh" {
continue
}
seat := h.Claim + "/" + h.Kind
if other, twice := meshSeats[seat]; twice {
refusals = append(refusals, fmt.Sprintf("the seat %s, held once for the mesh, is held on %s and %s", h.Claim, other, node))
}
meshSeats[seat] = node
}
}
if len(hubs) > 1 {
refusals = append(refusals, fmt.Sprintf("the private network has one hub, and %s are each placed as it", strings.Join(hubs, ", ")))
}
return refusals
}
func sortedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
+172
View File
@@ -0,0 +1,172 @@
package delivery
import (
"errors"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// ADR 0549 §1, §2: the placement key and file, the value's refusals, its reading as the composition reads a node, what
// a change says, and the catalogue's check of every file.
func TestPlacement(t *testing.T) {
t.Run("key and file", func(t *testing.T) {
key, err := PlacementKey("g14")
if err != nil || key != "placement.g14" {
t.Fatalf("PlacementKey = %q, %v", key, err)
}
if n, err := ParsePlacementKey(key); err != nil || n != "g14" {
t.Fatalf("ParsePlacementKey = %q, %v", n, err)
}
for _, bad := range []string{"want.g14.x", "placement.", "placement.a.b"} {
if _, err := ParsePlacementKey(bad); err == nil {
t.Errorf("ParsePlacementKey(%q) took it", bad)
}
}
if _, err := PlacementKey("a.b"); err == nil {
t.Error("PlacementKey took a dotted node")
}
if PlacementBucket != "mesh-delivery-indexer_placement" || PlacementFile("ace") != "placement/ace.json" {
t.Fatalf("bucket %q, file %q", PlacementBucket, PlacementFile("ace"))
}
for p, want := range map[string]string{"placement/ace.json": "ace", "./placement/ace.json": "ace",
"placement/x/ace.json": "", "modules/ace.json": "", "placement/ace.yaml": ""} {
if n, ok := PlacementFileNode(p); n != want || ok != (want != "") {
t.Errorf("PlacementFileNode(%q) = %q, %v", p, n, ok)
}
}
})
g14 := Placement{Node: "g14", Revision: "abc",
Decided: NodeDecided{Endpoint: "g14.example:51820", Site: "home", Address: "10.77.0.2", Account: "op", Taken: []string{"zsh"}},
Modules: []PlacedModule{
{Module: "zsh"},
{Module: "app", Ports: map[string]int{"8080": 18080}, Bindings: map[string]Provider{"postgres": {Node: "novox", Module: "postgres"}}},
},
Pins: map[string]Provider{"shell": {Node: "g14", Module: "zsh"}},
Holds: []Holding{{Claim: "node-shell", Scope: "node", Module: "zsh"}},
}
novox := Placement{Node: "novox", Decided: NodeDecided{Hub: true, Address: "10.77.0.1"}, Modules: []PlacedModule{{Module: "postgres", Ports: map[string]int{"5432": 5432}}},
Holds: []Holding{{Claim: "mesh-store", Scope: "mesh", Module: "postgres"}}}
t.Run("value", func(t *testing.T) {
raw, err := EncodePlacement(g14)
if err != nil {
t.Fatal(err)
}
back, err := DecodePlacement(raw)
if err != nil || back.V != Version || !SamePlacement(back, g14) {
t.Fatalf("round trip: %+v, %v", back, err)
}
moved := g14
moved.Revision = "def"
if !SamePlacement(g14, moved) {
t.Error("a new revision alone counts as a change")
}
if _, err := DecodePlacement([]byte(`{"v":9,"node":"g14","modules":[]}`)); !errors.Is(err, ErrUnknownVersion) {
t.Errorf("version 9: %v", err)
}
for name, bad := range map[string]string{
"module twice": `{"v":1,"node":"g14","modules":[{"module":"a"},{"module":"a"}]}`,
"port not number": `{"v":1,"node":"g14","modules":[{"module":"a","ports":{"http":80}}]}`,
"host port zero": `{"v":1,"node":"g14","modules":[{"module":"a","ports":{"80":0}}]}`,
"binding no node": `{"v":1,"node":"g14","modules":[{"module":"a","bindings":{"pg":{"module":"pg"}}}]}`,
"pin no module": `{"v":1,"node":"g14","modules":[],"pins":{"pg":{"node":"x"}}}`,
"hold not placed": `{"v":1,"node":"g14","modules":[],"holds":[{"claim":"s","module":"a"}]}`,
"node not a name": `{"v":1,"node":"g.14","modules":[]}`,
"value not shaped": `{"v":1,"node":"g14","modules":{}}`,
} {
if _, err := DecodePlacement([]byte(bad)); err == nil {
t.Errorf("%s: taken", name)
}
}
})
t.Run("as the composition reads it", func(t *testing.T) {
if got := g14.Assigned(); !reflect.DeepEqual(got, []string{"zsh", "app"}) {
t.Errorf("Assigned = %v, want the file's order", got)
}
if got := g14.ChosenPins(); !reflect.DeepEqual(got, map[string]catalogue.Chosen{"shell": {Node: "g14", Module: "zsh"}}) {
t.Errorf("ChosenPins = %v", got)
}
if got := g14.Bound(); !reflect.DeepEqual(got, map[string]map[string]catalogue.Chosen{"app": {"postgres": {Node: "novox", Module: "postgres"}}}) {
t.Errorf("Bound = %v", got)
}
if got := g14.HostPorts(); !reflect.DeepEqual(got, map[string]map[int]int{"app": {8080: 18080}}) {
t.Errorf("HostPorts = %v", got)
}
if got := g14.Held(); !reflect.DeepEqual(got, []catalogue.Held{{Claim: "node-shell", Scope: "node", Node: "g14", Module: "zsh"}}) {
t.Errorf("Held = %v", got)
}
layered := PlacedModule{Module: "app", HasMesh: true, Layer: map[string]any{"expose": map[string]any{"80": "mesh"}}}
want := []catalogue.Layer{
{From: catalogue.MeshWideLayer, Values: map[string]any{}},
{From: "g14", Values: map[string]any{"mode": "x"}, Placement: map[string]any{"expose": map[string]any{"80": "mesh"}}},
}
if got := layered.Layers("g14", map[string]any{"mode": "x"}); !reflect.DeepEqual(got, want) {
t.Errorf("Layers = %#v, want %#v", got, want)
}
if got := (PlacedModule{Module: "a"}).Layers("g14", map[string]any{"k": 1}); len(got) != 1 || got[0].From != "g14" {
t.Errorf("values alone make the node's layer: %#v", got)
}
if got := (PlacedModule{Module: "a", HasLayer: true}).Layers("g14", nil); len(got) != 1 || got[0].Values == nil {
t.Errorf("an empty layer of the node's own is kept, with empty values: %#v", got)
}
if got := (PlacedModule{Module: "a"}).Layers("g14", nil); got != nil {
t.Errorf("no layer and no value is no layer: %#v", got)
}
empty := Placement{Node: "x"}
if empty.ChosenPins() != nil || empty.Bound() != nil || empty.HostPorts() != nil || len(empty.Held()) != 0 {
t.Error("an empty placement reads as something")
}
})
t.Run("what a change says", func(t *testing.T) {
after := g14
after.Revision = "def"
after.Modules = []PlacedModule{{Module: "app", Ports: map[string]int{"8080": 18081}}, {Module: "new"}}
got := ChangeOf(g14, after)
want := PlacementChange{Node: "g14", Revision: "def", Added: []string{"new"}, Removed: []string{"zsh"}, Changed: []string{"app"}}
if !reflect.DeepEqual(got, want) {
t.Errorf("ChangeOf = %+v, want %+v", got, want)
}
pinned := g14
pinned.Pins = nil
if got := ChangeOf(g14, pinned); !reflect.DeepEqual(got.Changed, []string{"app", "zsh"}) {
t.Errorf("a pin change changes every module on the node: %+v", got)
}
if got := ChangeOf(Placement{}, novox); !reflect.DeepEqual(got.Added, []string{"postgres"}) || got.Node != "novox" {
t.Errorf("a new node: %+v", got)
}
})
t.Run("the catalogue's check", func(t *testing.T) {
held := func(m string) bool { return m != "ghost" }
enrolled := func(n string) bool { return n != "stranger" }
if r := CheckPlacements(map[string]Placement{"g14": g14, "novox": novox}, held, enrolled); len(r) != 0 {
t.Fatalf("a sound placement is refused: %v", r)
}
clash := g14
clash.Modules = append([]PlacedModule{{Module: "ghost", Ports: map[string]int{"80": 18080}}}, g14.Modules...)
clash.Modules[2] = PlacedModule{Module: "app", Ports: map[string]int{"8080": 18080}, Bindings: map[string]Provider{"postgres": {Node: "ace", Module: "postgres"}}}
clash.Pins = map[string]Provider{"shell": {Node: "g14", Module: "bash"}}
twice := novox
twice.Node = "stranger"
twice.Decided.Address = "10.77.0.2"
r := CheckPlacements(map[string]Placement{"g14": clash, "novox": novox, "stranger": twice}, held, enrolled)
for _, want := range []string{"holds no module ghost", "ghost and app are both published on host port 18080",
"bound to ace/postgres", "pin of shell names g14/bash", "stranger is not an enrolled node",
"mesh-store, held once for the mesh, is held on novox and stranger",
"g14 and stranger both have the address 10.77.0.2", "novox, stranger are each placed as it"} {
found := false
for _, line := range r {
found = found || strings.Contains(line, want)
}
if !found {
t.Errorf("no refusal says %q; got %v", want, r)
}
}
})
}
+16
View File
@@ -537,6 +537,11 @@ type Forge struct {
Act string `json:"act,omitempty"` Act string `json:"act,omitempty"`
Words string `json:"words,omitempty"` Words string `json:"words,omitempty"`
Open []OpenPull `json:"open,omitempty"` Open []OpenPull `json:"open,omitempty"`
// Placements are, on a merged message of the catalogue repository, every placement file of the merge commit, the
// bytes exact, by node (PlacementFile; novox/hq ADR 0549 §2), and PlacementsRead says they were read, so a catalogue
// whose last placement file was removed is told from a message that read none (older writers, another repository).
Placements map[string]string `json:"placements,omitempty"`
PlacementsRead bool `json:"placements-read,omitempty"`
} }
func (f Forge) check() error { func (f Forge) check() error {
@@ -549,6 +554,17 @@ func (f Forge) check() error {
if len(f.Open) > 0 { if len(f.Open) > 0 {
return fmt.Errorf("delivery: only an open message names open pull requests") return fmt.Errorf("delivery: only an open message names open pull requests")
} }
if (len(f.Placements) > 0 || f.PlacementsRead) && f.Kind != ForgeMerged {
return fmt.Errorf("delivery: only a merged message carries placement files")
}
if len(f.Placements) > 0 && !f.PlacementsRead {
return fmt.Errorf("delivery: a message with placement files must say it read them (placements-read)")
}
for node := range f.Placements {
if err := checkName("node", node); err != nil {
return fmt.Errorf("delivery: a placement file names no node: %w", err)
}
}
if err := checkRepository(f.Repository); err != nil { if err := checkRepository(f.Repository); err != nil {
return err return err
} }
+179 -1
View File
@@ -45,6 +45,86 @@ const (
// AskLanguages are the build seats' languages. // AskLanguages are the build seats' languages.
var AskLanguages = []string{AskGo, AskNode, AskImage} var AskLanguages = []string{AskGo, AskNode, AskImage}
// **An ask may name what its node needs** (issue 890, novox/hq ADR 0525 §1 as the placement record amends it). The seat's
// workers are shared, so the asker still chooses no node; a check ask whose packages' tests need something only some
// nodes have names it in Needs, and a holder whose node lacks one of them takes no part of the ask: it ends the ask's step
// refused as declined (StepDeclined) and drops the message, and the asker puts the ask again under a new id, so a node
// that has the need takes it. A need is a node seat whose holder installs what the tests read on its node; a package
// declares it beside its tests, and the build module's digest reports it (LevelDigest.Needs).
// NeedNodeBuildNode is the need of a package whose integration tests read the Node build module's installed toolchain
// (ADR 0533 §1): the node holds the seat node-build-node, so its toolchain is installed where the Go build module's
// integration level is given it.
const NeedNodeBuildNode = "node-build-node"
// AskNeeds are the needs an ask may name.
var AskNeeds = []string{NeedNodeBuildNode}
// CheckNeed refuses a need that is not one of AskNeeds.
func CheckNeed(need string) error {
for _, n := range AskNeeds {
if need == n {
return nil
}
}
return fmt.Errorf("delivery: %q is not a need an ask may name (%s)", need, strings.Join(AskNeeds, ", "))
}
// Lacks is what of needs has does not say the node has, in the order of needs; a nil has has nothing.
func Lacks(needs []string, has func(need string) bool) []string {
var out []string
for _, n := range needs {
if has == nil || !has(n) {
out = append(out, n)
}
}
return out
}
// PackageNeeds is one package and what its tests need of the node at a level.
type PackageNeeds struct {
Package string `json:"package"`
Needs []string `json:"needs"`
}
// StepDeclined is the result of the step of an ask a holder declined: the ask's needs, and those the holder's node lacks.
// The step's state is refused; the asker puts the ask again under a new id (DeclinedOf).
type StepDeclined struct {
Declined bool `json:"declined"`
Needs []string `json:"needs"`
Lacks []string `json:"lacks"`
}
// DeclinedOf says whether step is a declined ask's, and what its node lacked.
func DeclinedOf(step Step) (StepDeclined, bool) {
if step.State != StepRefused || len(step.Result) == 0 {
return StepDeclined{}, false
}
var d StepDeclined
if err := json.Unmarshal(step.Result, &d); err != nil || !d.Declined {
return StepDeclined{}, false
}
return d, true
}
// UnmetNeeds are the needs of the packages the levels left untested (LevelAnswer.Unmet), each once, sorted.
func UnmetNeeds(levels []LevelAnswer) []string {
seen := map[string]bool{}
var out []string
for _, l := range levels {
for _, u := range l.Unmet {
for _, n := range u.Needs {
if !seen[n] {
seen[n] = true
out = append(out, n)
}
}
}
}
sort.Strings(out)
return out
}
// AskRuns is how many times an ask is run before it is taken for poison (issue 1907): an ask whose run never ends (it // AskRuns is how many times an ask is run before it is taken for poison (issue 1907): an ask whose run never ends (it
// takes its node down, or its process dies with it) is delivered again by its stream to the next free node (ADR 0525 // takes its node down, or its process dies with it) is delivered again by its stream to the next free node (ADR 0525
// §6); the holder that takes it past AskRuns runs it no more, writes its step failed saying so, and ends it, so the // §6); the holder that takes it past AskRuns runs it no more, writes its step failed saying so, and ends it, so the
@@ -171,6 +251,8 @@ type StepAsk struct {
Build *BuildJob `json:"build,omitempty"` Build *BuildJob `json:"build,omitempty"`
Check *CheckJob `json:"check,omitempty"` Check *CheckJob `json:"check,omitempty"`
Compose *ComposeJob `json:"compose,omitempty"` Compose *ComposeJob `json:"compose,omitempty"`
// Needs are what the node that runs the ask must have (AskNeeds); a check ask's alone. Empty: any node.
Needs []string `json:"needs,omitempty"`
} }
// ComposeJob is a compose ask's job (novox/hq ADR 0482 clause 4, issue 1622): fetch the commit, build the composer // ComposeJob is a compose ask's job (novox/hq ADR 0482 clause 4, issue 1622): fetch the commit, build the composer
@@ -255,6 +337,9 @@ type CheckJob struct {
RunnerWhen string `json:"runner_when,omitempty"` RunnerWhen string `json:"runner_when,omitempty"`
// Run is a try's test pattern (go test -run): each level runs it alone, digesting nothing. // Run is a try's test pattern (go test -run): each level runs it alone, digesting nothing.
Run string `json:"run,omitempty"` Run string `json:"run,omitempty"`
// Packages, when set, are the only packages each level tests (issue 890, ADR 0552 §4): a placed ask names the
// packages the first ask left unmet, and the others are neither tested nor taken kept, so nothing is judged twice.
Packages []string `json:"packages,omitempty"`
} }
func (a StepAsk) check() error { func (a StepAsk) check() error {
@@ -290,6 +375,13 @@ func (a StepAsk) check() error {
return fmt.Errorf("delivery: the ask %s is a check on the delivery stream, which carries trunk builds alone", a.ID) return fmt.Errorf("delivery: the ask %s is a check on the delivery stream, which carries trunk builds alone", a.ID)
case a.Check != nil && a.Language == AskImage: case a.Check != nil && a.Language == AskImage:
return fmt.Errorf("delivery: the ask %s is a check of an image, which has no tests", a.ID) return fmt.Errorf("delivery: the ask %s is a check of an image, which has no tests", a.ID)
case len(a.Needs) > 0 && a.Check == nil:
return fmt.Errorf("delivery: the ask %s names needs and is no check, whose tests alone need a node", a.ID)
}
for _, n := range a.Needs {
if err := CheckNeed(n); err != nil {
return fmt.Errorf("the ask %s: %w", a.ID, err)
}
} }
if a.Build != nil { if a.Build != nil {
return a.Build.check(a.ID) return a.Build.check(a.ID)
@@ -350,6 +442,11 @@ func (c CheckJob) check(id string) error {
return err return err
} }
} }
for _, p := range c.Packages {
if p == "" || strings.ContainsAny(p, " \t\n") {
return fmt.Errorf("delivery: the check ask %s names a package %q that is no import path", id, p)
}
}
return checkFetchArgs(id, c.Fetch) return checkFetchArgs(id, c.Fetch)
} }
@@ -433,6 +530,9 @@ type LevelAnswer struct {
Digest *LevelDigest `json:"digest,omitempty"` Digest *LevelDigest `json:"digest,omitempty"`
Kept []KeptTaken `json:"kept,omitempty"` Kept []KeptTaken `json:"kept,omitempty"`
Tests []json.RawMessage `json:"tests,omitempty"` Tests []json.RawMessage `json:"tests,omitempty"`
// Unmet are the packages the level did not test on this node, because their tests need what it lacks (issue 890): the
// asker keeps what passed and asks them again of a node that has it (StepAsk.Needs).
Unmet []PackageNeeds `json:"unmet,omitempty"`
} }
// LevelDigest is the digest's answer as a level keeps it in the ask's step result: its packages a list of // LevelDigest is the digest's answer as a level keeps it in the ask's step result: its packages a list of
@@ -446,6 +546,8 @@ type LevelDigest struct {
Beside []string `json:"beside,omitempty"` Beside []string `json:"beside,omitempty"`
WithoutTests []string `json:"without_tests,omitempty"` WithoutTests []string `json:"without_tests,omitempty"`
JudgedBy string `json:"judged_by,omitempty"` JudgedBy string `json:"judged_by,omitempty"`
// Needs are the packages whose tests at this level need something of the node (issue 890), as they declare it.
Needs []PackageNeeds `json:"needs,omitempty"`
} }
// PackageDigest is one package's digest at a level. // PackageDigest is one package's digest at a level.
@@ -463,6 +565,7 @@ func (d *LevelDigest) UnmarshalJSON(raw []byte) error {
Beside []string `json:"beside"` Beside []string `json:"beside"`
WithoutTests []string `json:"without_tests"` WithoutTests []string `json:"without_tests"`
JudgedBy string `json:"judged_by"` JudgedBy string `json:"judged_by"`
Needs []PackageNeeds `json:"needs"`
} }
if err := json.Unmarshal(raw, &v); err != nil { if err := json.Unmarshal(raw, &v); err != nil {
return err return err
@@ -482,7 +585,20 @@ func (d *LevelDigest) UnmarshalJSON(raw []byte) error {
} }
} }
sort.Slice(pkgs, func(i, j int) bool { return pkgs[i].Package < pkgs[j].Package }) sort.Slice(pkgs, func(i, j int) bool { return pkgs[i].Package < pkgs[j].Package })
*d = LevelDigest{Level: v.Level, Packages: pkgs, Beside: v.Beside, WithoutTests: v.WithoutTests, JudgedBy: v.JudgedBy} *d = LevelDigest{Level: v.Level, Packages: pkgs, Beside: v.Beside, WithoutTests: v.WithoutTests, JudgedBy: v.JudgedBy, Needs: v.Needs}
return nil
}
// NeedsOf is what the package's tests need of the node at the digest's level; nil when it declares nothing.
func (d *LevelDigest) NeedsOf(pkg string) []string {
if d == nil {
return nil
}
for _, n := range d.Needs {
if n.Package == pkg {
return n.Needs
}
}
return nil return nil
} }
@@ -535,6 +651,11 @@ func ParseAskStepKey(key string) (node, askID string, err error) {
return node, askID, nil return node, askID, nil
} }
// StepBound is how long an asker waits for one ask's step once a build node took it (30 minutes): past it the asker has
// stopped waiting, so a key still running is nobody's. The build modules end such a key (buildstep.EndStale); the
// checker's, the builder's and the watchman's own copies of the number are to read this one (issue 1916).
const StepBound = 30 * time.Minute
// PausesState is the state a build module keeps its nodes' pause keys in. It has no age, unlike StepsState, whose keys // PausesState is the state a build module keeps its nodes' pause keys in. It has no age, unlike StepsState, whose keys
// are per step and dropped after their age: a pause kept among the steps would end by itself. // are per step and dropped after their age: a pause kept among the steps would end by itself.
const PausesState = "pauses" const PausesState = "pauses"
@@ -572,3 +693,60 @@ func DecodePause(data []byte) (Pause, error) {
} }
return p, nil return p, nil
} }
// CapacityKey is the key, in a build module's PausesState, where the module on node says the capacity it judges by
// now and why: `capacity.<node>` (issue 1806's follow-up). The pauses state has no age, so the value stands until the
// module writes another; a reader such as the delivery view's build_usage says it in place of the configured capacity.
func CapacityKey(node string) (string, error) {
if err := checkName("node", node); err != nil {
return "", err
}
return "capacity." + node, nil
}
// Capacity is a build seat's capacity on one node as its holder judges by it: the configured capacity (the module's
// setting `capacity`), bounded by the cores its node's build slice gives (its CPU quota, read live) over the cores one
// step needs (the module's setting `cores-per-step`). Effective is min(Configured, ByCPU) when a quota is known, else
// Configured; one step at least while Configured is above zero, so a node whose quota is under one step's need still
// builds, slowly.
type Capacity struct {
V int `json:"v"`
Effective int `json:"effective"`
Configured int `json:"configured"`
// ConfiguredFrom says where Configured was read: mesh-config's value as the config watch handed it, or the settings
// file.
ConfiguredFrom string `json:"configured-from,omitempty"`
// QuotaCores is the build slice's CPU quota in processors (cpu.max: 1200% is 12), 0 when none is known; QuotaFrom
// says where it was read (the cgroup, or the share the composer gave) or why none is known.
QuotaCores float64 `json:"quota-cores,omitempty"`
QuotaFrom string `json:"quota-from"`
// CoresPerStep is what one step of this module needs; ByCPU is floor(QuotaCores / CoresPerStep), at least 1, and 0
// when no quota is known.
CoresPerStep float64 `json:"cores-per-step"`
ByCPU int `json:"by-cpu,omitempty"`
// BuildHours is the node's build window (HH:MM-HH:MM, its local time and zone), absent when it builds at any hour;
// OutsideHours says the time judged fell outside it, so Effective is 0 and the next step waits for the window to open.
BuildHours string `json:"build-hours,omitempty"`
OutsideHours bool `json:"outside-hours,omitempty"`
// BuildHoursFault is why the node's build-hours value cannot be read, naming the value: it then builds at any hour.
BuildHoursFault string `json:"build-hours-fault,omitempty"`
// Why is the capacity in words: what bounds it.
Why string `json:"why"`
At time.Time `json:"at,omitzero"`
}
// EncodeCapacity is c as JSON at Version.
func EncodeCapacity(c Capacity) ([]byte, error) {
c.V = Version
c.At = c.At.UTC()
return json.Marshal(c)
}
// DecodeCapacity reads a capacity value; it refuses an unknown version (ErrUnknownVersion).
func DecodeCapacity(data []byte) (Capacity, error) {
var c Capacity
if err := decode(data, &c); err != nil {
return Capacity{}, err
}
return c, nil
}
+18
View File
@@ -56,6 +56,7 @@ func TestAskRefusesUnknownKindAndLanguage(t *testing.T) {
"recheck": func(a *StepAsk) { a.Kind = AskRecheck }, "recheck": func(a *StepAsk) { a.Kind = AskRecheck },
"commit": func(a *StepAsk) { a.Build.Fetch = map[string]any{"repository": "novox/x", "commit": "main"} }, "commit": func(a *StepAsk) { a.Build.Fetch = map[string]any{"repository": "novox/x", "commit": "main"} },
"path": func(a *StepAsk) { a.Build.Path = "" }, "path": func(a *StepAsk) { a.Build.Path = "" },
"needs": func(a *StepAsk) { a.Needs = []string{NeedNodeBuildNode} }, // issue 890: a build needs no node
} }
for name, change := range cases { for name, change := range cases {
a := buildAsk() a := buildAsk()
@@ -80,6 +81,23 @@ func TestAskRefusesUnknownKindAndLanguage(t *testing.T) {
if _, err := EncodeStepAsk(c); err == nil { if _, err := EncodeStepAsk(c); err == nil {
t.Error("a check of an image encoded") t.Error("a check of an image encoded")
} }
// Issue 890: a check names a known need and round-trips it; an unknown need is refused.
c = checkAsk()
c.Check.Packages = []string{"a b"}
if _, err := EncodeStepAsk(c); err == nil {
t.Error("a package that is no import path encoded")
}
c.Check.Packages = []string{"x/pkg/a"}
c.Needs = []string{"gpu"}
if _, err := EncodeStepAsk(c); err == nil {
t.Error("an unknown need encoded")
}
c.Needs = []string{NeedNodeBuildNode}
if raw, err := EncodeStepAsk(c); err != nil {
t.Errorf("a check needing %s: %v", NeedNodeBuildNode, err)
} else if back, err := DecodeStepAsk(raw); err != nil || len(back.Needs) != 1 || back.Needs[0] != NeedNodeBuildNode {
t.Errorf("the need did not round-trip: %+v %v", back.Needs, err)
}
raw, _ := EncodeStepAsk(buildAsk()) raw, _ := EncodeStepAsk(buildAsk())
for _, bad := range []string{ for _, bad := range []string{
strings.Replace(string(raw), `"kind":"delivery"`, `"kind":"queue"`, 1), strings.Replace(string(raw), `"kind":"delivery"`, `"kind":"queue"`, 1),
+8
View File
@@ -70,6 +70,14 @@ type Step struct {
Ended time.Time `json:"ended,omitzero"` Ended time.Time `json:"ended,omitzero"`
Reason string `json:"reason,omitempty"` Reason string `json:"reason,omitempty"`
Result json.RawMessage `json:"result,omitempty"` Result json.RawMessage `json:"result,omitempty"`
// What says what the step is, for a reader of the steps state: an ask's kind, repository, head and levels or
// module (issue 1890). Empty for a step a tool started by hand.
What string `json:"what,omitempty"`
// Phase is the part of its job a running ask's step is in (fetch, a level's digest, lint or tests), and PhaseAt
// when it entered it: the step key is written again at each phase, so a slow step says where it stands (issue 1890).
// A reader waiting for the end treats such a write as any running one.
Phase string `json:"phase,omitempty"`
PhaseAt time.Time `json:"phase-at,omitzero"`
} }
func (s Step) check() error { func (s Step) check() error {
+174
View File
@@ -0,0 +1,174 @@
package delivery
import (
"encoding/json"
"fmt"
"reflect"
"strings"
"time"
)
// **The merge turns** (novox/hq ADR 0548 clauses 3 and 4). An approved merge holds a turn on each repository it goes into:
// before it reads the checker's verdicts for the merge, the gate writes one key per repository in its bucket turns, naming
// the deliveries that wait in the order of their Approve, the verdict key each waits for and since when. The gate is the
// key's one writer; the checker is granted to read the bucket, reads it absent as empty, and claims no check of a repository
// a turn names, except the check of the waiting delivery's own heads (TurnHolds). The key goes when the last delivery in it
// merged or left.
const (
// TurnsBucket is the gate's bucket of merge turns.
TurnsBucket = GateModule + "_turns"
// TurnsHistory is the revisions the turns bucket keeps per key.
TurnsHistory = 5
)
// TurnsKey is the key, in TurnsBucket, of a repository's turn: `<owner>.<name>`, lowercase, each part escaped (EscapeName),
// so the key names the repository the checker's verdicts name as a trunk whatever its case.
func TurnsKey(repository string) (string, error) {
repository = strings.ToLower(repository)
if _, err := escapeRepository(repository); err != nil {
return "", err
}
owner, name, _ := strings.Cut(repository, "/")
return EscapeName(owner) + "." + EscapeName(name), nil
}
// ParseTurnsKey is the inverse of TurnsKey: the repository, lowercase.
func ParseTurnsKey(key string) (string, error) {
owner, name, ok := strings.Cut(key, ".")
if !ok || owner == "" || name == "" || strings.Contains(name, ".") {
return "", fmt.Errorf("delivery: %q is not a turns key, <owner>.<name>", key)
}
o, err := UnescapeName(owner)
if err != nil {
return "", fmt.Errorf("delivery: %q is not a turns key: %w", key, err)
}
n, err := UnescapeName(name)
if err != nil {
return "", fmt.Errorf("delivery: %q is not a turns key: %w", key, err)
}
repository := o + "/" + n
if again, err := TurnsKey(repository); err != nil || again != key {
return "", fmt.Errorf("delivery: %q is not a turns key: it does not name a lowercase <owner>/<name>", key)
}
return repository, nil
}
// Turn is the value of a turns key: the repository and the approved deliveries that wait to merge into it, in the order of
// their Approve. A turn with no delivery is no value: its key is deleted.
type Turn struct {
V int `json:"v"`
Repository string `json:"repository"`
Waiting []TurnWait `json:"waiting"`
}
// TurnWait is one approved delivery in a turn: its group (a lone pull request's branch), the hash of the heads the Approve
// covers (the checker's verdict key's hash, HeadsHash over its members), the Approve's ask key, when it was approved and since
// when it waits; and, while a check of another delivery runs on the repository, that check's verdict key, when the gate saw
// it start, and the step the checker says it is at.
type TurnWait struct {
Group string `json:"group"`
Heads string `json:"heads"`
Ask string `json:"ask,omitempty"`
Approved time.Time `json:"approved"`
Since time.Time `json:"since"`
// WaitsFor is the verdict key (`<group>.<heads hash>`) of the running check the merge waits for; empty when none runs.
WaitsFor string `json:"waits-for,omitempty"`
WaitsSince time.Time `json:"waits-since,omitzero"`
Step string `json:"step,omitempty"`
}
func (t Turn) check() error {
repository := strings.ToLower(t.Repository)
if repository != t.Repository {
return fmt.Errorf("delivery: the turn's repository %q is not lowercase", t.Repository)
}
if _, err := escapeRepository(repository); err != nil {
return err
}
if len(t.Waiting) == 0 {
return fmt.Errorf("delivery: the turn of %s names no delivery: its key is deleted instead", t.Repository)
}
seen := map[string]bool{}
for i, w := range t.Waiting {
if w.Group == "" {
return fmt.Errorf("delivery: a delivery in the turn of %s names no group", t.Repository)
}
if !isHex12(w.Heads) {
return fmt.Errorf("delivery: the heads hash %q in the turn of %s is not 12 lowercase hex digits", w.Heads, t.Repository)
}
if seen[w.Heads] {
return fmt.Errorf("delivery: the turn of %s names the heads %s twice", t.Repository, w.Heads)
}
seen[w.Heads] = true
if w.Approved.IsZero() || w.Since.IsZero() {
return fmt.Errorf("delivery: the delivery %s in the turn of %s says no time of its Approve or of its wait", w.Group, t.Repository)
}
if i > 0 && w.Approved.Before(t.Waiting[i-1].Approved) {
return fmt.Errorf("delivery: the turn of %s is not in the order of the Approve: %s was approved before %s", t.Repository, w.Group, t.Waiting[i-1].Group)
}
if w.WaitsFor != "" {
if _, _, err := ParseCheckerVerdictKey(w.WaitsFor); err != nil {
return fmt.Errorf("delivery: the delivery %s in the turn of %s waits for what is not a verdict key: %w", w.Group, t.Repository, err)
}
} else if !w.WaitsSince.IsZero() || w.Step != "" {
return fmt.Errorf("delivery: the delivery %s in the turn of %s says when and where a check runs, and names no check", w.Group, t.Repository)
}
}
return nil
}
// TurnHolds says whether a turn holds off the check of the heads: it names deliveries, and none of them of these heads. A
// turn never holds the check of heads that wait in it (the carry a waiting merge needs): two approved merges in one turn whose
// trunk moved would otherwise each hold the other's carry, and neither would ever merge (review of ADR 0548's build, round 1).
func TurnHolds(t Turn, heads string) bool {
if len(t.Waiting) == 0 {
return false
}
for _, w := range t.Waiting {
if w.Heads == heads {
return false
}
}
return true
}
// EncodeTurn is t as JSON at Version; it refuses a turn with no delivery, a repository that is not lowercase <owner>/<name>,
// a delivery without its group, heads hash or times, the same heads twice, an order other than the Approve's, and a check
// waited for that is not a verdict key.
func EncodeTurn(t Turn) ([]byte, error) {
if err := t.check(); err != nil {
return nil, err
}
t.V = Version
return json.Marshal(t)
}
// DecodeTurn reads a turn; it refuses an unknown version (ErrUnknownVersion) and what EncodeTurn refuses.
func DecodeTurn(data []byte) (Turn, error) {
var t Turn
if err := decode(data, &t); err != nil {
return Turn{}, err
}
if err := t.check(); err != nil {
return Turn{}, err
}
return t, nil
}
// SameTurn says whether two turns are the same value, so the gate writes only when they differ.
func SameTurn(a, b Turn) bool {
if len(a.Waiting) != len(b.Waiting) || a.Repository != b.Repository || a.V != b.V {
return false
}
for i := range a.Waiting {
x, y := a.Waiting[i], b.Waiting[i]
if !x.Approved.Equal(y.Approved) || !x.Since.Equal(y.Since) || !x.WaitsSince.Equal(y.WaitsSince) {
return false
}
x.Approved, y.Approved, x.Since, y.Since, x.WaitsSince, y.WaitsSince = time.Time{}, time.Time{}, time.Time{}, time.Time{}, time.Time{}, time.Time{}
if !reflect.DeepEqual(x, y) {
return false
}
}
return true
}
+149
View File
@@ -0,0 +1,149 @@
package delivery
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/seal"
)
// **The grants part's user list, sealed to the bus node** (novox/hq ADR 0547 clauses 7 and 8, issue 1945): the contract
// between the grants part (mesh-delivery modules/mesh-grants), which writes it, and the node-engine of the node holding
// mesh-broker (mesh-host), which follows it once the operator's take-over says so.
//
// - The grants part composes the bus's user list (the accounts block, buscompose.ComposeAccounts, with the vault's
// hashes), seals it to the bus node's sealing key (seal.Seal, the key the node says in its have of the bus module,
// UserListWritten.Key), and puts the box as one object of its object bucket GrantsUserListObjects, named by the
// list's digest (UserListObjectName).
// - It writes the key GrantsUserListKey of its state `user-list` (bucket GrantsUserListBucket): the GrantsUserList
// below, which names the object, the list's digest, its revision, its user count and the users left out, and which
// source the bus node follows (Follow: FollowGrants from take-over, FollowController in shadow and after hand-back).
// - The node-engine of the bus node watches that key alone. While it says FollowGrants it opens the object, checks
// the digest, writes the bus's user-list file (mode 0600) only when the file's digest differs, and says what it
// loaded in its have of the bus module (UserListWritten, Source GrantsUserListSource); otherwise it follows the
// controller's compositions as before (clause 8). Never both.
// The names of the grants part's user list on the bus.
const (
// GrantsUserListBucket is the grants part's state `user-list`.
GrantsUserListBucket = "mesh-grants_user-list"
// GrantsUserListKey is the one key of it the bus node reads.
GrantsUserListKey = "sealed"
// GrantsUserListObjects is the grants part's object bucket, where each sealed list is one object.
GrantsUserListObjects = "mesh-grants-user-list"
// GrantsUserListObjectStream is the bucket's stream.
GrantsUserListObjectStream = "OBJ_" + GrantsUserListObjects
// GrantsUserListSource is the source a have names for a list taken from the grants part.
GrantsUserListSource = GrantsUserListBucket + "." + GrantsUserListKey
)
// Which list the bus node follows.
const (
FollowController = "controller"
FollowGrants = "grants"
)
// GrantsUserList is the value of the grants part's key `user-list.sealed`.
type GrantsUserList struct {
V int `json:"v"`
// Follow is which list the bus node follows: FollowGrants from take-over, FollowController otherwise.
Follow string `json:"follow"`
// Digest is the list's digest, `sha256:<hex>` of the file's bytes as the bus loads them.
Digest string `json:"digest"`
// Object is the object's name in GrantsUserListObjects (UserListObjectName of Digest).
Object string `json:"object"`
// SealedTo is the fingerprint of the key the object is sealed to (SealingKeyID), and Node the node it holds.
SealedTo string `json:"sealed_to"`
Node string `json:"node"`
// Revision counts the lists the grants part wrote: one more for each new digest.
Revision uint64 `json:"revision"`
Users int `json:"users"`
// LeftOut are the users the list leaves out for want of a hash in the vault.
LeftOut []string `json:"left_out"`
At time.Time `json:"at"`
}
// EncodeGrantsUserList is g as JSON at Version; it refuses a value the bus node could not follow.
func EncodeGrantsUserList(g GrantsUserList) ([]byte, error) {
g.V = Version
if g.LeftOut == nil {
g.LeftOut = []string{}
}
if err := g.check(); err != nil {
return nil, err
}
return json.Marshal(g)
}
// DecodeGrantsUserList reads the key's value; it refuses another version and what Encode refuses.
func DecodeGrantsUserList(data []byte) (GrantsUserList, error) {
var g GrantsUserList
if err := decode(data, &g); err != nil {
return GrantsUserList{}, err
}
if err := g.check(); err != nil {
return GrantsUserList{}, err
}
return g, nil
}
func (g GrantsUserList) check() error {
if g.Follow != FollowGrants && g.Follow != FollowController {
return fmt.Errorf("delivery: the user list's follow %q is not %s or %s", g.Follow, FollowGrants, FollowController)
}
if g.Digest == "" {
return nil // a switch with no list written yet: the bus node keeps what it follows
}
if !strings.HasPrefix(g.Digest, "sha256:") || g.Object != UserListObjectName(g.Digest) {
return fmt.Errorf("delivery: the user list names object %q for digest %q; the object is the digest's hex", g.Object, g.Digest)
}
return nil
}
// UserListDigest is a list's digest as the have and the key say it, `sha256:<hex>` of its bytes.
func UserListDigest(content []byte) string {
sum := sha256.Sum256(content)
return "sha256:" + hex.EncodeToString(sum[:])
}
// UserListObjectName is the object a list of that digest is put under: the digest's hex.
func UserListObjectName(digest string) string { return strings.TrimPrefix(digest, "sha256:") }
// SealingKeyID is a short fingerprint of a public key in PEM, `sha256:<16 hex>`, to say which key a box is sealed to
// without carrying the key.
func SealingKeyID(publicPEM string) string {
sum := sha256.Sum256([]byte(strings.TrimSpace(publicPEM)))
return "sha256:" + hex.EncodeToString(sum[:8])
}
// SealUserList is the object a list is put as: the sealed box (seal.Seal) of its bytes to the bus node's key, as JSON.
func SealUserList(list []byte, nodePEM string) ([]byte, error) {
if !strings.Contains(nodePEM, "PUBLIC KEY") {
return nil, fmt.Errorf("delivery: the bus node's key is not a public key in PEM")
}
box, err := seal.Seal(string(list), nodePEM)
if err != nil {
return nil, fmt.Errorf("delivery: the user list could not be sealed: %w", err)
}
return json.Marshal(box)
}
// OpenUserList opens an object with open (the node's own key) and checks it is the list of digest.
func OpenUserList(object []byte, digest string, open func(seal.Box) ([]byte, error)) ([]byte, error) {
var box seal.Box
if err := json.Unmarshal(object, &box); err != nil {
return nil, fmt.Errorf("delivery: the object is no sealed box: %w", err)
}
list, err := open(box)
if err != nil {
return nil, err
}
if got := UserListDigest(list); got != digest {
return nil, fmt.Errorf("delivery: the opened list's digest is %s, and the key names %s", got, digest)
}
return list, nil
}
+16 -3
View File
@@ -8,7 +8,8 @@ import (
func hashOf(c byte) string { return "sha256:" + digestOf(c) } func hashOf(c byte) string { return "sha256:" + digestOf(c) }
// 1943/AC3: a want naming its fingerprint and the digest of the composition it allows round-trips, and so does one // 1943/AC3: a want naming its fingerprint and the digest of the composition it allows round-trips, and so does one
// carrying the answer to a failed backup; a want naming a fingerprint without a digest, a digest not of the sha256 // carrying the answer to a failed backup, and from ADR 0547's second milestone one naming the build alone or the yes
// a person gave; a digest not of the sha256
// form, a removed want naming a digest, an unknown backup answer and a rule past the seventh are refused. // form, a removed want naming a digest, an unknown backup answer and a rule past the seventh are refused.
func TestAWantNamesItsComposition(t *testing.T) { func TestAWantNamesItsComposition(t *testing.T) {
for _, w := range []Want{ for _, w := range []Want{
@@ -17,6 +18,9 @@ func TestAWantNamesItsComposition(t *testing.T) {
{FP: fp2of('a'), Composed: hashOf('1'), Backup: BackupSkip, Words: "the operator: go on without a backup"}, {FP: fp2of('a'), Composed: hashOf('1'), Backup: BackupSkip, Words: "the operator: go on without a backup"},
{FP: fp2of('a'), Composed: hashOf('1'), Newest: fp2of('b'), Rule: RuleComposed, Words: "no composition of b for ace yet"}, {FP: fp2of('a'), Composed: hashOf('1'), Newest: fp2of('b'), Rule: RuleComposed, Words: "no composition of b for ace yet"},
{Removed: true}, {Removed: true},
// ADR 0547 §2, §3 (issue 1126): from the second milestone a want names the build alone, and the yes a person gave.
{FP: fp2of('a'), Self: true},
{FP: fp2of('a'), Self: true, Yes: hashOf('3')},
} { } {
raw, err := EncodeWant(w) raw, err := EncodeWant(w)
if err != nil { if err != nil {
@@ -33,6 +37,9 @@ func TestAWantNamesItsComposition(t *testing.T) {
why string why string
}{ }{
{Want{FP: fp2of('a')}, "names no composition"}, {Want{FP: fp2of('a')}, "names no composition"},
{Want{FP: fp2of('a'), Self: true, Composed: hashOf('1')}, "names a composition"},
{Want{FP: fp2of('a'), Self: true, Yes: "sha256:12"}, "a want's yes allows"},
{Want{Removed: true, Self: true}, "removed want names no"},
{Want{FP: fp2of('a'), Composed: "sha256:12"}, "not sha256"}, {Want{FP: fp2of('a'), Composed: "sha256:12"}, "not sha256"},
{Want{Removed: true, Composed: hashOf('1')}, "removed want names no"}, {Want{Removed: true, Composed: hashOf('1')}, "removed want names no"},
{Want{FP: fp2of('a'), Composed: hashOf('1'), Backup: "later"}, `backup answer "later"`}, {Want{FP: fp2of('a'), Composed: hashOf('1'), Backup: "later"}, `backup answer "later"`},
@@ -45,7 +52,7 @@ func TestAWantNamesItsComposition(t *testing.T) {
} }
} }
if _, err := DecodeWant([]byte(`{"v":1,"fp":"` + fp2of('a') + `"}`)); err == nil || !strings.Contains(err.Error(), "names no composition") { if _, err := DecodeWant([]byte(`{"v":1,"fp":"` + fp2of('a') + `"}`)); err == nil || !strings.Contains(err.Error(), "names no composition") {
t.Errorf("a want of release 5, naming no composition, was read: %v", err) t.Errorf("a want of release 5, naming no composition and not the node's own, was read: %v", err)
} }
if RuleComposed != 7 { if RuleComposed != 7 {
t.Errorf("RuleComposed = %d, want the seventh rule", RuleComposed) t.Errorf("RuleComposed = %d, want the seventh rule", RuleComposed)
@@ -86,11 +93,17 @@ func TestTheNodesOwnWant(t *testing.T) {
if err != nil || back != w { if err != nil || back != w {
t.Errorf("node want came back %+v, %v", back, err) t.Errorf("node want came back %+v, %v", back, err)
} }
for _, bad := range []NodeWant{{}, {Composed: "sha256:1"}} { for _, bad := range []NodeWant{{}, {Composed: "sha256:1"}, {Self: true, Composed: hashOf('2')}} {
if _, err := EncodeNodeWant(bad); err == nil { if _, err := EncodeNodeWant(bad); err == nil {
t.Errorf("EncodeNodeWant(%+v) accepted", bad) t.Errorf("EncodeNodeWant(%+v) accepted", bad)
} }
} }
// ADR 0547 §2 (issue 1126): from the second milestone the node composes its own composition, and its want names none.
if raw, err := EncodeNodeWant(NodeWant{Self: true, Words: "the node composes its own"}); err != nil {
t.Errorf("a node want naming no composition: %v", err)
} else if back, err := DecodeNodeWant(raw); err != nil || back.Composed != "" {
t.Errorf("a node want naming no composition came back %+v, %v", back, err)
}
if _, err := DecodeNodeWant([]byte(`{"v":1,"composed":"` + hashOf('2') + `","fp":"` + fp2of('a') + `"}`)); err == nil { if _, err := DecodeNodeWant([]byte(`{"v":1,"composed":"` + hashOf('2') + `","fp":"` + fp2of('a') + `"}`)); err == nil {
t.Errorf("a node want naming a fingerprint was read") t.Errorf("a node want naming a fingerprint was read")
} }
+1 -1
View File
@@ -101,7 +101,7 @@ func (k *KeyKeeper) ensure(ctx context.Context) error {
} }
r := k.Run(ctx, nil, "ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-C", k.Account+"@"+k.Node+" (mesh)", "-f", k.path()) r := k.Run(ctx, nil, "ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-C", k.Account+"@"+k.Node+" (mesh)", "-f", k.path())
if r.Status != 0 || r.Err != "" { if r.Status != 0 || r.Err != "" {
return fmt.Errorf("ssh-keygen: %s", firstLine(r.Err+" "+r.Stderr)) return fmt.Errorf("ssh-keygen: %s", Said(r.Err+"\n"+r.Stderr))
} }
return nil return nil
} }
+58 -6
View File
@@ -9,6 +9,7 @@ import (
"os/exec" "os/exec"
"strings" "strings"
"time" "time"
"unicode/utf8"
) )
// Ran is what a command did: its output, its exit status, and why it never ran to an answer. // Ran is what a command did: its output, its exit status, and why it never ran to an answer.
@@ -43,7 +44,8 @@ func ExecRunnerFor(bound time.Duration) Runner {
// ExecRunnerClean is ExecRunnerFor whose commands get exactly the env argument as their environment, and nothing of // ExecRunnerClean is ExecRunnerFor whose commands get exactly the env argument as their environment, and nothing of
// this process's (issue 1497; novox/hq ADR 0466 §4: no registry password in a fetch's environment). The caller // this process's (issue 1497; novox/hq ADR 0466 §4: no registry password in a fetch's environment). The caller
// gives PATH and HOME itself. A directory is the command's own business (`git -C`, `go -C`). // gives PATH and HOME itself. A directory is the command's own business (`git -C`, `go -C`): it is started in the HOME
// it is given, or the machine's temporary directory, never in this process's (stableDir).
func ExecRunnerClean(bound time.Duration) Runner { func ExecRunnerClean(bound time.Duration) Runner {
return func(ctx context.Context, env []string, name string, args ...string) Ran { return func(ctx context.Context, env []string, name string, args ...string) Ran {
return execRun(ctx, bound, true, env, name, args...) return execRun(ctx, bound, true, env, name, args...)
@@ -62,6 +64,7 @@ func execRun(ctx context.Context, bound time.Duration, clean bool, env []string,
} else { } else {
cmd.Env = append(os.Environ(), env...) cmd.Env = append(os.Environ(), env...)
} }
cmd.Dir = stableDir(clean, env)
var out, errb bytes.Buffer var out, errb bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errb cmd.Stdout, cmd.Stderr = &out, &errb
err := cmd.Run() err := cmd.Run()
@@ -83,11 +86,60 @@ func execRun(ctx context.Context, bound time.Duration, clean bool, env []string,
return r return r
} }
func firstLine(s string) string { // stableDir is the directory a command starts in when it names none (novox/hq issue 734). **Never one that may be
for _, l := range strings.Split(s, "\n") { // deleted under it**: a module runs in the directory its build was unpacked in, which a new build replaces, and a
if l = strings.TrimSpace(l); l != "" { // command started there inherits it — `go version` then fails "getwd: no such file or directory" before it reads
return l // anything. A clean command starts in the HOME it is given, or the temporary directory; any other keeps this
// process's directory while it exists, and starts in the temporary directory once it is gone. "" is this process's.
func stableDir(clean bool, env []string) string {
if !clean {
if _, err := os.Getwd(); err == nil {
return ""
}
return os.TempDir()
}
for i := len(env) - 1; i >= 0; i-- {
if home, ok := strings.CutPrefix(env[i], "HOME="); ok {
if info, err := os.Stat(home); err == nil && info.IsDir() {
return home
}
break
} }
} }
return "" return os.TempDir()
}
// SaidBound bounds what Said keeps of a command's words, so a reason stays a line of a step's record and of a log.
const SaidBound = 600
// Said is what a command said of its failure in its own words, for a reason: every line that says something, trimmed
// and joined with "; ", bounded at SaidBound bytes. A line that is a bare prefix and nothing else (git writes "error:"
// alone before and after the forge's own message, "remote:" or "fatal:" likewise) is left out, so the cause is never
// cut off behind it: the first line alone of `git fetch` from a repository the account cannot read was "error:".
func Said(s string) string {
var keep []string
for _, l := range strings.Split(s, "\n") {
l = strings.TrimSpace(l)
if l == "" || barePrefix(l) {
continue
}
keep = append(keep, l)
}
out := strings.Join(keep, "; ")
if len(out) > SaidBound {
cut := SaidBound
for cut > 0 && !utf8.RuneStart(out[cut]) {
cut--
}
out = out[:cut] + " …"
}
return out
}
// barePrefix says whether a line is one word and a colon, such as "error:" or "remote:", with nothing after it.
func barePrefix(l string) bool {
if !strings.HasSuffix(l, ":") {
return false
}
return !strings.ContainsAny(strings.TrimSuffix(l, ":"), " \t:")
} }
+17 -1
View File
@@ -48,13 +48,29 @@ func TestExecRunnerForZeroIsCutByItsContext(t *testing.T) {
} }
} }
// 1497/AC4: ExecRunnerClean(0) runs env with only the given words: PATH and HOME, not the process's secret. // 1497/AC4: ExecRunnerClean(0) runs env with only the given words: PATH and HOME, not the process's secret. And issue
// 734: run from a process whose own directory was deleted under it (a module whose build a new one replaced), a clean
// command starts in its HOME, and any other in the temporary directory, rather than failing to find where it is.
func TestExecRunnerCleanGivesTheCommandOnlyTheEnvArgument(t *testing.T) { func TestExecRunnerCleanGivesTheCommandOnlyTheEnvArgument(t *testing.T) {
t.Setenv("MESH_TEST_SECRET", "x") t.Setenv("MESH_TEST_SECRET", "x")
r := ExecRunnerClean(0)(context.Background(), []string{"PATH=" + os.Getenv("PATH"), "HOME=/h"}, "env") r := ExecRunnerClean(0)(context.Background(), []string{"PATH=" + os.Getenv("PATH"), "HOME=/h"}, "env")
if r.Status != 0 || !strings.Contains(r.Stdout, "HOME=/h") || !strings.Contains(r.Stdout, "PATH=") || strings.Contains(r.Stdout, "MESH_TEST_SECRET") { if r.Status != 0 || !strings.Contains(r.Stdout, "HOME=/h") || !strings.Contains(r.Stdout, "PATH=") || strings.Contains(r.Stdout, "MESH_TEST_SECRET") {
t.Errorf("ran = %+v", r) t.Errorf("ran = %+v", r)
} }
home, gone := t.TempDir(), t.TempDir()
t.Chdir(gone)
if err := os.Remove(gone); err != nil {
t.Fatal(err)
}
r = ExecRunnerClean(0)(context.Background(), []string{"PATH=" + os.Getenv("PATH"), "HOME=" + home}, "pwd")
if r.Status != 0 || strings.TrimSpace(r.Stdout) != home {
t.Errorf("a clean command run from a deleted directory = %+v, want it started in its HOME %s", r, home)
}
r = ExecRunnerFor(0)(context.Background(), nil, "pwd")
if r.Status != 0 || strings.TrimSpace(r.Stdout) != os.TempDir() {
t.Errorf("a command run from a deleted directory = %+v, want it started in %s", r, os.TempDir())
}
} }
// 1497/AC5: ExecRunnerFor(0) is unchanged: the env argument is added to the process's environment. // 1497/AC5: ExecRunnerFor(0) is unchanged: the env argument is added to the process's environment.
+28
View File
@@ -0,0 +1,28 @@
package forgekey
import (
"strings"
"testing"
)
// git's own words survive as the reason: a bare "error:" or "fatal:" line is never the whole reason (the mirror of
// jschoubben/car-hunter "is not used: git -C: error:", 2026-10-26), every line that says something is kept, and the
// words are bounded.
func TestSaidKeepsTheCauseBehindABarePrefix(t *testing.T) {
forge := "error: \nerror: User: 22:nox-builder with Key: 30:mesh:ace.mesh-build-go is not authorized to read jschoubben/car-hunter.\nerror: \nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"
for _, c := range []struct{ name, in, want string }{
{"forge refusal", forge, "error: User: 22:nox-builder with Key: 30:mesh:ace.mesh-build-go is not authorized to read jschoubben/car-hunter.; fatal: Could not read from remote repository.; Please make sure you have the correct access rights; and the repository exists."},
{"one line", "fatal: couldn't find remote ref x\n", "fatal: couldn't find remote ref x"},
{"bare prefixes only", "error:\nremote:\n", ""},
{"a colon inside stays", " ENOENT \nfatal: bad: thing\n", "ENOENT; fatal: bad: thing"},
{"empty", "", ""},
} {
if got := Said(c.in); got != c.want {
t.Errorf("%s: Said = %q, want %q", c.name, got, c.want)
}
}
long := Said(strings.Repeat("é", SaidBound))
if !strings.HasSuffix(long, " …") || len(long) > SaidBound+len(" …") || !strings.HasPrefix(long, "é") {
t.Errorf("long: %d bytes, ends %q", len(long), long[len(long)-6:])
}
}
+143
View File
@@ -8,8 +8,11 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"sync"
"testing" "testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
) )
@@ -73,6 +76,75 @@ func TestReadAccountRefusesAFileMissingAField(t *testing.T) {
} }
}) })
t.Run("1790: a bucket missing at start is bound when the bus says it was created, and what was written meanwhile is kept", func(t *testing.T) {
bus := &laterBus{}
hear := &heldSubscriber{}
var said []string
logf := func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }
b := AwaitBucket(context.Background(), bus, hear, "m_hearing", "the health tool cannot say it", logf)
if hear.subject != "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_m_hearing" {
t.Fatalf("heard %q", hear.subject)
}
ready := make(chan jetstream.KeyValue, 1)
b.Ready(func(kv jetstream.KeyValue) { ready <- kv })
for _, v := range []string{"one", "two"} {
if _, err := b.Put(context.Background(), "dead-letters", []byte(v)); !errors.Is(err, ErrNotYet) {
t.Fatalf("a write while missing: %v", err)
}
}
if b.KV() != nil || len(ready) > 0 {
t.Fatal("bound before the bucket exists")
}
bus.create()
hear.deliver()
select {
case <-ready:
case <-time.After(5 * time.Second):
t.Fatal("not bound within 5s of the bus saying the bucket was created")
}
if got := bus.kv.puts; strings.Join(got, ",") != "dead-letters=two" {
t.Errorf("written when it appeared: %v, want the latest value only", got)
}
if _, err := b.Put(context.Background(), "dead-letters", []byte("three")); err != nil {
t.Errorf("a write once bound: %v", err)
}
if len(said) != 2 || !strings.Contains(said[0], "cannot be read yet") || !strings.Contains(said[1], "appeared") {
t.Errorf("said %q", said)
}
// The bus refuses the subscription to the creation: said once, loudly; a reconnect hears it again and binds.
refusedBus, refusing := &laterBus{}, &heldSubscriber{refused: true}
var loud []string
r := AwaitBucket(context.Background(), refusedBus, refusing, "m_reasks", "x",
func(f string, a ...any) { loud = append(loud, fmt.Sprintf(f, a...)) })
bound := make(chan struct{})
r.Ready(func(jetstream.KeyValue) { close(bound) })
if len(loud) != 2 || !strings.Contains(loud[0], "cannot hear the creation of m_reasks") ||
!strings.Contains(loud[0], "permissions violation") {
t.Fatalf("a refused subscription was said as %q", loud)
}
refusing.mu.Lock()
refusing.refused = false
refusing.mu.Unlock()
refusedBus.create()
refusing.status <- nats.CONNECTED
select {
case <-bound:
case <-time.After(5 * time.Second):
t.Fatal("not bound after the connection connected again")
}
if refusing.subscribed != 2 {
t.Errorf("subscribed %d times, want again at the reconnect", refusing.subscribed)
}
// Without the advisory heard, the next use binds it.
quiet := &laterBus{}
c := AwaitBucket(context.Background(), quiet, nil, "m_hearing", "x", nil)
quiet.create()
if c.KV() == nil {
t.Error("a use after the bucket appeared did not bind it")
}
})
for _, tc := range []struct { for _, tc := range []struct {
field string field string
values map[string]any values map[string]any
@@ -114,3 +186,74 @@ func writeAccount(t *testing.T, fields map[string]any) string {
type refusingBus struct{ err error } type refusingBus struct{ err error }
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) { return nil, r.err } func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) { return nil, r.err }
// laterBus has no bucket until create, and then one.
type laterBus struct {
mu sync.Mutex
kv *keptKV
}
func (l *laterBus) create() { l.mu.Lock(); l.kv = &keptKV{}; l.mu.Unlock() }
func (l *laterBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
l.mu.Lock()
defer l.mu.Unlock()
if l.kv == nil {
return nil, jetstream.ErrBucketNotFound
}
return l.kv, nil
}
// keptKV records each put as key=value.
type keptKV struct {
jetstream.KeyValue
puts []string
}
func (k *keptKV) Put(_ context.Context, key string, value []byte) (uint64, error) {
k.puts = append(k.puts, key+"="+string(value))
return uint64(len(k.puts)), nil
}
// heldSubscriber keeps the last subscription made, for the test to deliver to; refused, the bus refuses it as a
// *nats.Conn says it (the connection's last error after a round trip). status stands for the connection's CONNECTED.
type heldSubscriber struct {
mu sync.Mutex
subject string
handle nats.MsgHandler
subscribed int
refused bool
last error
status chan nats.Status
}
func (h *heldSubscriber) Subscribe(subject string, handle nats.MsgHandler) (*nats.Subscription, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.subject, h.handle = subject, handle
h.subscribed++
if h.refused {
h.last = fmt.Errorf("%w: Permissions Violation for Subscription to %q", nats.ErrPermissionViolation, subject)
}
return nil, nil
}
func (h *heldSubscriber) FlushTimeout(time.Duration) error { return nil }
func (h *heldSubscriber) LastError() error { h.mu.Lock(); defer h.mu.Unlock(); return h.last }
func (h *heldSubscriber) StatusChanged(...nats.Status) chan nats.Status {
h.mu.Lock()
defer h.mu.Unlock()
h.status = make(chan nats.Status, 1)
return h.status
}
func (h *heldSubscriber) RemoveStatusListener(chan nats.Status) {}
func (h *heldSubscriber) deliver() {
h.mu.Lock()
handle, subject := h.handle, h.subject
h.mu.Unlock()
handle(&nats.Msg{Subject: subject})
}
+267
View File
@@ -2,8 +2,13 @@ package process
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
) )
@@ -53,3 +58,265 @@ func OptionalBucket(ctx context.Context, js BucketOpener, name, without string,
func NotWatched(logf func(string, ...any), name string, err error, without string) { func NotWatched(logf func(string, ...any), name string, err error, without string) {
logf("the bucket %s cannot be watched (%v): %s", name, err, without) logf("the bucket %s cannot be watched (%v): %s", name, err, without)
} }
// ErrNotYet is a write to an awaited bucket that is not on the bus yet: the value is kept, the latest per key, and
// written once the bucket appears.
var ErrNotYet = errors.New("the bucket is not on the bus yet: the value is kept and written when it appears")
// Subscriber is the part of a connection that hears what the bus says: a *nats.Conn, or a test's fake.
type Subscriber interface {
Subscribe(subject string, handle nats.MsgHandler) (*nats.Subscription, error)
}
// createdAdvisory is the subject the bus says a bucket's stream was created on: the server's own advisory, which a
// principal is granted for each bucket it may bind (mesh-sdk buscompose StreamCreatedAdvisory, stateCreatedHeard).
func createdAdvisory(bucket string) string { return "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_" + bucket }
// bindWithin bounds one bind of an awaited bucket: one STREAM.INFO.
const bindWithin = 10 * time.Second
// AwaitedBucket is a bucket a part keeps or reads that may not be on the bus at its start, bound once it is (novox/hq
// issue 1790, met 2026-10-26: since the grants part creates a manifest's buckets after a delivery, ADR 0547, a module
// restarted by that delivery opened its new state `hearing` before it existed, got "bucket not found", and never looked
// again). It binds at AwaitBucket when it can; otherwise when the bus says the bucket's stream was created (heard on
// nc), and at each use before then (one STREAM.INFO, the cheap check), never on a timer. It says once that it waits and
// once that the bucket appeared.
type AwaitedBucket struct {
ctx context.Context
js BucketOpener
name string
without string
logf func(string, ...any)
nc Subscriber
mu sync.Mutex
kv jetstream.KeyValue
refusedSaid bool
stopStatus func()
pending map[string][]byte
order []string
ready []func(jetstream.KeyValue)
sub *nats.Subscription
}
// AwaitBucket opens the bucket name now if the bus holds it, and otherwise waits for it as AwaitedBucket says; without
// is what is not done while it is missing. nc may be nil: then only a use binds it. ctx is the part's life: a bind
// after it ends is not tried.
func AwaitBucket(ctx context.Context, js BucketOpener, nc Subscriber, name, without string, logf func(string, ...any)) *AwaitedBucket {
if logf == nil {
logf = func(string, ...any) {}
}
b := &AwaitedBucket{ctx: ctx, js: js, name: name, without: without, logf: logf, pending: map[string][]byte{}}
// Heard before the first try, so a creation between the try and the subscription is not missed.
if nc != nil {
b.nc = nc
b.hear()
b.followReconnects()
}
ctxB, cancel := context.WithTimeout(ctx, bindWithin)
kv, err := js.KeyValue(ctxB, name)
cancel()
if err == nil {
b.bound(kv, false)
return b
}
logf("the bucket %s cannot be read yet (%v): %s until it appears; it is bound when the bus says it was created, or at "+
"its next use", name, err, without)
return b
}
// KV is the bound bucket, or nil while it is not on the bus; while nil, the call tries to bind it once.
func (b *AwaitedBucket) KV() jetstream.KeyValue {
b.mu.Lock()
kv := b.kv
b.mu.Unlock()
if kv != nil {
return kv
}
return b.bind()
}
// Ready runs f with the bucket: now when it is bound, otherwise once it appears (on the goroutine that bound it).
func (b *AwaitedBucket) Ready(f func(jetstream.KeyValue)) {
b.mu.Lock()
kv := b.kv
if kv == nil {
b.ready = append(b.ready, f)
}
b.mu.Unlock()
if kv != nil {
f(kv)
}
}
// Put writes key, as jetstream.KeyValue's Put. While the bucket is missing it keeps value as key's latest and answers
// ErrNotYet; the kept values are written, in the order their keys were first kept, when it appears.
func (b *AwaitedBucket) Put(ctx context.Context, key string, value []byte) (uint64, error) {
kv := b.KV()
if kv == nil {
b.mu.Lock()
if b.kv == nil {
if _, ok := b.pending[key]; !ok {
b.order = append(b.order, key)
}
b.pending[key] = append([]byte(nil), value...)
b.mu.Unlock()
return 0, fmt.Errorf("%s: %w", b.name, ErrNotYet)
}
kv = b.kv
b.mu.Unlock()
}
return kv.Put(ctx, key, value)
}
// bind tries one STREAM.INFO of the bucket and, when it is there, binds it; it answers the bucket or nil.
func (b *AwaitedBucket) bind() jetstream.KeyValue {
if b.ctx.Err() != nil {
return nil
}
ctx, cancel := context.WithTimeout(b.ctx, bindWithin)
defer cancel()
kv, err := b.js.KeyValue(ctx, b.name)
if err != nil {
return nil
}
return b.bound(kv, true)
}
// bound keeps kv as the bucket (unless another bind did first), writes what was kept and runs what waited on it.
func (b *AwaitedBucket) bound(kv jetstream.KeyValue, appeared bool) jetstream.KeyValue {
b.mu.Lock()
if b.kv != nil {
kv = b.kv
b.mu.Unlock()
return kv
}
// The kept values are written before the bucket is handed to anyone, so no later write is overtaken by an earlier.
written, failed := 0, 0
for _, key := range b.order {
ctx, cancel := context.WithTimeout(b.ctx, bindWithin)
_, err := kv.Put(ctx, key, b.pending[key])
cancel()
if err != nil {
failed++
b.logf("the bucket %s appeared but the value kept for %s could not be written: %v", b.name, key, err)
continue
}
written++
}
b.kv, b.pending, b.order = kv, nil, nil
ready, sub, stop := b.ready, b.sub, b.stopStatus
b.ready, b.sub, b.stopStatus = nil, nil, nil
b.mu.Unlock()
if sub != nil {
_ = sub.Unsubscribe()
}
if stop != nil {
stop()
}
if appeared {
b.logf("the bucket %s appeared and is bound: %d kept value(s) written, %d not", b.name, written, failed)
}
for _, f := range ready {
f(kv)
}
return kv
}
// refusalReader is the part of a *nats.Conn that says whether the bus refused a subscription: the server's refusal comes
// back after a round trip as the connection's last error, naming the subject.
type refusalReader interface {
FlushTimeout(time.Duration) error
LastError() error
}
// statusFollower is the part of a *nats.Conn that says when it connected again, without taking its reconnect handler.
type statusFollower interface {
StatusChanged(statuses ...nats.Status) chan nats.Status
RemoveStatusListener(ch chan nats.Status)
}
// hear subscribes to the bucket's creation (again), and says once, loudly, when the bus refuses it: then only a use or
// a reconnect binds the bucket.
func (b *AwaitedBucket) hear() {
subject := createdAdvisory(b.name)
b.mu.Lock()
old, bound := b.sub, b.kv != nil
b.sub = nil
b.mu.Unlock()
if old != nil {
_ = old.Unsubscribe()
}
if bound {
return
}
checks, _ := b.nc.(refusalReader)
var before error
if checks != nil {
before = checks.LastError()
}
sub, err := b.nc.Subscribe(subject, func(*nats.Msg) { go b.bind() })
if err == nil && checks != nil && checks.FlushTimeout(bindWithin) == nil {
if after := checks.LastError(); after != nil && after != before && errors.Is(after, nats.ErrPermissionViolation) &&
strings.Contains(after.Error(), subject) {
err = after
}
}
if err != nil {
if sub != nil {
_ = sub.Unsubscribe()
}
b.mu.Lock()
said := b.refusedSaid
b.refusedSaid = true
b.mu.Unlock()
if !said {
b.logf("cannot hear the creation of %s: %v; it binds at its next use, or when the connection reconnects", b.name, err)
}
return
}
b.mu.Lock()
if b.kv != nil {
b.mu.Unlock()
if sub != nil {
_ = sub.Unsubscribe()
}
return
}
b.sub, b.refusedSaid = sub, false
b.mu.Unlock()
}
// followReconnects hears the bucket's creation again and tries a bind each time the connection connects again (a grant
// reaches the bus with a new user list; a lost subscription is made again), until the bucket is bound or the part ends.
func (b *AwaitedBucket) followReconnects() {
w, ok := b.nc.(statusFollower)
if !ok {
return
}
ch := w.StatusChanged(nats.CONNECTED)
b.mu.Lock()
if b.kv != nil {
b.mu.Unlock()
w.RemoveStatusListener(ch)
return
}
b.stopStatus = func() { w.RemoveStatusListener(ch) }
b.mu.Unlock()
go func() {
for {
select {
case <-b.ctx.Done():
w.RemoveStatusListener(ch)
return
case _, open := <-ch:
if !open {
return
}
b.hear()
b.KV()
}
}
}()
}
+4 -2
View File
@@ -57,8 +57,10 @@ func grantedSendProcess(t *testing.T) (url string, account Account) {
Users: []*server.User{ Users: []*server.User{
{Username: adminUser, Password: adminPass}, {Username: adminUser, Password: adminPass},
{Username: sendUser, Password: sendPassword, Permissions: &server.Permissions{ {Username: sendUser, Password: sendPassword, Permissions: &server.Permissions{
Publish: &server.SubjectPermission{Allow: sendGrants}, Publish: &server.SubjectPermission{Allow: sendGrants},
Subscribe: &server.SubjectPermission{Allow: []string{"_INBOX." + sendUser + ".>"}}, // And the creation of its bucket, as mesh-sdk's composer grants it since issue 1790 (stateCreatedHeard).
Subscribe: &server.SubjectPermission{Allow: []string{"_INBOX." + sendUser + ".>",
"$JS.EVENT.ADVISORY.STREAM.CREATED.KV_mesh-mail_sends"}},
}}, }},
}, },
} }
+31
View File
@@ -18,12 +18,40 @@ import (
// key, reads it, and updates it with the revision it read, then each succeeds and returns the new // key, reads it, and updates it with the revision it read, then each succeeds and returns the new
// revision; an update with a stale revision fails as stdio.ErrChanged; a create of an existing key fails // revision; an update with a stale revision fails as stdio.ErrChanged; a create of an existing key fails
// as stdio.ErrChanged. // as stdio.ErrChanged.
//
// 1790: and before that, the bucket is not on the bus when the process opens it (a delivery restarted it before the
// grants part created its new state): it is bound when the bus says the bucket was created, as the process's own
// account hears it, and the value written while it was missing is there.
func TestStateCreateGetUpdateOnAGrantedBucket(t *testing.T) { func TestStateCreateGetUpdateOnAGrantedBucket(t *testing.T) {
url, account := grantedSendProcess(t) url, account := grantedSendProcess(t)
_, adminJetStream := adminJS(t, url) _, adminJetStream := adminJS(t, url)
nc, err := nats.Connect(url, nats.UserInfo(sendUser, sendPassword), nats.CustomInboxPrefix("_INBOX."+sendUser))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
awaited := AwaitBucket(ctxFor(t), js, nc, "mesh-mail_sends", "nothing is kept", t.Logf)
appeared := make(chan struct{})
awaited.Ready(func(jetstream.KeyValue) { close(appeared) })
if _, err := awaited.Put(ctxFor(t), "hearing", []byte(`{"heard":false}`)); !errors.Is(err, ErrNotYet) {
t.Fatalf("a write before the bucket exists: %v, want ErrNotYet", err)
}
if _, err := awaited.Put(ctxFor(t), "hearing", []byte(`{"heard":true}`)); !errors.Is(err, ErrNotYet) {
t.Fatalf("a second write before the bucket exists: %v, want ErrNotYet", err)
}
if _, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "mesh-mail_sends"}); err != nil { if _, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "mesh-mail_sends"}); err != nil {
t.Fatalf("the admin could not make the bucket: %v", err) t.Fatalf("the admin could not make the bucket: %v", err)
} }
select {
case <-appeared:
case <-time.After(5 * time.Second):
t.Fatal("the bucket was not bound within 5s of its creation, with no use since")
}
conn, err := Connect(account) conn, err := Connect(account)
if err != nil { if err != nil {
@@ -31,6 +59,9 @@ func TestStateCreateGetUpdateOnAGrantedBucket(t *testing.T) {
} }
defer conn.Close() defer conn.Close()
s := conn.State("sends") s := conn.State("sends")
if kept, err := s.Get("hearing"); err != nil || kept == nil || string(kept.Value) != `{"heard":true}` {
t.Fatalf("the value written while the bucket was missing: %+v, %v; want the latest", kept, err)
}
rev, err := s.Create("key", map[string]any{"v": 1}) rev, err := s.Create("key", map[string]any{"v": 1})
if err != nil { if err != nil {
+11 -6
View File
@@ -38,7 +38,7 @@ interface World {
async function world( async function world(
names: string[], names: string[],
opts: { retire?: boolean; held?: string[]; del?: boolean; reenable?: boolean; stableForMs?: number } = {}, opts: { retire?: boolean; held?: string[]; del?: boolean; reenable?: boolean; stableForMs?: number; stablePasses?: number } = {},
): Promise<World> { ): Promise<World> {
const dir = await mkdtemp(join(tmpdir(), "retire-")); const dir = await mkdtemp(join(tmpdir(), "retire-"));
const receives = join(dir, "postgres-database.json"); const receives = join(dir, "postgres-database.json");
@@ -64,6 +64,7 @@ async function world(
}; };
w.p = runProviderLoop("postgres-database", adapter, { w.p = runProviderLoop("postgres-database", adapter, {
receives, everyMs: EVERY, node: "anchor", stableForMs: opts.stableForMs ?? 0, receives, everyMs: EVERY, node: "anchor", stableForMs: opts.stableForMs ?? 0,
...(opts.stablePasses ? { stablePasses: opts.stablePasses } : {}),
// Only retirement is these tests'; the standing (provisioner.failing, .recovered) is standing.test.ts's. // Only retirement is these tests'; the standing (provisioner.failing, .recovered) is standing.test.ts's.
announce: (event, body) => { if (event === RETIREMENT_EVENT) w.events.push(body); }, announce: (event, body) => { if (event === RETIREMENT_EVENT) w.events.push(body); },
}); });
@@ -109,8 +110,10 @@ test("the fifth pass and not the fourth retires", async () => {
// Driven by the clock: five passes 10 ms apart take at least 40 ms after the first sees the set. // Driven by the clock: five passes 10 ms apart take at least 40 ms after the first sees the set.
const w = await world(["a", "b", "c"]); const w = await world(["a", "b", "c"]);
try { try {
await w.ask(["b", "c"]); // The clock starts before the file is written: a pass may read it the moment it lands, before any line after the
// write runs, and a start taken after would then count fewer than four intervals.
const start = Date.now(); const start = Date.now();
await w.ask(["b", "c"]);
await until(() => w.retired.length === 1); await until(() => w.retired.length === 1);
assert.ok(Date.now() - start >= 4 * EVERY - 2, `retired after ${Date.now() - start} ms, before five passes`); assert.ok(Date.now() - start >= 4 * EVERY - 2, `retired after ${Date.now() - start} ms, before five passes`);
} finally { } finally {
@@ -155,10 +158,10 @@ test("five passes are not enough: the same set must also hold for stableForMs",
const HOLD = 400; const HOLD = 400;
const w = await world(["a", "b", "c"], { stableForMs: HOLD }); const w = await world(["a", "b", "c"], { stableForMs: HOLD });
try { try {
const start = Date.now(); // before the write, as in the fifth-pass test
await w.ask(["b", "c"]); await w.ask(["b", "c"]);
const start = Date.now();
await sleep(EVERY * 10); // well past five passes, well short of the hold await sleep(EVERY * 10); // well past five passes, well short of the hold
assert.deepEqual(w.retired, [], "retired after five passes, before the set held long enough"); if (Date.now() - start < HOLD) assert.deepEqual(w.retired, [], "retired after five passes, before the set held long enough");
await until(() => w.retired.length === 1, 3000); await until(() => w.retired.length === 1, 3000);
assert.ok(Date.now() - start >= HOLD - 5, `retired after ${Date.now() - start} ms, before ${HOLD} ms`); assert.ok(Date.now() - start >= HOLD - 5, `retired after ${Date.now() - start} ms, before ${HOLD} ms`);
assert.equal((await w.p.retirement()).stable_for_seconds, 0); // 400 ms rounds to 0 s assert.equal((await w.p.retirement()).stable_for_seconds, 0); // 400 ms rounds to 0 s
@@ -300,11 +303,13 @@ test("delete refuses an active consumer and deletes only the named retired one",
}); });
test("an addition acts on the first pass", async () => { test("an addition acts on the first pass", async () => {
const w = await world(["a"]); // Not gated by the stability a retirement waits for: with a hold of a minute and a thousand passes, only a pass that
// acts at once creates b. The deadline is a few passes, not three intervals of 10 ms: a scheduler stall must not fail it.
const w = await world(["a"], { stableForMs: 60_000, stablePasses: 1000 });
try { try {
const before = w.created.length; const before = w.created.length;
await w.ask(["a", "b"]); await w.ask(["a", "b"]);
await until(() => w.created.length > before, EVERY * 3); await until(() => w.created.length > before, 1000);
assert.equal(w.created.at(-1), "b"); assert.equal(w.created.at(-1), "b");
} finally { } finally {
w.done(); w.done();
+37 -21
View File
@@ -27,14 +27,18 @@ async function until(cond: () => boolean, ms = 4000): Promise<void> {
} }
} }
interface Said { event: string; body: Record<string, unknown> } interface Said { event: string; body: Record<string, unknown>; at: number }
interface World { interface World {
dir: string; dir: string;
ask(list: { as: string; node?: string; values?: Record<string, unknown> }[]): Promise<void>; ask(list: { as: string; node?: string; values?: Record<string, unknown> }[]): Promise<void>;
failing: Error | null; failing: Error | null;
holdsErr: Error | null; holdsErr: Error | null;
/** When set, holds fails only for these consumers (holdsErr's error). */
holdsErrFor: Set<string> | null;
created: string[]; created: string[];
/** How many times create was asked, failed or not. */
attempts: number;
/** Every event but the first success since start, which has its own test. */ /** Every event but the first success since start, which has its own test. */
said: Said[]; said: Said[];
all: Said[]; all: Said[];
@@ -44,7 +48,7 @@ interface World {
async function world(opts: { classify?: boolean; announce?: false; stableForMs?: number } = {}): Promise<World> { async function world(opts: { classify?: boolean; announce?: false; stableForMs?: number } = {}): Promise<World> {
const dir = await mkdtemp(join(tmpdir(), "standing-")); const dir = await mkdtemp(join(tmpdir(), "standing-"));
const receives = join(dir, "postgres-database.json"); const receives = join(dir, "postgres-database.json");
const w = { dir, failing: null, holdsErr: null, created: [], said: [], all: [] } as unknown as World; const w = { dir, failing: null, holdsErr: null, holdsErrFor: null, created: [], attempts: 0, said: [], all: [] } as unknown as World;
w.ask = async (list) => { w.ask = async (list) => {
const given = []; const given = [];
for (const g of list) { for (const g of list) {
@@ -55,13 +59,14 @@ async function world(opts: { classify?: boolean; announce?: false; stableForMs?:
}; };
const adapter: Adapter = { const adapter: Adapter = {
async create(p) { async create(p) {
w.attempts++;
if (w.failing) throw w.failing; if (w.failing) throw w.failing;
w.created.push(p.as); w.created.push(p.as);
}, },
async remove() {}, async remove() {},
async retire() {}, async retire() {},
async holds() { async holds(p) {
if (w.holdsErr) throw w.holdsErr; if (w.holdsErr && (!w.holdsErrFor || w.holdsErrFor.has(p.as))) throw w.holdsErr;
return true; return true;
}, },
...(opts.classify ? { classify: () => "its-own" } : {}), ...(opts.classify ? { classify: () => "its-own" } : {}),
@@ -73,8 +78,9 @@ async function world(opts: { classify?: boolean; announce?: false; stableForMs?:
failingAfterMs: FAILING_AFTER, sayAgainEveryMs: SAY_AGAIN, stableForMs: opts.stableForMs ?? 0, failingAfterMs: FAILING_AFTER, sayAgainEveryMs: SAY_AGAIN, stableForMs: opts.stableForMs ?? 0,
...(opts.announce === false ? {} : { ...(opts.announce === false ? {} : {
announce: (event: string, body: Record<string, unknown>) => { announce: (event: string, body: Record<string, unknown>) => {
w.all.push({ event, body }); const said = { event, body, at: Date.now() };
if (body.why !== "first-success") w.said.push({ event, body }); w.all.push(said);
if (body.why !== "first-success") w.said.push(said);
}, },
}), }),
}); });
@@ -92,10 +98,13 @@ test("a consumer failed for minutes is announced, naming it, its machine and the
const w = await world(); const w = await world();
try { try {
w.failing = new Error(`token request failed: 401 {"error":"invalid_grant"} for pw-a`); w.failing = new Error(`token request failed: 401 {"error":"invalid_grant"} for pw-a`);
// The loop stamps its own clock before it calls announce, so the test's stamp may read a few ms short: 10 ms of slack.
// "Not before failingAfterMs" and "not again too soon" are read from the clock at the moment each is said, not
// from a sleep: a sleep overruns under load and lets the very announcement it checks the absence of fall due.
const asked = Date.now();
await w.ask([{ as: "a", node: "anchor" }]); await w.ask([{ as: "a", node: "anchor" }]);
await sleep(FAILING_AFTER / 2);
assert.equal(events(w.said), "", "announced before failingAfterMs");
await until(() => w.said.length === 1); await until(() => w.said.length === 1);
assert.ok(w.said[0].at - asked >= FAILING_AFTER - 10, `announced after ${w.said[0].at - asked} ms, before failingAfterMs`);
const b = w.said[0].body; const b = w.said[0].body;
assert.equal(w.said[0].event, FAILING_EVENT); assert.equal(w.said[0].event, FAILING_EVENT);
assert.equal(b.consumer, "a"); assert.equal(b.consumer, "a");
@@ -103,15 +112,14 @@ test("a consumer failed for minutes is announced, naming it, its machine and the
assert.equal(b.class, CLASS_CREDENTIALS); assert.equal(b.class, CLASS_CREDENTIALS);
assert.equal(b.provider, "postgres-database"); assert.equal(b.provider, "postgres-database");
assert.equal(b["provider-node"], "home-server"); assert.equal(b["provider-node"], "home-server");
assert.ok((b.attempts as number) > 3, `attempts ${b.attempts}`); assert.ok((b.attempts as number) > 1, `attempts ${b.attempts}`);
assert.match(String(b.since), /^\d{4}-\d{2}-\d{2}T/); assert.match(String(b.since), /^\d{4}-\d{2}-\d{2}T/);
assert.doesNotMatch(String(b.error), /pw-a/, "the consumer's password travelled"); assert.doesNotMatch(String(b.error), /pw-a/, "the consumer's password travelled");
// Said again while it lasts, not every pass. // Said again while it lasts, not every pass.
await sleep(SAY_AGAIN / 2);
assert.equal(events(w.said), FAILING_EVENT, "repeated too soon");
await until(() => w.said.length === 2); await until(() => w.said.length === 2);
assert.equal(events(w.said), `${FAILING_EVENT},${FAILING_EVENT}`); assert.equal(events(w.said), `${FAILING_EVENT},${FAILING_EVENT}`);
assert.ok(w.said[1].at - w.said[0].at >= SAY_AGAIN - 10, `repeated after ${w.said[1].at - w.said[0].at} ms, before sayAgainEveryMs`);
// The first success takes it back. // The first success takes it back.
w.failing = null; w.failing = null;
@@ -127,15 +135,21 @@ test("a consumer failed for minutes is announced, naming it, its machine and the
test("one success between failures starts the run again", async () => { test("one success between failures starts the run again", async () => {
const w = await world(); const w = await world();
try { try {
w.failing = new Error("connection refused"); // Each failing run lasts two counted failed attempts and at least 0.6 of failingAfterMs of clock (long enough that a success which did not reset the run would be announced at the second run). A run the load
await w.ask([{ as: "a" }]); // stretched to failingAfterMs is a long run and may be announced, so only runs that stayed short are asserted.
await sleep(FAILING_AFTER * 0.6); const run = async (values?: Record<string, unknown>): Promise<number> => {
w.failing = null; const start = Date.now();
const base = w.attempts;
w.failing = new Error("connection refused");
await w.ask([{ as: "a", values }]);
await until(() => w.attempts >= base + 2 && Date.now() - start >= FAILING_AFTER * 0.6);
w.failing = null;
return Date.now() - start;
};
const first = await run();
await until(() => w.created.length === 1); await until(() => w.created.length === 1);
w.failing = new Error("connection refused"); const second = await run({ name: "changed" });
await w.ask([{ as: "a", values: { name: "changed" } }]); if (first < FAILING_AFTER && second < FAILING_AFTER) assert.equal(events(w.said), "", "two short runs are not one long one");
await sleep(FAILING_AFTER * 0.6);
assert.equal(events(w.said), "", "two short runs are not one long one");
} finally { } finally {
w.stop(); w.stop();
} }
@@ -197,9 +211,11 @@ test("a failing consumer the mesh stopped asking for is said recovered when it i
await until(() => w.created.length === 3); await until(() => w.created.length === 3);
w.holdsErr = new Error("boom"); w.holdsErr = new Error("boom");
await until(() => w.said.filter((s) => s.event === FAILING_EVENT).length >= 3); await until(() => w.said.filter((s) => s.event === FAILING_EVENT).length >= 3);
// c is no longer asked for while it is still failing; the others then recover. // c is no longer asked for while it is still failing; the others then recover. c keeps failing, so its only way
// to be said recovered is its retirement: clearing every failure at once raced the loop's read of the new list
// under load (a pass before that read recovered c as an ordinary recovery, with no why).
await w.ask([{ as: "a" }, { as: "b" }]); await w.ask([{ as: "a" }, { as: "b" }]);
w.holdsErr = null; w.holdsErrFor = new Set(["c"]);
await until(() => w.said.some((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c")); await until(() => w.said.some((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c"));
const c = w.said.find((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c")!; const c = w.said.find((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c")!;
assert.equal(c.body.why, "retired"); assert.equal(c.body.why, "retired");