Compare commits

...
Author SHA1 Message Date
jschoubben c104e7500d Merge pull request 'The grants part writes its list whole however large, one refused principal no longer freezes it, and stale watchman keys close (issue 774)' (#117) from fix/sweep-2026-10-30 into main
Reviewed-on: #117
2026-10-30 18:13:00 +00:00
jschoubben e0b56c83f9 Pin mesh-sdk a3718126, whose bus retry never asks again a call its callee answered (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c104e7500d21
mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged
mesh/check checking e0b56c83f94c on 33973d533825 | novox/mesh-delivery#117 in group fix/sweep-2026-10-30
2026-10-30 18:43:13 +01:00
jschoubben d4ac50c4b1 Pin mesh-sdk 206f794e, the fix group's final library head (issue 774) 2026-10-30 18:37:07 +01:00
jschoubben f756731700 Pin mesh-sdk ef1ec1c9, the fix group's reviewed library head (issue 774) 2026-10-30 18:27:08 +01:00
jschoubben 255f8b2aca Close a stuck key whose watched key aged out or was deleted unheard, so its condition clears (issue 774)
A key that ages out of its bucket (the build seats' asks pass the stream's 168 h) leaves no marker a watch hears,
and one deleted while the watchman was down is not heard either: its stuck key stood unmoved for ever, and 26
stuck-key conditions of asks from 2026-10-21 were open on 10-30.

Quick fix: mark stuck keys older than the stream's maximum age moved.
Thorough fix, chosen because it covers every bucket the watchman mirrors and any way a key goes: at the start's
end, a stuck key not moved whose bucket was read whole at this start (each build seat's steps, the assigner's
wants) and whose key was not read standing is written moved, which clears its finding, and the closing is said
in the log. A bucket that could not be read closes nothing. A key that ages out while the watchman runs is
closed at its next start: a KV age-out emits no event, and no timer is added.
2026-10-30 18:25:56 +01:00
jschoubben ff58b16cf9 Pin mesh-sdk 61b67d61, so mesh-grants writes through the size guard and every delivery module rides out a user-list reload (issue 774) 2026-10-30 18:23:08 +01:00
jschoubben 788969bcf8 Write a kept user through mesh-sdk's AccountsEntry, so its lines are ComposeAccounts' own (issue 774)
The grants part repeated the composer's entry lines to keep a refused principal's grants of the list in force;
mesh-sdk 61b67d6 exports them as buscompose.AccountsEntry, so the copy goes. The byte-for-byte test of the kept
entry still passes. Needs the mesh-sdk pin at 61b67d6 or later.
2026-10-30 18:20:28 +01:00
jschoubben ce054dc8dd Judge nothing and owe no status for a candidate outside the mesh (issue 774)
The forge carries the operator's own repositories (jschoubben/hal, jschoubben/novadock) beside the mesh's; the
indexer writes their pull requests as candidates, the gate never names their trunk, so the checker said each
"cannot be judged" and owed the status error on mesh/check, which the forge module wrote into repositories the
mesh's forge account may not write: 403, 119 times each (sweep 2026-10-30 item 7).

Quick fix: name the two repositories as not the mesh's in the checker.
Thorough fix, chosen because any repository on the forge can meet it: a candidate every pull request of which is
of a repository the gate's source.main names no trunk of and touches no module is outside the mesh. The checker
writes it no verdict and owes it no status, takes away a status owed before, says it once in the log, and lists it
in checker_status as outside_the_mesh. A new mesh repository's first pull request adds a module, so it is still
said untrunked and owed error (issue 1422). Nothing is outside before the gate's source is read.
2026-10-30 18:12:33 +01:00
jschoubben 94f555277c Never name the private network the mesh provides as a module missing its grant (issue 774)
mesh-wireguard is provided by the mesh itself (mesh-sdk delivery.ProvidedByMesh): no repository builds it, so the
indexer holds no fingerprint of it and the assigner never wants it by one; the controller composes it with the
node's own. The grants part named it "no fingerprint of the new form: the indexer holds no value of it" on every
pass, a warning nothing could clear (sweep 2026-10-30 item 6).

Quick fix: drop mesh-wireguard by name from the missing list.
Thorough fix, chosen as cheap and covering every module the mesh may provide later: MissingGrants passes over any
module ProvidedByMesh names. The controller says the same of it in mesh-control (its grants and switch lines).
2026-10-30 18:10:55 +01:00
jschoubben 031cfb8277 Compose the user list principal by principal, so one refused principal never freezes it (issue 774)
Met twice on 2026-10-30 (the engine's acts, then mesh-host_health): the composer refused one principal as a
second writer, and that refusal ended the whole composition, so the list in force stood for every principal
while changes piled up; the next list that composed moved 52 users at once (16:37:28).

Quick fix: leave the refused principal out and write the rest (locks that principal out of the bus).
Thorough fix, chosen because the kind was met twice and it hides every other change: ComposeEach composes each
principal on its own; a refused one keeps its grants of the list in force (the part's sealed grants), written as
the composer writes an entry; one the list in force does not carry stays out; a list that would drop one the
list in force carries, whose grants were not kept, is held, said. The refusal is raised as the warning
mesh-grants.principal-refused, by name and why, cleared once it composes, and listed in the pass
(principals_refused). The entry renderer mirrors buscompose.ComposeAccounts's lines and is pinned to them by the
test (the entry the bus loaded is written back byte for byte); it belongs in mesh-sdk as an exported entry writer.
2026-10-30 18:10:26 +01:00
jschoubben c1b28cc4fe Keep each sealed list's permissions one key per user, so no value outgrows the bucket (issue 774)
The sealed-grants value held every user of the last two lists in one value; at 61 users it was refused with
"message size exceeds maximum" (JetStream 10054, 16:37 and 16:55 on 2026-10-30), so the permissions of the
list in force were never kept and a grant judged against them waited.

Quick fix: keep only the newest list in the one value (halves it; it still grows with the mesh).
Thorough fix, chosen because the value grows with every placed module and the kind was met twice: one key per
user and list, sealed-grants.<digest hex>.<user>, and an index sealed-lists naming the kept lists and their users,
written only once every user's key is there; the third list's keys are taken away, and the old one-value key read
as a fallback and removed. mesh-sdk's kvlimit guard (7b02f63) refuses an oversized value before the put through
process.State.Put, which the part's own store (cmd/mesh-grants ownStore.Put) writes through once pinned.
2026-10-30 18:07:26 +01:00
jschoubben 33973d5338 Merge pull request 'mesh-grants grants each provisioner client every machine's plans (issue 774)' (#116) from fix/provisioner-reads-every-plan into main
Reviewed-on: #116
2026-10-30 15:53:41 +00:00
jschoubben f800d672f3 mesh-grants grants each provisioner client every machine's plans through mesh-sdk af66ccb, so g14 and shanks read their plans again (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@33973d533825
mesh/delivery-group group fix/provisioner-reads-every-plan merged: every member is merged
mesh/check being carried to f800d672f344 on 98594c990287 | from mesh-sdk 42988a734b3d
2026-10-30 16:41:02 +01:00
jschoubben 98594c9902 Merge pull request 'mesh-grants composes the user list without granting the engine's state to a tool runner (issue 774)' (#115) from fix/runner-carries-no-engine-state into main
Reviewed-on: #115
2026-10-30 15:35:58 +00:00
jschoubben 57193ab2e4 Pin mesh-grants to the mesh-sdk that composes no user from the node-engine's manifest, so the whole user list composes again (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@98594c990287
mesh/delivery-group group fix/runner-carries-no-engine-state merged: every member is merged
mesh/check checking 57193ab2e474 on f7cd4698db7e | novox/mesh-delivery#115 in group fix/runner-carries-no-engine-state
Second meeting of the engine-state-in-runner kind; thorough: the runner
never carries the engine module. The live placement names mesh-host,
and with its health state the runner was a second writer, so every
grant since 13:54 was held back. The seeded-placement test now carries
the engine's manifest and refuses any runner write of mesh-host_*.
2026-10-30 16:32:43 +01:00
jschoubben f7cd4698db Merge pull request 'mesh-grants writes its memberships bare through the mesh-sdk codec (issue 774)' (#114) from fix/memberships-shape into main
Reviewed-on: #114
2026-10-30 14:42:13 +00:00
jschoubben 180a3fe5db Pin mesh-sdk at its fix/memberships-shape head after review round 1 (issue 774)
mesh/check checking 180a3fe5db30 on ab210fc5ab40 | novox/mesh-delivery#114 in group fix/memberships-shape
mesh/delivery merged: source.main moved to novox/mesh-delivery@f7cd4698db7e
mesh/delivery-group group fix/memberships-shape merged: every member is merged
The decoder now refuses an old wrapper of a held-back membership; grants_membership says so for such a key.
2026-10-30 15:36:48 +01:00
jschoubben 4bb1970fef Write the grants part's memberships bare, through mesh-sdk's codec, so the tool runner serves again
The state memberships held the part's wrapper ({node, module, digest, membership, published, retired}) while every
reader takes the bare membership the assignments stream carries: novox's tool runner decoded an empty one and served
no tool. The part now writes membership.EncodeGrantsMembership of a membership published or unchanged, never one held
back. The pass record moves to the pass the part already keeps (user-list at pass): published or why not was there, the
retired seat verbs are added as Result.Retired; grants_membership reads both, so no new state is declared.

Issue 774 (the bus login list, L3). Quick fix: marshal m.Membership here. Thorough, chosen: one codec in mesh-sdk the
writer and the tool runner share, because a hand-built fixture agreed with the mistake. Pins mesh-sdk at the head of
its fix/memberships-shape.
2026-10-30 15:27:06 +01:00
mesh-tools ab210fc5ab Merge pull request 'Design 47's last milestones: watchman, grants gate, assigner wants and placement, builder log, view (issues 774, 1952, 1953, 1954, 1955)' (#113) from feat/774-l3-reland into main 2026-10-30 11:02:07 +00:00
jschoubben 8edc83950c Pin mesh-sdk 2da0de7, the release's sdk with main merged in (issue 774)
mesh/check checking 8edc83950cea on 5588875c3f21 | novox/mesh-delivery#113 in group feat/774-l3-reland
mesh/delivery merged: source.main moved to novox/mesh-delivery@ab210fc5ab40
mesh/delivery-group group feat/774-l3-reland merged: every member is merged
2026-10-30 01:12:57 +01:00
jschoubben 8e470f3af4 Merge origin/main (delivery 1, #112) into the release, keeping the release's mesh-sdk pins (issue 774) 2026-10-30 01:12:57 +01:00
jschoubben 7536d9d14e Pin mesh-sdk 6e4a301, the release's final head, which keeps the old provisioner client's needs grant for one release (issue 774) 2026-10-30 00:18:04 +01:00
jschoubben ce7032c49b Pin mesh-sdk 007b6fd, the release's final head (issue 774) 2026-10-30 00:05:25 +01:00
jschoubben 08ecf10ef3 Grants: write withheld hearings in mesh-sdk's layout, so the self-check reads them by the sdk's names (issue 1955) 2026-10-29 23:56:21 +01:00
jschoubben ba502479e8 Builder: ask and hear the seat mesh-artifact-sweep, the name mesh-sdk 7bd0f6c gives the store's sweep, so its held and artifacts-gone reach their holder (issue 1955, ADR 0571 §2) 2026-10-29 23:56:21 +01:00
jschoubben e564eaf75f Grants: write the hearings it withholds to its state withheld on change, so the self-check's probe D-heard reads them from the bus and not the controller's store (issue 1955) 2026-10-29 23:51:48 +01:00
jschoubben a54d1acf45 View: answer builder_history from the builder's log, each build's end with its duration from its start (issue 1955, ADR 0571 §1) 2026-10-29 23:48:14 +01:00
jschoubben e82ea89c23 Assigner: read the running commit of a reader of the manifests from the builder's log, and hold the buckets it starts without in one list so the start test skips them (issue 1955, ADR 0571 §1) 2026-10-29 23:48:14 +01:00
jschoubben 06b58bc83b Indexer: take directories and outside-built registrations from the builder's log, watched from its start, so no build record of the controller's is read (issue 1955, ADR 0571 §1) 2026-10-29 23:48:14 +01:00
jschoubben 17c0734fb2 Builder: keep each build's start and end in its own module log build-log, and answer builder_builds_since from it, so readers and mesh-catalog read the builder's history without a bare stream (issue 1955, ADR 0571 §1, §9) 2026-10-29 23:48:14 +01:00
jschoubben c045057ca0 Assigner: write who holds which seat from placement into seat-holders, only the keys a change moves, and answer it from placement's seats, so readers no longer ask the controller's seats (issue 1955, ADR 0571 §4)
Pins mesh-sdk f9a496d.
2026-10-29 23:14:28 +01:00
jschoubben b44f01f89c Builder: write each build's start and end to its own log on the bus, and ask the seat artifact-store's held and hear its artifacts-gone, the controller's only until the lease write, so build history and the store's sweep leave the controller (issue 1955, ADR 0571 §1, §2)
Pins mesh-sdk f9a496d.
2026-10-29 23:13:32 +01:00
jschoubben 229a3e9855 Assigner: from the lease write rule 7 reads a placed node's own composition digest from its have, an unplaced node gets no want and is said, and the controller's compositions are read only until then (issue 1955, ADR 0571 §6) 2026-10-29 23:10:31 +01:00
jschoubben e5192f55ee View: read no build record and no composition of the controller's: fingerprints against the builder's done keys alone, a move's judgement from the node agent's have (issue 1955, ADR 0571 §1, §6) 2026-10-29 23:08:46 +01:00
jschoubben ee2f1efe78 Watchman: hold a failing network part at the machines it points at, as the controller does, so a machine others cannot reach is said once there and not as each reporter's own (issue 774)
Pins mesh-sdk 6eae076.
2026-10-29 23:06:19 +01:00
jschoubben 8a91871de6 Watchman: raise machine-network, network-rewritten, network-unreachable, machine-units, relogin-needed and directory-used-as-found from each node's own health report, under the controller's keys, so these alerts outlive the controller's lease (issue 774)
Pins mesh-sdk 94d0de5.
2026-10-29 23:01:03 +01:00
jschoubben 64a5074d58 Assigner: lay each held value of the group's own pull requests at their heads over its module's values in the composition check, judged against the head's declaration and named, so a check reads the held values itself (issue 1953 D7, ADR 0565 §5) 2026-10-29 22:43:27 +01:00
jschoubben 7b7796b6fa Assigner: hold a want while a provision its module requires has no key yet, and say it once it stays without one past the delivery bound, so a module never starts without its database unseen (issue 1954 D1)
Pins mesh-sdk 66ff33d.
2026-10-29 22:41:44 +01:00
jschoubben f56eb62d8b Assigner: hold a want until every provision its module consumes on that node is provided, and say a failed one as a finding, so a module never moves ahead of what it needs (issue 1954 D1, ADR 0570 §4)
Pins mesh-sdk 6e39ed1.
2026-10-29 22:39:38 +01:00
jschoubben 78f03b7422 Watchman: say needs-operator from mesh-provisioner's provisions in waits, so the row the haves could not carry has its facts on the bus (issue 1954 D2)
Pins mesh-sdk 4a96793.
2026-10-29 22:37:39 +01:00
jschoubben 3aec7f9b02 Assigner: hold the seat mesh-wants, keep a person's upgrade policy and judge moves by it, say a policy split with the controller as a finding, and answer the bus step, so the policies and the bus step leave the controller (issue 1953 D4-D6) 2026-10-29 22:34:08 +01:00
jschoubben 2488d839ab Bridge, indexer and view: carry the catalogue's predefined list on its merged message, and keep it and each open pull request's config declarations in the indexer's states, so mesh-config reads them in place of the controller's mirror and pull-declares (issue 1953 D1-D3)
The view's tools read both states. Pins mesh-sdk 2d2aa7b.
2026-10-29 22:29:04 +01:00
jschoubben 2531cd42b6 Checker: hold that the change's own failing test ends its level there, as mesh-sdk's build asks now do, so a failing check costs only its own tests (issue 1289)
Pins mesh-sdk 2d2aa7b.
2026-10-29 22:24:50 +01:00
jschoubben 5235bb2e79 Assigner and grants part: record hand acts and warrants through the seat mesh-hand-acts, the controller's log only while that seat has no holder and the controller holds its lease (issue 1953 D8)
Pins mesh-sdk 2d2aa7b in the L5 modules.
2026-10-29 22:24:00 +01:00
jschoubben 3b6f3af38e Grants part: create a module's log with the max age it declares, so a log's entries age out as its manifest says (issue 1952)
Its nats fixture gives the tools bundle's account a secret the tool runtime can reach, as mesh-sdk 2e5cc4f asks. Pins mesh-sdk 2e5cc4f.
2026-10-29 22:19:35 +01:00
jschoubben 15a15cbda5 Watchman: observe S1, S9 and S21 as the controller does and stop the needs-operator row, so the keeper's compare can empty with no condition lost (issue 1952 WM2, WM3, WM5)
S1 is urgent after half an hour for a control node (the node running mesh-controller). S9 also hears the bus module's
slow consumers and closed connections, each closure counted once. S21 is one finding per heard machine from its failed
haves and its provisioner clients' reports (PROV_APPLIED). DEAD_LETTERS is counted again at a letter's age and every
five minutes while letters are held, on the watchman's own pulse. needs-operator waits for L6: its inputs are not on the
bus. Pins mesh-sdk b6ef01d.
2026-10-29 21:59:46 +01:00
jschoubben 7dbf7a19a1 Grants part: compose a principal for every own bus account, a tools bundle's with what it may remove, so the bus tools remove the controller's dead-letter consumer and the engine's acts on their own account (issue 1952 S5)
Pins mesh-sdk 7000bd5.
2026-10-29 21:56:13 +01:00
jschoubben e1fd3eb0f6 View: answer status from the buckets it reads and derive motion with mesh-sdk's MotionOf, so status leaves the controller and the keeper judges a delivery as the view does (issue 1952 V1) 2026-10-29 21:52:42 +01:00
jschoubben 73280f85c3 Assigner: write each condition it says as a finding too, and say it straight only until the take-over, so its holds reach the operator through the condition keeper and never twice (issue 1952 WM6) 2026-10-29 21:49:25 +01:00
jschoubben b8e8981730 Grants part: hear root-not-free from the seat the take-over key names, never both, so the switch to the condition keeper moves it with the router (issue 1952 G1)
Watchman: subscribe exactly what its grant lets it hear (buscompose.WatchmanHears). Both pin mesh-sdk 69fcb6c.
2026-10-29 21:46:29 +01:00
jschoubben da787ae360 Watchman: hold mesh-watchdogs and write its watchdogs as findings, so health can leave the controller (issue 1952 WM1-WM5)
The signals rows S1, S8, S9, S10, S11, S21 and the haves' unhealthy and needs-operator are pulses whose expiry is a finding;
stuck keys and presence are its own findings; it keeps the dead letters from the lease write on its own consumer, lists and
shows them, and refuses deliver and drop until the hand-act log's home (issue 1953). Tools read every state it keeps and
start resync. Pins mesh-sdk 2a04bf0.
2026-10-29 21:44:18 +01:00
jschoubben 013b7420f6 mesh-grants: hold a grant only while the loaded list lacks its permissions, read from the part's own record of the lists it sealed, so a release whose bus tools are down still moves (issue 774) 2026-10-29 21:18:21 +01:00
jschoubben c7fbc530da mesh-grants: compose the user list from the running and the trunk manifest, and write a grant only once the loaded list carries it, so L3's parts never use a subject the bus has not loaded (issue 774) 2026-10-29 21:15:55 +01:00
jschoubben 98ad1aa252 Pin mesh-sdk 22c9ddd in the modules L3 changes, the L3 re-land group's sdk head (issue 774) 2026-10-29 21:07:00 +01:00
jschoubben b01cf63237 Revert "Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#109) from revert/774-l3 into main"
This reverts commit c1b4533201, reversing
changes made to d8274204fb.
2026-10-29 21:05:14 +01:00
145 changed files with 10698 additions and 2198 deletions
+7
View File
@@ -0,0 +1,7 @@
# The modules of this repository that still read the controller's assignments (mesh-controller.assignments), each with
# why. mesh-grants' test (TestGrants, "assignments readers") refuses any other module.json that reads them, and an entry
# here whose module no longer does: the list only shrinks, and it is empty in the first change after the controller's
# lease is written to a holder that is no controller (novox/hq ADR 0549 §3, §6; issue 774). Each module named here
# uses the assignments only while the controller holds the lease (buscompose.LeaseHandedAway), placement alone after it.
mesh-delivery-assigner unioned with the placement until the lease write (assign.Placed with handedAway); placement alone after
mesh-grants unioned with the placement until the lease write (ReadInputs); placement alone after
@@ -119,11 +119,11 @@ func (p *passer) composeAtHead(ctx context.Context, m assign.Mesh, ch assign.Cha
if h := ch.Members[repository]; h != "" { if h := ch.Members[repository]; h != "" {
commits, member = []string{h}, true commits, member = []string{h}, true
} else { } else {
commits = p.w.runningCommits(ctx, kvGetter(p.kvs[delivery.BuildsBucket]), module) commits = p.w.runningCommits(module)
} }
if len(commits) == 0 { if len(commits) == 0 {
notJudged = append(notJudged, "not judged whether "+module+" reads the group's manifests: no build of it is assigned with a "+ notJudged = append(notJudged, "not judged whether "+module+" reads the group's manifests: the builder's log holds no done "+
"commit on record, and the group holds no member of "+repository) "trunk build of it, and the group holds no member of "+repository)
continue continue
} }
for _, commit := range commits { for _, commit := range commits {
@@ -106,7 +106,7 @@ func TestAGroupWithAnSDKMemberIsComposedByTheComposerAtItsHead(t *testing.T) {
b.putJSON(t, provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"build-1"}`)) b.putJSON(t, provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"build-1"}`))
running := strings.Repeat("3", 40) running := strings.Repeat("3", 40)
b.putJSON(t, provider.AssignmentsBucket, "novox.mesh-controller", []byte(`{"node":"novox","module":"mesh-controller","build":"c-1"}`)) b.putJSON(t, provider.AssignmentsBucket, "novox.mesh-controller", []byte(`{"node":"novox","module":"mesh-controller","build":"c-1"}`))
b.putJSON(t, delivery.BuildsBucket, delivery.BuildKey("mesh-controller", "c-1"), []byte(`{"id":"c-1","repository":"novox/mesh-controller","commit":"`+running+`"}`)) b.built(t, "mesh-controller", "novox/mesh-controller", running)
sdkHead := strings.Repeat("4", 40) sdkHead := strings.Repeat("4", 40)
heads := []delivery.Head{{Repository: "novox/mesh-sdk", Number: 80, Head: sdkHead}, {Repository: "novox/web", Number: 7, Head: strings.Repeat("2", 40)}} heads := []delivery.Head{{Repository: "novox/mesh-sdk", Number: 80, Head: sdkHead}, {Repository: "novox/web", Number: 7, Head: strings.Repeat("2", 40)}}
hash, _ := delivery.HeadsHash(heads) hash, _ := delivery.HeadsHash(heads)
@@ -21,6 +21,7 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce" "git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks" "git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose/judge" "git.novox.be/novox/mesh-sdk/go/compose/judge"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -82,6 +83,11 @@ func startBed(t *testing.T, leaveOut string) *bed {
} }
b.buckets[name] = kv b.buckets[name] = kv
} }
// The builder's module log, as the runtime raises it (ADR 0297): the running commits are read from it (ADR 0571 clause 1).
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(delivery.BuilderModule, delivery.BuilderLog),
Subjects: []string{buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog) + ".>"}, AllowDirect: true}); err != nil {
t.Fatal(err)
}
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil { if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -177,8 +183,8 @@ func saysWhatItKeeps(t *testing.T, b *bed, done <-chan error) {
// 1297/B2b: a bucket it reads that the bus does not hold stops the start, naming the bucket. // 1297/B2b: a bucket it reads that the bus does not hold stops the start, naming the bucket.
func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) { func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) {
for _, missing := range readBuckets { for _, missing := range readBuckets {
if missing == delivery.PlacementBucket { if startsWithout(missing) {
continue // it starts without it, below continue // it starts without it (the placement and the grants' source: main_test.go; the puts: below)
} }
b := startBed(t, missing) b := startBed(t, missing)
select { select {
@@ -264,6 +270,19 @@ func (b *bed) putJSON(t *testing.T, bucket, key string, value []byte) {
} }
} }
// built appends the end of a done trunk build of module at commit to the builder's log, as the builder does.
func (b *bed) built(t *testing.T, module, repository, commit string) {
t.Helper()
raw, err := delivery.EncodeBuildLogEntry(delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: module, FP: delivery.SourcePrefix2 + strings.Repeat("c", 64),
Repository: repository, Directory: delivery.RootPath, Commit: commit, Node: "novox", At: time.Now().UTC(), Outcome: delivery.BuildDone})
if err != nil {
t.Fatal(err)
}
if _, err := b.js.Publish(b.ctx, buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog)+"."+module, raw); err != nil {
t.Fatal(err)
}
}
// compose says the controller composed module for node at fp, with the digest d (a letter repeated) and the judgement of the // compose says the controller composed module for node at fp, with the digest d (a letter repeated) and the judgement of the
// move to it, in mesh-controller_composed. // move to it, in mesh-controller_composed.
func (b *bed) compose(t *testing.T, node, module, fp string, d byte, judgement delivery.Judgement) string { func (b *bed) compose(t *testing.T, node, module, fp string, d byte, judgement delivery.Judgement) string {
@@ -8,25 +8,32 @@ package main
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"io" "io"
"log" "log"
"os" "os"
"os/signal" "os/signal"
"slices" "slices"
"sync/atomic"
"syscall" "syscall"
"time" "time"
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce" "git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks" "git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose" "git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun" "git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provider" "git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-sdk/go/provisions"
"mesh-delivery-assigner/internal/assign"
) )
func main() { func main() {
@@ -52,8 +59,9 @@ func main() {
var readBuckets = []string{ var readBuckets = []string{
buscompose.ResolvedBucket, buscompose.ResolvedBucket,
provider.AssignmentsBucket, provider.AssignmentsBucket,
delivery.BuildsBucket,
delivery.ControllerGrantsBucket, delivery.ControllerGrantsBucket,
delivery.PartGrantsBucket,
buscompose.LeaseBucket,
delivery.ControllerComposedBucket, delivery.ControllerComposedBucket,
delivery.AnswersBucket, delivery.AnswersBucket,
delivery.BuilderBuildsBucket, delivery.BuilderBuildsBucket,
@@ -63,8 +71,19 @@ var readBuckets = []string{
delivery.PresenceBucket, delivery.PresenceBucket,
delivery.HaveBucket, delivery.HaveBucket,
delivery.StepsBucket(goBuildModule), delivery.StepsBucket(goBuildModule),
takeoverBucket,
provisions.ProvisionsBucket,
keptBucket,
} }
// readLogs are the logs of other parts the assigner reads, each named in its manifest's reads: the builder's log, whose
// newest done trunk build of a module names the commit a reader of the manifests runs at (novox/hq ADR 0571 clause 1).
var readLogs = []string{delivery.BuildLogRead}
// takeoverBucket is the condition keeper's take-over key's bucket (novox/hq ADR 0564 §9): whether the assigner's said
// conditions go straight to the router or only as findings (issue 1952 WM6).
var takeoverBucket = health.Bucket(health.KeeperModule, health.TakeoverState)
// goBuildModule is the Go build module, which takes the assigner's compose asks on its seat's check stream and writes // goBuildModule is the Go build module, which takes the assigner's compose asks on its seat's check stream and writes
// their steps in its steps state (novox/hq ADR 0482 clause 4). // their steps in its steps state (novox/hq ADR 0482 clause 4).
const goBuildModule = "mesh-build-go" const goBuildModule = "mesh-build-go"
@@ -73,7 +92,7 @@ const goBuildModule = "mesh-build-go"
// (ADR 0567 §1), named after the assigner as every module state is. // (ADR 0567 §1), named after the assigner as every module state is.
var ownBuckets = []string{delivery.AssignerAsksBucket, delivery.AssignerCompositionsBucket, delivery.AssignerDifferencesBucket, var ownBuckets = []string{delivery.AssignerAsksBucket, delivery.AssignerCompositionsBucket, delivery.AssignerDifferencesBucket,
delivery.AssignerPutsBucket, delivery.AssignerWantsBucket, delivery.PlacementBucket, delivery.PlacementProposalsBucket, delivery.AssignerPutsBucket, delivery.AssignerWantsBucket, delivery.PlacementBucket, delivery.PlacementProposalsBucket,
delivery.PlacementCandidatesBucket, delivery.PlacementPulsesBucket} delivery.PlacementCandidatesBucket, delivery.PlacementPulsesBucket, assign.FindingsBucket, assign.PoliciesBucket, delivery.SeatHoldersBucket}
// seatBuckets are the seat's states, new with it: until the controller has made them from the manifest and the bus grants // seatBuckets are the seat's states, new with it: until the controller has made them from the manifest and the bus grants
// them, the assigner starts without them and binds each once the bus says it was created (issue 1790's kind). // them, the assigner starts without them and binds each once the bus says it was created (issue 1790's kind).
@@ -83,7 +102,8 @@ var seatBuckets = []string{delivery.PlacementBucket, delivery.PlacementProposals
// keptStates is what the assigner keeps and who may read it (ADR 0495 §2-§3): its wants by the watchman and the view // keptStates is what the assigner keeps and who may read it (ADR 0495 §2-§3): its wants by the watchman and the view
// (and the node agent from release 6), its differences by the view, its compositions by the checker, its asks by the view // (and the node agent from release 6), its differences by the view, its compositions by the checker, its asks by the view
// and the teller, its puts by whoever follows one (ADR 0546), and the seat mesh-placement's placement, proposals and // and the teller, its puts by whoever follows one (ADR 0546), and the seat mesh-placement's placement, proposals and
// candidates by the node agents, the grants part, mesh-config and the view (ADR 0567 §9). // candidates by the node agents, the grants part, mesh-config and the view (ADR 0567 §9), and its placement by
// the builder, the checker and the indexer, through mesh-sdk's toolchain gate (issue 774, ADR 0549 §3).
var keptStates = []announce.State{ var keptStates = []announce.State{
{Name: delivery.PlacementState, Readable: true}, {Name: delivery.PlacementState, Readable: true},
{Name: delivery.PlacementProposalsState, Readable: true}, {Name: delivery.PlacementProposalsState, Readable: true},
@@ -94,6 +114,9 @@ var keptStates = []announce.State{
{Name: "differences", Readable: true}, {Name: "differences", Readable: true},
{Name: "puts", Readable: true}, {Name: "puts", Readable: true},
{Name: "wants", Readable: true}, {Name: "wants", Readable: true},
{Name: health.FindingsState, Readable: true},
{Name: delivery.AssignerPoliciesState, Readable: true},
{Name: delivery.SeatHoldersState, Readable: true},
} }
// buckets is every bucket the assigner opened, by name. // buckets is every bucket the assigner opened, by name.
@@ -105,8 +128,33 @@ type buckets map[string]jetstream.KeyValue
// without it and takes no put further until it starts again (ADR 0546). // without it and takes no put further until it starts again (ADR 0546).
func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any)) (buckets, error) { func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any)) (buckets, error) {
out := buckets{} out := buckets{}
// The controller's lease first (ADR 0549 §5, ADR 0561 §3): once it is handed away the seat's placement alone names
// where modules run, and the controller's assignments are no bucket the assigner needs. Not granted or not there, the
// controller holds it, as before.
away, why := false, ""
if kv, err := js.KeyValue(ctx, buscompose.LeaseBucket); err != nil {
logf("the bucket %s cannot be opened (%v): the controller is taken to hold its lease, so the assignments are read beside the placement", buscompose.LeaseBucket, err)
} else {
out[buscompose.LeaseBucket] = kv
e, gerr := kv.Get(ctx, buscompose.ControllerLeaseKey)
var raw []byte
if gerr == nil {
raw = e.Value()
}
if errors.Is(gerr, jetstream.ErrKeyNotFound) {
gerr = nil
}
away, why = buscompose.LeaseHandedAway(raw, raw != nil, gerr)
}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) { for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
if name == buscompose.LeaseBucket {
continue
}
kv, err := js.KeyValue(ctx, name) kv, err := js.KeyValue(ctx, name)
if name == provider.AssignmentsBucket && err != nil && away {
logf("the bucket %s cannot be opened (%v), and none is needed: %s, so the seat's placement alone is read", name, err, why)
continue
}
if name == delivery.AssignerPutsBucket && err != nil { if name == delivery.AssignerPutsBucket && err != nil {
logf("the bucket %s cannot be opened (%v): no put is taken further until the assigner starts again", name, err) logf("the bucket %s cannot be opened (%v): no put is taken further until the assigner starts again", name, err)
continue continue
@@ -117,6 +165,41 @@ func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err) logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue continue
} }
if name == delivery.SeatHoldersBucket && err != nil {
// The seat's state, new with ADR 0571 §4: bound once the bus says it was made; until then nobody is told who
// holds which seat here, and the controller's seats still answer.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == keptBucket && err != nil {
// mesh-config's held values (ADR 0565 §5): without them a composition lays none, said in the journal.
logf("the bucket %s cannot be opened (%v): no held value is laid on a composition until it is bound", name, err)
continue
}
if name == provisions.ProvisionsBucket && err != nil {
// The seat mesh-provisioner's state, with its own delivery (ADR 0570 §4): bound once the bus says it was made;
// until then no want waits for a provision, as before.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == assign.PoliciesBucket && err != nil {
// The seat mesh-wants' state, new with it (ADR 0565 §7): bound once the bus says it was made; until then no
// person's policy is read, so the controller's beside each composition holds, as before.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if (name == assign.FindingsBucket || name == takeoverBucket) && err != nil {
// New with health leaving the controller (issue 1952 WM6): without them the conditions are said straight to the
// router and kept as no finding, as before.
logf("the bucket %s cannot be opened (%v): the assigner's conditions are said straight to the router until it starts again", name, err)
continue
}
if sourceOfGrants(name) && err != nil {
// The grants part's grants and the controller's lease choose where grants are read from (issue 774): without
// them the controller's grants are read, as before the lease is handed away.
logf("the bucket %s cannot be opened (%v): the controller's grants are read until the assigner starts again", name, err)
continue
}
if err != nil { if err != nil {
return nil, process.EssentialErr("assigner", name, "write a want from the whole mesh", err) return nil, process.EssentialErr("assigner", name, "write a want from the whole mesh", err)
} }
@@ -125,6 +208,23 @@ func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any
return out, nil return out, nil
} }
// startsWithout says the assigner starts without the bucket when it cannot be opened (open says why for each): the seat's
// states and those new with a delivery of their own, bound once made, and the two that choose the grants' source.
func startsWithout(name string) bool {
switch name {
case delivery.SeatHoldersBucket, keptBucket, provisions.ProvisionsBucket, assign.PoliciesBucket, assign.FindingsBucket, takeoverBucket,
delivery.AssignerPutsBucket:
return true
}
return slices.Contains(seatBuckets, name) || sourceOfGrants(name)
}
// sourceOfGrants says the bucket is one of the two that choose the grants' source, the grants part's grants and the
// controller's lease: neither stops the assigner when it is not there (delivery.GrantSource).
func sourceOfGrants(name string) bool {
return name == delivery.PartGrantsBucket || name == buscompose.LeaseBucket
}
// run is the program: it returns when ctx ends or the bus cannot be used. // run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error { func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-assigner: ", log.LstdFlags) logger := log.New(logOut, "mesh-delivery-assigner: ", log.LstdFlags)
@@ -153,7 +253,71 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
} }
thisNode = account.Node thisNode = account.Node
logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs)) logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs))
return follow(ctx, js, conn, kvs, sayVia(ctx, js), emitting{said}, logger) findings := findingsOf(ctx, kvs, logger.Printf)
buildLog := func(handle func(stdio.LogEntry) error) error {
return said.ReadLog(delivery.BuilderModule, delivery.BuilderLog).Watch("", stdio.LogFromAll, handle)
}
return follow(ctx, js, conn, kvs, findings.Say(sayVia(ctx, js)), findings, emitting{said}, buildLog, logger)
}
// findingsOf is where the assigner's said conditions go as findings (issue 1952 WM6): its findings bucket, and the
// condition keeper's take-over key, watched, so a condition is said straight to the router only until the take-over. A
// bucket it cannot open is said: without its findings the conditions are said straight, as before; without the take-over
// key they are said straight too, the controller being followed.
func findingsOf(ctx context.Context, kvs buckets, logf func(string, ...any)) assign.Findings {
var f assign.Findings
if kv := kvs[assign.FindingsBucket]; kv != nil {
f.Put = func(key string, value []byte) error { _, err := kv.Put(ctx, key, value); return err }
f.Delete = func(key string) error { return kv.Delete(ctx, key) }
f.Keys = func() ([]string, error) {
keys, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
}
var keeper atomic.Bool
f.ToKeeper = keeper.Load
takeover := takeoverBucket
kv := kvs[takeover]
if kv == nil {
return f
}
w, err := kv.Watch(ctx, health.TakeoverKey)
if err != nil {
process.NotWatched(logf, takeover, err, "the controller is followed: the conditions are said straight to the router")
return f
}
go func() {
defer w.Stop()
for {
select {
case <-ctx.Done():
return
case e, ok := <-w.Updates():
if !ok {
logf("the watch of %s ended: the last source it named is followed until the assigner starts again", takeover)
return
}
if e == nil {
continue
}
var raw []byte
if e.Operation() == jetstream.KeyValuePut {
raw = e.Value()
}
t, err := health.DecodeTakeover(raw, raw != nil)
if err != nil {
logf("the take-over key cannot be read (%v): the controller is followed", err)
}
if was := keeper.Swap(t.Follow == health.FollowKeeper); was != keeper.Load() {
logf("the assigner's conditions are said by %s from now (the take-over key follows %s)", t.Seat(), t.Follow)
}
}
}
}()
return f
} }
// thisNode is the node this process runs on, as its credential names it: said with each of the seat's events. // thisNode is the node this process runs on, as its credential names it: said with each of the seat's events.
@@ -170,6 +334,8 @@ var watched = []string{
buscompose.ResolvedBucket, buscompose.ResolvedBucket,
provider.AssignmentsBucket, provider.AssignmentsBucket,
delivery.ControllerGrantsBucket, delivery.ControllerGrantsBucket,
delivery.PartGrantsBucket,
buscompose.LeaseBucket,
delivery.ControllerComposedBucket, delivery.ControllerComposedBucket,
delivery.AnswersBucket, delivery.AnswersBucket,
delivery.BuilderBuildsBucket, delivery.BuilderBuildsBucket,
@@ -185,11 +351,15 @@ var watched = []string{
delivery.AssignerCompositionsBucket, delivery.AssignerCompositionsBucket,
delivery.AssignerDifferencesBucket, delivery.AssignerDifferencesBucket,
delivery.AssignerWantsBucket, delivery.AssignerWantsBucket,
assign.PoliciesBucket,
provisions.ProvisionsBucket,
keptBucket,
delivery.SeatHoldersBucket,
} }
// follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after // follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after
// each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends. // each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends.
func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, kvs buckets, say func(asks.Condition) error, bus caller, logger *log.Logger) error { func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, kvs buckets, say func(asks.Condition) error, findings assign.Findings, bus caller, buildLog logWatch, logger *log.Logger) error {
w := newWorld() w := newWorld()
dirty := make(chan struct{}, 1) dirty := make(chan struct{}, 1)
wake := func() { wake := func() {
@@ -198,6 +368,22 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
default: default:
} }
} }
// The builder's log (ADR 0571 clause 1): every entry it keeps before the first pass, then each new one as it is appended;
// a module's newer done trunk build makes a pass, as a composition waiting on its commit is judged then. Where it cannot be
// read, a composition names no running reader of the manifests until the assigner starts again, and it is said.
if buildLog != nil {
var first atomic.Bool
if err := buildLog(func(le stdio.LogEntry) error {
if w.putBuilt(le.Value) && first.Load() {
wake()
}
return nil
}); err != nil {
logger.Printf("%s is not read (%v): no composition is judged against a running reader of the manifests until the assigner starts again",
delivery.BuildLogRead, err)
}
first.Store(true)
}
errs := make(chan error, len(watched)+2) errs := make(chan error, len(watched)+2)
settled := make(chan struct{}, len(watched)+2) settled := make(chan struct{}, len(watched)+2)
watch := func(name string, opts ...jetstream.WatchOpt) error { watch := func(name string, opts ...jetstream.WatchOpt) error {
@@ -233,7 +419,13 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
if e.Operation() == jetstream.KeyValuePut { if e.Operation() == jetstream.KeyValuePut {
value = e.Value() value = e.Value()
} }
before := w.grantsBucket()
w.putEntry(name, e.Key(), value, e.Revision(), e.Created()) w.putEntry(name, e.Key(), value, e.Revision(), e.Created())
if name == buscompose.LeaseBucket && w.grantsBucket() == before {
// The controller renews its lease every few seconds: only a change of where grants are read from
// makes a pass (delivery.GrantSource).
continue
}
wake() wake()
} }
} }
@@ -269,13 +461,13 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
return return
} }
if bus != nil { if bus != nil {
p.acts = newPutActs(bus, wake, logger.Printf) p.acts = newPutActs(bus, wake, logger.Printf, w.handedAway)
} }
logger.Printf("the bucket %s is open: puts are taken further", delivery.AssignerPutsBucket) logger.Printf("the bucket %s is open: puts are taken further", delivery.AssignerPutsBucket)
} }
started := 0 started := 0
for _, name := range watched { for _, name := range watched {
if kvs[name] == nil && slices.Contains(seatBuckets, name) { if kvs[name] == nil && (startsWithout(name) || name == provider.AssignmentsBucket) {
continue // not on the bus at this start (open said so) continue // not on the bus at this start (open said so)
} }
started++ started++
@@ -297,13 +489,21 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
name string name string
kv jetstream.KeyValue kv jetstream.KeyValue
} }
placementAppeared := make(chan appeared, len(seatBuckets)) placementAppeared := make(chan appeared, len(seatBuckets)+2)
for _, name := range seatBuckets { for _, name := range seatBuckets {
if kvs[name] == nil && js != nil { if kvs[name] == nil && js != nil {
process.AwaitBucket(ctx, js, nc, name, "the seat mesh-placement waits for it", process.AwaitBucket(ctx, js, nc, name, "the seat mesh-placement waits for it",
logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- appeared{name, kv} }) logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- appeared{name, kv} })
} }
} }
// And the two that choose where grants are read from, with the same delivery as the grants part that writes one.
for _, name := range []string{delivery.PartGrantsBucket, buscompose.LeaseBucket, assign.PoliciesBucket, provisions.ProvisionsBucket, keptBucket, delivery.SeatHoldersBucket} {
if kvs[name] == nil && js != nil {
name := name
process.AwaitBucket(ctx, js, nc, name, "until then the bucket is not read",
logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- appeared{name, kv} })
}
}
takePlacement := func(a appeared) error { takePlacement := func(a appeared) error {
if kvs[a.name] != nil { if kvs[a.name] != nil {
return nil // bound already, by the other of its two events return nil // bound already, by the other of its two events
@@ -321,13 +521,13 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
case <-settled: case <-settled:
} }
} }
p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, said: map[string]bool{}, answeredAt: map[string]uint64{}} p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, findings: findings, said: map[string]bool{}, answeredAt: map[string]uint64{}}
p.asks = newComposeAsks(ctx, buildrun.Asker{Publisher: buildrun.JetStreamAsks{JS: js}, Waiter: buildrun.JetStreamAsks{JS: js}}, wake, logger.Printf) p.asks = newComposeAsks(ctx, buildrun.Asker{Publisher: buildrun.JetStreamAsks{JS: js}, Waiter: buildrun.JetStreamAsks{JS: js}}, wake, logger.Printf)
if bus != nil && kvs[delivery.AssignerPutsBucket] != nil { if bus != nil && kvs[delivery.AssignerPutsBucket] != nil {
p.acts = newPutActs(bus, wake, logger.Printf) p.acts = newPutActs(bus, wake, logger.Printf, w.handedAway)
} }
if bus != nil { if bus != nil {
p.seat = newSeatActs(bus, wake, logger.Printf) p.seat = newSeatActs(bus, wake, logger.Printf, w.handedAway)
} }
if js != nil { if js != nil {
p.publish = streamPublisher(ctx, js, thisNode) p.publish = streamPublisher(ctx, js, thisNode)
@@ -1,6 +1,8 @@
package main package main
import ( import (
"mesh-delivery-assigner/internal/assign"
"context" "context"
"errors" "errors"
"fmt" "fmt"
@@ -17,8 +19,10 @@ import (
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks" "git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
) )
// 1297/B2a: the buckets the program opens are exactly the manifest's reads and states, as the broker names them // 1297/B2a: the buckets the program opens are exactly the manifest's reads and states, as the broker names them
@@ -38,8 +42,16 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
} }
var reads []string var reads []string
for _, r := range m.Reads { for _, r := range m.Reads {
if slices.Contains(readLogs, r) {
continue // a log, read through its own reader (ADR 0571 clause 1), not a bucket
}
reads = append(reads, strings.Replace(r, ".", "_", 1)) reads = append(reads, strings.Replace(r, ".", "_", 1))
} }
for _, l := range readLogs {
if !slices.Contains(m.Reads, l) {
t.Errorf("the program reads the log %s, the manifest does not name it", l)
}
}
var states []string var states []string
for _, s := range m.State { for _, s := range m.State {
states = append(states, m.Module+"_"+s.Name) states = append(states, m.Module+"_"+s.Name)
@@ -62,20 +74,30 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
} }
// ADR 0546: whatever keeps the puts' bucket from opening (not there, not granted, no answer in time), the assigner opens // ADR 0546: whatever keeps the puts' bucket from opening (not there, not granted, no answer in time), the assigner opens
// the rest and starts, writing wants; any other bucket it cannot open stops the start, named. // the rest and starts, writing wants; any other bucket it cannot open stops the start, named.
// Issue 774 (ADR 0549 §5): the controller's lease not granted, the assigner starts taking it as the controller's; the
// assignments refused while the controller holds it stop the start, and once it is handed away they are not needed.
handed := []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`)
for _, c := range []struct { for _, c := range []struct {
missing string missing string
err error err error
starts bool starts bool
lease []byte
said string
}{ }{
{delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true}, {delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true, nil, "no put is taken further"},
{delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true}, {delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true, nil, "no put is taken further"},
{delivery.AssignerPutsBucket, context.DeadlineExceeded, true}, {delivery.AssignerPutsBucket, context.DeadlineExceeded, true, nil, "no put is taken further"},
{delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false}, {delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false, nil, ""},
// Issue 774: the grants part's grants and the controller's lease choose the grants' source; neither stops the start.
{delivery.PartGrantsBucket, jetstream.ErrBucketNotFound, true, nil, "the controller's grants are read"},
{buscompose.LeaseBucket, errors.New("nats: permissions violation"), true, nil, "taken to hold its lease"},
{provider.AssignmentsBucket, errors.New("nats: permissions violation"), false, []byte(`{"instance":"novox/1"}`), ""},
{provider.AssignmentsBucket, errors.New("nats: permissions violation"), true, handed, "the seat's placement alone is read"},
} { } {
var logged string var logged string
kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err}, func(f string, a ...any) { logged = fmt.Sprintf(f, a...) }) kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err, lease: c.lease}, func(f string, a ...any) { logged += fmt.Sprintf(f, a...) })
switch { switch {
case c.starts && (err != nil || kvs[c.missing] != nil || len(kvs) != len(readBuckets)+len(ownBuckets)-1 || !strings.Contains(logged, "no put is taken further")): case c.starts && (err != nil || kvs[c.missing] != nil || len(kvs) != len(readBuckets)+len(ownBuckets)-1 || !strings.Contains(logged, c.said)):
t.Errorf("%s refused with %v: the start answered %v, %d buckets, logged %q", c.missing, c.err, err, len(kvs), logged) t.Errorf("%s refused with %v: the start answered %v, %d buckets, logged %q", c.missing, c.err, err, len(kvs), logged)
case !c.starts && (err == nil || !strings.Contains(err.Error(), "grant of "+c.missing)): // 1945: named with its grant case !c.starts && (err == nil || !strings.Contains(err.Error(), "grant of "+c.missing)): // 1945: named with its grant
t.Errorf("%s refused with %v: the start answered %v", c.missing, c.err, err) t.Errorf("%s refused with %v: the start answered %v", c.missing, c.err, err)
@@ -98,7 +120,7 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
kvs[delivery.ControllerComposedBucket].(*watchKV).entries = []kvEntry{{key: ckey, value: cd}} kvs[delivery.ControllerComposedBucket].(*watchKV).entries = []kvEntry{{key: ckey, value: cd}}
kvs[delivery.AssignerPutsBucket].(*watchKV).closed = true kvs[delivery.AssignerPutsBucket].(*watchKV).closed = true
kvs[delivery.AssignerWantsBucket].(*watchKV).wrote = wrote kvs[delivery.AssignerWantsBucket].(*watchKV).wrote = wrote
go follow(ctx, nil, nil, kvs, func(asks.Condition) error { return nil }, nil, log.New(io.Discard, "", 0)) go follow(ctx, nil, nil, kvs, func(asks.Condition) error { return nil }, assign.Findings{}, nil, nil, log.New(io.Discard, "", 0))
select { select {
case key := <-wrote: case key := <-wrote:
if key != "want.ace.node" { if key != "want.ace.node" {
@@ -114,13 +136,18 @@ type refusingJS struct {
jetstream.JetStream jetstream.JetStream
missing string missing string
err error err error
lease []byte // the controller's lease, when it holds one
} }
func (r refusingJS) KeyValue(_ context.Context, name string) (jetstream.KeyValue, error) { func (r refusingJS) KeyValue(_ context.Context, name string) (jetstream.KeyValue, error) {
if name == r.missing { if name == r.missing {
return nil, r.err return nil, r.err
} }
return &memKV{values: map[string][]byte{}}, nil kv := &memKV{values: map[string][]byte{}}
if name == buscompose.LeaseBucket && r.lease != nil {
kv.values[buscompose.ControllerLeaseKey] = r.lease
}
return kv, nil
} }
// watchKV is a memKV whose watch hands its entries and then the marker that all stands was handed, or ends at once when // watchKV is a memKV whose watch hands its entries and then the marker that all stands was handed, or ends at once when
@@ -2,6 +2,7 @@ package main
import ( import (
"context" "context"
"encoding/json"
"errors" "errors"
"maps" "maps"
"slices" "slices"
@@ -11,6 +12,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks" "git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
"git.novox.be/novox/mesh-sdk/go/sdkversion" "git.novox.be/novox/mesh-sdk/go/sdkversion"
"mesh-delivery-assigner/internal/assign" "mesh-delivery-assigner/internal/assign"
@@ -27,6 +29,16 @@ type passer struct {
// pass says a condition once and a restart says it again (the router keeps one per key). // pass says a condition once and a restart says it again (the router keeps one per key).
say func(asks.Condition) error say func(asks.Condition) error
said map[string]bool said map[string]bool
// findings is the assigner's findings bucket (its said conditions and policy-split, ADR 0564 §2, ADR 0565 §8); splits the
// modules whose policy-split finding stands, nil until the first pass, which deletes any a previous process left.
findings assign.Findings
splits map[string]bool
// failedProvisions are the failed provisions whose finding this process wrote (ADR 0570 §4).
failedProvisions map[string]bool
// missing are the required provisions with no key this process pulsed, and missingSaid those whose finding it wrote.
missing, missingSaid map[string]bool
// unplaced are the nodes whose unplaced finding this process wrote.
unplaced map[string]bool
// answeredAt is, by the id of a backup ask, how many values the have had when its answer was written into the want. // answeredAt is, by the id of a backup ask, how many values the have had when its answer was written into the want.
answeredAt map[string]uint64 answeredAt map[string]uint64
asks *composeAsks asks *composeAsks
@@ -129,6 +141,13 @@ func (p *passer) wants(ctx context.Context) {
} }
} }
wanted := p.w.wantsOf() wanted := p.w.wantsOf()
chosen, _ := assign.DecodePolicies(p.w.read(assign.PoliciesBucket))
provisionsOf := p.provisionsOf()
missing := map[string]bool{}
defer func() { p.provisionsMissing(ctx, missing) }()
defer p.unplacedFindings()
defer p.seatHolders(ctx)
defer p.policySplits(manifests, chosen)
frozen := p.sayPlacement(p.withCalls(p.w.placementDiffers())) frozen := p.sayPlacement(p.withCalls(p.w.placementDiffers()))
placed := p.w.placedNodes() placed := p.w.placedNodes()
puts := p.putsByModule() puts := p.putsByModule()
@@ -149,12 +168,17 @@ func (p *passer) wants(ctx context.Context) {
stale := map[string]neitherOne{} stale := map[string]neitherOne{}
left := map[string]string{} left := map[string]string{}
defer func() { p.sayNeither(neither, stale, left) }() defer func() { p.sayNeither(neither, stale, left) }()
// Where grants are read from, once for the pass: the grants part's while the lease is handed away (issue 774).
grants := p.w.grantsBucket()
for name := range names { for name := range names {
fp := trunk[name].FP2 fp := trunk[name].FP2
name := name name := name
m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name], m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Chosen: chosenOf(chosen, name), Provisions: provisionsOf(name), Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name],
Self: func(node string) bool { return placed[node] }, Self: func(node string) bool { return placed[node] },
Composed: func(node, fp string) (delivery.ComposedDigest, bool) { Composed: func(node, fp string) (delivery.ComposedDigest, bool) {
if placed[node] && p.w.handedAway() {
return p.w.ownComposed(node) // from the lease write: the digest its own have names (ADR 0571 §6)
}
if placed[node] { if placed[node] {
// The node composes its own (ADR 0547 §2, §4): no composition of the controller's is waited for, the want // The node composes its own (ADR 0547 §2, §4): no composition of the controller's is waited for, the want
// names the build alone, and the node judges the move itself. // names the build alone, and the node judges the move itself.
@@ -164,7 +188,7 @@ func (p *passer) wants(ctx context.Context) {
}, },
GrantedAt: func(fp string) bool { GrantedAt: func(fp string) bool {
key, err := delivery.ControllerGrantKey(name, fp) key, err := delivery.ControllerGrantKey(name, fp)
return err == nil && p.w.has(delivery.ControllerGrantsBucket, key) return err == nil && p.w.has(grants, key)
}, },
Restored: func(node string) bool { Restored: func(node string) bool {
key, err := delivery.RestoreAskKey(node, name, fp) key, err := delivery.RestoreAskKey(node, name, fp)
@@ -180,8 +204,8 @@ func (p *passer) wants(ctx context.Context) {
hold, _ := delivery.HoldAskKey(name, fp) hold, _ := delivery.HoldAskKey(name, fp)
m.Released = p.answeredWith(hold, "release") m.Released = p.answeredWith(hold, "release")
m.Kept = p.answeredWith(hold, "keep") m.Kept = p.answeredWith(hold, "keep")
m.Built, m.Held, m.Granted = p.w.built(name, fp), p.w.has(delivery.AssignerWantsBucket, held) && !m.Released, p.w.has(delivery.ControllerGrantsBucket, grant) m.Built, m.Held, m.Granted = p.w.built(name, fp), p.w.has(delivery.AssignerWantsBucket, held) && !m.Released, p.w.has(grants, grant)
if raw := p.w.value(delivery.ControllerGrantsBucket, grant); raw != nil { if raw := p.w.value(grants, grant); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil { if g, err := delivery.DecodeControllerGrant(raw); err == nil {
m.GrantedOn = func(node string) bool { return assign.GrantNamesNode(g, node, name) } m.GrantedOn = func(node string) bool { return assign.GrantNamesNode(g, node, name) }
} }
@@ -195,6 +219,11 @@ func (p *passer) wants(ctx context.Context) {
last[node] = p.w.lastHealthy(node, name, fp) last[node] = p.w.lastHealthy(node, name, fp)
} }
out := assign.Wants(m, nodes, wanted[name], last) out := assign.Wants(m, nodes, wanted[name], last)
for _, node := range m.Nodes {
for _, prov := range m.MissingProvisions(node) {
missing[node+"."+name+"."+prov] = true
}
}
p.yeses(ctx, name, m, wanted[name], &out) p.yeses(ctx, name, m, wanted[name], &out)
gatherNeither(neither, name, out) gatherNeither(neither, name, out)
leftNeither(left, m, wanted[name], out) leftNeither(left, m, wanted[name], out)
@@ -930,3 +959,275 @@ func (p *passer) yeses(ctx context.Context, module string, m assign.Module, befo
} }
} }
} }
// chosenOf is a person's choice for module, nil when none.
func chosenOf(chosen map[string]catalogue.PersonPolicy, module string) *catalogue.PersonPolicy {
if c, ok := chosen[module]; ok {
return &c
}
return nil
}
// policySplits writes the finding policy-split for each module whose policy at the assigner differs from one the controller
// writes beside a composition of it, and deletes it once they agree (ADR 0565 §8): written when a split begins, never again
// while it stands. The first pass after a start deletes every one a previous process left that no longer splits.
func (p *passer) policySplits(manifests map[string]catalogue.Manifest, chosen map[string]catalogue.PersonPolicy) {
if p.findings.Put == nil {
return
}
now := time.Now
if p.now != nil {
now = p.now
}
split := map[string]bool{}
theirs := map[string][]string{}
if !p.w.handedAway() { // the controller writes no policy from the lease write (ADR 0571 §6)
theirs = assign.ControllerPolicies(p.w.read(delivery.ControllerComposedBucket))
}
for _, s := range assign.Splits(manifests, chosen, theirs) {
split[s.Module] = true
if p.splits[s.Module] {
continue
}
value, err := assign.PolicySplitFinding(s, now())
if err == nil {
err = p.findings.Put(assign.PolicySplitKey(s.Module), value)
}
if err != nil {
p.logf("the policy split of %s is not written: %v", s.Module, err)
delete(split, s.Module)
}
}
gone := p.splits
if gone == nil && p.findings.Keys != nil {
gone = map[string]bool{}
if keys, err := p.findings.Keys(); err == nil {
for _, k := range keys {
if m, ok := strings.CutSuffix(strings.TrimPrefix(k, "module."), "."+assign.KindPolicySplit); ok && strings.HasPrefix(k, "module.") {
gone[m] = true
}
}
}
}
for m := range gone {
if !split[m] {
if err := p.findings.Delete(assign.PolicySplitKey(m)); err != nil {
p.logf("the policy split of %s is not deleted: %v", m, err)
split[m] = true // tried again at the next pass
}
}
}
p.splits = split
}
// provisionsOf answers, per module, the provisions it consumes on a node, by key (ADR 0570 §4), from the seat
// mesh-provisioner's state; nil while the state is not read (the seat not delivered yet), so no want waits on it. A failed
// provision is also said as the assigner's finding `provider.<key>.provision-failed` (a waiting one is the watchman's
// needs-operator), deleted once it is no longer failed.
func (p *passer) provisionsOf() func(module string) func(node string) map[string]provisions.Provision {
if p.kvs[provisions.ProvisionsBucket] == nil {
return func(string) func(string) map[string]provisions.Provision { return nil }
}
by := map[string]map[string]map[string]provisions.Provision{} // module, node, key
failed := map[string]provisions.Provision{}
for key, raw := range p.w.read(provisions.ProvisionsBucket) {
pr, err := provisions.Decode(raw)
if err != nil {
continue
}
node, module := pr.Consumer.Node, pr.Consumer.Module
if by[module] == nil {
by[module] = map[string]map[string]provisions.Provision{}
}
if by[module][node] == nil {
by[module][node] = map[string]provisions.Provision{}
}
by[module][node][key] = pr
if pr.State == provisions.Failed {
failed[key] = pr
}
}
p.provisionFindings(failed)
return func(module string) func(string) map[string]provisions.Provision {
return func(node string) map[string]provisions.Provision { return by[module][node] }
}
}
// provisionFindings writes the finding of each failed provision and deletes the ones no longer failed.
func (p *passer) provisionFindings(failed map[string]provisions.Provision) {
if p.findings.Put == nil {
return
}
now := time.Now
if p.now != nil {
now = p.now
}
for key, pr := range failed {
if p.failedProvisions[key] {
continue
}
value, err := assign.ProvisionFailedFinding(key, pr, now())
if err == nil {
err = p.findings.Put(assign.ProvisionFailedKey(key), value)
}
if err != nil {
p.logf("the failed provision %s is not said: %v", key, err)
continue
}
if p.failedProvisions == nil {
p.failedProvisions = map[string]bool{}
}
p.failedProvisions[key] = true
}
for key := range p.failedProvisions {
if _, still := failed[key]; still {
continue
}
if err := p.findings.Delete(assign.ProvisionFailedKey(key)); err != nil {
p.logf("the failed provision %s is not cleared: %v", key, err)
continue
}
delete(p.failedProvisions, key)
}
}
// MissingBound is how long a required provision may have no key before it is said: a module's delivery bound, the default
// when its manifest names none (ADR 0564 clause 13 (d)).
var MissingBound = delivery.DefaultDeliveryBound
// missingPulse is the pulse key of a provision with no key, `provision-missing.<node>.<module>.<provision>`.
func missingPulse(key string) string { return "provision-missing." + key }
// provisionsMissing is the loud end of a required provision with no key (the lead, 2026-10-29, ADR 0570 §4): each one is
// pulsed with the bound when it is first seen; once its pulse expired and it still has no key, the finding
// `provider.<node>.<module>.<provision>.provision-missing` is written; one keyed again ends both. The pulse's expiry, heard on
// the watch, is the pass that writes it: no timer.
func (p *passer) provisionsMissing(ctx context.Context, missing map[string]bool) {
pulses := p.kvs[delivery.PlacementPulsesBucket]
if p.missing == nil {
p.missing = map[string]bool{}
}
for key := range missing {
switch {
case !p.missing[key]:
p.missing[key] = true
if pulses != nil {
p.pulse(ctx, missingPulse(key), MissingBound)
}
case pulses == nil || !p.pulsing(missingPulse(key)):
if p.missingSaid[key] || p.findings.Put == nil {
continue
}
value, err := assign.ProvisionMissingFinding(key, MissingBound, p.clock())
if err == nil {
err = p.findings.Put(assign.ProvisionMissingKey(key), value)
}
if err != nil {
p.logf("the missing provision %s is not said: %v", key, err)
continue
}
if p.missingSaid == nil {
p.missingSaid = map[string]bool{}
}
p.missingSaid[key] = true
}
}
for key := range p.missing {
if missing[key] {
continue
}
delete(p.missing, key)
if pulses != nil && p.pulsing(missingPulse(key)) {
if err := pulses.Delete(ctx, missingPulse(key)); err == nil {
p.w.put(delivery.PlacementPulsesBucket, missingPulse(key), nil)
}
}
if p.missingSaid[key] && p.findings.Delete != nil {
if err := p.findings.Delete(assign.ProvisionMissingKey(key)); err != nil {
p.logf("the missing provision %s is not cleared: %v", key, err)
p.missing[key] = true
continue
}
delete(p.missingSaid, key)
}
}
}
// unplacedFindings writes, from the lease write, the finding `machine.<node>.unplaced` for each node that runs but no
// placement names, which gets no want (ADR 0571 §6), and deletes it once the node is placed.
func (p *passer) unplacedFindings() {
if p.findings.Put == nil {
return
}
now := []string(p.w.unplacedNodes())
is := map[string]bool{}
for _, node := range now {
is[node] = true
if p.unplaced[node] {
continue
}
value, err := assign.UnplacedFinding(node, p.clock())
if err == nil {
err = p.findings.Put(assign.UnplacedKey(node), value)
}
if err != nil {
p.logf("the unplaced node %s is not said: %v", node, err)
continue
}
if p.unplaced == nil {
p.unplaced = map[string]bool{}
}
p.unplaced[node] = true
}
for node := range p.unplaced {
if is[node] {
continue
}
if err := p.findings.Delete(assign.UnplacedKey(node)); err == nil {
delete(p.unplaced, node)
}
}
}
// seatHolders writes the seat mesh-placement's state seat-holders from every node's placement (ADR 0571 §4): one key per
// seat held once, `<seat>`, and one per node of a node seat, `<seat>.<node>`, written only where it differs, and a key no
// placement holds any more deleted. A pass follows each placement change and each manifest the indexer takes in, so no
// timer writes it.
func (p *passer) seatHolders(ctx context.Context) {
kv := p.kvs[delivery.SeatHoldersBucket]
if kv == nil {
return
}
placements := map[string]delivery.Placement{}
for key, value := range p.w.read(delivery.PlacementBucket) {
node, err := delivery.ParsePlacementKey(key)
if err != nil {
continue
}
if pl, err := delivery.DecodePlacement(value); err == nil && pl.Node == node {
placements[node] = pl
}
}
if len(placements) == 0 {
return // no placement read yet: nothing is known of who holds what, and nothing is deleted
}
want := delivery.SeatHoldersOf(placements)
have := p.w.read(delivery.SeatHoldersBucket)
for key, sh := range want {
value, err := json.Marshal(sh)
if err != nil || string(have[key]) == string(value) {
continue
}
p.put(ctx, delivery.SeatHoldersBucket, key, func() ([]byte, error) { return value, nil })
}
for key := range have {
if _, ok := want[key]; ok {
continue
}
if err := kv.Delete(ctx, key); err != nil {
p.logf("the seat holders %s are not deleted: %v", key, err)
continue
}
p.w.put(delivery.SeatHoldersBucket, key, nil)
}
}
@@ -13,6 +13,7 @@ import (
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks" "git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider" "git.novox.be/novox/mesh-sdk/go/provider"
@@ -31,6 +32,11 @@ func (m *memKV) Put(_ context.Context, key string, value []byte) (uint64, error)
return uint64(len(m.values)), nil return uint64(len(m.values)), nil
} }
func (m *memKV) Create(_ context.Context, key string, value []byte, _ ...jetstream.KVCreateOpt) (uint64, error) {
m.values[key] = value
return uint64(len(m.values)), nil
}
func (m *memKV) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error { func (m *memKV) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
delete(m.values, key) delete(m.values, key)
return nil return nil
@@ -135,6 +141,9 @@ func (m *memKV) Update(_ context.Context, key string, value []byte, revision uin
type memStore map[string]*memKV type memStore map[string]*memKV
func (s memStore) get(state, key string) ([]byte, uint64, bool, error) { func (s memStore) get(state, key string) ([]byte, uint64, bool, error) {
if s[state] == nil {
return nil, 0, false, nil
}
v, ok := s[state].values[key] v, ok := s[state].values[key]
return v, uint64(len(v)), ok, nil return v, uint64(len(v)), ok, nil
} }
@@ -183,7 +192,7 @@ func putOfTheBus(t *testing.T) {
p := newPasser(kvs, now) p := newPasser(kvs, now)
bus := &fakeBus{calls: make(chan string, 8)} bus := &fakeBus{calls: make(chan string, 8)}
woke := make(chan struct{}, 8) // each act's goroutine wakes a pass once it is done woke := make(chan struct{}, 8) // each act's goroutine wakes a pass once it is done
p.acts = newPutActs(bus, func() { woke <- struct{}{} }, func(string, ...any) {}) p.acts = newPutActs(bus, func() { woke <- struct{}{} }, func(string, ...any) {}, nil)
p.acts.snapshot = func(context.Context, string, string) (string, error) { p.acts.snapshot = func(context.Context, string, string) (string, error) {
return "novox's restore point of nats (streams) taken 22:01", nil return "novox's restore point of nats (streams) taken 22:01", nil
} }
@@ -284,7 +293,7 @@ func putOfTheBus(t *testing.T) {
if got.Step != delivery.PutWanted || err != nil || w.FP != fp || w.Composed != digest || !w.Put || !w.Hold { if got.Step != delivery.PutWanted || err != nil || w.FP != fp || w.Composed != digest || !w.Put || !w.Hold {
t.Fatalf("snapshotted, the put stands at %s (%s); the want %+v, %v", got.Step, got.Words, w, err) t.Fatalf("snapshotted, the put stands at %s (%s); the want %+v, %v", got.Step, got.Words, w, err)
} }
if call := <-bus.calls; !strings.Contains(call, "warranted") || !strings.Contains(call, a.Ask.ID) { if call := <-bus.calls; !strings.HasPrefix(call, "seat:mesh-hand-acts.warranted ") || !strings.Contains(call, a.Ask.ID) { // ADR 0565 §11: into the seat
t.Errorf("the warrant was recorded as %q", call) t.Errorf("the warrant was recorded as %q", call)
} }
world(delivery.AssignerWantsBucket, "want.novox.nats", wantsKV.values["want.novox.nats"]) world(delivery.AssignerWantsBucket, "want.novox.nats", wantsKV.values["want.novox.nats"])
@@ -873,6 +882,24 @@ func aPairOneSourceNamesAloneGetsNoWantAndOneCondition(t *testing.T) {
if m.Input == nil || len(m.Input.Nodes) != 1 || !slices.Equal(m.Input.Nodes[0].Assigned, []string{"a", "b", "c"}) || m.Input.Nodes[0].Place.Key != "k" { if m.Input == nil || len(m.Input.Nodes) != 1 || !slices.Equal(m.Input.Nodes[0].Assigned, []string{"a", "b", "c"}) || m.Input.Nodes[0].Place.Key != "k" {
t.Errorf("the check's input from the bus: %+v (%q)", m.Input, m.InputUnread) t.Errorf("the check's input from the bus: %+v (%q)", m.Input, m.InputUnread)
} }
// Issue 774 (ADR 0549 §5, §6): once the controller's lease is handed away the placement alone names where modules
// run: a pair in the placement only gets its want and no condition, and one in the assignments only is no module of
// the node. A lease that goes back to a controller brings the comparison back.
place("n2", "d")
assigned("g14", "e")
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`))
got, all := p.w.assigned()
if !slices.Equal(got["n2"], []string{"d"}) || !slices.Equal(got["g14"], []string{"a", "b", "c"}) || len(all) != 0 {
t.Errorf("handed away: assigned %v, assignments %v; want the placement alone", got, all)
}
if frozen := p.sayPlacement(p.w.placementDiffers()); len(frozen) != 0 || len(said) != 4 {
t.Errorf("handed away: frozen %v, said %+v; want no condition", frozen, said[4:])
}
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(`{"instance":"novox/1","epoch":58}`))
if frozen := p.sayPlacement(p.w.placementDiffers()); !frozen["n2.d"] || !frozen["g14.e"] {
t.Errorf("a controller's lease again: frozen %v; want both pairs held", frozen)
}
} }
// 1126 (ADR 0547 §2 to §4): a node the catalogue places composes its own: its wants name the build alone and its own want // 1126 (ADR 0547 §2 to §4): a node the catalogue places composes its own: its wants name the build alone and its own want
@@ -13,6 +13,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks" "git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"mesh-delivery-assigner/internal/assign" "mesh-delivery-assigner/internal/assign"
) )
@@ -46,14 +47,16 @@ type putActs struct {
snapped map[string]string snapped map[string]string
snapNo map[string]string snapNo map[string]string
warranted map[string]bool warranted map[string]bool
// rec records the put's warrant in the hand-act log (ADR 0565 §11).
rec handacts.Recorder
warrantNo map[string]string warrantNo map[string]string
said map[string]string said map[string]string
// snapshot takes the bus's snapshot on node; a variable so a test takes none. // snapshot takes the bus's snapshot on node; a variable so a test takes none.
snapshot func(ctx context.Context, node, module string) (string, error) snapshot func(ctx context.Context, node, module string) (string, error)
} }
func newPutActs(bus caller, wake func(), logf func(string, ...any)) *putActs { func newPutActs(bus caller, wake func(), logf func(string, ...any), leaseAway func() bool) *putActs {
a := &putActs{bus: bus, wake: wake, logf: logf, snapping: map[string]bool{}, snapped: map[string]string{}, snapNo: map[string]string{}, warranted: map[string]bool{}, warrantNo: map[string]string{}, a := &putActs{rec: handacts.Recorder{Ask: bus.Call, LeaseAway: leaseAway}, bus: bus, wake: wake, logf: logf, snapping: map[string]bool{}, snapped: map[string]string{}, snapNo: map[string]string{}, warranted: map[string]bool{}, warrantNo: map[string]string{},
said: map[string]string{}} said: map[string]string{}}
a.snapshot = a.snapshotTheBus a.snapshot = a.snapshotTheBus
return a return a
@@ -183,7 +186,10 @@ func (p *passer) putWorld(node, module string, put delivery.Put, manifest catalo
w.Composed, w.Granted = p.composedFor(node, module, put.FP) w.Composed, w.Granted = p.composedFor(node, module, put.FP)
if d := w.Composed; d != nil { if d := w.Composed; d != nil {
policy := d.Policy policy := d.Policy
if policy == "" { chosen, _ := assign.DecodePolicies(p.w.read(assign.PoliciesBucket))
if c, ok := chosen[module]; ok {
policy, _, _, _ = catalogue.Upgrade(manifest, &c) // a person's choice at the assigner (ADR 0565 §7)
} else if policy == "" {
policy, _, _ = catalogue.DerivedUpgrade(manifest) policy, _, _ = catalogue.DerivedUpgrade(manifest)
} }
w.Recorded = policy == catalogue.PolicyRecord w.Recorded = policy == catalogue.PolicyRecord
@@ -213,8 +219,7 @@ func (p *passer) putWorld(node, module string, put delivery.Put, manifest catalo
// composedFor is the composition of module for node at fp and whether the grant at fp names the node's accounts: the one // composedFor is the composition of module for node at fp and whether the grant at fp names the node's accounts: the one
// place a put reads them (ADR 0546 clause 4). Today their writer is the controller (mesh-controller_composed and // place a put reads them (ADR 0546 clause 4). Today their writer is the controller (mesh-controller_composed and
// mesh-controller_grants); when composing and grants move out of the controller into delivery's own parts, this reads them // mesh-controller_grants); the grants are the grants part's while the controller's lease is handed away (grantsBucket).
// from their new owner and nothing else changes.
func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest, bool) { func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest, bool) {
var out *delivery.ComposedDigest var out *delivery.ComposedDigest
if d, ok := p.w.composed(node, module, fp); ok { if d, ok := p.w.composed(node, module, fp); ok {
@@ -222,7 +227,7 @@ func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest,
} }
granted := false granted := false
if gkey, err := delivery.ControllerGrantKey(module, fp); err == nil { if gkey, err := delivery.ControllerGrantKey(module, fp); err == nil {
if raw := p.w.value(delivery.ControllerGrantsBucket, gkey); raw != nil { if raw := p.w.value(p.w.grantsBucket(), gkey); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil { if g, err := delivery.DecodeControllerGrant(raw); err == nil {
granted = g.NamesNode(node, module) granted = g.NamesNode(node, module)
} }
@@ -382,8 +387,9 @@ func (a *putActs) snapshotTheBus(ctx context.Context, node, module string) (stri
} }
} }
// warrant records once, in the controller's hand-act log, that the bus's want was written on the operator's warrant: the // warrant records once, in the hand-act log (the seat mesh-hand-acts', or the controller's while that seat has no holder and
// router's record of the teller's ask says who chose and how (ADR 0274). // the controller holds its lease, ADR 0565 §11), that the bus's want was written on the operator's warrant: the router's
// record of the teller's ask says who chose and how (ADR 0274).
func (a *putActs) warrant(id, ask string) { func (a *putActs) warrant(id, ask string) {
a.mu.Lock() a.mu.Lock()
if a.warranted[id] { if a.warranted[id] {
@@ -393,10 +399,10 @@ func (a *putActs) warrant(id, ask string) {
a.warranted[id] = true a.warranted[id] = true
a.mu.Unlock() a.mu.Unlock()
go func() { go func() {
if _, err := a.bus.Call("seat:"+catalogue.ControllerSeatName+".warranted", map[string]any{"asker": delivery.TellerModule, "ask": ask}); err != nil { if _, err := a.rec.RecordWarrant(delivery.TellerModule, ask); err != nil {
a.logf("the put's warrant %s is not recorded in the hand-act log: %v", ask, err) a.logf("the put's warrant %s is not recorded in the hand-act log: %v", ask, err)
a.mu.Lock() a.mu.Lock()
a.warrantNo[id] = fmt.Sprintf("the warrant %s is not recorded in the controller's hand-act log: %v", ask, err) a.warrantNo[id] = fmt.Sprintf("the warrant %s is not recorded in the hand-act log: %v", ask, err)
a.mu.Unlock() a.mu.Unlock()
a.wake() a.wake()
} }
@@ -21,6 +21,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/compose/frombus" "git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/compose/judge" "git.novox.be/novox/mesh-sdk/go/compose/judge"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-assigner/internal/assign" "mesh-delivery-assigner/internal/assign"
@@ -67,6 +68,8 @@ type seatActs struct {
export map[string]exportCheck export map[string]exportCheck
// calls is, by pair (node and module, or node and provision for a pin), what the last call for it came to. // calls is, by pair (node and module, or node and provision for a pin), what the last call for it came to.
calls map[string]string calls map[string]string
// rec records the seat's acts in the hand-act log (ADR 0565 §11).
rec handacts.Recorder
} }
// exportCheck is the export read again for one proposal: running, its answer, or why it was not read. // exportCheck is the export read again for one proposal: running, its answer, or why it was not read.
@@ -76,8 +79,9 @@ type exportCheck struct {
err string err string
} }
func newSeatActs(bus caller, wake func(), logf func(string, ...any)) *seatActs { func newSeatActs(bus caller, wake func(), logf func(string, ...any), leaseAway func() bool) *seatActs {
return &seatActs{bus: bus, wake: wake, logf: logf, export: map[string]exportCheck{}, calls: map[string]string{}} return &seatActs{bus: bus, wake: wake, logf: logf, export: map[string]exportCheck{}, calls: map[string]string{},
rec: handacts.Recorder{Ask: bus.Call, LeaseAway: leaseAway}}
} }
// exportOf is the export read again for a proposal, asked once in the background. // exportOf is the export read again for a proposal, asked once in the background.
@@ -114,26 +118,31 @@ func (a *seatActs) forget(id string) {
a.mu.Unlock() a.mu.Unlock()
} }
// record records once, in the controller's hand-act log, that placement was written on the operator's warrant (§5): the // record records once, in the hand-act log, that placement was written on the operator's warrant (§5): the router's
// router's record of the teller's ask says who chose and how (ADR 0274). // record of the teller's ask says who chose and how (ADR 0274). The log is the seat mesh-hand-acts' from its delivery, and
// the controller's while that seat has no holder and the controller holds its lease (ADR 0565 §11, handacts.Recorder).
func (a *seatActs) record(warrant string) { func (a *seatActs) record(warrant string) {
if a == nil || warrant == "" { if a == nil || warrant == "" {
return return
} }
ask, _, _ := strings.Cut(warrant, "@") ask, _, _ := strings.Cut(warrant, "@")
go func() { go func() {
if _, err := a.bus.Call("seat:"+catalogue.ControllerSeatName+".warranted", map[string]any{"asker": delivery.TellerModule, "ask": ask}); err != nil { if taken, err := a.rec.RecordWarrant(delivery.TellerModule, ask); err != nil {
a.logf("the placement's warrant %s is not recorded in the hand-act log: %v", ask, err) a.logf("the placement's warrant %s is not recorded in the hand-act log: %v", ask, err)
} else {
a.logf("the placement's warrant %s is recorded in the log of %s", ask, taken.Log)
} }
}() }()
} }
// handAct records a hand-back in the controller's hand-act log (§7). // handAct records a hand-back in the hand-act log (§7), as record does.
func (a *seatActs) handAct(prop delivery.PlacementProposal) { func (a *seatActs) handAct(prop delivery.PlacementProposal) {
go func() { go func() {
if _, err := a.bus.Call(handActCall, map[string]any{"what": "mesh-placement hand-back of " + strings.Join(prop.Nodes(), ", "), if taken, err := a.rec.Record(handacts.Request{What: "mesh-placement hand-back of " + strings.Join(prop.Nodes(), ", "),
"why": prop.Why, "cause": "asked by " + prop.By + ", approved by the operator"}); err != nil { Why: prop.Why, Cause: "asked by " + prop.By + ", approved by the operator"}); err != nil {
a.logf("the hand-back %s is not recorded in the hand-act log: %v", prop.ID, err) a.logf("the hand-back %s is not recorded in the hand-act log: %v", prop.ID, err)
} else {
a.logf("the hand-back %s is recorded in the log of %s", prop.ID, taken.Log)
} }
}() }()
} }
@@ -524,7 +533,7 @@ func (p *passer) layAgain(ctx context.Context, prop delivery.PlacementProposal,
held, recorded := p.lays(ctx) held, recorded := p.lays(ctx)
again, laid, findings := assign.Propose(assign.ProposeIn{ID: prop.ID, By: prop.By, Why: prop.Why, At: prop.At, Changes: prop.Changes, again, laid, findings := assign.Propose(assign.ProposeIn{ID: prop.ID, By: prop.By, Why: prop.Why, At: prop.At, Changes: prop.Changes,
Current: current, Revisions: revs, Held: held, Recorded: recorded, Current: current, Revisions: revs, Held: held, Recorded: recorded,
Seed: prop.Seed, ExportDigest: prop.ExportDigest}) Seed: prop.Seed, ExportDigest: prop.ExportDigest, Leased: p.w.handedAway()})
if len(findings) > 0 { if len(findings) > 0 {
p.endProposal(ctx, prop, why+", and laid over the keys now it is refused: "+strings.Join(findings, "; ")) p.endProposal(ctx, prop, why+", and laid over the keys now it is refused: "+strings.Join(findings, "; "))
return return
@@ -661,7 +670,7 @@ func (p *passer) followWrite(prop delivery.PlacementProposal, answer delivery.An
} }
p.seat.record(answer.Warrant) p.seat.record(answer.Warrant)
p.sayChanged(prop, answer, changes) p.sayChanged(prop, answer, changes)
if p.seat != nil && !assign.Leased && !prop.Seed { if p.seat != nil && !p.w.handedAway() && !prop.Seed {
p.seat.callAll(assign.ControllerCalls(before, written, prop.ID, answer.Warrant)) p.seat.callAll(assign.ControllerCalls(before, written, prop.ID, answer.Warrant))
} }
} }
@@ -729,7 +738,7 @@ func (p *passer) proposeSeed(ctx context.Context, was delivery.PlacementProposal
now := p.clock() now := p.clock()
seed, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: was.By, Why: was.Why, At: now, seed, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: was.By, Why: was.Why, At: now,
Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{}, Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{},
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export)}) Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export), Leased: p.w.handedAway()})
if len(findings) > 0 { if len(findings) > 0 {
p.logf("the new export is not proposed as the seed: %s", strings.Join(findings, "; ")) p.logf("the new export is not proposed as the seed: %s", strings.Join(findings, "; "))
return return
@@ -4,11 +4,13 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"slices"
"sort" "sort"
"strings" "strings"
"time" "time"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -214,7 +216,7 @@ func (s *seatTools) doPropose(args map[string]any, by string) (any, error) {
} }
now := s.now() now := s.now()
p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now, p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now,
Changes: changes, Current: current, Revisions: revs, Held: held, Recorded: recorded}) Changes: changes, Current: current, Revisions: revs, Held: held, Recorded: recorded, Leased: s.leased()})
if len(findings) > 0 { if len(findings) > 0 {
return nil, fmt.Errorf("the proposal is refused: %s", strings.Join(findings, "; ")) return nil, fmt.Errorf("the proposal is refused: %s", strings.Join(findings, "; "))
} }
@@ -298,7 +300,7 @@ func (s *seatTools) doAdopt(args map[string]any, by string) (any, error) {
now := s.now() now := s.now()
p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now, p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now,
Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{}, Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{},
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export)}) Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export), Leased: s.leased()})
if len(findings) > 0 { if len(findings) > 0 {
return nil, fmt.Errorf("the seed is refused: %s", strings.Join(findings, "; ")) return nil, fmt.Errorf("the seed is refused: %s", strings.Join(findings, "; "))
} }
@@ -308,14 +310,20 @@ func (s *seatTools) doAdopt(args map[string]any, by string) (any, error) {
return proposed{Proposal: p, Follow: followProposal(p.ID)}, nil return proposed{Proposal: p, Follow: followProposal(p.ID)}, nil
} }
// handActCall records a hand act in the controller's hand-act log. // leased says the controller's lease is handed away (mesh-sdk buscompose.LeaseHandedAway, ADR 0561 §3; issue 774): the lease
const handActCall = "seat:" + catalogue.ControllerSeatName + ".hand-act" // write of ADR 0547 that ends keeping the controller in step (ADR 0567 §8). A lease not read, a controller's, expired or
// unreadable, is not handed away.
func (s *seatTools) leased() bool {
raw, _, found, err := s.st.get(buscompose.ControllerLease, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, err)
return away
}
// doHandBack is the verb hand-back (§7): the operator's alone, so it is asked as a proposal is. It writes a hand-back // doHandBack is the verb hand-back (§7): the operator's alone, so it is asked as a proposal is. It writes a hand-back
// proposal naming every key at its revision; the assigner raises its authorising ask and, on its warrant, deletes the keys, // proposal naming every key at its revision; the assigner raises its authorising ask and, on its warrant, deletes the keys,
// records the hand act and says placement-changed; every node returns to the controller's compositions. // records the hand act and says placement-changed; every node returns to the controller's compositions.
func (s *seatTools) doHandBack(args map[string]any, by string) (any, error) { func (s *seatTools) doHandBack(args map[string]any, by string) (any, error) {
if assign.Leased { if s.leased() {
return nil, errors.New("placement is the mesh's alone since the lease write: there is no controller to hand it back to") return nil, errors.New("placement is the mesh's alone since the lease write: there is no controller to hand it back to")
} }
why := arg(args, "why") why := arg(args, "why")
@@ -368,6 +376,9 @@ type historyEntry struct {
// doPlacement is the verb placement (§10, §6): one node's key, one module's nodes, or every key; with history, the // doPlacement is the verb placement (§10, §6): one node's key, one module's nodes, or every key; with history, the
// changes in order from the revisions the state keeps. // changes in order from the revisions the state keeps.
func (s *seatTools) doPlacement(args map[string]any) (any, error) { func (s *seatTools) doPlacement(args map[string]any) (any, error) {
if arg(args, "seats") == "true" {
return s.seats(arg(args, "node"))
}
all, revs, err := s.placements() all, revs, err := s.placements()
if err != nil { if err != nil {
return nil, err return nil, err
@@ -531,3 +542,32 @@ func (s *seatTools) tools() []stdio.Tool {
} }
return out return out
} }
// seats is the verb placement with seats (ADR 0571 §4): who holds each seat and on which node, as the state seat-holders
// holds it; with node, the keys of seats held on that node.
func (s *seatTools) seats(node string) (any, error) {
keys, err := s.st.keys(delivery.SeatHoldersState)
if err != nil {
return nil, fmt.Errorf("seat-holders cannot be listed: %w", err)
}
sort.Strings(keys)
out := []delivery.SeatHolders{}
for _, key := range keys {
raw, _, found, err := s.st.get(delivery.SeatHoldersState, key)
if err != nil {
return nil, fmt.Errorf("seat-holders %s cannot be read: %w", key, err)
}
if !found {
continue
}
var sh delivery.SeatHolders
if json.Unmarshal(raw, &sh) != nil {
continue
}
if node != "" && sh.Node != node && !slices.ContainsFunc(sh.Holders, func(h delivery.SeatHolder) bool { return h.Node == node }) {
continue
}
out = append(out, sh)
}
return map[string]any{"seats": out, "holds": len(keys)}, nil
}
@@ -1,6 +1,7 @@
package main package main
import ( import (
"context"
"encoding/json" "encoding/json"
"os" "os"
"slices" "slices"
@@ -9,8 +10,12 @@ import (
"time" "time"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
"mesh-delivery-assigner/internal/assign"
) )
func (s memStore) del(state, key string) error { func (s memStore) del(state, key string) error {
@@ -142,9 +147,250 @@ func TestTheSeatsVerbs(t *testing.T) {
if hb := got.(proposed).Proposal; err != nil || !hb.HandBack || hb.Base["anchor"] != uint64(len(raw)) || len(st[delivery.PlacementState].values) != 1 { if hb := got.(proposed).Proposal; err != nil || !hb.HandBack || hb.Base["anchor"] != uint64(len(raw)) || len(st[delivery.PlacementState].values) != 1 {
t.Errorf("hand-back = %+v, %v", got, err) t.Errorf("hand-back = %+v, %v", got, err)
} }
// Issue 774 (ADR 0567 §8): once the controller's lease is handed away a change its verbs cannot take is proposed, and
// there is no controller to hand placement back to.
st[buscompose.ControllerLease] = &memKV{values: map[string][]byte{buscompose.ControllerLeaseKey: []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`)}}
if _, err := s.doPropose(map[string]any{"why": "hold", "changes": `[{"op":"hold","node":"anchor","module":"web","claim":"node-shell","scope":"node"}]`}, "op"); err != nil {
t.Errorf("a hold after the lease write: %v", err)
}
if _, err := s.doHandBack(map[string]any{"why": "back"}, "op"); err == nil || !strings.Contains(err.Error(), "since the lease write") {
t.Errorf("a hand-back after the lease write: %v", err)
}
if tools := s.tools(); len(tools) != 7 || tools[2].Name != "mesh-placement.propose" || tools[2].Run == nil || tools[2].Input == nil { if tools := s.tools(); len(tools) != 7 || tools[2].Name != "mesh-placement.propose" || tools[2].Run == nil || tools[2].Input == nil {
t.Errorf("the seat's verbs as served: %+v", tools) t.Errorf("the seat's verbs as served: %+v", tools)
} }
t.Run("1953/D4-D6: the seat mesh-wants' upgrade and bus", wantsVerbs)
t.Run("1955/L7 (ADR 0571 §6): rule 7 reads the controller's composition until the lease write, the node's own after; an unplaced node is said", func(t *testing.T) {
w := newWorld()
fp := delivery.SourcePrefix2 + strings.Repeat("2", 64)
ckey, _ := delivery.ComposedKey("ace", "web", fp)
cv, _ := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("c", 64), Judgement: delivery.JudgedNeither,
Policy: catalogue.PolicyRoll, PolicyFrom: catalogue.FromDefault})
w.putEntry(delivery.ControllerComposedBucket, ckey, cv, 1, time.Now())
pl, _ := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: "ace", Modules: []delivery.PlacedModule{{Module: "web"}}})
pkey, _ := delivery.PlacementKey("ace")
w.putEntry(delivery.PlacementBucket, pkey, pl, 1, time.Now())
for node, digest := range map[string]string{"ace": "sha256:" + strings.Repeat("a", 64), "g14": ""} {
hn, _ := delivery.EncodeHaveNode(delivery.HaveNode{Composed: digest})
hk, _ := delivery.HaveNodeKey(node)
w.putEntry(delivery.HaveBucket, hk, hn, 1, time.Now())
}
if d, ok := w.composed("ace", "web", fp); !ok || d.Digest != "sha256:"+strings.Repeat("c", 64) || len(w.unplacedNodes()) != 0 {
t.Fatalf("before the lease write: %+v %v, unplaced %v", d, ok, w.unplacedNodes())
}
w.putEntry(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(`{"handed-to":"mesh-vault","handed-by":"operator"}`), 2, time.Now())
if d, ok := w.composed("ace", "web", fp); !ok || d.Digest != "sha256:"+strings.Repeat("a", 64) {
t.Errorf("after the lease write, ace: %+v %v", d, ok)
}
if _, ok := w.composed("g14", "web", fp); ok || !slices.Equal(w.unplacedNodes(), []string{"g14"}) {
t.Errorf("after the lease write, g14 unplaced: composed %v, unplaced %v", ok, w.unplacedNodes())
}
findings := map[string][]byte{}
p := &passer{w: w, logf: func(string, ...any) {}, findings: assign.Findings{
Put: func(k string, v []byte) error { findings[k] = v; return nil },
Delete: func(k string) error { delete(findings, k); return nil }}}
p.unplacedFindings()
if _, ok := findings[assign.UnplacedKey("g14")]; !ok || len(findings) != 1 {
t.Fatalf("findings = %v", findings)
}
pl2, _ := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: "g14", Modules: []delivery.PlacedModule{{Module: "web"}}})
pkey2, _ := delivery.PlacementKey("g14")
w.putEntry(delivery.PlacementBucket, pkey2, pl2, 1, time.Now())
p.unplacedFindings()
if len(findings) != 0 {
t.Errorf("placed again: findings %v", findings)
}
})
t.Run("1955/L7 (ADR 0571 §4): seat-holders is written from placement, only the keys a change moves, and answered by placement seats", func(t *testing.T) {
writes := 0
kv := &countingKV{memKV: memKV{values: map[string][]byte{}}, writes: &writes}
p := &passer{w: newWorld(), logf: t.Logf, kvs: buckets{delivery.SeatHoldersBucket: kv}}
place := func(node string, holds ...delivery.Holding) {
pl, err := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: node, Modules: []delivery.PlacedModule{{Module: "web"}, {Module: "nats"}}, Holds: holds})
if err != nil {
t.Fatal(err)
}
k, _ := delivery.PlacementKey(node)
p.w.putEntry(delivery.PlacementBucket, k, pl, 1, time.Now())
}
place("ace", delivery.Holding{Claim: "mesh-broker", Scope: "mesh", Module: "nats"}, delivery.Holding{Claim: "node-shell", Scope: "node", Module: "web"})
place("g14", delivery.Holding{Claim: "node-shell", Scope: "node", Module: "web"})
ctx := context.Background()
p.seatHolders(ctx)
if writes != 3 || kv.values["mesh-broker"] == nil || kv.values["node-shell.ace"] == nil || kv.values["node-shell.g14"] == nil {
t.Fatalf("%d writes: %v", writes, kv.values)
}
p.seatHolders(ctx)
if writes != 3 {
t.Errorf("an unchanged placement wrote %d keys", writes-3)
}
place("g14") // g14 holds nothing now
p.seatHolders(ctx)
if writes != 3 || kv.values["node-shell.g14"] != nil || kv.values["node-shell.ace"] == nil {
t.Errorf("after g14's change: %d writes, %v", writes, kv.values)
}
st := memStore{delivery.SeatHoldersState: &memKV{values: kv.values}}
out, err := newSeatTools(st, nil).seats("ace")
if err != nil || len(out.(map[string]any)["seats"].([]delivery.SeatHolders)) != 2 {
t.Errorf("seats on ace = %+v, %v", out, err)
}
})
t.Run("1954/D1: a failed provision is the assigner's finding until it is provided", func(t *testing.T) {
findings := map[string][]byte{}
p := &passer{w: newWorld(), logf: func(string, ...any) {}, kvs: buckets{provisions.ProvisionsBucket: nil}, findings: assign.Findings{
Put: func(k string, v []byte) error { findings[k] = v; return nil },
Delete: func(k string) error { delete(findings, k); return nil }}}
failed, _ := provisions.Encode(provisions.Provision{Provision: "db", Consumer: provisions.End{Module: "photos", Node: "ace"},
Provider: provisions.End{Module: "postgres", Node: "novox"}, State: provisions.Failed, Why: "the role exists already"})
p.w.putEntry(provisions.ProvisionsBucket, "ace.photos.db", failed, 1, time.Now())
p.kvs[provisions.ProvisionsBucket] = &memKV{}
of := p.provisionsOf()
if got := of("photos")("ace"); got["ace.photos.db"].State != provisions.Failed {
t.Fatalf("provisions = %+v", got)
}
if _, ok := findings[assign.ProvisionFailedKey("ace.photos.db")]; !ok || len(findings) != 1 {
t.Fatalf("findings = %v", findings)
}
provided, _ := provisions.Encode(provisions.Provision{Provision: "db", Consumer: provisions.End{Module: "photos", Node: "ace"},
Provider: provisions.End{Module: "postgres", Node: "novox"}, State: provisions.Provided})
p.w.putEntry(provisions.ProvisionsBucket, "ace.photos.db", provided, 2, time.Now())
p.provisionsOf()
if len(findings) != 0 {
t.Errorf("findings after provided = %v", findings)
}
// A required provision with no key: pulsed with the bound first, said once the pulse expired, ended once keyed.
ctx := context.Background()
p.kvs[delivery.PlacementPulsesBucket] = &memKV{values: map[string][]byte{}}
gone := map[string]bool{"ace.photos.s3": true}
p.provisionsMissing(ctx, gone)
if !p.pulsing("provision-missing.ace.photos.s3") || len(findings) != 0 {
t.Fatalf("first seen: pulsing %v, findings %v", p.pulsing("provision-missing.ace.photos.s3"), findings)
}
p.provisionsMissing(ctx, gone)
if len(findings) != 0 {
t.Fatal("said before its bound")
}
p.w.put(delivery.PlacementPulsesBucket, "provision-missing.ace.photos.s3", nil) // the server expired it
p.provisionsMissing(ctx, gone)
if _, ok := findings[assign.ProvisionMissingKey("ace.photos.s3")]; !ok {
t.Fatalf("past its bound: findings %v", findings)
}
p.provisionsMissing(ctx, map[string]bool{})
if len(findings) != 0 {
t.Errorf("keyed again: findings %v", findings)
}
})
}
// wantsVerbs (novox/hq ADR 0565 §7, §8, §9): upgrade answers each module's policy and its source; a person's choice is
// written with who and why, recorded as a hand act in the seat's log and said as policy-changed, and default takes it back;
// the bus is refused; compare lists the modules the controller writes another policy for; the process writes policy-split
// for a split and deletes it once they agree; bus names the node behind the indexer's newest.
func wantsVerbs(t *testing.T) {
fpOld, fpNew := delivery.SourcePrefix2+strings.Repeat("1", 64), delivery.SourcePrefix2+strings.Repeat("2", 64)
st := memStore{}
for _, state := range []string{modulesState, manifestsState, composedState, policiesState, haveState, putsState, "findings"} {
st[state] = &memKV{values: map[string][]byte{}}
}
manifest := func(name, extra string) {
raw := `{"module":"` + name + `","version":"1"` + extra + `}`
digest := delivery.ManifestDigest([]byte(raw))
st[manifestsState].values[digest], _ = delivery.EncodeManifest(delivery.Manifest{V: delivery.Version, Module: name, JSON: raw})
st[modulesState].values[name], _ = delivery.EncodeModule(delivery.Module{V: delivery.Version, FP2: fpNew, Manifest: digest, Repository: "novox/x", Commit: "c"})
}
manifest("web", "")
manifest("photos", `,"upgrade":{"policy":"record","why":"a person checks the albums"}`)
manifest("nats", `,"provides":[{"name":"mesh-bus","scope":"mesh"}]`)
for node, fp := range map[string]string{"anchor": fpOld, "laptop": fpNew} {
st[haveState].values[node+".nats"], _ = delivery.EncodeHave(delivery.Have{FP: fp, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
}
composed := func(module, policy string) {
key, _ := delivery.ComposedKey("anchor", module, fpNew)
st[composedState].values[key], _ = json.Marshal(delivery.ComposedDigest{V: delivery.Version, Digest: "sha256:" + strings.Repeat("c", 64),
Judgement: delivery.JudgedNeither, Policy: policy, PolicyFrom: catalogue.FromPerson, PolicyWhy: "chosen at the controller"})
}
composed("web", catalogue.PolicyRecord)
composed("photos", catalogue.PolicyRecord)
var asked []string
var said []string
w := newWantsTools(st, func(key string, body any) (json.RawMessage, error) {
asked = append(asked, key)
return json.RawMessage(`{"ok":true}`), nil
}, func(event, id string, body any) error { said = append(said, event+" "+id); return nil })
policyOf := func(module string) policyRow {
out, err := w.doUpgrade(map[string]any{"module": module}, "op")
if err != nil {
t.Fatal(err)
}
return out.(map[string]any)["modules"].([]policyRow)[0]
}
if p := policyOf("photos"); p.Policy != catalogue.PolicyRecord || p.From != catalogue.FromModule {
t.Errorf("photos = %+v", p)
}
if p := policyOf("nats"); p.Policy != catalogue.PolicyRecord || p.From != catalogue.FromBus {
t.Errorf("nats = %+v", p)
}
// compare: web rolls at the assigner and records at the controller.
out, _ := w.doUpgrade(map[string]any{"compare": "true"}, "op")
if splits := out.(map[string]any)["splits"].([]assign.Split); len(splits) != 1 || splits[0].Module != "web" || out.(map[string]any)["findings"] == nil {
t.Errorf("splits = %+v", splits)
}
// The process says the split as its finding, and deletes it once they agree.
findings := map[string][]byte{}
p := &passer{w: newWorld(), logf: func(string, ...any) {}, findings: assign.Findings{
Put: func(k string, v []byte) error { findings[k] = v; return nil },
Delete: func(k string) error { delete(findings, k); return nil },
Keys: func() ([]string, error) { return []string{"module.gone.policy-split"}, nil }}}
for k, v := range st[composedState].values {
p.w.putEntry(delivery.ControllerComposedBucket, k, v, 1, time.Now())
}
man := map[string]catalogue.Manifest{}
for _, name := range []string{"web", "photos"} {
raw, _, _, _ := st.get(modulesState, name)
mod, _ := delivery.DecodeModule(raw)
mraw, _, _, _ := st.get(manifestsState, mod.Manifest)
kept, _ := delivery.DecodeManifest(mraw)
man[name], _ = catalogue.ParseManifest([]byte(kept.JSON))
}
findings["module.gone.policy-split"] = []byte("{}")
p.policySplits(man, nil)
if _, ok := findings[assign.PolicySplitKey("web")]; !ok || len(findings) != 1 {
t.Errorf("findings = %v, want web's split alone (a stale one deleted)", findings)
}
// a person's record for web: written, recorded, said; the split is gone.
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "record", "why": "the operator checks it"}, "person.jochen"); err != nil {
t.Fatal(err)
}
chosen, _ := assign.DecodePolicies(st[policiesState].values)
if c := chosen["web"]; c.Policy != catalogue.PolicyRecord || c.By != "person.jochen" || !slices.Contains(asked, "seat:mesh-hand-acts.hand-act") ||
len(said) != 1 || !strings.HasPrefix(said[0], "policy-changed policy-changed.web.") {
t.Errorf("chosen %+v, asked %v, said %v", c, asked, said)
}
p.policySplits(man, chosen)
if len(findings) != 0 {
t.Errorf("findings after the choice = %v", findings)
}
if _, err := w.doUpgrade(map[string]any{"module": "nats", "policy": "roll", "why": "x"}, "op"); err == nil {
t.Error("the bus was given a policy")
}
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "roll"}, "op"); err == nil {
t.Error("a policy without why was set")
}
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "default", "why": "back to the module"}, "op"); err != nil || len(st[policiesState].values) != 0 {
t.Errorf("default left %v (%v)", st[policiesState].values, err)
}
// bus: the node behind the indexer's newest.
b, err := w.doBus(nil)
if err != nil {
t.Fatal(err)
}
nodes := b.(map[string]any)["nodes"].([]busNode)
if b.(map[string]any)["module"] != "nats" || len(nodes) != 2 || !nodes[0].Behind || nodes[1].Behind || !slices.Contains(asked, "seat:mesh-hand-acts.hand-acts") {
t.Errorf("bus = %+v, asked %v", b, asked)
}
if tools := w.tools(); len(tools) != 2 || tools[0].Name != "mesh-wants.upgrade" || tools[1].Run == nil {
t.Errorf("the seat's verbs as served: %+v", tools)
}
} }
// readManifest is the module's manifest, read as the controller reads it. // readManifest is the module's manifest, read as the controller reads it.
@@ -155,3 +401,14 @@ func readManifest() (catalogue.Manifest, error) {
} }
return catalogue.ParseManifest(raw) return catalogue.ParseManifest(raw)
} }
// countingKV is a memKV that counts its puts.
type countingKV struct {
memKV
writes *int
}
func (c *countingKV) Put(ctx context.Context, key string, value []byte) (uint64, error) {
*c.writes++
return c.memKV.Put(ctx, key, value)
}
@@ -351,7 +351,12 @@ func serveTools() error {
time.Sleep(wait) time.Sleep(wait)
} }
}() }()
return stdio.Serve("", append(s.tools(), newSeatTools(runtimeStore{}, stdio.Ask).tools()...)) tools := append(s.tools(), newSeatTools(runtimeStore{}, stdio.Ask).tools()...)
tools = append(tools, newWantsTools(runtimeStore{}, stdio.Ask, func(e, id string, b any) error {
// The seat's event, said on the seat: a claimed seat's event is never the module's own.
return stdio.SeatPublish(wantsEventSubject(e), b, id)
}).tools()...)
return stdio.Serve("", tools)
} }
// shortFP is a fingerprint's first twelve hex digits. // shortFP is a fingerprint's first twelve hex digits.
@@ -0,0 +1,300 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/health"
"mesh-delivery-assigner/internal/assign"
)
// The seat mesh-wants' verbs (novox/hq ADR 0565 §7, §8, §9, §15; issue 1953 D4-D6), served through the tool runner beside the
// put tools and mesh-placement's verbs. `upgrade` reads and sets a person's upgrade policy, kept in the state `policies`
// (each set recorded as a hand act and said as policy-changed), and with compare lists every module whose policy differs from
// the one the controller writes beside its compositions; `bus` says what a bus upgrade would do. The assigner's process reads
// `policies` when it judges a move and writes the finding policy-split while a split stands.
// The states the verbs read, as the runtime names them.
const (
policiesState = delivery.AssignerPoliciesState
manifestsState = delivery.IndexerModule + ".manifests"
composedState = "mesh-controller.composed"
// PolicyChangedEvent is the seat's event of a person's policy set or taken back.
PolicyChangedEvent = "policy-changed"
// busStepCause is the cause the bus's step is recorded under in the hand-act log.
busStepCause = "bus-upgrade"
)
// wantsTools serves the seat's verbs over a store, asking seats through ask and saying events through emit.
type wantsTools struct {
st seatStore
ask func(key string, body any) (json.RawMessage, error)
emit func(event, id string, body any) error
now func() time.Time
}
// wantsEventSubject is the subject of one of the seat's events.
func wantsEventSubject(event string) string {
return "mesh.seat." + catalogue.WantsSeat + ".event." + event
}
func newWantsTools(st seatStore, ask func(string, any) (json.RawMessage, error), emit func(event, id string, body any) error) *wantsTools {
return &wantsTools{st: st, ask: ask, emit: emit, now: time.Now}
}
// leased says the controller's lease is handed away.
func (s *wantsTools) leased() bool {
raw, _, found, err := s.st.get(buscompose.ControllerLease, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, err)
return away
}
// values reads every key of a state.
func (s *wantsTools) values(state string) (map[string][]byte, error) {
keys, err := s.st.keys(state)
if err != nil {
return nil, fmt.Errorf("%s cannot be listed: %w", state, err)
}
out := map[string][]byte{}
for _, k := range keys {
raw, _, found, err := s.st.get(state, k)
if err != nil {
return nil, fmt.Errorf("%s %s cannot be read: %w", state, k, err)
}
if found {
out[k] = raw
}
}
return out, nil
}
// manifests is every module the indexer holds, with its trunk manifest and fingerprint.
func (s *wantsTools) manifests() (map[string]catalogue.Manifest, map[string]delivery.Module, error) {
mods, err := s.values(modulesState)
if err != nil {
return nil, nil, err
}
out, held := map[string]catalogue.Manifest{}, map[string]delivery.Module{}
for name, raw := range mods {
m, err := delivery.DecodeModule(raw)
if err != nil {
continue
}
held[name] = m
if m.Manifest == "" {
continue
}
mraw, _, found, err := s.st.get(manifestsState, m.Manifest)
if err != nil || !found {
continue
}
kept, err := delivery.DecodeManifest(mraw)
if err != nil {
continue
}
if man, err := catalogue.ParseManifest([]byte(kept.JSON)); err == nil {
out[name] = man
}
}
return out, held, nil
}
// policyRow is one module's policy as upgrade answers it.
type policyRow struct {
Module string `json:"module"`
Policy string `json:"policy"`
From string `json:"from"`
Why string `json:"why,omitempty"`
By string `json:"by,omitempty"`
}
// doUpgrade is the verb upgrade: read every module's policy or one's; with policy and why set a person's choice, or take it
// back with default; with compare list the splits.
func (s *wantsTools) doUpgrade(args map[string]any, by string) (any, error) {
manifests, _, err := s.manifests()
if err != nil {
return nil, err
}
raw, err := s.values(policiesState)
if err != nil {
return nil, err
}
chosen, unreadable := assign.DecodePolicies(raw)
if arg(args, "compare") == "true" {
composed, err := s.values(composedState)
if err != nil {
return nil, err
}
splits := assign.Splits(manifests, chosen, assign.ControllerPolicies(composed))
if splits == nil {
splits = []assign.Split{}
}
out := map[string]any{"splits": splits, "empty": len(splits) == 0}
// The assigner's findings, its said conditions and its policy splits as the condition keeper reads them (§8, §15).
if found, err := s.values(health.FindingsState); err != nil {
out["findings-unread"] = err.Error()
} else {
keys := []string{}
for k := range found {
keys = append(keys, k)
}
sort.Strings(keys)
out["findings"] = keys
}
return out, nil
}
module, policy := arg(args, "module"), arg(args, "policy")
if policy != "" {
return s.setPolicy(module, policy, arg(args, "why"), by, manifests)
}
var rows []policyRow
for name, m := range manifests {
if module != "" && name != module {
continue
}
p, from, why, who := assign.PolicyOf(m, chosen)
rows = append(rows, policyRow{Module: name, Policy: p, From: from, Why: why, By: who})
}
if module != "" && len(rows) == 0 {
return nil, fmt.Errorf("the indexer holds no manifest of %s", module)
}
sort.Slice(rows, func(i, j int) bool { return rows[i].Module < rows[j].Module })
return map[string]any{"modules": rows, "unreadable": unreadable}, nil
}
// setPolicy writes a person's choice of a module's policy, or deletes it for default, records the act in the hand-act log
// and says policy-changed. The bus is never rolled: a choice for it is refused.
func (s *wantsTools) setPolicy(module, policy, why, by string, manifests map[string]catalogue.Manifest) (any, error) {
if module == "" {
return nil, errors.New("a policy is set for one module: name it")
}
m, ok := manifests[module]
if !ok {
return nil, fmt.Errorf("the indexer holds no manifest of %s", module)
}
if catalogue.ProvidesBus(m) {
return nil, fmt.Errorf("%s is the bus, which is never rolled whoever says so (novox/hq ADR 0236): its step is the put on the operator's warrant", module)
}
if why == "" {
return nil, errors.New("a person's policy says why")
}
if by == "" || by == stdio.Unproven {
return nil, errors.New("a person's policy names who chose it, and the runtime proved no caller")
}
what := fmt.Sprintf("mesh-wants.upgrade %s %s", module, policy)
if policy == "default" {
if err := s.st.del(policiesState, module); err != nil {
return nil, fmt.Errorf("the choice for %s cannot be taken back: %w", module, err)
}
} else {
p := catalogue.PersonPolicy{V: catalogue.PersonPolicyVersion, Module: module, Policy: policy, Why: why, By: by, At: s.now().UTC()}
if err := p.Check(); err != nil {
return nil, err
}
raw, _ := json.Marshal(p)
if err := s.st.put(policiesState, module, raw); err != nil {
return nil, fmt.Errorf("the choice for %s cannot be written: %w", module, err)
}
}
rec := handacts.Recorder{Ask: s.ask, LeaseAway: s.leased}
taken, rerr := rec.Record(handacts.Request{What: what, Why: why, Cause: "upgrade-policy", Verb: "mesh-wants.upgrade"})
recorded := taken.Log
if rerr != nil {
recorded = "not recorded: " + rerr.Error()
}
body := map[string]any{"module": module, "policy": policy, "why": why, "by": by}
said := ""
if s.emit != nil {
id := PolicyChangedEvent + "." + module + "." + strconv.FormatInt(s.now().UnixNano(), 10)
if err := s.emit(PolicyChangedEvent, id, body); err != nil {
said = "not said: " + err.Error()
}
}
return map[string]any{"set": body, "hand-act": recorded, "event": said}, nil
}
// busNode is one node's bus as bus answers it.
type busNode struct {
Node string `json:"node"`
FP string `json:"fp"`
Behind bool `json:"behind"`
}
// doBus is the verb bus: the fingerprint of the bus module each node runs against the indexer's newest, the nodes behind,
// the last bus steps of the hand-act log and the assigner's puts of the bus. Read-only.
func (s *wantsTools) doBus(map[string]any) (any, error) {
manifests, held, err := s.manifests()
if err != nil {
return nil, err
}
bus := ""
for name, m := range manifests {
if catalogue.ProvidesBus(m) {
bus = name
}
}
if bus == "" {
return nil, errors.New("the indexer holds no module that provides the bus")
}
newest := held[bus].FP2
haves, err := s.values(haveState)
if err != nil {
return nil, err
}
var nodes []busNode
for key, raw := range haves {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode || module != bus {
continue
}
h, err := delivery.DecodeHave(raw)
if err != nil || h.Phase == delivery.PhaseRemoved {
continue
}
nodes = append(nodes, busNode{Node: node, FP: h.FP, Behind: newest != "" && h.FP != newest})
}
sort.Slice(nodes, func(i, j int) bool { return nodes[i].Node < nodes[j].Node })
out := map[string]any{"module": bus, "newest": newest, "nodes": nodes}
if steps, err := s.ask("seat:"+handacts.Seat+".hand-acts", map[string]any{"cause": busStepCause, "days": 90}); err != nil {
out["steps-unread"] = err.Error()
} else {
out["steps"] = steps
}
puts, err := s.values(putsState)
if err != nil {
out["puts-unread"] = err.Error()
} else {
mine := map[string]json.RawMessage{}
for k, v := range puts {
if _, module, err := delivery.ParsePutKey(k); err == nil && module == bus {
mine[k] = v
}
}
out["puts"] = mine
}
return out, nil
}
// tools are the seat's verbs as the runtime serves them, each named <seat>.<verb>, with the seat's own schemas.
func (s *wantsTools) tools() []stdio.Tool {
seat, _ := catalogue.SeatNamed(catalogue.WantsSeat)
run := map[string]func(map[string]any) (any, error){
"upgrade": func(a map[string]any) (any, error) { return s.doUpgrade(a, stdio.CallerOf(a)) },
"bus": s.doBus,
}
var out []stdio.Tool
for _, v := range seat.Serves {
out = append(out, stdio.Tool{Name: catalogue.WantsSeat + "." + v.Name, Description: v.Description, Input: v.Input, Run: run[v.Name]})
}
return out
}
@@ -13,6 +13,7 @@ import (
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose" "git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose" "git.novox.be/novox/mesh-sdk/go/compose"
@@ -51,6 +52,8 @@ type world struct {
changes map[string]uint64 changes map[string]uint64
// meta is, by bucket and key, the revision and the time of the value held now, where the watch said them (putEntry). // meta is, by bucket and key, the revision and the time of the value held now, where the watch said them (putEntry).
meta map[string]entryMeta meta map[string]entryMeta
// trunkBuilt is, by module, its newest done trunk build in the builder's log (putBuilt).
trunkBuilt map[string]delivery.BuildLogEntry
} }
// entryMeta is a value's revision in its bucket and when it was written. // entryMeta is a value's revision in its bucket and when it was written.
@@ -68,7 +71,7 @@ type parsed struct {
func newWorld() *world { func newWorld() *world {
return &world{values: map[string]map[string][]byte{}, manifests: map[string]parsed{}, healthy: map[string][]string{}, changes: map[string]uint64{}, return &world{values: map[string]map[string][]byte{}, manifests: map[string]parsed{}, healthy: map[string][]string{}, changes: map[string]uint64{},
meta: map[string]entryMeta{}} meta: map[string]entryMeta{}, trunkBuilt: map[string]delivery.BuildLogEntry{}}
} }
// putEntry is put, keeping the value's revision and the time it was written. // putEntry is put, keeping the value's revision and the time it was written.
@@ -195,21 +198,38 @@ func (w *world) manifest(ctx context.Context, get getter, digest string) parsed
// assigned is each node's modules both the controller's assignments and the seat mesh-placement's placement name, in name order // assigned is each node's modules both the controller's assignments and the seat mesh-placement's placement name, in name order
// (ADR 0549 §5; the assignments alone before any placement was read), with every open assignment by node and module. // (ADR 0549 §5; the assignments alone before any placement was read), with every open assignment by node and module.
// Once the controller's lease is handed away it is the seat's placement alone, with no assignment (ADR 0549 §5, §6).
func (w *world) assigned() (map[string][]string, map[string]assignment) { func (w *world) assigned() (map[string][]string, map[string]assignment) {
byAssignment, all := w.assignedByController() // A lease handed away before any placement was read keeps the assignments as they last stood, never an empty mesh:
// the lease write's own check refuses that order (ADR 0547 §11).
placed, seeded := w.placed() placed, seeded := w.placed()
both, _ := assign.Placed(byAssignment, placed, seeded) byAssignment, all := w.assignedByController()
if away := w.handedAway(); away && seeded {
both, _ := assign.Placed(byAssignment, placed, seeded, away)
return both, map[string]assignment{}
}
both, _ := assign.Placed(byAssignment, placed, seeded, false)
return both, all return both, all
} }
// placementDiffers is every pair the controller's assignments and the seat's placement do not both name. // placementDiffers is every pair the controller's assignments and the seat's placement do not both name: none once
// the controller's lease is handed away, since the placement is then the one source.
func (w *world) placementDiffers() []assign.Pair { func (w *world) placementDiffers() []assign.Pair {
byAssignment, _ := w.assignedByController()
placed, seeded := w.placed() placed, seeded := w.placed()
_, differ := assign.Placed(byAssignment, placed, seeded) byAssignment, _ := w.assignedByController()
_, differ := assign.Placed(byAssignment, placed, seeded, w.handedAway())
return differ return differ
} }
// handedAway says the controller's lease is handed to a holder that is no controller (mesh-sdk buscompose.LeaseHandedAway,
// ADR 0561 §3): from then the seat mesh-placement's placement alone names where modules run (ADR 0549 §5). A lease not read (its
// bucket not granted or not there), a controller's, expired or unreadable, is not handed away.
func (w *world) handedAway() bool {
raw, found := w.read(buscompose.LeaseBucket)[buscompose.ControllerLeaseKey]
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, nil)
return away
}
// placed is each node's modules in the seat's placement (ADR 0567 §1), and whether any placement key was read: a key // placed is each node's modules in the seat's placement (ADR 0567 §1), and whether any placement key was read: a key
// that does not read places nothing on its node and is said by the indexer's own read tool. // that does not read places nothing on its node and is said by the indexer's own read tool.
func (w *world) placed() (map[string][]string, bool) { func (w *world) placed() (map[string][]string, bool) {
@@ -415,6 +435,11 @@ func (r worldReader) Served(ctx context.Context, node, module string, keys []str
func (w *world) change(ctx context.Context, manifests getter, c delivery.Candidate, trunk map[string]delivery.Module) (assign.Change, string) { func (w *world) change(ctx context.Context, manifests getter, c delivery.Candidate, trunk map[string]delivery.Module) (assign.Change, string) {
ch := assign.Change{Group: c.Group, Heads: c.HeadsHash, Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{}, ch := assign.Change{Group: c.Group, Heads: c.HeadsHash, Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{},
Twice: map[string][]string{}, Digests: map[string]string{}, Members: map[string]string{}} Twice: map[string][]string{}, Digests: map[string]string{}, Members: map[string]string{}}
heads := map[string]string{}
for _, p := range c.Pulls {
heads[fmt.Sprintf("%s#%d", p.Repository, p.Number)] = p.Head.Head
}
ch.Held = assign.HeldOf(w.read(keptBucket), heads)
definedIn := map[string]map[string]bool{} definedIn := map[string]map[string]bool{}
for _, p := range c.Pulls { for _, p := range c.Pulls {
if p.Repository == assign.SDKRepository { if p.Repository == assign.SDKRepository {
@@ -546,6 +571,16 @@ func (w *world) value(bucket, key string) []byte {
return w.values[bucket][key] return w.values[bucket][key]
} }
// grantsBucket is the bucket the grants are read from now (delivery.GrantSource): the grants part's while the controller's
// lease is handed away, as the lease bucket's key says it (buscompose.LeaseHandedAway), the controller's in every other
// case, a lease not held here included.
func (w *world) grantsBucket() string {
raw := w.value(buscompose.LeaseBucket, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, raw != nil, nil)
bucket, _ := delivery.GrantSource(away)
return bucket
}
// has says whether a bucket holds a key now. // has says whether a bucket holds a key now.
func (w *world) has(bucket, key string) bool { func (w *world) has(bucket, key string) bool {
w.mu.Lock() w.mu.Lock()
@@ -572,7 +607,17 @@ func (w *world) built(module, fp string) bool {
// composed is the controller's composition of module for node at fp: its digest and the judgement of the move to it // composed is the controller's composition of module for node at fp: its digest and the judgement of the move to it
// (mesh-controller_composed `<node>.<module>.<hex>`), and whether it exists and can be read. // (mesh-controller_composed `<node>.<module>.<hex>`), and whether it exists and can be read.
//
// From the lease write the controller's copy is no longer read (ADR 0571 §6, a named exception of clause 10 until then): a
// placed node composes its own, and its composition is the digest its own have names; an unplaced node has none, so it gets
// no want (and the pass says so, unplacedFindings).
func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool) { func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool) {
if w.handedAway() {
if !w.placedNodes()[node] {
return delivery.ComposedDigest{}, false
}
return w.ownComposed(node)
}
key, err := delivery.ComposedKey(node, module, fp) key, err := delivery.ComposedKey(node, module, fp)
if err != nil { if err != nil {
return delivery.ComposedDigest{}, false return delivery.ComposedDigest{}, false
@@ -590,6 +635,9 @@ func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool
// nodeComposed is, by node, the digest of the node's own composition (mesh-controller_composed `<node>.node`). // nodeComposed is, by node, the digest of the node's own composition (mesh-controller_composed `<node>.node`).
func (w *world) nodeComposed() map[string]delivery.ComposedDigest { func (w *world) nodeComposed() map[string]delivery.ComposedDigest {
out := map[string]delivery.ComposedDigest{} out := map[string]delivery.ComposedDigest{}
if w.handedAway() {
return out // the controller composes no node from the lease write (ADR 0571 §6)
}
for key, value := range w.read(delivery.ControllerComposedBucket) { for key, value := range w.read(delivery.ControllerComposedBucket) {
node, _, _, isNode, err := delivery.ParseComposedKey(key) node, _, _, isNode, err := delivery.ParseComposedKey(key)
if err != nil || !isNode { if err != nil || !isNode {
@@ -621,29 +669,72 @@ func (w *world) changed(bucket, key string) uint64 {
return w.changes[bucket+"|"+key] return w.changes[bucket+"|"+key]
} }
// runningCommits is, of every open assignment of module, the commit of the build it names, read from the controller's // logWatch follows the builder's log: handle each entry it keeps, then each new one (process.LogReader.Watch).
// build record (`mesh-controller_builds` `<module>.build.<id>`), distinct and sorted: the commits a reader of the type logWatch func(handle func(stdio.LogEntry) error) error
// manifests runs at (novox/hq ADR 0482 clause 5, issue 1623). A build whose record names no full commit is left out.
func (w *world) runningCommits(ctx context.Context, get getter, module string) []string { // putBuilt takes one entry of the builder's log in: a done trunk build's end (no ref) naming a full commit, newer than the
_, all := w.assigned() // module's last, becomes the module's running commit. It answers whether the commit changed.
seen := map[string]bool{} func (w *world) putBuilt(raw []byte) bool {
for _, a := range all { e, err := delivery.DecodeBuildLogEntry(raw)
if a.Module != module || a.Build == "" { if err != nil || e.Event != delivery.BuildEndedEvent || e.Outcome != delivery.BuildDone || e.Ref != "" || len(e.Commit) != 40 {
continue return false
}
value, err := get(ctx, delivery.BuildKey(module, a.Build))
if err != nil {
continue
}
var rec delivery.BuildRecord
if json.Unmarshal(value, &rec) != nil || len(rec.Commit) != 40 {
continue
}
seen[rec.Commit] = true
} }
out := make([]string, 0, len(seen)) w.mu.Lock()
for c := range seen { defer w.mu.Unlock()
out = append(out, c) have, ok := w.trunkBuilt[e.Module]
if ok && !e.At.After(have.At) {
return false
}
w.trunkBuilt[e.Module] = e
return !ok || have.Commit != e.Commit
}
// runningCommits is the commit of module's newest done trunk build in the builder's log (ADR 0571 clause 1): the commit a
// reader of the manifests runs at (novox/hq ADR 0482 clause 5, issue 1623); none when the log holds no such build.
func (w *world) runningCommits(module string) []string {
w.mu.Lock()
defer w.mu.Unlock()
if e, ok := w.trunkBuilt[module]; ok {
return []string{e.Commit}
}
return nil
}
// keptBucket is mesh-config's `kept`: the approved held values of open pull requests (ADR 0565 §5).
var keptBucket = "mesh-config_" + delivery.MeshConfigKeptState
// ownComposed is a placed node's own composition, as the digest its own have names (`<node>.node` of the haves, Composed):
// what rule 7 reads from the lease write. A node whose have names none is not composed yet.
func (w *world) ownComposed(node string) (delivery.ComposedDigest, bool) {
key, err := delivery.HaveNodeKey(node)
if err != nil {
return delivery.ComposedDigest{}, false
}
raw, ok := w.read(delivery.HaveBucket)[key]
if !ok {
return delivery.ComposedDigest{}, false
}
h, err := delivery.DecodeHaveNode(raw)
if err != nil || h.Composed == "" {
return delivery.ComposedDigest{}, false
}
return delivery.ComposedDigest{V: delivery.Version, Digest: h.Composed, Judgement: delivery.JudgedNeither,
Words: "the node composes its own"}, true
}
// unplacedNodes are, from the lease write, the nodes whose engine has a have and that no placement names: they get no
// want, since nothing composes them.
func (w *world) unplacedNodes() []string {
if !w.handedAway() {
return nil
}
placed := w.placedNodes()
var out []string
for key := range w.read(delivery.HaveBucket) {
node, _, isNode, err := delivery.ParseHaveKey(key)
if err == nil && isNode && !placed[node] {
out = append(out, node)
}
} }
sort.Strings(out) sort.Strings(out)
return out return out
@@ -6,7 +6,9 @@ import (
"slices" "slices"
"strings" "strings"
"testing" "testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign" "mesh-delivery-assigner/internal/assign"
@@ -125,4 +127,40 @@ func theWorldCountsTheValuesAKeyTakes(t *testing.T) {
if got := w.changed(delivery.HaveBucket, "gone"); got != 0 { if got := w.changed(delivery.HaveBucket, "gone"); got != 0 {
t.Errorf("changed = %d for a key never seen", got) t.Errorf("changed = %d for a key never seen", got)
} }
// Issue 774: grants are read from the grants part's bucket only while the controller's lease is handed away: a grant
// only the part holds lets a node's put through then, and with a controller's lease (or none, or an unreadable one) it
// holds it, a grant only the controller holds being the one that counts instead.
fp := delivery.SourcePrefix2 + strings.Repeat("d", 64)
gkey, _ := delivery.ControllerGrantKey("web", fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{Accounts: []string{"ace.web"}})
for _, row := range []struct {
name, lease string
inPart, inController bool
bucket string
granted bool
}{
{"handed away, the part's grant", `{"handed-to":"the operator's hand-over"}`, true, false, delivery.PartGrantsBucket, true},
{"handed away, only the controller's grant", `{"handed-to":"x"}`, false, true, delivery.PartGrantsBucket, false},
{"a controller holds it, only the part's grant", `{"instance":"novox/1","epoch":3}`, true, false, delivery.ControllerGrantsBucket, false},
{"a controller holds it, its own grant", `{"instance":"novox/1"}`, false, true, delivery.ControllerGrantsBucket, true},
{"no lease read", ``, true, false, delivery.ControllerGrantsBucket, false},
{"an unreadable lease", `[1]`, true, false, delivery.ControllerGrantsBucket, false},
} {
p := newPasser(buckets{}, time.Now())
if row.lease != "" {
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(row.lease))
}
if row.inPart {
p.w.put(delivery.PartGrantsBucket, gkey, grant)
}
if row.inController {
p.w.put(delivery.ControllerGrantsBucket, gkey, grant)
}
if got := p.w.grantsBucket(); got != row.bucket {
t.Errorf("%s: grants are read from %s, want %s", row.name, got, row.bucket)
}
if _, granted := p.composedFor("ace", "web", fp); granted != row.granted {
t.Errorf("%s: granted = %v, want %v", row.name, granted, row.granted)
}
}
} }
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-assigner
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 h1:Tt16HJk8JZRce1i3JzzDSNdCWOrv7dw8FQ/bckNQ5Ig= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24 h1:s0XXZtR7CFPBbnaDygWnE4/Q827iEFt0YfEzujk4Q9g=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -1,12 +1,14 @@
package assign package assign
import ( import (
"slices"
"strings" "strings"
"testing" "testing"
"time" "time"
"git.novox.be/novox/mesh-sdk/go/asks" "git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
) )
// 1297/B9 (the assigner's half): the ask about a held fingerprint is a valid ask of the router and of mesh-sdk's layout at // 1297/B9 (the assigner's half): the ask about a held fingerprint is a valid ask of the router and of mesh-sdk's layout at
@@ -73,7 +75,13 @@ func TestTheRestoreAskIsValid(t *testing.T) {
// 1906/D3: the condition for a held fingerprint names the module, the node and the check's words, under the assigner's own // 1906/D3: the condition for a held fingerprint names the module, the node and the check's words, under the assigner's own
// name (ADR 0437 clause 9), and is cleared under the same key. // name (ADR 0437 clause 9), and is cleared under the same key.
func TestTheHeldConditionNamesModuleNodeAndWords(t *testing.T) { func TestTheAssignersConditions(t *testing.T) {
t.Run("the held condition names module, node and words", heldConditionNamesModuleNodeAndWords)
t.Run("a lifted hold clears its condition saying why", liftedHoldClearsItsConditionSayingWhy)
t.Run("1952/WM6: each said condition is a finding, its clear deletes it, said straight only until the take-over", saidConditionsAreFindings)
}
func heldConditionNamesModuleNodeAndWords(t *testing.T) {
c := HeldCondition("mail", fNew, "ace", "connection refused", true, true) c := HeldCondition("mail", fNew, "ace", "connection refused", true, true)
if _, err := asks.EncodeCondition(c); err != nil { if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err) t.Fatalf("the condition is refused: %v", err)
@@ -87,7 +95,7 @@ func TestTheHeldConditionNamesModuleNodeAndWords(t *testing.T) {
} }
// 1906/H2: a hold lifted by itself clears the held condition under its key, saying why. // 1906/H2: a hold lifted by itself clears the held condition under its key, saying why.
func TestALiftedHoldClearsItsConditionSayingWhy(t *testing.T) { func liftedHoldClearsItsConditionSayingWhy(t *testing.T) {
c := LiftedCondition("mail", fNew) c := LiftedCondition("mail", fNew)
if _, err := asks.EncodeCondition(c); err != nil { if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err) t.Fatalf("the condition is refused: %v", err)
@@ -97,6 +105,48 @@ func TestALiftedHoldClearsItsConditionSayingWhy(t *testing.T) {
} }
} }
// 1952/WM6 (ADR 0564 §2, issue 1952 note 16): every condition the assigner says is a finding in its own bucket, under
// `<scope>.<id>.<kind>`, written at the raise and deleted at the clear; it is said straight to the router while the take-over
// key follows the controller, and only as a finding from the take-over.
func saidConditionsAreFindings(t *testing.T) {
at := time.Date(2026, 10, 29, 21, 0, 0, 0, time.UTC)
for _, c := range []struct {
raised, cleared asks.Condition
key string
}{
{HeldCondition("mail", fNew, "ace", "refused", true, true), LiftedCondition("mail", fNew), "module.mail." + strings.TrimPrefix(fNew, delivery.SourcePrefix2) + ".held"},
{NodeFailedCondition("ace", "mail", fNew, NodeFailure{Words: "x", Others: []string{"g14"}}, true), NodeFailedCondition("ace", "mail", fNew, NodeFailure{}, false), "module.ace.mail.failed-alone"},
{NeitherCondition("ace", "mail", Holding{Rule: 3, Words: "w"}, true), NeitherCondition("ace", "mail", Holding{}, false), "module.ace.mail.neither"},
{PlacementCondition(Pair{Node: "ace", Module: "mail", Only: OnlyAssignments}, true), PlacementCondition(Pair{Node: "ace", Module: "mail"}, false), "module.ace.mail.placement"},
{SeatWaitsCondition([]string{"placement"}), SeatWaitsCondition(nil), "module.mesh-delivery-assigner.seat-states"},
{BackupCondition("ace", "mail", fNew, "disk full", true), BackupCondition("ace", "mail", fNew, "", false), "module.ace.mail." + strings.TrimPrefix(fNew, delivery.SourcePrefix2) + ".backup"},
} {
puts, deletes, said := map[string]health.Finding{}, []string{}, 0
keeper := false
f := Findings{Put: func(k string, v []byte) error {
got, err := health.DecodeFinding(v)
puts[k] = got
return err
}, Delete: func(k string) error { deletes = append(deletes, k); return nil }, ToKeeper: func() bool { return keeper }, Now: func() time.Time { return at }}
say := f.Say(func(asks.Condition) error { said++; return nil })
if err := say(c.raised); err != nil {
t.Fatalf("%s: %v", c.key, err)
}
got, ok := puts[c.key]
if !ok || got.Severity != health.Severity(c.raised.Severity) || got.Headline != c.raised.Headline || got.Observer != delivery.AssignerModule ||
got.Subject.Machine != c.raised.Subject.Machine || said != 1 {
t.Fatalf("%s: finding %+v (written %v), said %d", c.key, got, puts, said)
}
keeper = true
if err := say(c.cleared); err != nil || !slices.Equal(deletes, []string{c.key}) || said != 1 {
t.Fatalf("%s: deleted %v, said %d after the take-over (%v)", c.key, deletes, said, err)
}
}
if _, _, _, err := FindingOf(asks.Condition{Key: "another.x", State: asks.StateRaised}, at); err == nil {
t.Error("a condition not the assigner's is a finding")
}
}
// 1906/D4: the record ask and the backup ask are valid asks of the router and of mesh-sdk's layout at their keys, answerable in // 1906/D4: the record ask and the backup ask are valid asks of the router and of mesh-sdk's layout at their keys, answerable in
// the view, and name the move and the node. // the view, and name the move and the node.
func TestTheRecordAndBackupAsksAreValid(t *testing.T) { func TestTheRecordAndBackupAsksAreValid(t *testing.T) {
@@ -5,6 +5,7 @@
package assign package assign
import ( import (
"encoding/json"
"fmt" "fmt"
"slices" "slices"
"sort" "sort"
@@ -59,6 +60,63 @@ type Change struct {
// Members the head of each member by repository: what a composer at the group's mesh-sdk reads (issues 1622, 1623). // Members the head of each member by repository: what a composer at the group's mesh-sdk reads (issues 1622, 1623).
Digests map[string]string Digests map[string]string
Members map[string]string Members map[string]string
// Held are the approved held values of the group's pull requests at the heads judged (mesh-config's `kept`, novox/hq ADR
// 0565 §5, ADR 0502 §4): each laid over its module's values on its node in the change's composition, and named.
Held []delivery.KeptValue
}
// HeldOf are the held values kept for the pull requests the candidate names, each at the head it names: one of another
// pull request, or at another head, is never laid (ADR 0502 §4). kept is mesh-config's `kept` by key.
func HeldOf(kept map[string][]byte, heads map[string]string) []delivery.KeptValue {
var out []delivery.KeptValue
for _, raw := range kept {
var k delivery.KeptValue
if json.Unmarshal(raw, &k) != nil || k.Node == "" || k.Module == "" || k.Key == "" {
continue
}
if head, ok := heads[k.Pull]; ok && head == k.Head {
out = append(out, k)
}
}
sort.Slice(out, func(i, j int) bool {
a, b := out[i], out[j]
return a.Pull+a.Node+a.Module+a.Key < b.Pull+b.Node+b.Module+b.Key
})
return out
}
// layHeld lays each held value over its module's values on its node, as a layer of its own after the rest, and answers
// the words that name each; judged first against the head's declaration of its key (catalogue.ValueProblems), a value the
// declaration refuses, or a key it does not declare, is a failure.
func layHeld(in *compose.Input, held []delivery.KeptValue) (said, fails []string) {
for _, k := range held {
var v any
if err := json.Unmarshal(k.Value, &v); err != nil {
said = append(said, fmt.Sprintf("the held value %s of %s on %s for %s cannot be read, so it is not laid: %v", k.Key, k.Module, k.Node, k.Pull, err))
continue
}
d, declared := in.Catalogue[k.Module].Settings[k.Key]
if !declared {
fails = append(fails, fmt.Sprintf("the held value %s of %s on %s for %s: the head declares no such key", k.Key, k.Module, k.Node, k.Pull))
continue
}
if problems := catalogue.ValueProblems(d, v); len(problems) > 0 {
fails = append(fails, fmt.Sprintf("the held value %s of %s on %s for %s does not hold to the head's declaration: %s", k.Key, k.Module,
k.Node, k.Pull, strings.Join(problems, "; ")))
continue
}
at := slices.IndexFunc(in.Nodes, func(n compose.Node) bool { return n.Name == k.Node })
if at < 0 {
said = append(said, fmt.Sprintf("the held value %s of %s for %s names %s, which runs no module of the mesh: not laid", k.Key, k.Module, k.Pull, k.Node))
continue
}
if in.Nodes[at].Settings == nil {
in.Nodes[at].Settings = map[string][]catalogue.Layer{}
}
in.Nodes[at].Settings[k.Module] = append(in.Nodes[at].Settings[k.Module], catalogue.Layer{From: "held for " + k.Pull, Values: map[string]any{k.Key: v}})
said = append(said, fmt.Sprintf("held value laid: %s of %s on %s, held for %s at %.12s, approved by %s (novox/hq ADR 0502 §4)", k.Key, k.Module, k.Node, k.Pull, k.Head, k.Approved))
}
return said, fails
} }
// Touched is the machines a change touches (ADR 0446 clause 3): those running a module the change moves, adds or // Touched is the machines a change touches (ADR 0446 clause 3): those running a module the change moves, adds or
@@ -171,6 +229,11 @@ func Compose(m Mesh, c Change, sdk string, took time.Duration) delivery.Composit
delete(changed, name) delete(changed, name)
} }
after := input(changed, m.Assigned) after := input(changed, m.Assigned)
laid, heldFails := layHeld(&after, c.Held)
v.Warnings = append(v.Warnings, laid...)
for _, f := range heldFails {
failAt("", f)
}
for _, node := range machines { for _, node := range machines {
_, before := compose.Resolve(trunk, node) _, before := compose.Resolve(trunk, node)
_, now := compose.Resolve(after, node) _, now := compose.Resolve(after, node)
@@ -1,7 +1,9 @@
package assign package assign
import ( import (
"encoding/json"
"slices" "slices"
"strconv"
"strings" "strings"
"testing" "testing"
@@ -38,6 +40,32 @@ func TestAChangeOfOneModuleComposesItsMachine(t *testing.T) {
if _, err := delivery.EncodeCompositionVerdict(v); err != nil { if _, err := delivery.EncodeCompositionVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err) t.Errorf("the verdict cannot be written: %v", err)
} }
// 1953/D7 (ADR 0565 §5, ADR 0502 §4): the group's own pull request's held value at its head is laid and named, judged
// against the head's declaration; one of another pull request or another head is not laid.
web, err := catalogue.ParseManifest([]byte(`{"module":"web","version":"2","config":{"port":{"type":"number","default":80,"description":"the port, a number","runtime":true}}}`))
if err != nil {
t.Fatal(err)
}
kept := map[string][]byte{}
for i, k := range []delivery.KeptValue{
{Pull: "novox/x#1", Head: "h1", Node: "g14", Module: "web", Key: "port", Value: json.RawMessage(`81`), Approved: "op"},
{Pull: "novox/x#2", Head: "h2", Node: "g14", Module: "web", Key: "port", Value: json.RawMessage(`82`)},
{Pull: "novox/x#1", Head: "old", Node: "g14", Module: "web", Key: "other", Value: json.RawMessage(`1`)},
} {
kept[strconv.Itoa(i)], _ = json.Marshal(k)
}
held := HeldOf(kept, map[string]string{"novox/x#1": "h1"})
if len(held) != 1 || string(held[0].Value) != "81" {
t.Fatalf("held = %+v, want #1's at its head alone", held)
}
c = Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"web": web}, Held: held}
if v := Compose(meshOf(), c, "x", 0); v.State != delivery.CompositionPass || !hasWarning(v, "held value laid: port of web on g14, held for novox/x#1") {
t.Errorf("composition = %+v", v)
}
c.Held[0].Value = json.RawMessage(`"eighty"`)
if v := Compose(meshOf(), c, "x", 0); v.State != delivery.CompositionFail || !strings.Contains(v.Words, "does not hold to the head's declaration") {
t.Errorf("a held value the declaration refuses: %+v", v)
}
} }
// 1297/AC2: a change that removes a module a machine runs fails, naming the machine and the module. // 1297/AC2: a change that removes a module a machine runs fails, naming the machine and the module.
@@ -0,0 +1,105 @@
package assign
import (
"errors"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
// The assigner's said conditions become findings (novox/hq ADR 0564 §2, issue 1952 WM6, note 16): every condition it says
// under its own name (ADR 0437 §9) — a hold and its lift, a node failing alone, a module on neither list, a backup, a pair
// placed by one source, the seat waiting for its states — is also a finding in its own bucket `findings`, written at its
// raise and deleted at its clear, which the condition keeper raises and clears. Until the operator's take-over the router
// hears it straight from the assigner, as before; from the take-over the keeper's condition is the one said, so the
// assigner no longer says it straight (never both).
// FindingsBucket is the assigner's findings.
var FindingsBucket = health.FindingsBucket(delivery.AssignerModule)
// FindingOf is a said condition as a finding: its key `<scope>.<id>.<kind>`, from the condition's own key
// `mesh-delivery-assigner.<kind>.<id…>` (the subject's id when the key names none), its words unchanged. raised is false for
// a clear, whose key is to be deleted; value is nil then.
func FindingOf(c asks.Condition, at time.Time) (key string, value []byte, raised bool, err error) {
rest, ok := strings.CutPrefix(c.Key, delivery.AssignerModule+".")
if !ok || rest == "" {
return "", nil, false, fmt.Errorf("the condition %q is not the assigner's", c.Key)
}
kind, id, _ := strings.Cut(rest, ".")
if id == "" {
id = c.Subject.ID
}
scope := c.Subject.Scope
if scope == "" {
scope = health.ScopeModule
}
key = health.FindingKey(scope, id, kind)
if c.State == asks.StateCleared {
return key, nil, false, nil
}
if c.State != asks.StateRaised {
return "", nil, false, fmt.Errorf("the condition %s is neither raised nor cleared", c.Key)
}
summary := c.Summary
if summary == "" {
summary = c.Headline
}
var actions []health.Action
for _, a := range c.Actions {
actions = append(actions, health.Action{Label: a.Label, Verb: a.Verb, Machine: a.Machine, Arguments: a.Arguments, Level: string(a.Level)})
}
value, err = health.EncodeFinding(health.Finding{Key: key, Kind: kind, Subject: health.Subject{Scope: scope, ID: id, Machine: c.Subject.Machine},
Severity: health.Severity(c.Severity), Summary: summary, Headline: c.Headline, Explanation: c.Explanation, Resolved: c.Resolved,
Needs: c.Needs, Actions: actions, Evidence: []health.Evidence{{At: at.UTC(), Said: summary}}, Source: "assigner." + kind,
Observer: delivery.AssignerModule, At: at.UTC()})
return key, value, true, err
}
// Findings is where the assigner's findings go: the bucket's put and delete, and whether the keeper's conditions are the
// ones said (the take-over key names the keeper).
type Findings struct {
Put func(key string, value []byte) error
Delete func(key string) error
Keys func() ([]string, error)
ToKeeper func() bool
Now func() time.Time
}
// Say is say with each condition also written as a finding (or its finding deleted), and said straight to the router only
// while the take-over key follows the controller. A finding that cannot be written is the error, beside the say's.
func (f Findings) Say(say func(asks.Condition) error) func(asks.Condition) error {
return func(c asks.Condition) error {
var errs []error
if f.Put != nil {
now := time.Now
if f.Now != nil {
now = f.Now
}
key, value, raised, err := FindingOf(c, now())
switch {
case err != nil:
errs = append(errs, err)
case raised:
if err := f.Put(key, value); err != nil {
errs = append(errs, fmt.Errorf("the finding %s cannot be written: %w", key, err))
}
default:
if err := f.Delete(key); err != nil {
errs = append(errs, fmt.Errorf("the finding %s cannot be deleted: %w", key, err))
}
}
}
if f.ToKeeper == nil || !f.ToKeeper() {
if say != nil {
if err := say(c); err != nil {
errs = append(errs, err)
}
}
}
return errors.Join(errs...)
}
}
@@ -2,6 +2,7 @@ package assign
import ( import (
"fmt" "fmt"
"slices"
"sort" "sort"
"strings" "strings"
@@ -15,7 +16,8 @@ import (
// removed on one source's word, and it raises a condition naming the pair until both agree. After an approved proposal // removed on one source's word, and it raises a condition naming the pair until both agree. After an approved proposal
// the assigner keeps the controller in step itself, calling its assign, unassign, pin and unpin; a call refused or not // the assigner keeps the controller in step itself, calling its assign, unassign, pin and unpin; a call refused or not
// yet answered leaves the pair differing, and the condition names the call (Pair.Call). Before the first placement key // yet answered leaves the pair differing, and the condition names the call (Pair.Call). Before the first placement key
// is written the controller's assignments are the one source, as before. // is written the controller's assignments are the one source, as before. From the lease write (the controller's lease
// handed away) the placement is the one source.
// Pair is a module on a node that one source names and the other does not; Only says which: OnlyAssignments or // Pair is a module on a node that one source names and the other does not; Only says which: OnlyAssignments or
// OnlyPlacement. // OnlyPlacement.
@@ -32,8 +34,17 @@ const (
) )
// Placed is, by node, the modules both sources name, in name order, and every pair one names alone, by node then // Placed is, by node, the modules both sources name, in name order, and every pair one names alone, by node then
// module. With no placement read yet (seeded false) it is the assignments, and no pair differs. // module. With no placement read yet (seeded false) it is the assignments, and no pair differs. Once the controller's
func Placed(assigned, placed map[string][]string, seeded bool) (map[string][]string, []Pair) { // lease is handed away (handedAway, ADR 0549 §5, §6) it is the placement alone, in name order, and no pair differs: the
// assignments are then a source of nothing.
func Placed(assigned, placed map[string][]string, seeded, handedAway bool) (map[string][]string, []Pair) {
if handedAway && seeded {
out := make(map[string][]string, len(placed))
for node, modules := range placed {
out[node] = slices.Sorted(slices.Values(modules))
}
return out, nil
}
if !seeded { if !seeded {
return assigned, nil return assigned, nil
} }
@@ -0,0 +1,208 @@
package assign
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
// A person's upgrade policy is the assigner's (novox/hq ADR 0565 §7, §8, issue 1953 D4, D5): kept in its state `policies`, one
// key per module, set and taken back through `mesh-wants.upgrade`. Every reader takes one derivation, mesh-sdk's
// catalogue.Upgrade: a person's choice over the module's word, its irreplaceable data and the default, the bus never rolled.
// Until the lease write the controller still writes a policy beside each composition's digest; where it differs from the
// assigner's, the assigner says so as its finding `policy-split`, naming mesh-wants.upgrade as the remedy.
// PoliciesBucket is the assigner's state `policies`.
var PoliciesBucket = delivery.AssignerModule + "_" + delivery.AssignerPoliciesState
// DecodePolicies reads the policies state's values by key (the module); a value that cannot be read is left out, named.
func DecodePolicies(values map[string][]byte) (map[string]catalogue.PersonPolicy, []string) {
out := map[string]catalogue.PersonPolicy{}
var bad []string
for key, raw := range values {
var p catalogue.PersonPolicy
if err := json.Unmarshal(raw, &p); err != nil || p.V != catalogue.PersonPolicyVersion || p.Check() != nil || p.Module != key {
bad = append(bad, key)
continue
}
out[key] = p
}
sort.Strings(bad)
return out, bad
}
// PolicyOf is a module's policy and where it comes from, with the person's choice for it when there is one.
func PolicyOf(m catalogue.Manifest, chosen map[string]catalogue.PersonPolicy) (policy, from, why, by string) {
var c *catalogue.PersonPolicy
if p, ok := chosen[m.Module]; ok {
c = &p
}
return catalogue.Upgrade(m, c)
}
// ControllerPolicies is, by module, every policy the controller writes beside a composition's digest of it.
func ControllerPolicies(composed map[string][]byte) map[string][]string {
out := map[string][]string{}
for key, raw := range composed {
_, module, _, isNode, err := delivery.ParseComposedKey(key)
if err != nil || isNode {
continue
}
d, err := delivery.DecodeComposedDigest(raw)
if err != nil || d.Policy == "" {
continue
}
if !contains(out[module], d.Policy) {
out[module] = append(out[module], d.Policy)
sort.Strings(out[module])
}
}
return out
}
func contains(list []string, s string) bool {
for _, v := range list {
if v == s {
return true
}
}
return false
}
// Split is a module whose policy at the assigner differs from one the controller writes beside a composition of it.
type Split struct {
Module string `json:"module"`
Policy string `json:"policy"`
From string `json:"from"`
Controller []string `json:"controller"`
}
// Splits is every module of manifests the controller writes a policy for that differs from the assigner's, sorted.
func Splits(manifests map[string]catalogue.Manifest, chosen map[string]catalogue.PersonPolicy, controller map[string][]string) []Split {
var out []Split
for module, theirs := range controller {
m, ok := manifests[module]
if !ok {
continue
}
policy, from, _, _ := PolicyOf(m, chosen)
for _, p := range theirs {
if p != policy {
out = append(out, Split{Module: module, Policy: policy, From: from, Controller: theirs})
break
}
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Module < out[j].Module })
return out
}
// KindPolicySplit is the assigner's finding of a split policy (ADR 0565 §8).
const KindPolicySplit = "policy-split"
// PolicySplitKey is a module's policy-split finding's key.
func PolicySplitKey(module string) string {
return health.FindingKey(health.ScopeModule, module, KindPolicySplit)
}
// PolicySplitFinding is the finding of one split, in mesh-sdk's layout.
func PolicySplitFinding(s Split, at time.Time) ([]byte, error) {
summary := fmt.Sprintf("%s's upgrade policy is %s at the assigner (%s) and %s at the controller", s.Module, s.Policy, s.From,
strings.Join(s.Controller, " and "))
return health.EncodeFinding(health.Finding{Key: PolicySplitKey(s.Module), Kind: KindPolicySplit,
Subject: health.Subject{Scope: health.ScopeModule, ID: s.Module}, Severity: health.Warning, Summary: summary,
Headline: clipHeadline(s.Module + "'s upgrade policy is split"),
Needs: "set it with mesh-wants.upgrade, so the assigner and the controller agree",
Explanation: asks.NeedsYou + "set it with mesh-wants.upgrade, so the assigner and the controller agree." + " The controller still " +
"writes a policy beside each composition until it hands its lease away; a choice made at its upgrade verb after the copy " +
"is not the assigner's.",
Resolved: s.Module + "'s upgrade policy agrees again", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindPolicySplit, Observer: delivery.AssignerModule, At: at.UTC()})
}
func clipHeadline(s string) string {
if len([]rune(s)) <= 60 {
return s
}
return string([]rune(s)[:60])
}
// KindProvisionFailed is the assigner's finding of a provision its module's want waits for that failed (ADR 0570 §4).
const KindProvisionFailed = "provision-failed"
// ProvisionFailedKey is a failed provision's finding's key, `provider.<node>.<consumer>.<provision>[.<local>].provision-failed`.
func ProvisionFailedKey(key string) string {
return health.FindingKey(health.ScopeProvider, key, KindProvisionFailed)
}
// ProvisionFailedFinding is the finding of a failed provision, naming its consumer, its provider and why.
func ProvisionFailedFinding(key string, p provisions.Provision, at time.Time) ([]byte, error) {
summary := fmt.Sprintf("%s for %s on %s failed at %s on %s: %s, so %s is not moved there until it is provided", p.Provision,
p.Consumer.Module, p.Consumer.Node, p.Provider.Module, p.Provider.Node, p.Why, p.Consumer.Module)
return health.EncodeFinding(health.Finding{Key: ProvisionFailedKey(key), Kind: KindProvisionFailed,
Subject: health.Subject{Scope: health.ScopeProvider, ID: key, Machine: p.Consumer.Node, Also: also(p.Provider.Node, p.Consumer.Node)},
Severity: health.Warning, Summary: summary, Headline: clipHeadline(p.Consumer.Module + " waits for a provision that failed"),
Explanation: asks.NothingToDo + " " + summary + ". It clears once the provider provides it.",
Resolved: p.Provision + " for " + p.Consumer.Module + " is provided", Evidence: []health.Evidence{{At: at.UTC(), Said: p.Why}},
Source: "assigner." + KindProvisionFailed, Observer: delivery.AssignerModule, At: at.UTC()})
}
func also(provider, consumer string) []string {
if provider == "" || provider == consumer {
return nil
}
return []string{provider}
}
// KindProvisionMissing is the assigner's finding of a required provision that has had no key for longer than its bound.
const KindProvisionMissing = "provision-missing"
// ProvisionMissingKey is its key, `provider.<node>.<module>.<provision>.provision-missing`.
func ProvisionMissingKey(key string) string {
return health.FindingKey(health.ScopeProvider, key, KindProvisionMissing)
}
// ProvisionMissingFinding is the finding of a required provision with no key past bound (key `<node>.<module>.<provision>`).
func ProvisionMissingFinding(key string, bound time.Duration, at time.Time) ([]byte, error) {
parts := strings.SplitN(key, ".", 3)
if len(parts) != 3 {
return nil, fmt.Errorf("%q is not <node>.<module>.<provision>", key)
}
node, module, provision := parts[0], parts[1], parts[2]
summary := fmt.Sprintf("%s on %s requires %s and mesh-provisioner has composed no key for it in %s, so %s is not moved there: "+
"no provider of it may be placed", module, node, provision, bound, module)
return health.EncodeFinding(health.Finding{Key: ProvisionMissingKey(key), Kind: KindProvisionMissing,
Subject: health.Subject{Scope: health.ScopeProvider, ID: key, Machine: node},
Severity: health.Warning, Summary: summary, Headline: clipHeadline(module + " on " + node + " lacks " + provision),
Explanation: asks.NothingToDo + " " + summary + ". It clears once the provision has its key.",
Resolved: provision + " for " + module + " on " + node + " has its key", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindProvisionMissing, Observer: delivery.AssignerModule, At: at.UTC()})
}
// KindUnplaced is, from the lease write, a node that runs but no placement names (ADR 0571 §6).
const KindUnplaced = "unplaced"
// UnplacedKey is its finding's key, `machine.<node>.unplaced`.
func UnplacedKey(node string) string {
return health.FindingKey(health.ScopeMachine, node, KindUnplaced)
}
// UnplacedFinding is the finding of a node no placement names since the lease write.
func UnplacedFinding(node string, at time.Time) ([]byte, error) {
summary := fmt.Sprintf("%s runs, but no placement names it: since the controller handed its lease away nothing composes it, "+
"so the assigner writes it no want", node)
return health.EncodeFinding(health.Finding{Key: UnplacedKey(node), Kind: KindUnplaced,
Subject: health.Subject{Scope: health.ScopeMachine, ID: node, Machine: node}, Severity: health.Warning, Summary: summary,
Headline: clipHeadline(node + " has no placement"), Needs: "place it with mesh-placement.propose.",
Explanation: asks.NeedsYou + "place it with mesh-placement.propose. " + summary + ".",
Resolved: node + " is placed", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindUnplaced, Observer: delivery.AssignerModule, At: at.UTC()})
}
@@ -15,10 +15,6 @@ import (
// `placement`, changed only by a proposal (§2) the nodes it touches answer (§3), judged (§4) and written on the // `placement`, changed only by a proposal (§2) the nodes it touches answer (§3), judged (§4) and written on the
// operator's Approve at the revision it was laid over (§5). What here decides; the tools and the passes do. // operator's Approve at the revision it was laid over (§5). What here decides; the tools and the passes do.
// Leased says whether the lease write of ADR 0547 has happened, after which the controller's assignments are read no
// more (§8). Not before L3: until then the holder keeps the controller in step and refuses what its verbs cannot take.
var Leased = false
// ProposeIn is what a proposal is laid over: its id, who and why, its changes; every node's current placement and the // ProposeIn is what a proposal is laid over: its id, who and why, its changes; every node's current placement and the
// revision of its key; which modules the indexer holds and which nodes have a record; and, for the seed, the export's // revision of its key; which modules the indexer holds and which nodes have a record; and, for the seed, the export's
// digest. // digest.
@@ -32,6 +28,10 @@ type ProposeIn struct {
Recorded func(node string) bool Recorded func(node string) bool
Seed bool Seed bool
ExportDigest string ExportDigest string
// Leased says the lease write of ADR 0547 has happened (the controller's lease handed away, mesh-sdk
// buscompose.LeaseHandedAway; issue 774), after which the controller's assignments are read no more (§8): until then
// the holder keeps the controller in step and refuses what its verbs cannot take.
Leased bool
} }
// Propose lays a proposal over the current keys (§2): the proposal (state answering, each changed node's base revision), // Propose lays a proposal over the current keys (§2): the proposal (state answering, each changed node's base revision),
@@ -58,7 +58,7 @@ func Propose(in ProposeIn) (delivery.PlacementProposal, map[string]*delivery.Pla
findings = append(findings, fmt.Sprintf("change %d: the seed is adopt's alone, and never mixed with other changes", i+1)) findings = append(findings, fmt.Sprintf("change %d: the seed is adopt's alone, and never mixed with other changes", i+1))
continue continue
} }
if !Leased { if !in.Leased {
if why := c.BeforeLease(); why != "" { if why := c.BeforeLease(); why != "" {
findings = append(findings, fmt.Sprintf("change %d: %s", i+1, why)) findings = append(findings, fmt.Sprintf("change %d: %s", i+1, why))
} }
@@ -60,18 +60,18 @@ func TestThePlacementSeatDecides(t *testing.T) {
if _, _, f := Propose(in(delivery.PlacementProposalChange{Op: delivery.OpConfig, Node: "anchor", Module: "web", Values: map[string]any{"puid": 1}})); len(f) != 0 { if _, _, f := Propose(in(delivery.PlacementProposalChange{Op: delivery.OpConfig, Node: "anchor", Module: "web", Values: map[string]any{"puid": 1}})); len(f) != 0 {
t.Errorf("config values before the lease write: %v", f) t.Errorf("config values before the lease write: %v", f)
} }
Leased = true
for _, c := range []delivery.PlacementProposalChange{ for _, c := range []delivery.PlacementProposalChange{
{Op: delivery.OpPlace, Node: "laptop", Module: "web", Ports: map[string]int{"80": 8080}, Bindings: map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}}, {Op: delivery.OpPlace, Node: "laptop", Module: "web", Ports: map[string]int{"80": 8080}, Bindings: map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}},
{Op: delivery.OpConfig, Node: "anchor", Module: "web", Layer: map[string]any{"places": []any{}}}, {Op: delivery.OpConfig, Node: "anchor", Module: "web", Layer: map[string]any{"places": []any{}}},
{Op: delivery.OpDecide, Node: "laptop", Field: "site", Value: json.RawMessage(`"attic"`)}, {Op: delivery.OpDecide, Node: "laptop", Field: "site", Value: json.RawMessage(`"attic"`)},
{Op: delivery.OpHold, Node: "anchor", Module: "store", Claim: "node-shell", Scope: "node"}, {Op: delivery.OpHold, Node: "anchor", Module: "store", Claim: "node-shell", Scope: "node"},
} { } {
if _, _, f := Propose(in(c)); len(f) != 0 { leased := in(c)
leased.Leased = true
if _, _, f := Propose(leased); len(f) != 0 {
t.Errorf("after the lease write %s is refused: %v", c.Op, f) t.Errorf("after the lease write %s is refused: %v", c.Op, f)
} }
} }
Leased = false
// A key moved while the warrant wrote: what is left to write is the other nodes' changes, to be laid and asked again. // A key moved while the warrant wrote: what is left to write is the other nodes' changes, to be laid and asked again.
if rest, ok := Rest(p, map[string]*delivery.Placement{"anchor": laid["anchor"]}); !ok || len(rest.Changes) != 1 || rest.Changes[0].Node != "laptop" || if rest, ok := Rest(p, map[string]*delivery.Placement{"anchor": laid["anchor"]}); !ok || len(rest.Changes) != 1 || rest.Changes[0].Node != "laptop" ||
@@ -10,6 +10,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
) )
// Module is what the six rules read of one module (ADR 0429 §6, ADR 0528 clause 3): its newest fingerprint F at the // Module is what the six rules read of one module (ADR 0429 §6, ADR 0528 clause 3): its newest fingerprint F at the
@@ -29,7 +30,12 @@ type Module struct {
// 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go was wanted with no account on the bus). Nil: the grant alone. // 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go was wanted with no account on the bus). Nil: the grant alone.
GrantedOn func(node string) bool GrantedOn func(node string) bool
Manifest catalogue.Manifest Manifest catalogue.Manifest
Nodes []string // Provisions is the seat mesh-provisioner's provisions this module consumes on a node, by key (ADR 0570 §4): the want
// waits until each is provided. Nil: the seat's state is not read (not delivered yet), and nothing waits on it.
Provisions func(node string) map[string]provisions.Provision
// Chosen is a person's choice of the module's upgrade policy (the assigner's `policies`, ADR 0565 §7); nil: none.
Chosen *catalogue.PersonPolicy
Nodes []string
// Composed answers the controller's composition of the module for a node at a fingerprint (mesh-controller_composed): its // Composed answers the controller's composition of the module for a node at a fingerprint (mesh-controller_composed): its
// digest and the judgement of the move to it, and whether it exists. Nil: none exists (ADR 0539 §1, §6). // digest and the judgement of the move to it, and whether it exists. Nil: none exists (ADR 0539 §1, §6).
Composed func(node, fp string) (delivery.ComposedDigest, bool) Composed func(node, fp string) (delivery.ComposedDigest, bool)
@@ -124,9 +130,54 @@ func holds(m Module, node string, n Nodes, ordering bool) (bool, int, string) {
if _, ok := m.composition(node, m.FP); !ok { if _, ok := m.composition(node, m.FP); !ok {
return false, delivery.RuleComposed, fmt.Sprintf("the controller has not composed %s for %s at %s", m.Name, node, short(m.FP)) return false, delivery.RuleComposed, fmt.Sprintf("the controller has not composed %s for %s at %s", m.Name, node, short(m.FP))
} }
if words, waiting := m.provisionWaiting(node); waiting {
return false, delivery.RuleProvisions, words
}
return true, 0, "" return true, 0, ""
} }
// provisionWaiting says why the module's want on node waits for a provision (ADR 0570 §4): a provision it requires that has
// no key yet, or the first of its keys that is not provided. Nil Provisions (the seat's state not on the bus) waits for none.
func (m Module) provisionWaiting(node string) (string, bool) {
if m.Provisions == nil {
return "", false
}
if missing := m.MissingProvisions(node); len(missing) > 0 {
return fmt.Sprintf("provision %s has no key yet for %s on %s", missing[0], m.Name, node), true
}
keyed := m.Provisions(node)
for _, key := range slices.Sorted(maps.Keys(keyed)) {
if p := keyed[key]; p.State != provisions.Provided {
words := fmt.Sprintf("the provision %s of %s on %s is %s, not provided", key, m.Name, node, p.State)
if p.Why != "" {
words += ": " + p.Why
}
return words, true
}
}
return "", false
}
// MissingProvisions are the provisions the module requires (not optionally) that have no key on node yet, sorted; none
// while the seat's state is not read.
func (m Module) MissingProvisions(node string) []string {
if m.Provisions == nil {
return nil
}
have := map[string]bool{}
for _, p := range m.Provisions(node) {
have[p.Provision] = true
}
var out []string
for _, r := range m.Manifest.Requires {
if !have[r] && !m.Manifest.OptionalRequires[r] && !slices.Contains(out, r) {
out = append(out, r)
}
}
slices.Sort(out)
return out
}
// providedHealthy is rule 5 for one provision: some module providing it runs healthy on the node, or, assigned // providedHealthy is rule 5 for one provision: some module providing it runs healthy on the node, or, assigned
// elsewhere only, at its canary. A provision no module provides is not a module's to judge (the mesh or the node // elsewhere only, at its canary. A provision no module provides is not a module's to judge (the mesh or the node
// offers it) and holds. // offers it) and holds.
@@ -639,9 +690,14 @@ func (m Module) busBar(before delivery.Want, had bool, running string) (prev str
return prev, catalogue.ProvidesBus(m.Manifest) && prev != m.FP return prev, catalogue.ProvidesBus(m.Manifest) && prev != m.FP
} }
// recorded says whether the module is of policy record: as the controller holds it beside the composition's digest, the // recorded says whether the module is of policy record: a person's choice at the assigner over everything but the bus (ADR
// operator's own choice included (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it. // 0565 §7, catalogue.Upgrade); for a module with none, as the controller holds it beside the composition's digest until the
// lease write (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it.
func (m Module) recorded(d delivery.ComposedDigest) bool { func (m Module) recorded(d delivery.ComposedDigest) bool {
if m.Chosen != nil {
policy, _, _, _ := catalogue.Upgrade(m.Manifest, m.Chosen)
return policy == catalogue.PolicyRecord
}
if d.Policy != "" { if d.Policy != "" {
return d.Policy == delivery.PolicyRecord return d.Policy == delivery.PolicyRecord
} }
@@ -8,6 +8,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
) )
func fpOf(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) } func fpOf(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
@@ -69,6 +70,12 @@ func TestEachRuleHoldsTheWantBack(t *testing.T) {
delivery.RuleComposed: func(m *Module, _ *Nodes) { delivery.RuleComposed: func(m *Module, _ *Nodes) {
m.Composed = func(string, string) (delivery.ComposedDigest, bool) { return delivery.ComposedDigest{}, false } m.Composed = func(string, string) (delivery.ComposedDigest, bool) { return delivery.ComposedDigest{}, false }
}, },
// 1954/D1 (ADR 0570 §4): a provision of the module on that node still asked holds its want.
delivery.RuleProvisions: func(m *Module, _ *Nodes) {
m.Provisions = func(node string) map[string]provisions.Provision {
return map[string]provisions.Provision{node + ".mail.database": {Provision: "database", State: provisions.Asked}, node + ".mail.oidc": {Provision: "oidc", State: provisions.Provided}}
}
},
} { } {
m, n := ready() m, n := ready()
breakIt(&m, &n) breakIt(&m, &n)
@@ -82,6 +89,29 @@ func TestEachRuleHoldsTheWantBack(t *testing.T) {
if w := out.Wants["g14"]; w.FP != "" && (w.FP != fOld || w.Newest != fNew || w.Rule != rule) { if w := out.Wants["g14"]; w.FP != "" && (w.FP != fOld || w.Newest != fNew || w.Rule != rule) {
t.Errorf("rule %d broken: g14's want is %+v", rule, w) t.Errorf("rule %d broken: g14's want is %+v", rule, w)
} }
if rule == delivery.RuleProvisions && (!strings.Contains(out.Rules["g14"].Words, "g14.mail.database") || !strings.Contains(out.Rules["g14"].Words, "asked")) {
t.Errorf("the provision's hold does not name its key and state: %s", out.Rules["g14"].Words)
}
}
// 1954/D1: every provision provided, the want moves; with the seat's state not read, nothing waits on it.
for _, all := range []func(string) map[string]provisions.Provision{
func(node string) map[string]provisions.Provision {
return map[string]provisions.Provision{node + ".mail.database": {Provision: "database", State: provisions.Provided}}
}, nil} {
m, n := ready()
m.Provisions = all
if out := Wants(m, n, wantedOld(), nil); out.Wants["g14"].FP != fNew {
t.Errorf("provided provisions held g14's want: %+v (%s)", out.Wants["g14"], out.Rules["g14"].Words)
}
}
// 1954/D1 (the lead, 2026-10-29): a required provision with no key yet holds the want, named.
{
m, n := ready()
m.Provisions = func(string) map[string]provisions.Provision { return map[string]provisions.Provision{} }
if out := Wants(m, n, wantedOld(), nil); out.Rules["g14"].Rule != delivery.RuleProvisions ||
!strings.Contains(out.Rules["g14"].Words, "provision database has no key yet") || len(m.MissingProvisions("g14")) != 1 {
t.Errorf("a provision with no key: %+v", out.Rules["g14"])
}
} }
// 1960: rule 5 holds no want, and no module is left without a first want for a provider down or a node offline; the // 1960: rule 5 holds no want, and no module is left without a first want for a provider down or a node offline; the
// provider down is said (Unreached). novox on 2026-10-29: route-proxy down held de-spiegel and link2pay, minio not // provider down is said (Unreached). novox on 2026-10-29: route-proxy down held de-spiegel and link2pay, minio not
@@ -802,6 +832,12 @@ func TestTheRecordPolicyIsTheOneTheControllerHolds(t *testing.T) {
if out := Wants(m, n, wanted, nil); len(out.Record) == 0 { if out := Wants(m, n, wanted, nil); len(out.Record) == 0 {
t.Errorf("a hand-made composition fell through the manifest's record policy: %+v", out) t.Errorf("a hand-made composition fell through the manifest's record policy: %+v", out)
} }
// 1953/D4 (ADR 0565 §7): a person's choice at the assigner is over the controller's beside the composition.
with(delivery.PolicyRecord, "person")
m.Chosen = &catalogue.PersonPolicy{V: catalogue.PersonPolicyVersion, Module: m.Name, Policy: catalogue.PolicyRoll, Why: "w", By: "op"}
if out := Wants(m, n, wanted, nil); len(out.Record) != 0 || out.Wants["ace"].FP != fNew {
t.Errorf("a person's roll at the assigner was not honoured over the controller's record: %+v", out)
}
} }
// Review round 1, item 2 (ADR 0539 §15): a node that runs the bus with no readable want compares the new fingerprint with the one // Review round 1, item 2 (ADR 0539 §15): a node that runs the bus with no readable want compares the new fingerprint with the one
@@ -33,9 +33,9 @@ func TestManifest(t *testing.T) {
} }
reads := slices.Clone(m.Reads) reads := slices.Clone(m.Reads)
slices.Sort(reads) slices.Sort(reads)
want := []string{"mesh-build-go.steps", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-controller.grants", want := []string{"mesh-build-go.steps", "mesh-conditions.takeover", "mesh-config.kept", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.composed", "mesh-controller.grants", "mesh-controller.lease",
"mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-builder.build-log", "mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "node-engine.have"} "mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-grants.grants", "mesh-provisioner.provisions", "node-engine.have"}
if !reflect.DeepEqual(reads, want) { if !reflect.DeepEqual(reads, want) {
t.Errorf("reads = %v, want exactly %v", reads, want) t.Errorf("reads = %v, want exactly %v", reads, want)
} }
@@ -44,7 +44,7 @@ func TestManifest(t *testing.T) {
states[s.Name] = s.History states[s.Name] = s.History
} }
if want := map[string]int{"asks": 5, "compositions": 5, "differences": 64, "puts": 20, "wants": 20, if want := map[string]int{"asks": 5, "compositions": 5, "differences": 64, "puts": 20, "wants": 20,
"placement": 64, "proposals": 5, "candidates": 5, "pulses": 0}; !reflect.DeepEqual(states, want) { "placement": 64, "proposals": 5, "candidates": 5, "pulses": 0, "findings": 5, "policies": 20, "seat-holders": 5}; !reflect.DeepEqual(states, want) {
t.Errorf("state = %v, want %v", states, want) t.Errorf("state = %v, want %v", states, want)
} }
if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 { if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 {
@@ -63,9 +63,14 @@ func TestManifest(t *testing.T) {
// ADR 0567 §1, §6-§10: it holds the seat mesh-placement, serving its seven verbs; keeps the seat's events on its stream // ADR 0567 §1, §6-§10: it holds the seat mesh-placement, serving its seven verbs; keeps the seat's events on its stream
// placement for 30 days; calls the controller's export, its four verbs that keep its assignments in step and its // placement for 30 days; calls the controller's export, its four verbs that keep its assignments in step and its
// hand-act log; and reads mesh-config's resolved values to compose every node a proposal touches. // hand-act log; and reads mesh-config's resolved values to compose every node a proposal touches.
if len(m.Claims) != 1 || m.Claims[0].Name != catalogue.PlacementSeat || m.Claims[0].At() != catalogue.ScopeMesh || // ADR 0565 §7, §9 (issue 1953 D4): and the seat mesh-wants, serving upgrade and bus.
!slices.Equal(m.Claims[0].Serves, []string{"placement", "proposals", "propose", "withdraw", "again", "adopt", "hand-back"}) { if len(m.Claims) != 2 || m.Claims[0].Name != catalogue.PlacementSeat || m.Claims[0].At() != catalogue.ScopeMesh ||
t.Errorf("the assigner claims %+v, want mesh-placement with its seven verbs", m.Claims) !slices.Equal(m.Claims[0].Serves, []string{"placement", "proposals", "propose", "withdraw", "again", "adopt", "hand-back"}) ||
m.Claims[1].Name != catalogue.WantsSeat || m.Claims[1].At() != catalogue.ScopeMesh || !slices.Equal(m.Claims[1].Serves, []string{"upgrade", "bus"}) {
t.Errorf("the assigner claims %+v, want mesh-placement with its seven verbs and mesh-wants with its two", m.Claims)
}
if seat, _ := catalogue.SeatNamed(catalogue.WantsSeat); catalogue.CanHold(m, seat) != nil {
t.Errorf("the assigner cannot hold mesh-wants: %v", catalogue.CanHold(m, seat))
} }
if seat, _ := catalogue.SeatNamed(catalogue.PlacementSeat); catalogue.CanHold(m, seat) != nil { if seat, _ := catalogue.SeatNamed(catalogue.PlacementSeat); catalogue.CanHold(m, seat) != nil {
t.Errorf("the assigner cannot hold the seat: %v", catalogue.CanHold(m, seat)) t.Errorf("the assigner cannot hold the seat: %v", catalogue.CanHold(m, seat))
@@ -79,7 +84,8 @@ func TestManifest(t *testing.T) {
} }
if want := []string{"seat:mesh-controller.assign", "seat:mesh-controller.hand-act", "seat:mesh-controller.pin", if want := []string{"seat:mesh-controller.assign", "seat:mesh-controller.hand-act", "seat:mesh-controller.pin",
"seat:mesh-controller.placement-export", "seat:mesh-controller.unassign", "seat:mesh-controller.unpin", "seat:mesh-controller.placement-export", "seat:mesh-controller.unassign", "seat:mesh-controller.unpin",
"seat:mesh-controller.warranted", "seat:node-backup.backed-up", "seat:node-backup.now"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) { "seat:mesh-controller.warranted", "seat:mesh-hand-acts.hand-act", "seat:mesh-hand-acts.hand-acts", "seat:mesh-hand-acts.warranted", // ADR 0565 §9, §11: the seat, the controller's only while it has no holder
"seat:node-backup.backed-up", "seat:node-backup.now"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) {
t.Errorf("invokes = %v, want %v", m.Invokes, want) t.Errorf("invokes = %v, want %v", m.Invokes, want)
} }
if !slices.Equal(m.Emits, []string{"put"}) { if !slices.Equal(m.Emits, []string{"put"}) {
+21 -2
View File
@@ -15,21 +15,34 @@
"adopt", "adopt",
"hand-back" "hand-back"
] ]
},
{
"name": "mesh-wants",
"scope": "mesh",
"serves": [
"upgrade",
"bus"
]
} }
], ],
"reads": [ "reads": [
"mesh-build-go.steps", "mesh-build-go.steps",
"mesh-conditions.takeover",
"mesh-config.kept",
"mesh-config.resolved", "mesh-config.resolved",
"mesh-controller.assignments", "mesh-controller.assignments",
"mesh-controller.builds",
"mesh-controller.composed", "mesh-controller.composed",
"mesh-controller.grants", "mesh-controller.grants",
"mesh-controller.lease",
"mesh-delivery-builder.build-log",
"mesh-delivery-builder.builds", "mesh-delivery-builder.builds",
"mesh-delivery-indexer.candidates", "mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-indexer.modules",
"mesh-delivery-teller.answers", "mesh-delivery-teller.answers",
"mesh-delivery-watchman.presence", "mesh-delivery-watchman.presence",
"mesh-grants.grants",
"mesh-provisioner.provisions",
"node-engine.have" "node-engine.have"
], ],
"uses": [ "uses": [
@@ -44,6 +57,9 @@
"seat:mesh-controller.unassign", "seat:mesh-controller.unassign",
"seat:mesh-controller.unpin", "seat:mesh-controller.unpin",
"seat:mesh-controller.warranted", "seat:mesh-controller.warranted",
"seat:mesh-hand-acts.hand-act",
"seat:mesh-hand-acts.hand-acts",
"seat:mesh-hand-acts.warranted",
"seat:node-backup.backed-up", "seat:node-backup.backed-up",
"seat:node-backup.now" "seat:node-backup.now"
], ],
@@ -64,7 +80,10 @@
{ "name": "compositions", "history": 5 }, { "name": "compositions", "history": 5 },
{ "name": "differences", "history": 64 }, { "name": "differences", "history": 64 },
{ "name": "puts", "history": 20 }, { "name": "puts", "history": 20 },
{ "name": "wants", "history": 20 } { "name": "wants", "history": 20 },
{ "name": "findings", "history": 5 },
{ "name": "policies", "history": 20 },
{ "name": "seat-holders", "history": 5 }
], ],
"streams": [ "streams": [
{ {
@@ -26,7 +26,7 @@ type queueAsker func(key string, value []byte) (json.RawMessage, error)
// Tools are the builder's tools: stuck reads, askAgain writes the reasks value, queue writes a queues value and waits for // Tools are the builder's tools: stuck reads, askAgain writes the reasks value, queue writes a queues value and waits for
// the builder's read, buildAgain writes a reasks value of mode build and waits for the builder's answer. // the builder's read, buildAgain writes a reasks value of mode build and waits for the builder's answer.
func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue, buildAgain queueAsker, now func() time.Time) []stdio.Tool { func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue, buildAgain queueAsker, since build.LogSince, now func() time.Time) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) } str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) }
return []stdio.Tool{ return []stdio.Tool{
{Name: build.StuckTool, {Name: build.StuckTool,
@@ -94,6 +94,20 @@ func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, q
return map[string]any{"key": key, "revision": r.Revision, "asks": r.Asks, "answer": r.Answer, return map[string]any{"key": key, "revision": r.Revision, "asks": r.Asks, "answer": r.Answer,
"next": build.StuckTool + " shows the key and its asks while they wait"}, nil "next": build.StuckTool + " shows the key and its asks while they wait"}, nil
}}, }},
{Name: delivery.BuildsSinceTool,
Description: "Every build that ended since a time, oldest first, from the builder's log build-log (kept 30 days): per build its module, " +
"fingerprint, source revision, repository, directory, commit and ref, the build node, when it ended, its outcome (done, failed or " +
"queued), the manifest built, the bases it was built against and each artifact made with its digest and reference. more says the " +
"log held more than one answer carries: ask again from the last build's time. What mesh-catalog registers a build from. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"since"}, "properties": map[string]any{
"since": map[string]any{"type": "string", "description": "RFC 3339 time; the builds that ended at or after it"}}},
Run: func(args map[string]any) (any, error) {
at, err := time.Parse(time.RFC3339Nano, str(args, "since"))
if err != nil {
return nil, fmt.Errorf("since %q is not an RFC 3339 time: %w", str(args, "since"), err)
}
return build.BuildsSince(since, at)
}},
{Name: build.AnnounceAgainTool, {Name: build.AnnounceAgainTool,
Description: "Announce a done build again (take in again, compose again): the builder writes the done key of module and fingerprint " + Description: "Announce a done build again (take in again, compose again): the builder writes the done key of module and fingerprint " +
"again, unchanged, as a new revision of the same key (its content, artifacts and manifest stay identical), so the controller's " + "again, unchanged, as a new revision of the same key (its content, artifacts and manifest stay identical), so the controller's " +
@@ -250,7 +264,7 @@ func main() {
a, err := build.DecodeReAsk(v) a, err := build.DecodeReAsk(v)
return err == nil && !a.Answered.IsZero() return err == nil && !a.Answered.IsZero()
}) })
if err := stdio.Serve("", Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), buildAgain, time.Now)); err != nil { if err := stdio.Serve("", Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), buildAgain, stdio.Log(delivery.BuilderLog).Since, time.Now)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] %v\n", err) fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] %v\n", err)
os.Exit(1) os.Exit(1)
} }
@@ -23,6 +23,7 @@ import (
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/buildstep" "git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
@@ -268,6 +269,12 @@ func startBedWith(t *testing.T, closed bool) *bed {
mk(delivery.ManifestsBucket, delivery.ManifestsHistory) mk(delivery.ManifestsBucket, delivery.ManifestsHistory)
b.builds = mk(delivery.BuilderBuildsBucket, delivery.BuilderBuildsHistory) b.builds = mk(delivery.BuilderBuildsBucket, delivery.BuilderBuildsHistory)
b.instances = mk(instancesBucket, 1) b.instances = mk(instancesBucket, 1)
// The builder's module log, as the runtime raises it (ADR 0297): its stream on the log's subjects.
logStream := buscompose.LogStreamName(delivery.BuilderModule, delivery.BuilderLog)
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: logStream, Subjects: []string{buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog) + ".>"},
AllowDirect: true}); err != nil {
t.Fatal(err)
}
b.fake = &fakeBuildModules{js: js, ctx: ctx, calls: map[string]int{}} b.fake = &fakeBuildModules{js: js, ctx: ctx, calls: map[string]int{}}
b.fake.closed.Store(closed) b.fake.closed.Store(closed)
b.fake.serve(t, "mesh-build-go", "n1", "n2") b.fake.serve(t, "mesh-build-go", "n1", "n2")
@@ -280,6 +287,29 @@ func startBedWith(t *testing.T, closed bool) *bed {
return b return b
} }
// logged is every entry of module's key in the builder's log, read through process.Conn.ReadLog on the controller's user.
func (b *bed) logged(t *testing.T, module string) []delivery.BuildLogEntry {
t.Helper()
conn, err := process.Connect(process.Account{URL: b.srvURL, Module: "mesh-delivery-view", User: "ctl", Password: "pw"})
if err != nil {
t.Fatal(err)
}
defer conn.Close()
entries, err := conn.ReadLog(delivery.BuilderModule, delivery.BuilderLog).Since(time.Unix(0, 0), module)
if err != nil {
t.Fatal(err)
}
var out []delivery.BuildLogEntry
for _, le := range entries {
e, err := delivery.DecodeBuildLogEntry(le.Value)
if err != nil {
t.Fatal(err)
}
out = append(out, e)
}
return out
}
// report writes a node's toolchains key of a build module and says so on the module's stream, as the module does. // report writes a node's toolchains key of a build module and says so on the module's stream, as the module does.
func (b *bed) report(t *testing.T, module, node, language, version, recipe string) { func (b *bed) report(t *testing.T, module, node, language, version, recipe string) {
t.Helper() t.Helper()
@@ -406,6 +436,13 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
if v.Toolchains["go"].Version != "go1.27.1" || v.Toolchains["typescript"].Version != "tsc5" { if v.Toolchains["go"].Version != "go1.27.1" || v.Toolchains["typescript"].Version != "tsc5" {
t.Errorf("toolchains = %+v", v.Toolchains) t.Errorf("toolchains = %+v", v.Toolchains)
} }
// 1955/L7 (ADR 0571 §1): the build's start and end are in the module log build-log under the module's key, read as a
// reader with the log's read reads them, the end naming the manifest and what was made.
ended := b.logged(t, "netcheck")
if len(ended) != 2 || ended[0].Event != delivery.BuildStartedEvent || ended[1].Event != delivery.BuildEndedEvent ||
ended[1].Manifest != digest || len(ended[1].Made) != 2 || ended[1].Node != "n1" {
t.Errorf("logged = %+v, want the start and the end of netcheck's build", ended)
}
for tool, want := range map[string]int{"mesh-build-go.build_go_fetch": 1, "mesh-build-go.build_go_build": 1, "mesh-build-node.build_node_fetch": 1, "mesh-build-node.build_node_build": 1} { for tool, want := range map[string]int{"mesh-build-go.build_go_fetch": 1, "mesh-build-go.build_go_build": 1, "mesh-build-node.build_node_fetch": 1, "mesh-build-node.build_node_build": 1} {
if got := b.fake.count(tool); got != want { if got := b.fake.count(tool); got != want {
t.Errorf("%s called %d times, want %d (on one node, ADR 0478 clause 4)", tool, got, want) t.Errorf("%s called %d times, want %d (on one node, ADR 0478 clause 4)", tool, got, want)
@@ -120,6 +120,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
Manifests: manifestReader{manifests}, Manifests: manifestReader{manifests},
Logf: logger.Printf, Logf: logger.Printf,
Stored: storedVia(calls.Call), Stored: storedVia(calls.Call),
Node: cred.Node,
} }
// A candidate's preview is merged onto the trunk head the indexer previewed it onto: source.main at the candidate's base // A candidate's preview is merged onto the trunk head the indexer previewed it onto: source.main at the candidate's base
// revision (the gate's source state). The source's revisions are watched with their history (a direct get of one revision // revision (the gate's source state). The source's revisions are watched with their history (a direct get of one revision
@@ -129,6 +130,13 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
trunks := &trunkSource{ctx: ctx, js: js, heads: &sourceHeads{}, moved: b.TrunkMoved, errs: errs, logf: logger.Printf} trunks := &trunkSource{ctx: ctx, js: js, heads: &sourceHeads{}, moved: b.TrunkMoved, errs: errs, logf: logger.Printf}
trunks.start() trunks.start()
b.Trunk = trunks.head b.Trunk = trunks.head
// The builder's log (ADR 0571 §1): each build's start and end in its module log `build-log`, read by the view, the
// indexer and the assigner through their reads, and by mesh-catalog through the tool builder_builds_since.
b.Log = func(e delivery.BuildLogEntry) {
if err := appendLog(context.WithoutCancel(ctx), js, cred, e); err != nil {
logger.Printf("%v", err)
}
}
b.Said = func(e build.AskedAgain) { b.Said = func(e build.AskedAgain) {
if err := emitAskedAgain(context.WithoutCancel(ctx), js, cred, e); err != nil { if err := emitAskedAgain(context.WithoutCancel(ctx), js, cred, e); err != nil {
logger.Printf("%v", err) logger.Printf("%v", err)
@@ -153,11 +161,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
} }
// The toolchains first, so a build that starts from the first modules value finds the nodes. // The toolchains first, so a build that starts from the first modules value finds the nodes.
watch := newToolchainWatch(ctx, js, tools, b.NodesChanged, logger) watch := newToolchainWatch(ctx, js, tools, b.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's // Only the toolchain reports of nodes the catalogue places the build module on count (issue 1940, ADR 0549 §3): the
// assignments are settled first, so the toolchains settle through the gate already knowing them. // placement is settled first, so the toolchains settle through the gate already knowing it.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf) gate := buildrun.NewToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, b.NodesChanged, logger) placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, b.NodesChanged, logger.Printf)
if err != nil { if err != nil {
return err return err
} }
@@ -254,7 +262,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return nil return nil
case err := <-errs: case err := <-errs:
return err return err
case err := <-assignErrs: case err := <-placementErrs:
return err return err
case err := <-watch.errs: case err := <-watch.errs:
return err return err
@@ -8,6 +8,8 @@ import (
"strings" "strings"
"time" "time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
@@ -138,3 +140,26 @@ func announcedAgain(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e
logf("%s.%s: the answer could not be written: %v", build.ReAsksState, e.Key(), err) logf("%s.%s: the answer could not be written: %v", build.ReAsksState, e.Key(), err)
} }
} }
// appendLog appends one entry to the builder's module log `build-log` (ADR 0297, ADR 0571 §1) under the module's key
// (delivery.BuildLogKey), on the log's own subject: a module's program appends to the logs it keeps on its own credential.
// The id makes a repeated append of one entry one entry.
func appendLog(ctx context.Context, js jetstream.JetStream, cred process.Account, e delivery.BuildLogEntry) error {
data, err := delivery.EncodeBuildLogEntry(e)
if err != nil {
return err
}
key, err := delivery.BuildLogKey(e.Module)
if err != nil {
return err
}
msg := nats.NewMsg(buscompose.LogSubject(cred.Module, delivery.BuilderLog) + "." + key)
msg.Data = data
id := e.Event + "." + e.Module + "." + e.FP + "." + e.At.Format(time.RFC3339Nano)
pub, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if _, err := js.PublishMsg(pub, msg, jetstream.WithMsgID(id)); err != nil {
return fmt.Errorf("logging %s of %s at %s: %w", e.Event, e.Module, e.FP, err)
}
return nil
}
@@ -0,0 +1,73 @@
package main
import (
"bytes"
"context"
"errors"
"log"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// 1945 (a part exits at start for a bucket it works without): a bus that refuses every toolchains bucket leaves the
// watch unstarted and the part running, said; a refused bucket it cannot work without is an error naming the bucket and
// its grant. The gate over the seat's placement, and its refusal, are mesh-sdk's (buildrun.ToolchainGate,
// issue 774).
func TestARefusedBucketIsSaidAndThePartGoesOn(t *testing.T) {
set := &buildrun.ToolchainSet{}
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-build-node_toolchains")}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "builder", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
// The gate's source (the previews' trunk heads): refused at start, the builder goes on, and a candidate asking a trunk
// head is told why and opens it again then.
src := &trunkSource{ctx: context.Background(), js: refusingBus{refuseOpen: true, err: refused.err}, heads: &sourceHeads{}, errs: make(chan error, 1), logf: logger.Printf}
src.start()
if _, err := src.head(context.Background(), 7, "novox/mesh-catalog"); err == nil || !strings.Contains(err.Error(), delivery.SourceBucket) {
t.Errorf("a trunk head with the source refused: %v, want why", err)
}
if src.opened {
t.Error("a refused source counts as opened, so it is never opened again")
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
@@ -1,211 +0,0 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -1,228 +0,0 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
// 1945 (a part exits at start for a bucket it works without): a bus that refuses the assignments' watch, or every
// toolchains bucket, leaves the watches unstarted and the part running, said; a refused bucket it cannot work without
// is an error naming the bucket and its grant.
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-controller_assignments")}
if _, err := watchAssignments(context.Background(), refused, newToolchainGate(nil, set.Put, t.Logf), func(context.Context) {}, logger); err != nil {
t.Errorf("a refused watch of the assignments stopped the start: %v", err)
}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "builder", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
// The gate's source (the previews' trunk heads): refused at start, the builder goes on, and a candidate asking a trunk
// head is told why and opens it again then.
src := &trunkSource{ctx: context.Background(), js: refusingBus{refuseOpen: true, err: refused.err}, heads: &sourceHeads{}, errs: make(chan error, 1), logf: logger.Printf}
src.start()
if _, err := src.head(context.Background(), 7, "novox/mesh-catalog"); err == nil || !strings.Contains(err.Error(), delivery.SourceBucket) {
t.Errorf("a trunk head with the source refused: %v, want why", err)
}
if src.opened {
t.Error("a refused source counts as opened, so it is never opened again")
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
@@ -9,7 +9,9 @@ import (
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"mesh-delivery-builder/internal/build" "mesh-delivery-builder/internal/build"
) )
@@ -18,17 +20,27 @@ import (
// consumer of artifacts-gone (mesh-controller internal/broker EventsStream). // consumer of artifacts-gone (mesh-controller internal/broker EventsStream).
const eventsStream = "EVENTS" const eventsStream = "EVENTS"
// SeatHeld is the seat mesh-artifact-sweep's verb held (ADR 0571 §2).
const SeatHeld = "seat:" + catalogue.ArtifactSweepSeat + ".held"
// toolCaller calls a tool or a seat verb as the builder's own account (mesh-sdk process.Conn.Call). // toolCaller calls a tool or a seat verb as the builder's own account (mesh-sdk process.Conn.Call).
type toolCaller func(key string, body any) (json.RawMessage, error) type toolCaller func(key string, body any) (json.RawMessage, error)
// storedVia is build.Builder's Stored: it asks the controller's verb held (mesh-sdk delivery.HeldVerb; novox/hq issue // storedVia is build.Builder's Stored: it asks the controller's verb held (mesh-sdk delivery.HeldVerb; novox/hq issue
// 1937) and answers the references the store no longer holds. One it could not ask about, with none gone, is an error: // 1937) and answers the references the store no longer holds. One it could not ask about, with none gone, is an error:
// the key is then left done, and the controller's take-in still refuses a gone blob. // the key is then left done, and the controller's take-in still refuses a gone blob.
//
// It asks the seat mesh-artifact-sweep's `held` (novox/hq ADR 0571 §2), and the controller's while that seat has no holder that
// answers (a named exception of clause 10, until the lease write).
func storedVia(call toolCaller) func(ctx context.Context, references []string) ([]string, error) { func storedVia(call toolCaller) func(ctx context.Context, references []string) ([]string, error) {
return func(_ context.Context, references []string) ([]string, error) { return func(_ context.Context, references []string) ([]string, error) {
raw, err := call(delivery.HeldVerb, map[string]any{"references": references}) body := map[string]any{"references": references}
raw, err := call(SeatHeld, body)
if handacts.NoResponder(err) {
raw, err = call(delivery.HeldVerb, body)
}
if err != nil { if err != nil {
return nil, fmt.Errorf("asking %s: %w", delivery.HeldVerb, err) return nil, fmt.Errorf("asking held: %w", err)
} }
a, err := delivery.DecodeHeldAnswer(raw, references) a, err := delivery.DecodeHeldAnswer(raw, references)
if err != nil { if err != nil {
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-builder
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 h1:Tt16HJk8JZRce1i3JzzDSNdCWOrv7dw8FQ/bckNQ5Ig= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24 h1:s0XXZtR7CFPBbnaDygWnE4/Q827iEFt0YfEzujk4Q9g=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -7,6 +7,7 @@ import (
"fmt" "fmt"
"maps" "maps"
"sort" "sort"
"strconv"
"strings" "strings"
"sync" "sync"
"time" "time"
@@ -68,6 +69,11 @@ type Builder struct {
Askings int Askings int
// Said is told each ask asked again (the builder's event build.asked-again). Nil tells nobody. // Said is told each ask asked again (the builder's event build.asked-again). Nil tells nobody.
Said func(AskedAgain) Said func(AskedAgain)
// Log is told each build's start and end, the entries of the builder's log (novox/hq ADR 0571 §1). Nil tells nobody.
Log func(delivery.BuildLogEntry)
// Now is the clock of the log; nil is time.Now. Node is the node the builder runs on, which asked the build.
Now func() time.Time
Node string
once sync.Once once sync.Once
sem chan struct{} sem chan struct{}
@@ -429,6 +435,7 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
return return
} }
b.handClaimed(key, cl.Rev) b.handClaimed(key, cl.Rev)
b.logBuild(LogStarted(module, m, b.refOf(module, m.FP2), b.Node, b.now()))
if cl.Unasked != nil { if cl.Unasked != nil {
b.logf("%v", cl.Unasked) b.logf("%v", cl.Unasked)
} }
@@ -478,6 +485,7 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
b.logf("%s: the build could not be written: %v", cl.Key, err) b.logf("%s: the build could not be written: %v", cl.Key, err)
return return
} }
b.logBuild(LogEnded(module, m, b.refOf(module, m.FP2), b.Node, b.now(), result))
if result.State == delivery.BuildFailed { if result.State == delivery.BuildFailed {
b.logf("%s: failed: %s", cl.Key, result.Error) b.logf("%s: failed: %s", cl.Key, result.Error)
} }
@@ -872,3 +880,60 @@ func needsFetch(tasks []Task) bool {
} }
return false return false
} }
func (b *Builder) now() time.Time {
if b.Now != nil {
return b.Now()
}
return time.Now()
}
// logBuild tells the log an entry, said in the journal when it cannot be one.
func (b *Builder) logBuild(e delivery.BuildLogEntry) {
if b.Log == nil {
return
}
if _, err := delivery.EncodeBuildLogEntry(e); err != nil {
b.logf("%s at %s: no log entry: %v", e.Module, e.FP, err)
return
}
b.Log(e)
}
// refOf is the ref a candidate's build of module's fingerprint is built for, "" for the trunk's.
func (b *Builder) refOf(module, fp string) string {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return ""
}
b.mu.Lock()
defer b.mu.Unlock()
return b.via[key].Ref
}
// LogStarted is the log's entry of a build of module's fingerprint starting (ADR 0571 §1): the module, the fingerprint, the
// source revision, where it is built from (repository, directory, commit, the candidate's ref), the node of the builder
// that asked it and the time.
func LogStarted(module string, m delivery.Module, ref, by string, at time.Time) delivery.BuildLogEntry {
dir := m.Path
if dir == "" {
dir = delivery.RootPath
}
return delivery.BuildLogEntry{Event: delivery.BuildStartedEvent, Module: module, FP: m.FP2, Source: strconv.FormatUint(m.Src, 10),
Repository: m.Repository, Directory: dir, Commit: m.Commit, Ref: ref, Node: by, At: at.UTC()}
}
// LogEnded is the log's entry of that build ending: the outcome (done, failed or queued), the manifest built, the bases it
// was built against and each artifact made, as the done key has them: what mesh-catalog registers a build from.
func LogEnded(module string, m delivery.Module, ref, by string, at time.Time, result delivery.BuilderBuild) delivery.BuildLogEntry {
e := LogStarted(module, m, ref, by, at)
e.Event, e.Outcome, e.Manifest = delivery.BuildEndedEvent, result.State, m.Manifest
if result.Manifest != "" {
e.Manifest = result.Manifest
}
if m.Inputs2 != nil {
e.Against = append([]string(nil), m.Inputs2.Bases...)
}
e.Made = append([]delivery.BuiltArtifact(nil), result.Artifacts...)
return e
}
@@ -5,7 +5,6 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
@@ -13,9 +12,11 @@ import (
"testing" "testing"
"time" "time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buildstep" "git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
) )
// buildModules simulates the two build modules on the nodes: a long tool answers a step and key at once and writes the // buildModules simulates the two build modules on the nodes: a long tool answers a step and key at once and writes the
@@ -221,7 +222,42 @@ var (
func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7, ADR 0478 clauses 1, 4 func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7, ADR 0478 clauses 1, 4
r := newRig(t, goBundle, archiveA, tsBundle) r := newRig(t, goBundle, archiveA, tsBundle)
var logged []delivery.BuildLogEntry
var lmu sync.Mutex
r.b.Node, r.b.Log = "novox", func(e delivery.BuildLogEntry) { lmu.Lock(); logged = append(logged, e); lmu.Unlock() }
v := r.consider(t, "modules/m") v := r.consider(t, "modules/m")
// 1955/L7 (ADR 0571 §1): the build is one start and one end in the builder's log, every field named, its duration read
// from the two.
if len(logged) != 2 || logged[0].Event != delivery.BuildStartedEvent || logged[1].Event != delivery.BuildEndedEvent ||
logged[1].Outcome != delivery.BuildDone || len(logged[1].Made) != 3 || logged[1].Made[0].Reference == "" ||
logged[1].Manifest != manifestD || logged[0].Manifest != "" || logged[0].Repository != "novox/mesh-catalog" ||
logged[0].Directory != "modules/m" || logged[0].Commit != "7a74cee83f88e6585a5511c3cc53bdf0f5303d20" || logged[0].Node != "novox" ||
logged[0].FP != fp1 || logged[0].Module != "m" {
t.Fatalf("logged = %+v", logged)
}
for _, e := range logged {
if _, err := delivery.EncodeBuildLogEntry(e); err != nil {
t.Errorf("an entry the log refuses: %v", err)
}
}
if _, ok := delivery.BuildDuration(logged[0], logged[1]); !ok {
t.Error("no duration from the two entries")
}
// builder_builds_since (ADR 0571 clause 9) answers the end alone from the log's page, and says when the log holds more.
var page stdio.LogPage
for i, e := range logged {
raw, _ := delivery.EncodeBuildLogEntry(e)
page.Entries = append(page.Entries, stdio.LogEntry{Sequence: uint64(i + 1), Key: e.Module, Value: raw, At: e.At})
}
page.Next = 3
since, err := BuildsSince(func(time.Time, string, int) (*stdio.LogPage, error) { return &page, nil }, logged[0].At)
if err != nil || len(since.Builds) != 1 || since.Builds[0].Event != delivery.BuildEndedEvent || len(since.Builds[0].Made) != 3 || !since.More {
t.Errorf("builds since = %+v, %v, want the one end and more", since, err)
}
page.Entries[0].Value = []byte(`{"v":0}`)
if _, err := BuildsSince(func(time.Time, string, int) (*stdio.LogPage, error) { return &page, nil }, logged[0].At); err == nil {
t.Error("an entry that does not read was skipped, want it refused")
}
if v.State != delivery.BuildDone || v.By != "novox.builder" || v.Manifest != manifestD || len(v.Artifacts) != 3 { if v.State != delivery.BuildDone || v.By != "novox.builder" || v.Manifest != manifestD || len(v.Artifacts) != 3 {
t.Fatalf("build = %+v, want done by novox.builder with three artifacts", v) t.Fatalf("build = %+v, want done by novox.builder with three artifacts", v)
} }
@@ -0,0 +1,43 @@
package build
import (
"fmt"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// BuildsSinceLimit is how many entries one answer of BuildsSinceTool reads of the log; More says the log held more.
const BuildsSinceLimit = 500
// LogSince is one page of the builder's log from a time on: stdio.Log(BuilderLog).Since.
type LogSince func(at time.Time, keys string, limit int) (*stdio.LogPage, error)
// BuildsSince answers delivery.BuildsSinceTool (ADR 0571 clause 9): every ended build in the builder's log from since on,
// oldest first, read through the runtime's log verbs; More when the page was not the log's last, so the caller asks again
// from the last build's time. An entry the log holds that does not read is refused, naming its sequence: a reader that
// skipped it would register too little.
func BuildsSince(read LogSince, since time.Time) (delivery.BuildsSince, error) {
out := delivery.BuildsSince{Since: since.UTC(), Builds: []delivery.BuildLogEntry{}}
page, err := read(since, "", BuildsSinceLimit)
if err != nil {
return out, fmt.Errorf("the builder's log %s from %s: %w", delivery.BuilderLog, since.Format(time.RFC3339), err)
}
if page == nil {
return out, nil
}
var entries []delivery.BuildLogEntry
for _, le := range page.Entries {
e, err := delivery.DecodeBuildLogEntry(le.Value)
if err != nil {
return out, fmt.Errorf("the builder's log %s, entry %d: %w", delivery.BuilderLog, le.Sequence, err)
}
entries = append(entries, e)
}
if ended := delivery.EndedSince(entries, since); ended != nil {
out.Builds = ended
}
out.More = page.Next != 0
return out, nil
}
+22 -10
View File
@@ -10,6 +10,8 @@ import (
"slices" "slices"
"strings" "strings"
"testing" "testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
) )
// The controller's manifest reader is mesh-controller internal/catalogue (Manifest, manifest.go: Consumes :521, // The controller's manifest reader is mesh-controller internal/catalogue (Manifest, manifest.go: Consumes :521,
@@ -40,10 +42,14 @@ type manifest struct {
Emits []string `json:"emits"` Emits []string `json:"emits"`
Consumes []string `json:"consumes"` Consumes []string `json:"consumes"`
State []state `json:"state"` State []state `json:"state"`
Reads []string `json:"reads"` Logs []struct {
Uses []string `json:"uses"` Name string `json:"name"`
Invokes []string `json:"invokes"` MaxAgeDays int `json:"max-age-days"`
Config map[string]struct { } `json:"logs"`
Reads []string `json:"reads"`
Uses []string `json:"uses"`
Invokes []string `json:"invokes"`
Config map[string]struct {
Type string `json:"type"` Type string `json:"type"`
Description string `json:"description"` Description string `json:"description"`
Pattern string `json:"pattern"` Pattern string `json:"pattern"`
@@ -92,8 +98,8 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
} }
// 1887/A4c: the builder chooses no node, so it reads neither presence nor have, invokes no build module's tool and // 1887/A4c: the builder chooses no node, so it reads neither presence nor have, invokes no build module's tool and
// holds no capacity; it puts asks on the three build seats and reads their steps (ADR 0525 §1, §4, §5). // holds no capacity; it puts asks on the three build seats and reads their steps (ADR 0525 §1, §4, §5).
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-image.steps", "mesh-build-node.steps", "mesh-build-node.toolchains", "mesh-controller.assignments", if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-image.steps", "mesh-build-node.steps", "mesh-build-node.toolchains",
"mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules"}; !reflect.DeepEqual(sorted(m.Reads), want) { "mesh-delivery-assigner.placement", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want) t.Errorf("reads = %v, want %v", m.Reads, want)
} }
if want := []string{"node-build-go", "node-build-image", "node-build-node"}; !reflect.DeepEqual(sorted(m.Uses), want) { if want := []string{"node-build-go", "node-build-image", "node-build-node"}; !reflect.DeepEqual(sorted(m.Uses), want) {
@@ -101,12 +107,14 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
} }
// 1937/AC3: the builder asks the controller whether the artifact store still holds a done key's blobs (held), and no // 1937/AC3: the builder asks the controller whether the artifact store still holds a done key's blobs (held), and no
// other tool; it holds no capacity. // other tool; it holds no capacity.
if want := []string{"seat:mesh-controller.held"}; !reflect.DeepEqual(m.Invokes, want) || len(m.Config) != 0 { // 1955/L7 (ADR 0571 §2): the seat mesh-artifact-sweep's held, the controller's only while the seat has no holder (clause 10).
if want := []string{"seat:mesh-artifact-sweep.held", "seat:mesh-controller.held"}; !reflect.DeepEqual(sorted(m.Invokes), want) || len(m.Config) != 0 {
t.Errorf("invokes = %v, config = %v, want %v and no config", m.Invokes, m.Config, want) t.Errorf("invokes = %v, config = %v, want %v and no config", m.Invokes, m.Config, want)
} }
// The builder is woken by a build module's report of a toolchain, as the indexer is: the bucket may not exist when it // The builder is woken by a build module's report of a toolchain, as the indexer is: the bucket may not exist when it
// starts; and by the controller's artifacts-gone, a take-in refused for a gone blob (1937/AC4). // starts; and by the controller's artifacts-gone, a take-in refused for a gone blob (1937/AC4).
if want := []string{"mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-controller.artifacts-gone"}; !reflect.DeepEqual(sorted(m.Consumes), want) { // 1955/L7: and the seat mesh-artifact-sweep's, the controller's until the lease write (its take-in sweeps until then).
if want := []string{"mesh-artifact-sweep.artifacts-gone", "mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-controller.artifacts-gone"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
t.Errorf("consumes = %v, want %v", m.Consumes, want) t.Errorf("consumes = %v, want %v", m.Consumes, want)
} }
if b, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 || b != "${dir:state}/broker" { if b, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 || b != "${dir:state}/broker" {
@@ -116,13 +124,17 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
t.Errorf("secrets-owner = %q", m.SecretsOwner) t.Errorf("secrets-owner = %q", m.SecretsOwner)
} }
// 1945: the builder's tools (builder_seat_queue: note 62), served by its tools bundle, and the event it says when it asks an ask again. // 1945: the builder's tools (builder_seat_queue: note 62), served by its tools bundle, and the event it says when it asks an ask again.
if want := []string{"builder_announce_again", "builder_ask_again", "builder_build_again", "builder_seat_queue", "builder_stuck"}; !reflect.DeepEqual(sorted(m.Tools), want) { if want := []string{"builder_announce_again", "builder_ask_again", "builder_build_again", "builder_builds_since", "builder_seat_queue", "builder_stuck"}; !reflect.DeepEqual(sorted(m.Tools), want) {
t.Errorf("tools = %v, want %v", m.Tools, want) t.Errorf("tools = %v, want %v", m.Tools, want)
} }
if want := []string{"build.asked-again"}; !reflect.DeepEqual(m.Emits, want) { if want := []string{"build.asked-again"}; !reflect.DeepEqual(m.Emits, want) {
t.Errorf("emits = %v, want %v", m.Emits, want) t.Errorf("emits = %v, want %v", m.Emits, want)
} }
for _, k := range []string{"streams", "logs", "claims", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} { // 1955/L7 (ADR 0571 §1): its module log, each build's start and end, kept 30 days; no bare stream.
if len(m.Logs) != 1 || m.Logs[0].Name != delivery.BuilderLog || m.Logs[0].MaxAgeDays != delivery.BuilderLogDays {
t.Errorf("logs = %+v, want %s kept %d days", m.Logs, delivery.BuilderLog, delivery.BuilderLogDays)
}
for _, k := range []string{"streams", "claims", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
if _, ok := top[k]; ok { if _, ok := top[k]; ok {
t.Errorf("the manifest declares %q: the builder has none", k) t.Errorf("the manifest declares %q: the builder has none", k)
} }
+7 -2
View File
@@ -6,6 +6,7 @@
"builder_announce_again", "builder_announce_again",
"builder_ask_again", "builder_ask_again",
"builder_build_again", "builder_build_again",
"builder_builds_since",
"builder_seat_queue", "builder_seat_queue",
"builder_stuck" "builder_stuck"
], ],
@@ -13,11 +14,13 @@
"build.asked-again" "build.asked-again"
], ],
"consumes": [ "consumes": [
"mesh-artifact-sweep.artifacts-gone",
"mesh-build-go.toolchain.reported", "mesh-build-go.toolchain.reported",
"mesh-build-node.toolchain.reported", "mesh-build-node.toolchain.reported",
"mesh-controller.artifacts-gone" "mesh-controller.artifacts-gone"
], ],
"invokes": [ "invokes": [
"seat:mesh-artifact-sweep.held",
"seat:mesh-controller.held" "seat:mesh-controller.held"
], ],
"state": [ "state": [
@@ -27,13 +30,16 @@
{ "name": "queues", "history": 1 }, { "name": "queues", "history": 1 },
{ "name": "reasks", "history": 1 } { "name": "reasks", "history": 1 }
], ],
"logs": [
{ "name": "build-log", "max-age-days": 30 }
],
"reads": [ "reads": [
"mesh-build-go.steps", "mesh-build-go.steps",
"mesh-build-go.toolchains", "mesh-build-go.toolchains",
"mesh-build-image.steps", "mesh-build-image.steps",
"mesh-build-node.steps", "mesh-build-node.steps",
"mesh-build-node.toolchains", "mesh-build-node.toolchains",
"mesh-controller.assignments", "mesh-delivery-assigner.placement",
"mesh-delivery-gate.source", "mesh-delivery-gate.source",
"mesh-delivery-indexer.candidates", "mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests", "mesh-delivery-indexer.manifests",
@@ -44,7 +50,6 @@
"node-build-image", "node-build-image",
"node-build-node" "node-build-node"
], ],
"own-secrets": { "own-secrets": {
"broker": "${dir:state}/broker" "broker": "${dir:state}/broker"
}, },
@@ -170,6 +170,21 @@ func TryTools(tr Trying) []stdio.Tool {
} }
} }
// Seat is the seat the checker holds (novox/hq ADR 0547 §10, issue 774); its verbs are served as tools named <seat>.<verb>.
const Seat = "mesh-delivery-checker"
// SeatTryTools are the try's tools as the seat's verbs: try, try-read and try-end, run by the same code as checker_try,
// checker_try_read and checker_try_end, so the answer is the same. The module tools stay beside them.
func SeatTryTools(tr Trying) []stdio.Tool {
verbs := map[string]string{check.TryTool: "try", check.TryReadTool: "try-read", check.TryEndTool: "try-end"}
var out []stdio.Tool
for _, t := range TryTools(tr) {
t.Name = Seat + "." + verbs[t.Name]
out = append(out, t)
}
return out
}
// Tools are the read tools over the checker's states. // Tools are the read tools over the checker's states.
func Tools(r check.Reader) []stdio.Tool { func Tools(r check.Reader) []stdio.Tool {
return []stdio.Tool{ return []stdio.Tool{
@@ -308,7 +323,7 @@ func main() {
_, err := verdicts.Update(key, json.RawMessage(value), revision) _, err := verdicts.Update(key, json.RawMessage(value), revision)
return err return err
}} }}
tools := append(append(Tools(r), TryTools(trying)...), RecoveryTools(r, written)...) tools := append(append(append(Tools(r), TryTools(trying)...), SeatTryTools(trying)...), RecoveryTools(r, written)...)
if err := stdio.Serve("", logged(tools)); err != nil { if err := stdio.Serve("", logged(tools)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-checker-tools] %v\n", err) fmt.Fprintf(os.Stderr, "[mesh-delivery-checker-tools] %v\n", err)
os.Exit(1) os.Exit(1)
@@ -88,6 +88,24 @@ func TestTheTryToolsAskAndReadATry(t *testing.T) { // ADR 0501 §5, the agreed i
if err != nil || !strings.Contains(string(raw.(json.RawMessage)), `"state":"queued"`) { if err != nil || !strings.Contains(string(raw.(json.RawMessage)), `"state":"queued"`) {
t.Errorf("read = %s, %v", raw, err) t.Errorf("read = %s, %v", raw, err)
} }
// The seat's verbs answer as the tools do (ADR 0547 §10, issue 774): the same ask is the same try, and the read is alike.
verbs := SeatTryTools(Trying{Tries: store, Ender: store, Now: func() time.Time { return time.Unix(1, 5) },
Trunks: func() (map[string]string, error) {
return map[string]string{"novox/mesh-delivery": strings.Repeat("a", 40)}, nil
}})
if len(verbs) != 3 || verbs[0].Name != Seat+".try" || verbs[1].Name != Seat+".try-read" || verbs[2].Name != Seat+".try-end" {
t.Fatalf("verbs = %+v", verbs)
}
again, err := verbs[0].Run(map[string]any{"repository": "novox/mesh-delivery", "head": strings.Repeat("1", 40), "trunk": "main"})
if err != nil || again.(map[string]any)["id"] != id {
t.Errorf("verb try = %v, %v, want the waiting try %s", again, err, id)
}
if viaVerb, err := verbs[1].Run(map[string]any{"id": id}); err != nil || string(viaVerb.(json.RawMessage)) != string(raw.(json.RawMessage)) {
t.Errorf("verb try-read = %s, %v; tool = %s", viaVerb, err, raw)
}
if _, err := verbs[0].Run(map[string]any{"repository": "novox/hq", "head": strings.Repeat("1", 40)}); err == nil {
t.Error("the verb tried a repository nothing checks")
}
// Issue 734: a try whose ask was lost is ended by hand with why, and its head is asked anew. // Issue 734: a try whose ask was lost is ended by hand with why, and its head is asked anew.
ended, err := tools[2].Run(map[string]any{"id": id, "why": "its ask was lost"}) ended, err := tools[2].Run(map[string]any{"id": id, "why": "its ask was lost"})
if err != nil || ended.(map[string]any)["state"] != "error" { if err != nil || ended.(map[string]any)["state"] != "error" {
@@ -208,11 +208,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// The toolchains first, so a check that starts from the first candidate finds the nodes. // The toolchains first, so a check that starts from the first candidate finds the nodes.
watch := newToolchainWatch(ctx, js, tools, ch.NodesChanged, logger) watch := newToolchainWatch(ctx, js, tools, ch.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's // Only the toolchain reports of nodes the catalogue places the build module on count (issue 1940, ADR 0549 §3): the
// assignments are settled first, so the toolchains settle through the gate already knowing them. // placement is settled first, so the toolchains settle through the gate already knowing it.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf) gate := buildrun.NewToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, ch.NodesChanged, logger) placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, ch.NodesChanged, logger.Printf)
if err != nil { if err != nil {
return err return err
} }
@@ -362,7 +362,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return nil return nil
case err := <-errs: case err := <-errs:
return err return err
case err := <-assignErrs: case err := <-placementErrs:
return err return err
case err := <-watch.errs: case err := <-watch.errs:
return err return err
@@ -0,0 +1,62 @@
package main
import (
"bytes"
"context"
"errors"
"log"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// 1945 (a part exits at start for a bucket it works without): a bus that refuses every toolchains bucket leaves the
// watch unstarted and the part running, said; a refused bucket it cannot work without is an error naming the bucket and
// its grant. The gate over the seat's placement, and its refusal, are mesh-sdk's (buildrun.ToolchainGate,
// issue 774).
func TestARefusedBucketIsSaidAndThePartGoesOn(t *testing.T) {
set := &buildrun.ToolchainSet{}
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-build-node_toolchains")}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "checker", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
@@ -1,211 +0,0 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -1,218 +0,0 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
// 1945 (a part exits at start for a bucket it works without): a bus that refuses the assignments' watch, or every
// toolchains bucket, leaves the watches unstarted and the part running, said; a refused bucket it cannot work without
// is an error naming the bucket and its grant.
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-controller_assignments")}
if _, err := watchAssignments(context.Background(), refused, newToolchainGate(nil, set.Put, t.Logf), func(context.Context) {}, logger); err != nil {
t.Errorf("a refused watch of the assignments stopped the start: %v", err)
}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "checker", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-checker
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 h1:Tt16HJk8JZRce1i3JzzDSNdCWOrv7dw8FQ/bckNQ5Ig= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24 h1:s0XXZtR7CFPBbnaDygWnE4/Q827iEFt0YfEzujk4Q9g=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -151,6 +151,31 @@ func IsUntrunked(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && v.Failed.Machine == "checker" && strings.HasPrefix(v.Failed.Words, UntrunkedWords) return v.State == delivery.VerdictFail && v.Failed.Machine == "checker" && strings.HasPrefix(v.Failed.Words, UntrunkedWords)
} }
// **A candidate outside the mesh is not the checker's** (sweep 2026-10-30 item 7): the forge carries repositories the mesh
// does not deliver (an operator's own), whose pull requests the indexer writes as candidates like any. Their trunk the
// gate never names, so each was judged "cannot be judged" and owed the status error on mesh/check, which the forge
// module wrote 119 times per head into a repository the mesh's forge account may not write (403). A candidate every pull
// request of which is of a repository the gate's source.main names no trunk head of, and touches no module, is outside
// the mesh: judged nothing, owed nothing on the forge, said once in the log and in checker_status (outside_the_mesh). A
// new repository of the mesh is not outside: its first pull request adds a module, and is said untrunked (issue 1422).
// OutsideWords say why a candidate is outside the mesh.
const OutsideWords = "outside the mesh: the gate's source.main names no trunk head of its repositories and it touches no module, so the checker judges nothing and owes no status on the forge"
// Outside says every pull request of c is of a repository trunk names no head of and touches no module. Never with
// trunk unknown (nil): before the gate's source is read nothing is outside.
func Outside(c delivery.Candidate, trunk map[string]string) bool {
if trunk == nil || len(c.Pulls) == 0 {
return false
}
for _, p := range c.Pulls {
if trunk[p.Repository] != "" || len(p.Modules) > 0 || len(p.Fps) > 0 {
return false
}
}
return true
}
// leftOf are the pull requests of a candidate whose repository has no trunk head known, with Trunk empty, in repository and // leftOf are the pull requests of a candidate whose repository has no trunk head known, with Trunk empty, in repository and
// number order; nil when every repository's is known. // number order; nil when every repository's is known.
func leftOf(c delivery.Candidate, trunk map[string]string) []Pull { func leftOf(c delivery.Candidate, trunk map[string]string) []Pull {
@@ -104,6 +104,8 @@ type Checker struct {
held map[string]string // by verdict key: the turn that holds its check off, as last said held map[string]string // by verdict key: the turn that holds its check off, as last said
live map[string]liveCheck // by verdict key: the check running in this process, ended when its heads go live map[string]liveCheck // by verdict key: the check running in this process, ended when its heads go
settling map[string]*window // by candidates key: the window its heads settle in settling map[string]*window // by candidates key: the window its heads settle in
// outsideSaid are the candidates outside the mesh said in the log, by group and heads.
outsideSaid map[string]bool
} }
// liveCheck is a check running in this process: the candidates key of its group and the end of its run. // liveCheck is a check running in this process: the candidates key of its group and the end of its run.
@@ -160,6 +162,7 @@ func (ch *Checker) init() {
ch.known, ch.inflight, ch.again, ch.tried = map[string]delivery.Candidate{}, map[string]bool{}, map[string]bool{}, map[string]bool{} ch.known, ch.inflight, ch.again, ch.tried = map[string]delivery.Candidate{}, map[string]bool{}, map[string]bool{}, map[string]bool{}
ch.nodesFail, ch.held = map[string]uint64{}, map[string]string{} ch.nodesFail, ch.held = map[string]uint64{}, map[string]string{}
ch.live, ch.settling = map[string]liveCheck{}, map[string]*window{} ch.live, ch.settling = map[string]liveCheck{}, map[string]*window{}
ch.outsideSaid = map[string]bool{}
}) })
} }
@@ -327,6 +330,10 @@ func (ch *Checker) heads() map[string]string {
func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidate) { func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidate) {
heads := ch.heads() heads := ch.heads()
if Outside(c, heads) {
ch.outside(ctx, c)
return
}
if left := leftOf(c, heads); len(left) > 0 { if left := leftOf(c, heads); len(left) > 0 {
// A pull request the checker cannot claim (its repository's trunk head is not known) leaves its whole group // A pull request the checker cannot claim (its repository's trunk head is not known) leaves its whole group
// unjudged, said in a failure verdict (issue 1422, issue 1733): the gate asks the Approve only on a verdict, and a // unjudged, said in a failure verdict (issue 1422, issue 1733): the gate asks the Approve only on a verdict, and a
@@ -464,6 +471,39 @@ func (ch *Checker) untrunked(ctx context.Context, c delivery.Candidate, left []P
} }
} }
// outside takes a candidate outside the mesh (case.go Outside): no verdict, and every status owed for its pull requests
// taken away, so the forge module stops writing one into a repository the mesh's forge account may not write. Said once
// per heads in the log.
func (ch *Checker) outside(ctx context.Context, c delivery.Candidate) {
var names []string
for _, p := range c.Pulls {
names = append(names, fmt.Sprintf("%s#%d@%s", p.Repository, p.Number, short12(p.Head.Head)))
key, err := delivery.CheckerOwedKey(p.Repository, p.Number, p.Head.Head)
if err != nil {
continue
}
_, _, found, err := ch.Owed.Get(ctx, key)
if err != nil || !found {
continue
}
d, ok := ch.Owed.(Deleter)
if !ok {
continue
}
if err := d.Delete(ctx, key); err != nil {
ch.logf("%s: the status owed of a pull request outside the mesh could not be taken away: %v", key, err)
}
}
said := c.Group + "@" + c.HeadsHash
ch.mu.Lock()
first := !ch.outsideSaid[said]
ch.outsideSaid[said] = true
ch.mu.Unlock()
if first {
ch.logf("candidates of %s: %s %s", c.Group, strings.Join(names, ", "), OutsideWords)
}
}
// The kinds of a check (ADR 0536 §4): a head never judged and a head asked again after a failure of the nodes are asked on // The kinds of a check (ADR 0536 §4): a head never judged and a head asked again after a failure of the nodes are asked on
// the build seats' check stream; a carry, a verdict moved to a newer trunk or sibling commit by its package digests (ADR 0548 // the build seats' check stream; a carry, a verdict moved to a newer trunk or sibling commit by its package digests (ADR 0548
// §5), on the recheck stream, pulled after. // §5), on the recheck stream, pulled after.
@@ -48,6 +48,13 @@ func pullOf(repo string, number int, head string) delivery.CandidatePull {
return delivery.CandidatePull{Head: delivery.Head{Repository: repo, Number: number, Head: head}, Modules: []string{}} return delivery.CandidatePull{Head: delivery.Head{Repository: repo, Number: number, Head: head}, Modules: []string{}}
} }
// newRepoPullOf is a pull request of a new repository of the mesh: its trunk not named yet, its first module added.
func newRepoPullOf(repo string, number int, head string) delivery.CandidatePull {
p := pullOf(repo, number, head)
p.Modules = []string{"mod-u"}
return p
}
// judgeFunc is a Judge written as a function; it counts its calls. // judgeFunc is a Judge written as a function; it counts its calls.
type judgeFunc struct { type judgeFunc struct {
mu sync.Mutex mu sync.Mutex
@@ -249,7 +256,7 @@ func TestARepositoryWithNoTrunkCannotBeJudgedLoudlyUntilItsTrunkIsKnown(t *testi
var logged []string var logged []string
r.ch.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) } r.ch.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
const unknown = "novox/unknown" const unknown = "novox/unknown"
cand := candidateOf(t, "feat/y", pullOf(unknown, 4, head1)) cand := candidateOf(t, "feat/y", newRepoPullOf(unknown, 4, head1))
key := keyOf(t, "feat/y", cand) key := keyOf(t, "feat/y", cand)
r.ch.Candidate(ctx, "feat/y", cand) r.ch.Candidate(ctx, "feat/y", cand)
r.ch.Wait() r.ch.Wait()
@@ -305,7 +312,7 @@ func TestARepositoryWithNoTrunkCannotBeJudgedLoudlyUntilItsTrunkIsKnown(t *testi
want string // the verdict's state, and for a failure the repository it names want string // the verdict's state, and for a failure the repository it names
}{ }{
{name: "a member the checker cannot claim", pulls: []delivery.CandidatePull{pullOf(repoA, 9, head2), pullOf("novox/other", 2, head2)}, want: "novox/other"}, {name: "a member the checker cannot claim", pulls: []delivery.CandidatePull{pullOf(repoA, 9, head2), pullOf("novox/other", 2, head2)}, want: "novox/other"},
{name: "a running verdict left for heads that cannot be judged", pulls: []delivery.CandidatePull{pullOf(unknown, 5, head2)}, running: true, want: unknown}, {name: "a running verdict left for heads that cannot be judged", pulls: []delivery.CandidatePull{newRepoPullOf(unknown, 5, head2)}, running: true, want: unknown},
{name: "a verdict key deleted", pulls: []delivery.CandidatePull{pullOf(repoA, 5, head2)}, deleted: true, judged: 2, want: delivery.VerdictPass}, {name: "a verdict key deleted", pulls: []delivery.CandidatePull{pullOf(repoA, 5, head2)}, deleted: true, judged: 2, want: delivery.VerdictPass},
} { } {
t.Run(row.name, func(t *testing.T) { t.Run(row.name, func(t *testing.T) {
@@ -566,3 +573,43 @@ func TestTheForgeBridgeIsToldOnceForEachChangeOfAStatus(t *testing.T) { // 1683
t.Errorf("told %v after a check that changed nothing", told) t.Errorf("told %v after a check that changed nothing", told)
} }
} }
// Sweep 2026-10-30 item 7: a pull request of a repository the mesh does not deliver (an operator's own, on the same
// forge) touches no module and its trunk the gate never names. It is outside the mesh: no verdict, no status owed (one
// owed before is taken away, so the forge module stops writing it into a repository the mesh may not write), said once.
func TestACandidateOutsideTheMeshIsJudgedNothingAndOwesNothing(t *testing.T) {
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
var logged []string
r.ch.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
const theirs = "someone/own-app"
okey, _ := delivery.CheckerOwedKey(theirs, 55, head1)
before, _ := delivery.EncodeCheckerOwed(delivery.Owed{Act: delivery.OwedStatus, Context: delivery.StatusContextCheck, State: "error", Text: "cannot be judged"})
if _, err := r.owed.Create(ctx, okey, before); err != nil {
t.Fatal(err)
}
cand := candidateOf(t, "feat/theirs", pullOf(theirs, 55, head1))
key := keyOf(t, "feat/theirs", cand)
r.ch.Candidate(ctx, "feat/theirs", cand)
r.ch.Wait()
r.ch.Reconsider(ctx)
r.ch.Wait()
if _, _, found, _ := r.verdicts.Get(ctx, key); found || j.count() != 0 {
t.Errorf("a candidate outside the mesh was judged (%d) or given a verdict (%v)", j.count(), found)
}
if s := owedState(t, r, theirs, 55, head1); s != "" {
t.Errorf("%s#55 still owes %q on the forge", theirs, s)
}
if n := countWith(logged, "outside the mesh"); n != 1 {
t.Errorf("said %d times in the log, want once: %q", n, logged)
}
ckey, _ := delivery.CandidatesKey("feat/theirs")
st, err := Reader{CandidateState: fakeState{ckey: cand}.read(), VerdictState: fakeState(r.verdicts.vals).read(),
Trunks: func() (map[string]string, error) { return r.trunks.Heads(), nil }}.Status()
if err != nil {
t.Fatal(err)
}
if st.Outside["feat/theirs"] == "" || len(st.CannotBeJudged) != 0 || len(st.Unjudged) != 0 {
t.Errorf("status: outside %v, cannot be judged %v, unjudged %v", st.Outside, st.CannotBeJudged, st.Unjudged)
}
}
@@ -425,9 +425,9 @@ func TestANodePackageIsNeverKeptAndRunsEveryCheck(t *testing.T) { // 1288/AC4, 1
} }
} }
// 1289/AC1, AC2 as issue 1921 changed it: the change's own failing test runs first, the level's other packages still run // 1289/AC1, AC2 (R4c, mesh-sdk buildrun at 0b12b62): the change's own tests run first, and when one fails the level ends
// after it (so one check names every failure), and no later level runs. // there: no other package of the level and no later level runs, so the rest of the suite is not paid for.
func TestTheChangesOwnFailingTestRunsFirstThenTheRestAndNoLaterLevel(t *testing.T) { func TestTheChangesOwnFailingTestEndsTheLevelThere(t *testing.T) {
b := newBench(t) b := newBench(t)
b.fetch.Changed = []string{"modules/m/internal/b/b_test.go"} b.fetch.Changed = []string{"modules/m/internal/b/b_test.go"}
b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"m/internal/a": digA, "m/internal/b": digB}} b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"m/internal/a": digA, "m/internal/b": digB}}
@@ -443,8 +443,8 @@ func TestTheChangesOwnFailingTestRunsFirstThenTheRestAndNoLaterLevel(t *testing.
t.Fatalf("verdict = %+v", out.Verdict) t.Fatalf("verdict = %+v", out.Verdict)
} }
tests := b.steps.tool("test") tests := b.steps.tool("test")
if len(tests) != 2 || strings.Join(packagesOf(tests[0]), ",") != "m/internal/b" || strings.Join(packagesOf(tests[1]), ",") != "m/internal/a" { if len(tests) != 1 || strings.Join(packagesOf(tests[0]), ",") != "m/internal/b" {
t.Fatalf("test calls = %+v, want m/internal/b, then m/internal/a, each once", tests) t.Fatalf("test calls = %+v, want m/internal/b alone: its failure ends the level", tests)
} }
if n := len(b.steps.tool("digest")); n != 1 { if n := len(b.steps.tool("digest")); n != 1 {
t.Errorf("%d digest calls, want the unit level's alone (the check stopped)", n) t.Errorf("%d digest calls, want the unit level's alone (the check stopped)", n)
@@ -335,6 +335,8 @@ type Status struct {
// CannotBeJudged are the candidates no pull request of which can be judged, by group, with why: the gate's source.main // CannotBeJudged are the candidates no pull request of which can be judged, by group, with why: the gate's source.main
// names no trunk head of their repositories (issue 1422). They are judged once it does; until then their status is error. // names no trunk head of their repositories (issue 1422). They are judged once it does; until then their status is error.
CannotBeJudged map[string]string `json:"cannot_be_judged"` CannotBeJudged map[string]string `json:"cannot_be_judged"`
// Outside are the candidates outside the mesh, by group, with why (case.go Outside): judged nothing, owed nothing.
Outside map[string]string `json:"outside_the_mesh"`
// Lost are the candidates whose verdict is running while no step of theirs can still run and their newest step ended, by // Lost are the candidates whose verdict is running while no step of theirs can still run and their newest step ended, by
// group, with that step: nothing will judge them (the lost check of 2026-10-28); checker_ask_again asks one again. // group, with that step: nothing will judge them (the lost check of 2026-10-28); checker_ask_again asks one again.
Lost map[string]string `json:"lost"` Lost map[string]string `json:"lost"`
@@ -415,7 +417,7 @@ func (r Reader) Status() (Status, error) {
} }
st.Unjudged = []string{} st.Unjudged = []string{}
st.Waiting = map[string][]string{KindNew: {}, KindNodes: {}, KindTrunk: {}} st.Waiting = map[string][]string{KindNew: {}, KindNodes: {}, KindTrunk: {}}
st.HeldByTurn, st.CannotBeJudged = map[string]string{}, map[string]string{} st.HeldByTurn, st.CannotBeJudged, st.Outside = map[string]string{}, map[string]string{}, map[string]string{}
turns := r.turns(&st) turns := r.turns(&st)
var heads map[string]string var heads map[string]string
if r.Trunks == nil { if r.Trunks == nil {
@@ -466,6 +468,10 @@ func (r Reader) Status() (Status, error) {
continue continue
} }
c, raw := row.c, row.raw c, raw := row.c, row.raw
if Outside(c, heads) {
st.Outside[c.Group] = OutsideWords
continue
}
if heads != nil && len(leftOf(c, heads)) > 0 { if heads != nil && len(leftOf(c, heads)) > 0 {
// Nothing of it can be judged now, whatever its verdict says (issue 1422): never counted as waiting. // Nothing of it can be judged now, whatever its verdict says (issue 1422): never counted as waiting.
var repos []string var repos []string
@@ -17,3 +17,8 @@ type KV interface {
// Keys are every key of the bucket. // Keys are every key of the bucket.
Keys(ctx context.Context) ([]string, error) Keys(ctx context.Context) ([]string, error)
} }
// Deleter is a KV that can take a key away; a KV that cannot leaves it.
type Deleter interface {
Delete(ctx context.Context, key string) error
}
+13 -3
View File
@@ -38,6 +38,11 @@ type stream struct {
} }
type manifest struct { type manifest struct {
Claims []struct {
Name string `json:"name"`
Scope string `json:"scope"`
Serves []string `json:"serves"`
} `json:"claims"`
Streams []stream `json:"streams"` Streams []stream `json:"streams"`
Tools []string `json:"tools"` Tools []string `json:"tools"`
Module string `json:"module"` Module string `json:"module"`
@@ -96,8 +101,8 @@ func TestManifestDeclarations(t *testing.T) { // 1678/AC1
if want := []state{{Name: "asks", History: 1}, {Name: "hearing", History: 1}, {Name: "kept", History: 1}, {Name: "owed", History: 5}, {Name: "previews", History: 5}, {Name: "tries", History: 1}, {Name: "verdicts", History: 5}}; !reflect.DeepEqual(m.State, want) { if want := []state{{Name: "asks", History: 1}, {Name: "hearing", History: 1}, {Name: "kept", History: 1}, {Name: "owed", History: 5}, {Name: "previews", History: 5}, {Name: "tries", History: 1}, {Name: "verdicts", History: 5}}; !reflect.DeepEqual(m.State, want) {
t.Errorf("state = %+v, want %+v", m.State, want) t.Errorf("state = %+v, want %+v", m.State, want)
} }
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-node.steps", "mesh-build-node.toolchains", "mesh-controller.assignments", if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-node.steps", "mesh-build-node.toolchains",
"mesh-delivery-assigner.compositions", "mesh-delivery-builder.builds", "mesh-delivery-gate.source", "mesh-delivery-gate.turns", "mesh-delivery-indexer.candidates", "mesh-delivery-assigner.compositions", "mesh-delivery-assigner.placement", "mesh-delivery-builder.builds", "mesh-delivery-gate.source", "mesh-delivery-gate.turns", "mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules", "mesh-delivery-watchman.presence", "node-engine.have"}; !reflect.DeepEqual(sorted(m.Reads), want) { "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules", "mesh-delivery-watchman.presence", "node-engine.have"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want) t.Errorf("reads = %v, want %v", m.Reads, want)
} }
@@ -129,7 +134,12 @@ func TestManifestDeclarations(t *testing.T) { // 1678/AC1
if want := []string{"checker_ask_again", "checker_kept", "checker_previews", "checker_pull", "checker_status", "checker_try", "checker_try_end", "checker_try_read", "checker_verdicts"}; !reflect.DeepEqual(sorted(m.Tools), want) { if want := []string{"checker_ask_again", "checker_kept", "checker_previews", "checker_pull", "checker_status", "checker_try", "checker_try_end", "checker_try_read", "checker_verdicts"}; !reflect.DeepEqual(sorted(m.Tools), want) {
t.Errorf("tools = %v, want %v", m.Tools, want) t.Errorf("tools = %v, want %v", m.Tools, want)
} }
for _, k := range []string{"logs", "emits", "claims", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} { // The try is the checker's own seat's verbs (ADR 0547 §10, issue 774), served beside the tools by the same code.
if len(m.Claims) != 1 || m.Claims[0].Name != "mesh-delivery-checker" || m.Claims[0].Scope != "mesh" ||
!reflect.DeepEqual(m.Claims[0].Serves, []string{"try", "try-read", "try-end"}) {
t.Errorf("claims = %+v, want the mesh seat mesh-delivery-checker serving try, try-read and try-end", m.Claims)
}
for _, k := range []string{"logs", "emits", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
if _, ok := top[k]; ok { if _, ok := top[k]; ok {
t.Errorf("the manifest declares %q: the checker has none", k) t.Errorf("the manifest declares %q: the checker has none", k)
} }
+12 -1
View File
@@ -2,6 +2,17 @@
"module": "mesh-delivery-checker", "module": "mesh-delivery-checker",
"version": "1", "version": "1",
"slug": "checker", "slug": "checker",
"claims": [
{
"name": "mesh-delivery-checker",
"scope": "mesh",
"serves": [
"try",
"try-read",
"try-end"
]
}
],
"consumes": [ "consumes": [
"mesh-build-go.toolchain.reported", "mesh-build-go.toolchain.reported",
"mesh-build-node.toolchain.reported" "mesh-build-node.toolchain.reported"
@@ -23,8 +34,8 @@
"mesh-build-go.toolchains", "mesh-build-go.toolchains",
"mesh-build-node.steps", "mesh-build-node.steps",
"mesh-build-node.toolchains", "mesh-build-node.toolchains",
"mesh-controller.assignments",
"mesh-delivery-assigner.compositions", "mesh-delivery-assigner.compositions",
"mesh-delivery-assigner.placement",
"mesh-delivery-builder.builds", "mesh-delivery-builder.builds",
"mesh-delivery-gate.source", "mesh-delivery-gate.source",
"mesh-delivery-gate.turns", "mesh-delivery-gate.turns",
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-forge-bridge
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 h1:ycu/sNieTdLpD3bzhZOCXdzcoqXUOxPRmqWV019REpo= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24 h1:s0XXZtR7CFPBbnaDygWnE4/Q827iEFt0YfEzujk4Q9g=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -46,6 +46,13 @@ var ErrNotFound = errors.New("the forge has no such file")
// says is not there is ErrNotFound, any other failure any other error (issue 1448). // says is not there is ErrNotFound, any other failure any other error (issue 1448).
type Files func(owner, repo, path, ref string) ([]byte, error) type Files func(owner, repo, path, ref string) ([]byte, error)
// CatalogRepository is the catalogue, whose merged message carries its predefined list, and PredefinedFile that list's
// file at its root (novox/hq ADR 0565 §2, ADR 0348 §7).
const (
CatalogRepository = "novox/mesh-catalog"
PredefinedFile = "predefined.json"
)
// LockFiles are the lock files read beside a module's manifest, by name relative to its directory (novox/hq ADR 0465 // LockFiles are the lock files read beside a module's manifest, by name relative to its directory (novox/hq ADR 0465
// clause 1: go.sum for a Go module, package-lock.json for a Node one). // clause 1: go.sum for a Go module, package-lock.json for a Node one).
var LockFiles = []string{"go.sum", "package-lock.json"} var LockFiles = []string{"go.sum", "package-lock.json"}
@@ -315,6 +322,18 @@ func merged(subject string, p pull, trees Trees, files Files, kept Kept) (Messag
} }
f := delivery.Forge{Kind: delivery.ForgeMerged, Repository: repository, Branch: p.Head, Number: p.Number, f := delivery.Forge{Kind: delivery.ForgeMerged, Repository: repository, Branch: p.Head, Number: p.Number,
Head: p.HeadSHA, Merge: p.MergeSHA, Root: root, Trees: dirs, Manifests: manifests, Locks: locks} Head: p.HeadSHA, Merge: p.MergeSHA, Root: root, Trees: dirs, Manifests: manifests, Locks: locks}
if repository == CatalogRepository {
// The catalogue's predefined list at the merge (novox/hq ADR 0565 §2): the indexer keeps it for mesh-config.
raw, err := files(p.Owner, p.Repo, PredefinedFile, p.MergeSHA)
switch {
case errors.Is(err, ErrNotFound):
f.PredefinedRead = true
case err != nil:
return Message{}, fmt.Errorf("%w: %s: %s of %s at the merge commit %s: %v", ErrRetry, subject, PredefinedFile, repository, p.MergeSHA, err)
default:
f.Predefined = json.RawMessage(raw)
}
}
body, err := delivery.EncodeForge(f) body, err := delivery.EncodeForge(f)
if err != nil { if err != nil {
return Message{}, unreadable(subject, "%v", err) return Message{}, unreadable(subject, "%v", err)
@@ -97,6 +97,27 @@ func TestMergedCarriesManifestsAndLocks(t *testing.T) { // 1448/AC1
t.Errorf("the forge was asked for %s", c.path) t.Errorf("the forge was asked for %s", c.path)
} }
} }
// 1953/D1 (ADR 0565 §2): the catalogue's merged message carries its predefined.json at the merge, or says it holds
// none; another repository's asks the forge for none.
if !f.PredefinedRead || len(f.Predefined) != 0 {
t.Errorf("a merge of the catalogue with no predefined.json does not say so: %s", msg.Data)
}
forge.have[PredefinedFile] = []byte(`{"font-size":{"type":"integer","value":13}}`)
msg, _ = Convert(mergedSubject, mergedBody(map[string]any{"repo": "mesh-catalog"}), treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get)
if f, err = delivery.DecodeForge(msg.Data); err != nil || string(f.Predefined) != string(forge.have[PredefinedFile]) || f.PredefinedRead {
t.Errorf("predefined = %s (%v)", f.Predefined, err)
}
forge.calls = nil
msg, _ = Convert(mergedSubject, mergedBody(nil), treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get)
for _, c := range forge.calls {
if c.path == PredefinedFile {
t.Errorf("another repository's merge asked for %s", c.path)
}
}
forge.errs = map[string]error{PredefinedFile: errors.New("forge down")}
if _, err := Convert(mergedSubject, mergedBody(map[string]any{"repo": "mesh-catalog"}), treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get); !errors.Is(err, ErrRetry) {
t.Errorf("a forge that cannot say is %v, want a retry", err)
}
// Issue 1126: a merge reads the forge only for the module directories whose tree changed. // Issue 1126: a merge reads the forge only for the module directories whose tree changed.
t.Run("a merge reads only the directories whose tree changed", mergeReadsOnlyChangedDirs) t.Run("a merge reads only the directories whose tree changed", mergeReadsOnlyChangedDirs)
} }
@@ -19,6 +19,7 @@ import (
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
) )
@@ -44,21 +45,10 @@ accounts {
// manifest's state (mesh-delivery-indexer_candidates, history 10) and mesh-controller_builds holding build records, and // manifest's state (mesh-delivery-indexer_candidates, history 10) and mesh-controller_builds holding build records, and
// no mesh-delivery-gate_source bucket. // no mesh-delivery-gate_source bucket.
func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10 func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10
js, ctx, candidates, _ := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) { js, ctx, candidates, _ := startIndexer(t, func(ctx context.Context, builds buildLog) {
// Build records in the controller's layout (mesh-sdk go/delivery/testdata/build-record-build-agent.json, captured from // The builder's log entries of two builds' ends (ADR 0571 clause 1), as the builder appends them.
// the controller's encoder), under delivery.BuildKey, with the newest pointer beside them. builds.end(t, "novox/mesh-delivery", "a", "modules/a")
for _, b := range []struct{ repo, module, path, id string }{ builds.end(t, "novox/mesh-catalog", "c", "modules/c")
{"novox/mesh-delivery", "a", "modules/a", "build-1"}, {"novox/mesh-catalog", "c", "modules/c", "build-2"},
} {
rec, _ := json.Marshal(map[string]any{"id": b.id, "repository": b.repo, "ref": "main", "module": b.module,
"commit": "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", "path": b.path, "at": "2026-10-19T09:38:00Z"})
if _, err := builds.Put(ctx, delivery.BuildKey(b.module, b.id), rec); err != nil {
t.Fatal(err)
}
if _, err := builds.Put(ctx, delivery.NewestKey(b.module), []byte(b.id)); err != nil {
t.Fatal(err)
}
}
}) })
h1, h2 := "0123456789abcdef0123456789abcdef01234567", "89abcdef0123456789abcdef0123456789abcdef" h1, h2 := "0123456789abcdef0123456789abcdef01234567", "89abcdef0123456789abcdef0123456789abcdef"
for _, f := range []delivery.Forge{ for _, f := range []delivery.Forge{
@@ -99,7 +89,7 @@ func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10
// startIndexer starts the bus, the stream FORGE with the indexer's durable consumer, the buckets, lets put fill // startIndexer starts the bus, the stream FORGE with the indexer's durable consumer, the buckets, lets put fill
// mesh-controller_builds, and runs the program. mesh-delivery-gate_source is never created. // mesh-controller_builds, and runs the program. mesh-delivery-gate_source is never created.
func startIndexer(t *testing.T, put func(context.Context, jetstream.KeyValue), before ...func(context.Context, jetstream.JetStream)) (jetstream.JetStream, context.Context, jetstream.KeyValue, jetstream.KeyValue) { func startIndexer(t *testing.T, put func(context.Context, buildLog), before ...func(context.Context, jetstream.JetStream)) (jetstream.JetStream, context.Context, jetstream.KeyValue, jetstream.KeyValue) {
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "i") dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "i")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -162,11 +152,12 @@ func startIndexer(t *testing.T, put func(context.Context, jetstream.KeyValue), b
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.ManifestsBucket, History: delivery.ManifestsHistory}); err != nil { if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.ManifestsBucket, History: delivery.ManifestsHistory}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
builds, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.BuildsBucket, History: 5}) // The builder's module log, as the runtime raises it (ADR 0297), filled by put before the indexer starts.
if err != nil { if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(delivery.BuilderModule, delivery.BuilderLog),
Subjects: []string{buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog) + ".>"}, AllowDirect: true}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
put(ctx, builds) put(ctx, buildLog{ctx: ctx, js: js})
for _, b := range before { for _, b := range before {
b(ctx, js) b(ctx, js)
} }
@@ -185,15 +176,32 @@ func startIndexer(t *testing.T, put func(context.Context, jetstream.KeyValue), b
return js, ctx, candidates, modules return js, ctx, candidates, modules
} }
// buildLog appends entries to the builder's log as the builder does.
type buildLog struct {
ctx context.Context
js jetstream.JetStream
}
// end appends the end of a done trunk build of module from repo's directory path.
func (l buildLog) end(t *testing.T, repo, module, path string) {
t.Helper()
raw, err := delivery.EncodeBuildLogEntry(delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: module, FP: "src2:" + module,
Repository: repo, Directory: path, Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", Node: "n1",
At: time.Date(2026, 10, 19, 9, 38, 0, 0, time.UTC), Outcome: delivery.BuildDone})
if err != nil {
t.Fatal(err)
}
if _, err := l.js.Publish(l.ctx, buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog)+"."+module, raw); err != nil {
t.Fatal(err)
}
}
// TestIndexerWritesModulesAtAMerge: a module's record and its newest pointer are in mesh-controller_builds; a merged // TestIndexerWritesModulesAtAMerge: a module's record and its newest pointer are in mesh-controller_builds; a merged
// message on FORGE makes modules.<m> within 5 s, the fingerprint of the merge's tree and the recorded inputs. // message on FORGE makes modules.<m> within 5 s, the fingerprint of the merge's tree and the recorded inputs.
func TestIndexerWritesModulesAtAMerge(t *testing.T) { // 1239/AC9; I1, I3: the modules are the merged message's, no build record names them (issue 1851) func TestIndexerWritesModulesAtAMerge(t *testing.T) { // 1239/AC9; I1, I3: the modules are the merged message's, no build record names them (issue 1851)
// The bucket holds only build-agent's record, which no fingerprint reads since R4g (ADR 0530 §1). // The bucket holds only build-agent's record, which no fingerprint reads since R4g (ADR 0530 §1).
js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) { js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds buildLog) {
rec := `{"id":"build-1","repository":"novox/mesh-controller","ref":"main","module":"build-agent","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/build-agent","source-fingerprint":"src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7","at":"2026-10-19T09:38:00Z"}` builds.end(t, "novox/mesh-controller", "build-agent", "modules/build-agent")
if _, err := builds.Put(ctx, delivery.BuildKey("build-agent", "build-1"), []byte(rec)); err != nil {
t.Fatal(err)
}
}) })
const merge = "89abcdef0123456789abcdef0123456789abcdef" const merge = "89abcdef0123456789abcdef0123456789abcdef"
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-delivery", Branch: "feat/x", Number: 12, data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-delivery", Branch: "feat/x", Number: 12,
@@ -229,15 +237,8 @@ func TestIndexerKeepsManifestsAtAMerge(t *testing.T) { // 1444/AC4, 1449/AC4
t.Fatal(err) t.Fatal(err)
} }
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381" const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
const inputs = `{"Module":"avahi","Tree":"oldtree","Bases":null,"Contexts":null,"Toolchains":{"code":"go sha256:ccc 0123456789abcdef"},"Unpinned":""}` js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds buildLog) {
js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) { builds.end(t, "novox/mesh-catalog", "avahi", "modules/avahi")
rec := `{"id":"build-1","repository":"novox/mesh-catalog","ref":"main","module":"avahi","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/avahi","at":"2026-10-19T09:38:00Z","source-inputs":` + inputs + `}`
if _, err := builds.Put(ctx, delivery.BuildKey("avahi", "build-1"), []byte(rec)); err != nil {
t.Fatal(err)
}
if _, err := builds.Put(ctx, delivery.NewestKey("avahi"), []byte("build-1")); err != nil {
t.Fatal(err)
}
}) })
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-catalog", Branch: "feat/x", Number: 301, data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-catalog", Branch: "feat/x", Number: 301,
Head: "0123456789abcdef0123456789abcdef01234567", Merge: "89abcdef0123456789abcdef0123456789abcdef", Root: "r0", Head: "0123456789abcdef0123456789abcdef01234567", Merge: "89abcdef0123456789abcdef0123456789abcdef", Root: "r0",
@@ -287,15 +288,8 @@ func TestIndexerWritesAGoModuleAgainWhenTheBuildModuleReportsItsToolchain(t *tes
t.Fatal(err) t.Fatal(err)
} }
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381" const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
const inputs = `{"Module":"avahi","Tree":"oldtree","Bases":null,"Contexts":null,"Toolchains":{"code":"go sha256:ccc 0123456789abcdef"},"Unpinned":""}` js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds buildLog) {
js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) { builds.end(t, "novox/mesh-catalog", "avahi", "modules/avahi")
rec := `{"id":"build-1","repository":"novox/mesh-catalog","ref":"main","module":"avahi","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/avahi","at":"2026-10-19T09:38:00Z","source-inputs":` + inputs + `}`
if _, err := builds.Put(ctx, delivery.BuildKey("avahi", "build-1"), []byte(rec)); err != nil {
t.Fatal(err)
}
if _, err := builds.Put(ctx, delivery.NewestKey("avahi"), []byte("build-1")); err != nil {
t.Fatal(err)
}
}) })
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-catalog", Branch: "feat/x", Number: 301, data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-catalog", Branch: "feat/x", Number: 301,
Head: "0123456789abcdef0123456789abcdef01234567", Merge: "89abcdef0123456789abcdef0123456789abcdef", Root: "r0", Head: "0123456789abcdef0123456789abcdef01234567", Merge: "89abcdef0123456789abcdef0123456789abcdef", Root: "r0",
@@ -362,16 +356,9 @@ func TestIndexerGivesACandidateItsFingerprintWhenTheCheckerCreatesThePreviewsBuc
t.Fatal(err) t.Fatal(err)
} }
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381" const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
const inputs = `{"Module":"avahi","Tree":"oldtree","Bases":null,"Contexts":null,"Toolchains":{"code":"go sha256:ccc 0123456789abcdef"},"Unpinned":""}`
const trunk, head = "5555555555555555555555555555555555555555", "0123456789abcdef0123456789abcdef01234567" const trunk, head = "5555555555555555555555555555555555555555", "0123456789abcdef0123456789abcdef01234567"
js, ctx, candidates, _ := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) { js, ctx, candidates, _ := startIndexer(t, func(ctx context.Context, builds buildLog) {
rec := `{"id":"build-1","repository":"novox/mesh-catalog","ref":"main","module":"avahi","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/avahi","at":"2026-10-19T09:38:00Z","source-inputs":` + inputs + `}` builds.end(t, "novox/mesh-catalog", "avahi", "modules/avahi")
if _, err := builds.Put(ctx, delivery.BuildKey("avahi", "build-1"), []byte(rec)); err != nil {
t.Fatal(err)
}
if _, err := builds.Put(ctx, delivery.NewestKey("avahi"), []byte("build-1")); err != nil {
t.Fatal(err)
}
}, func(ctx context.Context, js jetstream.JetStream) { }, func(ctx context.Context, js jetstream.JetStream) {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: PreviewsStream, Subjects: []string{previewWrittenSubject}}); err != nil { if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: PreviewsStream, Subjects: []string{previewWrittenSubject}}); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -451,7 +438,7 @@ func TestAMergeOfMeshSDKChangingOnlyGoLeavesItsNamedTreeEqual(t *testing.T) {
} }
objects := map[string]string{"src": "1111111111111111111111111111111111111111", "test": "2222222222222222222222222222222222222222", objects := map[string]string{"src": "1111111111111111111111111111111111111111", "test": "2222222222222222222222222222222222222222",
"package.json": "3333333333333333333333333333333333333333", "package-lock.json": "4444444444444444444444444444444444444444", "tsconfig.json": "5555555555555555555555555555555555555555"} "package.json": "3333333333333333333333333333333333333333", "package-lock.json": "4444444444444444444444444444444444444444", "tsconfig.json": "5555555555555555555555555555555555555555"}
js, ctx, _, modules := startIndexer(t, func(context.Context, jetstream.KeyValue) {}) js, ctx, _, modules := startIndexer(t, func(context.Context, buildLog) {})
publish := func(merge, root string, objects map[string]string) { publish := func(merge, root string, objects map[string]string) {
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-sdk", Branch: "feat/x", Number: 70, data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-sdk", Branch: "feat/x", Number: 70,
Head: "0123456789abcdef0123456789abcdef01234567", Merge: merge, Root: root, Trees: map[string]string{"": root}, Head: "0123456789abcdef0123456789abcdef01234567", Merge: merge, Root: root, Trees: map[string]string{"": root},
@@ -518,7 +505,7 @@ func TestIndexerJudgesAnUnreadableVerdictAgainAtStart(t *testing.T) { // 1923/AC
} }
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381" const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
digest := delivery.ManifestDigest(manifest) digest := delivery.ManifestDigest(manifest)
_, ctx, _, modules := startIndexer(t, func(context.Context, jetstream.KeyValue) {}, func(ctx context.Context, js jetstream.JetStream) { _, ctx, _, modules := startIndexer(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
mkey, err := delivery.ManifestsKey(digest) mkey, err := delivery.ManifestsKey(digest)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -578,23 +565,39 @@ func TestIndexerJudgesAnUnreadableVerdictAgainAtStart(t *testing.T) { // 1923/AC
} }
} }
// TestIndexerCountsOnlyTheAssignedBuildNodesToolchains (issue 1940): the toolchains bucket holds the report of n2, where // TestIndexerCountsOnlyThePlacedBuildNodesToolchains (issue 1940, issue 774): the toolchains bucket holds the report of
// mesh-build-go was unassigned (its assignment ended) at another version than n1's, as ace's v26.12.0 stayed beside the // n2, where the catalogue no longer places mesh-build-go, at another version than n1's, as ace's v26.12.0 stayed beside
// v26.12.1 of the nodes mesh-build-node runs on. The indexer gives the merged module a fingerprint from n1's toolchain // the v26.12.1 of the nodes mesh-build-node runs on. The indexer gives the merged module a fingerprint from n1's
// alone; when the controller assigns mesh-build-go to n2 again, its report counts and the builders disagree (the guard // toolchain alone; when the placement puts mesh-build-go on n2 again, its report written since counts and the builders
// stays); when the assignment ends again, the fingerprint comes back. No restart, no timer: the assignments' watch. // disagree (the guard stays); when it is taken off again, the fingerprint comes back. No restart, no timer: the
func TestIndexerCountsOnlyTheAssignedBuildNodesToolchains(t *testing.T) { // placement's watch (ADR 0549 §3), not the controller's assignments.
func TestIndexerCountsOnlyThePlacedBuildNodesToolchains(t *testing.T) {
manifest, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json") manifest, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381" const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
var assignments jetstream.KeyValue var placement, toolchains jetstream.KeyValue
js, ctx, _, modules := startIndexer(t, func(context.Context, jetstream.KeyValue) {}, func(ctx context.Context, js jetstream.JetStream) { place := func(ctx context.Context, t *testing.T, node string, modules ...string) {
p := delivery.Placement{Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
key, _ := delivery.PlacementKey(node)
value, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
if _, err := placement.Put(ctx, key, value); err != nil {
t.Fatal(err)
}
}
js, ctx, _, modules := startIndexer(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.ToolchainsBucket("mesh-build-go"), History: 1}) kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.ToolchainsBucket("mesh-build-go"), History: 1})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
toolchains = kv
for node, version := range map[string]string{"n1": "go1.27.3", "n2": "go1.27.2"} { for node, version := range map[string]string{"n1": "go1.27.3", "n2": "go1.27.2"} {
key, _ := delivery.ToolchainKey(node, "go") key, _ := delivery.ToolchainKey(node, "go")
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "go", Version: version, Recipe: "1", At: time.Now()}) value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "go", Version: version, Recipe: "1", At: time.Now()})
@@ -605,18 +608,12 @@ func TestIndexerCountsOnlyTheAssignedBuildNodesToolchains(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
} }
// The controller's assignments mirror, in its layout (mesh-controller internal/chain Assignment). // The seat mesh-placement's placement, as the assigner writes it (ADR 0567): n1 places mesh-build-go, n2 nothing.
if assignments, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: "mesh-controller_assignments", History: 1}); err != nil { if placement, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.PlacementBucket, History: delivery.PlacementHistory}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
for key, value := range map[string]string{ place(ctx, t, "n1", "mesh-build-go")
"n1.mesh-build-go": `{"node":"n1","module":"mesh-build-go","assigned":"2026-10-20T08:00:00Z"}`, place(ctx, t, "n2")
"n2.mesh-build-go": `{"node":"n2","module":"mesh-build-go","assigned":"2026-10-20T08:00:00Z","ended":"2026-10-23T08:00:00Z"}`,
} {
if _, err := assignments.Put(ctx, key, []byte(value)); err != nil {
t.Fatal(err)
}
}
}) })
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-catalog", Branch: "feat/x", Number: 302, data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-catalog", Branch: "feat/x", Number: 302,
Head: "0123456789abcdef0123456789abcdef01234567", Merge: "89abcdef0123456789abcdef0123456789abcdef", Root: "r0", Head: "0123456789abcdef0123456789abcdef01234567", Merge: "89abcdef0123456789abcdef0123456789abcdef", Root: "r0",
@@ -647,16 +644,22 @@ func TestIndexerCountsOnlyTheAssignedBuildNodesToolchains(t *testing.T) {
if !agreed(held) { if !agreed(held) {
t.Fatalf("within 5 s avahi has fp2 %q, none2 %q, toolchains %v: want a fingerprint of n1's go1.27.3 alone", held.FP2, held.None2, held.Toolchains) t.Fatalf("within 5 s avahi has fp2 %q, none2 %q, toolchains %v: want a fingerprint of n1's go1.27.3 alone", held.FP2, held.None2, held.Toolchains)
} }
if _, err := assignments.Put(ctx, "n2.mesh-build-go", []byte(`{"node":"n2","module":"mesh-build-go","assigned":"2026-10-24T08:00:00Z"}`)); err != nil { place(ctx, t, "n2", "mesh-build-go")
key, _ := delivery.ToolchainKey("n2", "go")
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "go", Version: "go1.27.2", Recipe: "1", At: time.Now().Add(time.Second)})
if err != nil {
t.Fatal(err) t.Fatal(err)
} }
// n2's report, written since its assignment, counts: the builders disagree. A fingerprint already computed is kept if _, err := toolchains.Put(ctx, key, value); err != nil {
t.Fatal(err)
}
// n2's report, written since its placement, counts: the builders disagree. A fingerprint already computed is kept
// (issue 1940), and only a new one needs the agreement: a merge moving avahi gets none until they agree. // (issue 1940), and only a new one needs the agreement: a merge moving avahi gets none until they agree.
time.Sleep(500 * time.Millisecond) time.Sleep(500 * time.Millisecond)
if e, err := modules.Get(ctx, "avahi"); err != nil { if e, err := modules.Get(ctx, "avahi"); err != nil {
t.Fatal(err) t.Fatal(err)
} else if m, err := delivery.DecodeModule(e.Value()); err != nil || m.FP2 != held.FP2 { } else if m, err := delivery.DecodeModule(e.Value()); err != nil || m.FP2 != held.FP2 {
t.Fatalf("after n2's assignment avahi has fp2 %q, none2 %q: want its fingerprint %q kept", m.FP2, m.None2, held.FP2) t.Fatalf("after n2's placement avahi has fp2 %q, none2 %q: want its fingerprint %q kept", m.FP2, m.None2, held.FP2)
} }
moved := strings.Replace(string(data), `"newtree"`, `"movedtree"`, 1) moved := strings.Replace(string(data), `"newtree"`, `"movedtree"`, 1)
if moved == string(data) { if moved == string(data) {
@@ -669,12 +672,10 @@ func TestIndexerCountsOnlyTheAssignedBuildNodesToolchains(t *testing.T) {
return m.FP2 == "" && strings.HasPrefix(m.None2, "builders disagree: go1.27.2 recipe 1 on n2") return m.FP2 == "" && strings.HasPrefix(m.None2, "builders disagree: go1.27.2 recipe 1 on n2")
} }
if m := read(disagree); !disagree(m) { if m := read(disagree); !disagree(m) {
t.Fatalf("within 5 s of a merge after n2's assignment avahi has fp2 %q, none2 %q: want the builders to disagree", m.FP2, m.None2) t.Fatalf("within 5 s of a merge after n2's placement avahi has fp2 %q, none2 %q: want the builders to disagree", m.FP2, m.None2)
}
if _, err := assignments.Put(ctx, "n2.mesh-build-go", []byte(`{"node":"n2","module":"mesh-build-go","ended":"2026-10-24T09:00:00Z"}`)); err != nil {
t.Fatal(err)
} }
place(ctx, t, "n2")
if m := read(agreed); !agreed(m) { if m := read(agreed); !agreed(m) {
t.Fatalf("within 5 s of n2's unassignment avahi has fp2 %q, none2 %q", m.FP2, m.None2) t.Fatalf("within 5 s of n2's placement taken away avahi has fp2 %q, none2 %q", m.FP2, m.None2)
} }
} }
@@ -10,20 +10,22 @@ package main
import ( import (
"context" "context"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"log" "log"
"os" "os"
"os/signal" "os/signal"
"sync/atomic"
"syscall" "syscall"
"time" "time"
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-indexer/internal/index" "mesh-delivery-indexer/internal/index"
@@ -273,28 +275,15 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// Its taken state (ADR 0556 clause 1) is bound when it appears, as every state a delivery adds (issue 1790): until then // Its taken state (ADR 0556 clause 1) is bound when it appears, as every state a delivery adds (issue 1790): until then
// a merge's taken key is said and not written, and the rest of the indexer runs. // a merge's taken key is said and not written, and the rest of the indexer runs.
taken := process.AwaitBucket(ctx, js, conn, delivery.TakenBucket, "no repository's taken key is written", logger.Printf) taken := process.AwaitBucket(ctx, js, conn, delivery.TakenBucket, "no repository's taken key is written", logger.Printf)
// The controller's build records only name the directory of a module whose key names none (fillPaths): without them // The control records (ADR 0565 §2), new with this delivery and bound when they appear: until then nothing is written to them.
// such a module waits for its path, every other is indexed (issue 1945). predefined := process.AwaitBucket(ctx, js, conn, index.PredefinedBucket, "no predefined list is written", logger.Printf)
builds := process.OptionalBucket(ctx, js, delivery.BuildsBucket, "a module whose key names no directory is not given one from the controller's build records until the indexer starts again", logger.Printf) configCandidates := process.AwaitBucket(ctx, js, conn, index.ConfigCandidatesBucket, "no pull request's config declarations are written", logger.Printf)
// The builder's log (ADR 0571 clause 1), read on the indexer's own credential (the calls connection) through its declared read: every entry it
// keeps at the start, then each new one as the builder appends it (no timer). It names the directory of a module whose key
// names none (fillPaths) and the places a registered module was built from; no build moves a fingerprint (ADR 0530 §1).
// Where the log cannot be read such a module waits for its path, every other is indexed (issue 1945), and it is said.
known := index.NewBuilds() known := index.NewBuilds()
// The controller's build records name the directory of a module whose key names none (fillPaths); no build of any
// module moves a fingerprint since no fingerprint takes the builder (ADR 0530 §1).
each := func(e jetstream.KeyValueEntry) {
if opOf(e) {
known.Put(e.Key(), e.Value())
} else {
known.Delete(e.Key())
}
}
errs := make(chan error, 1) errs := make(chan error, 1)
var buildsWatch jetstream.KeyWatcher
if builds != nil {
if w, err := settle(ctx, builds, each); err != nil {
process.NotWatched(logger.Printf, delivery.BuildsBucket, err, "a module whose key names no directory is not given one until the indexer starts again")
} else {
buildsWatch = w
}
}
// The indexer asks the forge for the objects of a manifest's named paths as its own account (ADR 0499 clause 3). // The indexer asks the forge for the objects of a manifest's named paths as its own account (ADR 0499 clause 3).
calls, err := process.Connect(cred, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true)) calls, err := process.Connect(cred, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
@@ -302,9 +291,29 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err return err
} }
defer calls.Close() defer calls.Close()
ix := &index.Indexer{Store: kvStore{candidates}, Modules: kvStore{modules}, Manifests: kvStore{manifests}, Taken: awaitedStore{taken}, Builds: known, Logf: logger.Printf, ix := &index.Indexer{Store: kvStore{candidates}, Modules: kvStore{modules}, Manifests: kvStore{manifests}, Taken: awaitedStore{taken}, Builds: known,
Predefined: awaitedStore{predefined}, ConfigCandidates: awaitedStore{configCandidates}, Logf: logger.Printf,
Named: namedVia(calls.Call), Trunk: trunkVia(calls.Call), Contains: containsVia(calls.Call), Read: readVia(calls.Call), Named: namedVia(calls.Call), Trunk: trunkVia(calls.Call), Contains: containsVia(calls.Call), Read: readVia(calls.Call),
ReadOpen: openVia(calls.Call)} ReadOpen: openVia(calls.Call)}
// The log's entries at the start are read before any fingerprint is computed or registered module indexed (below); each
// new one indexes the module it names when that module is registered and the bucket holds none (registered.go).
var started atomic.Bool
if err := calls.ReadLog(delivery.BuilderModule, delivery.BuilderLog).Watch("", stdio.LogFromAll, func(le stdio.LogEntry) error {
if err := known.Put(le.Value); err != nil {
logger.Printf("%s entry %d is not read: %v", delivery.BuildLogRead, le.Sequence, err)
return nil
}
if started.Load() {
if err := ix.IndexRegisteredModule(ctx, le.Key); err != nil {
logger.Printf("the registered module %s is not indexed: %v", le.Key, err)
}
}
return nil
}); err != nil {
logger.Printf("%s is not read (%v): a module whose key names no directory is not given one, and no registered module is "+
"indexed from it, until the indexer starts again", delivery.BuildLogRead, err)
}
started.Store(true)
// The gate's source bucket (mesh-delivery-gate_source) comes with the gate (release 3e, novox/hq ADR 0450 §1): its key // The gate's source bucket (mesh-delivery-gate_source) comes with the gate (release 3e, novox/hq ADR 0450 §1): its key
// main is the mesh's source revision, which stamps the modules (issue 1344). Where it does not exist yet the indexer does // main is the mesh's source revision, which stamps the modules (issue 1344). Where it does not exist yet the indexer does
// not wait for it (that needs a timer, ADR 0437 §8): it says so, and is started again with the gate. // not wait for it (that needs a timer, ADR 0437 §8): it says so, and is started again with the gate.
@@ -321,14 +330,14 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
go func() { sourceErrs <- follow(ctx, sw, each) }() go func() { sourceErrs <- follow(ctx, sw, each) }()
} }
} }
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's // Only the toolchain reports of nodes the catalogue places the build module on count (issue 1940, ADR 0549 §3): the
// assignments are settled first, so the toolchains settle through the gate already knowing them. // placement is settled first, so the toolchains settle through the gate already knowing it.
gate := newToolchainGate(sortedModules(), ix.PutToolchain, logger.Printf) gate := buildrun.NewToolchainGate(sortedModules(), ix.PutToolchain, logger.Printf)
assignErrs, err := watchAssignments(ctx, js, gate, func(ctx context.Context) { placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, func(ctx context.Context) {
if err := ix.Recompute(ctx); err != nil { if err := ix.Recompute(ctx); err != nil {
logger.Printf("writing the modules again after an assignment of a build module changed: %v", err) logger.Printf("writing the modules again after the placement of a build module changed: %v", err)
} }
}, logger) }, logger.Printf)
if err != nil { if err != nil {
return err return err
} }
@@ -355,28 +364,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err := ix.RecomputeAtStart(ctx); err != nil { if err := ix.RecomputeAtStart(ctx); err != nil {
logger.Printf("computing the new-form fingerprints at start: %v", err) logger.Printf("computing the new-form fingerprints at start: %v", err)
} }
// A module registered in the catalogue from its own repository gets its manifest indexed when the controller's records // A module registered in the catalogue from its own repository gets its manifest indexed when the builder's log names
// name it and the bucket holds none (registered.go); the records that follow are heard below. // it and the bucket holds none (registered.go); the entries that follow are heard above.
if err := ix.IndexRegisteredAll(ctx); err != nil { if err := ix.IndexRegisteredAll(ctx); err != nil {
logger.Printf("indexing the registered modules at start: %v", err) logger.Printf("indexing the registered modules at start: %v", err)
} }
if buildsWatch != nil {
go func() {
errs <- follow(ctx, buildsWatch, func(e jetstream.KeyValueEntry) {
each(e)
if !opOf(e) {
return
}
var r delivery.BuildRecord
if json.Unmarshal(e.Value(), &r) != nil || r.Module == "" {
return
}
if err := ix.IndexRegisteredModule(ctx, r.Module); err != nil {
logger.Printf("the registered module %s is not indexed: %v", r.Module, err)
}
})
}()
}
// A person takes a registered module in now with indexer_take_in, which writes the takeins state (issue 1945): watched, // A person takes a registered module in now with indexer_take_in, which writes the takeins state (issue 1945): watched,
// as every other input. A module the forge could not read is also read again by itself on the next source revision and // as every other input. A module the forge could not read is also read again by itself on the next source revision and
// merged message (index.RetryUnread). // merged message (index.RetryUnread).
@@ -442,7 +434,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err return err
case err := <-sourceErrs: case err := <-sourceErrs:
return err return err
case err := <-assignErrs: case err := <-placementErrs:
return err return err
case err := <-tools.errs: case err := <-tools.errs:
return err return err
@@ -1,211 +0,0 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -1,165 +0,0 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"fmt"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-indexer
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 h1:Tt16HJk8JZRce1i3JzzDSNdCWOrv7dw8FQ/bckNQ5Ig= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24 h1:s0XXZtR7CFPBbnaDygWnE4/Q827iEFt0YfEzujk4Q9g=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -1,72 +1,63 @@
package index package index
import ( import (
"encoding/json" "fmt"
"maps" "maps"
"slices" "slices"
"strings" "strings"
"sync" "sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
// Builds holds the controller's build records (mesh-controller_builds) for the two things the indexer still reads from // Builds holds the builder's log (mesh-delivery-builder.build-log, novox/hq ADR 0571 clause 1) for what the indexer reads
// them until release R4g, when build-agent goes (the lead's decision of R4e, 2026-10-22): // of past builds: the directory of a module whose modules key has none (PathOf; issue 1614), filled once and changing no
// - the Builder input of the builder's own module, which build-agent builds (TrunkOf; novox/hq ADR 0453 clause 5, // fingerprint, and the places a module registered from its own repository was built from (RegistrationsOf). It is filled
// issue 1727); // from the log's entries at the indexer's start (every one the log keeps, 30 days) and then each new one as the builder
// - the directory of a module whose modules key has none (PathOf; issue 1614), filled once and changing no fingerprint. // appends it; no entry is ever taken out, since only age takes one from the log.
// //
// It lists no module and maps no directory to a module: the indexer's modules are those its own messages and bucket name // It lists no module and maps no directory to a module: the indexer's modules are those its own messages and bucket name
// (ADR 0501 clause 6, issue 1851). Safe for use from a watch and from the message handler at once. // (ADR 0501 clause 6, issue 1851). Safe for use from a watch and from the message handler at once.
type Builds struct { type Builds struct {
mu sync.Mutex mu sync.Mutex
records map[string]delivery.BuildRecord // by bucket key records map[string]delivery.BuildRecord // by the entry's event, module, fingerprint and time
} }
// NewBuilds is an empty Builds. // NewBuilds is an empty Builds.
func NewBuilds() *Builds { return &Builds{records: map[string]delivery.BuildRecord{}} } func NewBuilds() *Builds { return &Builds{records: map[string]delivery.BuildRecord{}} }
// Put takes a key of the builds bucket and its value. Only a build record of a module (key `<module>.build.<id>`, // Put takes one entry of the builder's log. A build's end is an outcome: done registers, failed or queued does not; its
// delivery.BuildKey) counts; the newest-pointer keys and requests that knew no module are not records of one. // start is a request, which registers nothing either. An entry that does not read, or names no module or repository, is
func (b *Builds) Put(key string, value []byte) { // refused and said.
if !strings.Contains(key, ".build.") { func (b *Builds) Put(value []byte) error {
return e, err := delivery.DecodeBuildLogEntry(value)
if err != nil {
return err
} }
var r delivery.BuildRecord if e.Module == "" || e.Repository == "" {
if err := json.Unmarshal(value, &r); err != nil || r.Module == "" || r.Repository == "" { return fmt.Errorf("a build log entry naming no module or repository: %s", value)
return }
r := delivery.BuildRecord{ID: e.At.UTC().Format(time.RFC3339Nano) + "." + e.FP, Repository: e.Repository, Ref: e.Ref, Module: e.Module,
Path: e.Directory, Commit: e.Commit, At: e.At}
switch {
case e.Event != delivery.BuildEndedEvent:
r.State = e.Event
case e.Outcome == delivery.BuildFailed:
r.Failed = "the build failed"
case e.Outcome != delivery.BuildDone:
r.State = e.Outcome
} }
b.mu.Lock() b.mu.Lock()
defer b.mu.Unlock() defer b.mu.Unlock()
b.records[key] = r b.records[e.Event+"."+e.Module+"."+r.ID] = r
} return nil
// Delete takes a key out.
func (b *Builds) Delete(key string) {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.records, key)
} }
func newer(a, b delivery.BuildRecord) bool { func newer(a, b delivery.BuildRecord) bool {
return a.At.After(b.At) || (a.At.Equal(b.At) && a.ID > b.ID) return a.At.After(b.At) || (a.At.Equal(b.At) && a.ID > b.ID)
} }
// TrunkOf is the newest successful trunk build record of a module, in whatever repository it was built from; nil when
// there is none (issue 1727).
func (b *Builds) TrunkOf(module string) *delivery.BuildRecord {
var best *delivery.BuildRecord
b.mu.Lock()
defer b.mu.Unlock()
for _, r := range b.records {
if r.Module == module && r.IsTrunkBuild() && (best == nil || newer(r, *best)) {
c := r
best = &c
}
}
return best
}
// PathOf is the directory of a module of the repository as its newest build record of that repository names it, its // PathOf is the directory of a module of the repository as its newest build record of that repository names it, its
// newest successful trunk build's first ("." for the repository's root), and false when no record of the repository // newest successful trunk build's first ("." for the repository's root), and false when no record of the repository
// names the module (issue 1614). // names the module (issue 1614).
@@ -79,7 +70,7 @@ func (b *Builds) PathOf(module, repository string) (string, bool) {
if r.Module != module || !delivery.SameRepository(r.Repository, repository) { if r.Module != module || !delivery.SameRepository(r.Repository, repository) {
continue continue
} }
t := r.IsTrunkBuild() t := registered(r) && trunkLike[r.Ref]
if best == nil || (t && !trunk) || (t == trunk && newer(r, *best)) { if best == nil || (t && !trunk) || (t == trunk && newer(r, *best)) {
c := r c := r
best, trunk = &c, t best, trunk = &c, t
@@ -91,15 +82,14 @@ func (b *Builds) PathOf(module, repository string) (string, bool) {
return pathOf(strings.Trim(best.Path, "/")), true return pathOf(strings.Trim(best.Path, "/")), true
} }
// Registration is a module registered in the catalogue from its own repository, as the controller's newest outcome record // Registration is a module registered in the catalogue from its own repository, as the builder log.s newest end of a build
// of it names it: the repository, the directory of its manifest there ("" or "." for the root) and the commit it was // of it names it: the repository, the directory of its manifest there ("" or "." for the root) and the commit it was
// built at. // built at.
type Registration struct { type Registration struct {
Module, Repository, Path, Commit, Ref string Module, Repository, Path, Commit, Ref string
} }
// registered is whether the record is an outcome that did not fail: what registers a module. Unlike IsTrunkBuild it // registered is whether the record is an outcome that did not fail: what registers a module. It asks for no particular branch: a
// asks for no recorded source inputs, which no build records since ADR 0530 clause 1, and for no particular branch: a
// repository's trunk is named main, master, or what its owner chose (the registered photos, de-spiegel and link2pay were // repository's trunk is named main, master, or what its owner chose (the registered photos, de-spiegel and link2pay were
// built from nox-mesh, and every record of them names that ref). // built from nox-mesh, and every record of them names that ref).
func registered(r delivery.BuildRecord) bool { func registered(r delivery.BuildRecord) bool {
@@ -48,6 +48,8 @@ type Indexer struct {
Modules Store Modules Store
// Manifests is the manifests bucket; nil writes no manifests (issue 1449). // Manifests is the manifests bucket; nil writes no manifests (issue 1449).
Manifests Store Manifests Store
// Predefined and ConfigCandidates are the control records' states (control_records.go, ADR 0565 §2); nil writes none.
Predefined, ConfigCandidates Store
// Taken is the taken bucket: per repository the last merge taken in (taken.go, ADR 0556); nil writes none. // Taken is the taken bucket: per repository the last merge taken in (taken.go, ADR 0556); nil writes none.
Taken Store Taken Store
// Now is the clock of a taken value; nil is time.Now. // Now is the clock of a taken value; nil is time.Now.
@@ -121,6 +123,14 @@ func (ix *Indexer) ApplyMessage(ctx context.Context, f delivery.Forge) (Applied,
if err := ix.writeTakenKnown(ctx, f, w.asked); err != nil { if err := ix.writeTakenKnown(ctx, f, w.asked); err != nil {
return applied, err return applied, err
} }
// The control records (ADR 0565 §2): the catalogue's predefined list, and the open pull requests' config
// declarations judged again against the trunk this merge moved.
if err := ix.writePredefined(ctx, f); err != nil {
return applied, err
}
if err := ix.configCandidatesAfterMerge(ctx, f, writtenModules(w.written)); err != nil {
return applied, err
}
} }
ix.kickRetry(ctx) // a registered module the forge could not read is read again on a merge, in the background (issue 1945) ix.kickRetry(ctx) // a registered module the forge could not read is read again on a merge, in the background (issue 1945)
} }
@@ -134,6 +144,12 @@ func (ix *Indexer) ApplyMessage(ctx context.Context, f delivery.Forge) (Applied,
if held, err = ix.heldModules(ctx); err != nil { if held, err = ix.heldModules(ctx); err != nil {
return applied, err return applied, err
} }
if err := ix.configCandidateOfPull(ctx, f, held); err != nil {
if !errors.Is(err, ErrUnreadable) {
return applied, err
}
ix.logf("%s#%d: its config declarations are not written: %v", f.Repository, f.Number, err)
}
} }
if trunkLike[f.Branch] { if trunkLike[f.Branch] {
ix.logf("%s#%d: its branch %q is no delivery group, so no candidate", f.Repository, f.Number, f.Branch) ix.logf("%s#%d: its branch %q is no delivery group, so no candidate", f.Repository, f.Number, f.Branch)
@@ -0,0 +1,292 @@
package index
import (
"context"
"encoding/json"
"errors"
"fmt"
"reflect"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// The indexer's two control records (novox/hq ADR 0565 §2, issue 1953 D2, D3): the catalogue's predefined list, read from the
// forge bridge's merged message, and each open pull request's config declarations, read from its pull message's manifests
// against the trunk's manifests the indexer holds. mesh-config reads both in place of the controller's mirror and its
// `pull-declares`; a key's write is its event, nothing polls.
// The two states' buckets.
var (
PredefinedBucket = delivery.IndexerModule + "_" + delivery.IndexerPredefinedState
ConfigCandidatesBucket = delivery.IndexerModule + "_" + delivery.IndexerConfigCandidatesState
)
// writePredefined keeps the predefined list a merged message carries, unless the key names a later merge: a merge older
// than the one it names never replaces it (ADR 0348 §7). The same list at the same merge writes nothing (ADR 0289 §1).
func (ix *Indexer) writePredefined(ctx context.Context, f delivery.Forge) error {
if ix.Predefined == nil || len(f.Predefined) == 0 {
return nil
}
return ix.notYet(PredefinedBucket, write(ctx, ix.Predefined, delivery.PredefinedListKey, func(value []byte, found bool) ([]byte, bool, error) {
if found {
var cur delivery.PredefinedKept
if err := json.Unmarshal(value, &cur); err == nil {
if cur.Commit == f.Merge && string(cur.List) == string(f.Predefined) {
return nil, false, nil
}
older, err := ix.older(ctx, f.Repository, f.Merge, cur.Commit)
if err != nil {
return nil, false, err
}
if older {
ix.logf("%s: the predefined list of %.12s is older than the one kept (%.12s): left", f.Repository, f.Merge, cur.Commit)
return nil, false, nil
}
}
}
next, err := json.Marshal(delivery.PredefinedKept{V: delivery.Version, Commit: f.Merge, At: ix.now(), List: f.Predefined})
return next, false, err
}))
}
// notYet says a state not on the bus yet (it comes with this delivery, bound when it appears) and lets the message go:
// what it would have written is written at the next message that carries it.
func (ix *Indexer) notYet(bucket string, err error) error {
if errors.Is(err, process.ErrNotYet) {
ix.logf("the state %s is not on the bus yet: nothing is written to it for this message", bucket)
return nil
}
return err
}
func (ix *Indexer) now() time.Time {
if ix.Now != nil {
return ix.Now()
}
return time.Now()
}
// PullName is a pull request as mesh-config names it, `owner/repository#number`.
func PullName(repository string, number int) string { return repository + "#" + strconv.Itoa(number) }
// DeclarationsOf is a manifest's config declarations by key, in the layout of the controller's declared-config mirror: the
// config form's `config`, else the settings form's `settings`, each default and enum encoded the one way.
func DeclarationsOf(raw string) (map[string]delivery.ConfigDeclaration, error) {
var m struct {
Config map[string]delivery.ConfigDeclaration `json:"config"`
Settings map[string]delivery.ConfigDeclaration `json:"settings"`
}
if err := json.Unmarshal([]byte(raw), &m); err != nil {
return nil, err
}
declared := m.Config
if declared == nil {
declared = m.Settings
}
out := map[string]delivery.ConfigDeclaration{}
for key, d := range declared {
for _, field := range []*json.RawMessage{&d.Default, &d.Enum} {
if len(*field) == 0 {
continue
}
var v any
if err := json.Unmarshal(*field, &v); err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
canonical, err := json.Marshal(v)
if err != nil {
return nil, err
}
*field = canonical
}
out[key] = d
}
return out, nil
}
// changedDeclarations are the declarations head adds or changes against trunk.
func changedDeclarations(head, trunk map[string]delivery.ConfigDeclaration) map[string]delivery.ConfigDeclaration {
out := map[string]delivery.ConfigDeclaration{}
for key, d := range head {
if before, there := trunk[key]; !there || !reflect.DeepEqual(before, d) {
out[key] = d
}
}
return out
}
// trunkManifest is the manifest the indexer holds for a module on the trunk, "" when it holds none.
func (ix *Indexer) trunkManifest(ctx context.Context, held map[string]delivery.Module, module string) (string, error) {
m, ok := held[module]
if !ok || m.Manifest == "" || ix.Manifests == nil {
return "", nil
}
raw, _, found, err := ix.Manifests.Get(ctx, m.Manifest)
if err != nil || !found {
return "", err
}
kept, err := delivery.DecodeManifest(raw)
if err != nil {
return "", err
}
return kept.JSON, nil
}
// candidateDeclares are, per module of a pull request's head, the declarations its manifest adds or changes against the
// trunk's (an empty set for a module that adds none): heads are the head's manifests by module.
func (ix *Indexer) candidateDeclares(ctx context.Context, held map[string]delivery.Module, heads map[string]string) (map[string]map[string]delivery.ConfigDeclaration, error) {
out := map[string]map[string]delivery.ConfigDeclaration{}
for module, raw := range heads {
head, err := DeclarationsOf(raw)
if err != nil {
return nil, fmt.Errorf("%w: the manifest of %s at the head: %v", ErrUnreadable, module, err)
}
trunkRaw, err := ix.trunkManifest(ctx, held, module)
if err != nil {
return nil, err
}
trunk := map[string]delivery.ConfigDeclaration{}
if trunkRaw != "" {
if trunk, err = DeclarationsOf(trunkRaw); err != nil {
return nil, fmt.Errorf("the trunk's manifest of %s: %w", module, err)
}
}
if changed := changedDeclarations(head, trunk); len(changed) > 0 {
out[module] = changed
}
}
return out, nil
}
// writeConfigCandidate writes a pull request's key of config-candidates from its head's manifests by module: written when
// what it says differs (its time apart), never for the same head and declarations twice.
func (ix *Indexer) writeConfigCandidate(ctx context.Context, repository string, number int, head string, held map[string]delivery.Module, heads map[string]string) error {
if ix.ConfigCandidates == nil {
return nil
}
pull := PullName(repository, number)
key, err := delivery.ConfigCandidateKey(pull)
if err != nil {
return fmt.Errorf("%w: %v", ErrUnreadable, err)
}
declares, err := ix.candidateDeclares(ctx, held, heads)
if err != nil {
return err
}
return ix.notYet(ConfigCandidatesBucket, write(ctx, ix.ConfigCandidates, key, func(value []byte, found bool) ([]byte, bool, error) {
if found {
var cur delivery.ConfigCandidate
if json.Unmarshal(value, &cur) == nil && cur.Head == head && reflect.DeepEqual(cur.Declares, declares) {
return nil, false, nil
}
}
next, err := json.Marshal(delivery.ConfigCandidate{V: delivery.Version, Pull: pull, Head: head, Declares: declares, At: ix.now()})
return next, false, err
}))
}
// dropConfigCandidate deletes a pull request's key: it merged or closed.
func (ix *Indexer) dropConfigCandidate(ctx context.Context, repository string, number int) error {
if ix.ConfigCandidates == nil {
return nil
}
key, err := delivery.ConfigCandidateKey(PullName(repository, number))
if err != nil {
return nil
}
return ix.notYet(ConfigCandidatesBucket, write(ctx, ix.ConfigCandidates, key, func(_ []byte, found bool) ([]byte, bool, error) { return nil, found, nil }))
}
// configCandidateOfPull is a pull message's part in config-candidates: a head whose manifests the message carries is written,
// a closed one deleted. A head held again without its directories leaves the key as it is.
func (ix *Indexer) configCandidateOfPull(ctx context.Context, f delivery.Forge, held map[string]delivery.Module) error {
if f.Action == delivery.PullClosed {
return ix.dropConfigCandidate(ctx, f.Repository, f.Number)
}
if !f.DirsKnown {
return nil
}
heads := map[string]string{}
for dir, raw := range f.Manifests {
heads[moduleOfDir(held, f, dir)] = raw
}
return ix.writeConfigCandidate(ctx, f.Repository, f.Number, f.Head, held, heads)
}
// configCandidatesAfterMerge writes again every open pull request's key that names a module the merge changed, since what
// it adds is judged against the trunk (ADR 0565 §2), from the head's manifests its candidate keeps; and deletes the merged
// pull request's.
func (ix *Indexer) configCandidatesAfterMerge(ctx context.Context, f delivery.Forge, changed []string) error {
if ix.ConfigCandidates == nil {
return nil
}
if err := ix.dropConfigCandidate(ctx, f.Repository, f.Number); err != nil {
return err
}
if len(changed) == 0 || ix.Store == nil || ix.Manifests == nil {
return nil
}
held, err := ix.heldModules(ctx)
if err != nil {
return err
}
if _, err := ix.ConfigCandidates.Keys(ctx); errors.Is(err, process.ErrNotYet) {
return nil
}
groups, err := ix.Store.Keys(ctx)
if err != nil {
return err
}
var errs []error
for _, g := range groups {
raw, _, found, err := ix.Store.Get(ctx, g)
if err != nil || !found {
continue
}
c, err := delivery.DecodeCandidate(raw)
if err != nil {
continue
}
for _, p := range c.Pulls {
if !touchesAny(p, changed) {
continue
}
heads := map[string]string{}
for module, fp := range p.Fps {
mraw, _, found, err := ix.Manifests.Get(ctx, fp.Manifest)
if err != nil || !found {
continue
}
if kept, err := delivery.DecodeManifest(mraw); err == nil {
heads[module] = kept.JSON
}
}
if err := ix.writeConfigCandidate(ctx, p.Repository, p.Number, p.Head.Head, held, heads); err != nil {
errs = append(errs, err)
}
}
}
return errors.Join(errs...)
}
func touchesAny(p delivery.CandidatePull, modules []string) bool {
for _, m := range modules {
if _, ok := p.Fps[m]; ok {
return true
}
}
return false
}
// writtenModules are the modules a merged message wrote, from its written keys (`modules.<module>`).
func writtenModules(written []string) []string {
var out []string
for _, k := range written {
out = append(out, strings.TrimPrefix(k, "modules."))
}
return out
}
@@ -0,0 +1,88 @@
package index
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// 1953/D2, D3 (novox/hq ADR 0565 §2): the indexer keeps the catalogue's predefined list from the merged message, a merge older
// than the one kept never replacing it; and each open pull request's config declarations its head adds or changes against the
// trunk's manifest, an empty set when none, written only when they change and deleted when it merges or closes.
func TestTheControlRecords(t *testing.T) {
ctx := context.Background()
at := time.Date(2026, 10, 29, 22, 0, 0, 0, time.UTC)
t.Run("the predefined list, never replaced by an older merge", func(t *testing.T) {
ix, _, _ := manifestsIndexer(t)
pre := newMem()
ix.Predefined, ix.Now = pre, func() time.Time { return at }
f := mergedMsg(catalogueRepo, map[string]string{"modules/a": "t1"})
f.Predefined = json.RawMessage(`{"font-size":{"type":"integer","value":13}}`)
if err := ix.Apply(ctx, f); err != nil {
t.Fatal(err)
}
var kept delivery.PredefinedKept
if err := json.Unmarshal(pre.vals[delivery.PredefinedListKey], &kept); err != nil || kept.Commit != merge || string(kept.List) != string(f.Predefined) {
t.Fatalf("kept = %+v (%v)", kept, err)
}
writes := pre.writes
if err := ix.Apply(ctx, f); err != nil || pre.writes != writes {
t.Errorf("the same list at the same merge wrote again (%v)", err)
}
older := f
older.Merge, older.Predefined = strings.Repeat("e", 40), json.RawMessage(`{}`)
ix.Contains = func(context.Context, string, string, string) (bool, error) { return true, nil }
if err := ix.writePredefined(ctx, older); err != nil || pre.writes != writes {
t.Errorf("an older merge replaced the list (%v)", err)
}
ix.Predefined = &memStore{unbound: process.ErrNotYet}
if err := ix.writePredefined(ctx, f); err != nil {
t.Errorf("a state not on the bus yet stopped the message: %v", err)
}
})
t.Run("a pull request's config declarations against the trunk", func(t *testing.T) {
ix, _, _ := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs))
cands := newMem()
ix.ConfigCandidates, ix.Now = cands, func() time.Time { return at }
trunk := `{"module":"avahi","version":"1","config":{"port":{"type":"integer","default":5353,"why":"mDNS"}}}`
if err := ix.Apply(ctx, withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "t1"}), map[string]string{"modules/avahi": trunk}, nil)); err != nil {
t.Fatal(err)
}
key := "novox.mesh-catalog.7"
head := strings.Replace(trunk, `"why":"mDNS"}`, `"why":"mDNS"},"name":{"type":"string","default":"x","why":"shown"}`, 1)
p := withManifests(pull(catalogueRepo, 7, "feat/y", h1, "modules/avahi"), map[string]string{"modules/avahi": head}, nil)
if err := ix.Apply(ctx, p); err != nil {
t.Fatal(err)
}
var c delivery.ConfigCandidate
if err := json.Unmarshal(cands.vals[key], &c); err != nil || c.Pull != "novox/mesh-catalog#7" || c.Head != h1 || len(c.Declares["avahi"]) != 1 ||
string(c.Declares["avahi"]["name"].Default) != `"x"` {
t.Fatalf("candidate = %+v (%v)", c, err)
}
writes := cands.writes
if err := ix.Apply(ctx, p); err != nil || cands.writes != writes {
t.Errorf("the same head wrote again (%v)", err)
}
same := withManifests(pull(catalogueRepo, 7, "feat/y", strings.Repeat("d", 40), "modules/avahi"), map[string]string{"modules/avahi": trunk}, nil)
if err := ix.Apply(ctx, same); err != nil {
t.Fatal(err)
}
c = delivery.ConfigCandidate{}
if err := json.Unmarshal(cands.vals[key], &c); err != nil || len(c.Declares) != 0 || c.Declares == nil {
t.Errorf("a head declaring nothing new = %+v, want an empty set", c)
}
closed := pull(catalogueRepo, 7, "feat/y", strings.Repeat("d", 40))
closed.Action = delivery.PullClosed
if err := ix.Apply(ctx, closed); err != nil {
t.Fatal(err)
}
if _, ok := cands.vals[key]; ok {
t.Error("a closed pull request's declarations stayed")
}
})
}
@@ -2,7 +2,6 @@ package index
import ( import (
"context" "context"
"encoding/json"
"strings" "strings"
"testing" "testing"
@@ -85,14 +84,7 @@ func (m *memStore) Delete(_ context.Context, key string, rev uint64) error {
// the controller's encoder; copied here, the vendored SDK does not carry testdata) has it, with the repository, module and path changed. // the controller's encoder; copied here, the vendored SDK does not carry testdata) has it, with the repository, module and path changed.
func buildRecord(t *testing.T, repo, module, path, at string) []byte { func buildRecord(t *testing.T, repo, module, path, at string) []byte {
t.Helper() t.Helper()
raw := []byte(`{"id":"build-1792395447088936520","repository":"novox/mesh-controller","ref":"main","module":"build-agent","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/build-agent","source-fingerprint":"src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7","at":"2026-10-19T09:38:00Z"}`) return logged(t, "build-"+module, module, repo, path, "", "", at)
var m map[string]any
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
m["repository"], m["module"], m["path"], m["at"] = repo, module, path, at
b, _ := json.Marshal(m)
return b
} }
func pull(repo string, number int, branch, head string, dirs ...string) delivery.Forge { func pull(repo string, number int, branch, head string, dirs ...string) delivery.Forge {
@@ -104,9 +96,9 @@ func newIndexer(t *testing.T) (*Indexer, *memStore, *strings.Builder) {
t.Helper() t.Helper()
st, logs := newMem(), &strings.Builder{} st, logs := newMem(), &strings.Builder{}
b := NewBuilds() b := NewBuilds()
b.Put(delivery.BuildKey("a", "build-1"), buildRecord(t, "novox/mesh-delivery", "a", "modules/a", "2026-10-19T09:00:00Z")) _ = b.Put(buildRecord(t, "novox/mesh-delivery", "a", "modules/a", "2026-10-19T09:00:00Z"))
b.Put(delivery.BuildKey("c", "build-2"), buildRecord(t, "novox/mesh-catalog", "c", "modules/c", "2026-10-19T09:00:00Z")) _ = b.Put(buildRecord(t, "novox/mesh-catalog", "c", "modules/c", "2026-10-19T09:00:00Z"))
b.Put(delivery.BuildKey("odd", "build-3"), buildRecord(t, "novox/mesh-delivery", "odd-name", "modules/odd", "2026-10-19T09:00:00Z")) _ = b.Put(buildRecord(t, "novox/mesh-delivery", "odd-name", "modules/odd", "2026-10-19T09:00:00Z"))
return &Indexer{Store: st, Builds: b, Logf: func(f string, a ...any) { return &Indexer{Store: st, Builds: b, Logf: func(f string, a ...any) {
logs.WriteString(strings.TrimSpace(strings.ReplaceAll(f, "\n", " ")) + "\n") logs.WriteString(strings.TrimSpace(strings.ReplaceAll(f, "\n", " ")) + "\n")
_ = a _ = a
@@ -2,6 +2,7 @@ package index
import ( import (
"testing" "testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
@@ -23,13 +24,10 @@ const (
) )
func liveRecord(id, module, repository, ref, commit, path, at, source string) [2]any { func liveRecord(id, module, repository, ref, commit, path, at, source string) [2]any {
raw := `{"id":"` + id + `","repository":"` + repository + `",` when, _ := time.Parse(time.RFC3339, at)
if ref != "" { raw, _ := delivery.EncodeBuildLogEntry(delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: module, FP: source, Repository: repository,
raw += `"ref":"` + ref + `",` Directory: path, Commit: commit, Ref: ref, Node: "n1", At: when, Outcome: delivery.BuildDone})
} return [2]any{module, raw}
raw += `"module":"` + module + `","commit":"` + commit + `","path":"` + path + `","at":"` + at + `","source-fingerprint":"` + source +
`","source-inputs":{"Module":"` + module + `",` + liveInputsTail + `}`
return [2]any{delivery.BuildKey(module, id), []byte(raw)}
} }
func liveBuilds() [][2]any { func liveBuilds() [][2]any {
@@ -42,7 +42,12 @@ func withManifests(f delivery.Forge, manifests map[string]string, locks map[stri
return f return f
} }
func TestAMergedManifestIsKeptByItsDigestAndNamedByTheModule(t *testing.T) { // 1449/AC1 func TestAMergedManifestIsKeptOnceByItsDigestAndNamedByTheModule(t *testing.T) {
t.Run("kept by its digest and named by the module", mergedManifestIsKeptByItsDigestAndNamedByTheModule)
t.Run("the same manifest applied again is written once", sameManifestAppliedAgainIsWrittenOnce)
}
func mergedManifestIsKeptByItsDigestAndNamedByTheModule(t *testing.T) { // 1449/AC1
avahi := catalogueManifest(t, "avahi") avahi := catalogueManifest(t, "avahi")
ix, mods, manifests := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs)) ix, mods, manifests := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs))
f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi}, nil) f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi}, nil)
@@ -63,7 +68,7 @@ func TestAMergedManifestIsKeptByItsDigestAndNamedByTheModule(t *testing.T) { //
} }
} }
func TestTheSameManifestAppliedAgainIsWrittenOnce(t *testing.T) { // 1449/AC2 func sameManifestAppliedAgainIsWrittenOnce(t *testing.T) { // 1449/AC2
avahi := catalogueManifest(t, "avahi") avahi := catalogueManifest(t, "avahi")
ix, _, manifests := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs)) ix, _, manifests := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs))
f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi}, nil) f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi}, nil)
@@ -2,12 +2,12 @@ package index
import ( import (
"context" "context"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"path" "path"
"strings" "strings"
"testing" "testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
@@ -15,29 +15,38 @@ import (
const merge = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" const merge = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
// record is a build record as the controller writes it (mesh-sdk go/delivery/testdata/build-record-build-agent.json is // logged is one entry of the builder's log (mesh-sdk go/delivery BuildLogEntry, as the builder appends it): a build of
// the captured shape): SourceInputs under its Go field names, no json keys. inputs is that object's JSON. // module from repo's directory path at ref ("" for the trunk), its outcome ("" done, "failed", or "asked" for a build only
func record(t *testing.T, id, module, repo, path, ref, state, at, inputs string) (string, []byte) { // started), at the time at; id makes its fingerprint.
func logged(t *testing.T, id, module, repo, path, ref, outcome, at string) []byte {
t.Helper() t.Helper()
// an outcome has no state key (mesh-controller internal/chain/records.go:57-59); state asked is a request when, err := time.Parse(time.RFC3339, at)
raw := `{"id":"` + id + `",` if err != nil {
switch state {
case "":
case "failed": // a failure is an outcome with failed set and no state (mesh-controller internal/chain/records.go:109)
raw += `"failed":"the build failed",`
default:
raw += `"state":"` + state + `",`
}
raw += `"repository":"` + repo + `","ref":"` + ref + `","module":"` + module + `","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"` + path + `","at":"` + at + `"`
if inputs != "" {
raw += `,"source-fingerprint":"src1:ignored","source-inputs":` + inputs
}
raw += `}`
var check delivery.BuildRecord
if err := json.Unmarshal([]byte(raw), &check); err != nil {
t.Fatal(err) t.Fatal(err)
} }
return delivery.BuildKey(module, id), []byte(raw) if path == "" {
path = delivery.RootPath
}
e := delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: module, FP: "src2:" + id, Repository: repo, Directory: path,
Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", Ref: ref, Node: "n1", At: when, Outcome: delivery.BuildDone}
switch outcome {
case "":
case "failed":
e.Outcome = delivery.BuildFailed
default:
e.Event, e.Outcome = delivery.BuildStartedEvent, ""
}
raw, err := delivery.EncodeBuildLogEntry(e)
if err != nil {
t.Fatal(err)
}
return raw
}
// record is logged under the module's log key, as the tests' tables carry it.
func record(t *testing.T, id, module, repo, path, ref, state, at, _ string) (string, []byte) {
t.Helper()
return module, logged(t, id, module, repo, path, ref, state, at)
} }
func modulesIndexer(t *testing.T, recs ...[2]any) (*Indexer, *memStore) { func modulesIndexer(t *testing.T, recs ...[2]any) (*Indexer, *memStore) {
@@ -45,7 +54,9 @@ func modulesIndexer(t *testing.T, recs ...[2]any) (*Indexer, *memStore) {
mods := newMem() mods := newMem()
ix := &Indexer{Store: newMem(), Modules: mods, Builds: NewBuilds()} ix := &Indexer{Store: newMem(), Modules: mods, Builds: NewBuilds()}
for _, r := range recs { for _, r := range recs {
ix.Builds.Put(r[0].(string), r[1].([]byte)) if err := ix.Builds.Put(r[1].([]byte)); err != nil {
t.Fatal(err)
}
} }
return ix, mods return ix, mods
} }
@@ -302,14 +302,14 @@ func TestARecordOfAPullRequestDoesNotReplaceTheRegisteredRef(t *testing.T) { //
rec(t, "b2", "photos", repo, "", "nox-mesh", "", "2026-10-02T09:00:00Z", ""), rec(t, "b2", "photos", repo, "", "nox-mesh", "", "2026-10-02T09:00:00Z", ""),
rec(t, "b3", "photos", repo, "", "feat/x", "", "2026-10-09T09:00:00Z", ""), rec(t, "b3", "photos", repo, "", "feat/x", "", "2026-10-09T09:00:00Z", ""),
} { } {
b.Put(r[0].(string), r[1].([]byte)) _ = b.Put(r[1].([]byte))
} }
got := b.RegistrationsOf("photos") got := b.RegistrationsOf("photos")
if len(got) != 1 || got[0].Ref != "nox-mesh" { if len(got) != 1 || got[0].Ref != "nox-mesh" {
t.Fatalf("registrations = %+v, want the ref most records name, not the newer feat/x", got) t.Fatalf("registrations = %+v, want the ref most records name, not the newer feat/x", got)
} }
k, v := record(t, "b4", "photos", repo, "", "main", "", "2026-10-03T09:00:00Z", "") k, v := record(t, "b4", "photos", repo, "", "main", "", "2026-10-03T09:00:00Z", "")
b.Put(k, v) // a trunk's ref beats a branch of its own _, _ = k, b.Put(v) // a trunk's ref beats a branch of its own
if got := b.RegistrationsOf("photos"); len(got) != 1 || got[0].Ref != "main" { if got := b.RegistrationsOf("photos"); len(got) != 1 || got[0].Ref != "main" {
t.Fatalf("registrations = %+v", got) t.Fatalf("registrations = %+v", got)
} }
@@ -46,13 +46,10 @@ const (
) )
func agentBuild(id, src, at string) [2]any { func agentBuild(id, src, at string) [2]any {
commit := "cada7e3c" when, _ := time.Parse(time.RFC3339, at)
if id == "build-1792587792086548572" { raw, _ := delivery.EncodeBuildLogEntry(delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: "build-agent", FP: src, Repository: agentCloneURL,
commit = "f10fbda1" Directory: "modules/build-agent", Commit: "cada7e3c", Node: "n1", At: when, Outcome: delivery.BuildDone})
} return [2]any{"build-agent", raw}
raw := `{"id":"` + id + `","repository":"` + agentCloneURL + `","ref":"main","module":"build-agent","commit":"` + commit + `",` +
`"path":"modules/build-agent","at":"` + at + `","source-fingerprint":"` + src + `","source-inputs":` + aInputs + `}`
return [2]any{delivery.BuildKey("build-agent", id), []byte(raw)}
} }
type mod struct { type mod struct {
@@ -184,7 +181,7 @@ func TestBasesAreComputedFirstAndNoFingerprintTakesTheBuilder(t *testing.T) { //
t.Errorf("other: fp2 = %q none2 = %q, want none2 naming the base ts-base", m.FP2, m.None2) t.Errorf("other: fp2 = %q none2 = %q, want none2 naming the base ts-base", m.FP2, m.None2)
} }
a := agentBuild("build-1792587792086548572", agentSrcY, "2026-10-21T15:03:00Z") a := agentBuild("build-1792587792086548572", agentSrcY, "2026-10-21T15:03:00Z")
ix.Builds.Put(a[0].(string), a[1].([]byte)) _ = ix.Builds.Put(a[1].([]byte))
if err := ix.Recompute(context.Background()); err != nil { if err := ix.Recompute(context.Background()); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -86,10 +86,10 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
t.Errorf("emits = %v, want none", m.Emits) t.Errorf("emits = %v, want none", m.Emits)
} }
// 1126 note 76: heard keeps the last 64 FORGE messages the indexer handled, for indexer_heard. // 1126 note 76: heard keeps the last 64 FORGE messages the indexer handled, for indexer_heard.
if want := []state{{Name: "candidates", History: 10}, {Name: "heard", History: 64}, {Name: "manifests", History: 1}, {Name: "modules", History: 20}, {Name: "taken", History: 1}, {Name: "takeins", History: 1}}; !reflect.DeepEqual(m.State, want) { if want := []state{{Name: "candidates", History: 10}, {Name: "heard", History: 64}, {Name: "manifests", History: 1}, {Name: "modules", History: 20}, {Name: "predefined", History: 5}, {Name: "config-candidates", History: 5}, {Name: "taken", History: 1}, {Name: "takeins", History: 1}}; !reflect.DeepEqual(m.State, want) {
t.Errorf("state = %+v, want %+v", m.State, want) t.Errorf("state = %+v, want %+v", m.State, want)
} }
if want := []string{"mesh-build-go.toolchains", "mesh-build-node.toolchains", "mesh-controller.assignments", "mesh-controller.builds", "mesh-delivery-checker.previews", "mesh-delivery-gate.source"}; !reflect.DeepEqual(sorted(m.Reads), want) { if want := []string{"mesh-build-go.toolchains", "mesh-build-node.toolchains", "mesh-delivery-assigner.placement", "mesh-delivery-builder.build-log", "mesh-delivery-checker.previews", "mesh-delivery-gate.source"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want) t.Errorf("reads = %v, want %v", m.Reads, want)
} }
// "Reads FORGE" is a consumes of the indexer's stream: the broker grants only buckets by reads (mesh-control // "Reads FORGE" is a consumes of the indexer's stream: the broker grants only buckets by reads (mesh-control
+4 -2
View File
@@ -16,14 +16,16 @@
{ "name": "heard", "history": 64 }, { "name": "heard", "history": 64 },
{ "name": "manifests", "history": 1 }, { "name": "manifests", "history": 1 },
{ "name": "modules", "history": 20 }, { "name": "modules", "history": 20 },
{ "name": "predefined", "history": 5 },
{ "name": "config-candidates", "history": 5 },
{ "name": "taken", "history": 1 }, { "name": "taken", "history": 1 },
{ "name": "takeins", "history": 1 } { "name": "takeins", "history": 1 }
], ],
"reads": [ "reads": [
"mesh-build-go.toolchains", "mesh-build-go.toolchains",
"mesh-build-node.toolchains", "mesh-build-node.toolchains",
"mesh-controller.assignments", "mesh-delivery-assigner.placement",
"mesh-controller.builds", "mesh-delivery-builder.build-log",
"mesh-delivery-checker.previews", "mesh-delivery-checker.previews",
"mesh-delivery-gate.source" "mesh-delivery-gate.source"
], ],
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-teller
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 h1:ycu/sNieTdLpD3bzhZOCXdzcoqXUOxPRmqWV019REpo= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24 h1:s0XXZtR7CFPBbnaDygWnE4/Q827iEFt0YfEzujk4Q9g=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -19,7 +19,11 @@ import (
func serve(t *testing.T) string { func serve(t *testing.T) string {
t.Helper() t.Helper()
m := view.New() m := view.New()
m.Apply(view.Entry{Bucket: view.AssignmentsBucket, Key: "a.x", Value: []byte(`{"node":"a","module":"x","build":"b1"}`)}) placed, err := delivery.EncodePlacement(delivery.Placement{Node: "a", Modules: []delivery.PlacedModule{{Module: "x"}}})
if err != nil {
t.Fatal(err)
}
m.Apply(view.Entry{Bucket: delivery.PlacementBucket, Key: "placement.a", Value: placed})
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "c") dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "c")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -8,6 +8,7 @@ import (
"fmt" "fmt"
"os" "os"
"strings" "strings"
"time"
"mesh-delivery-view/internal/view" "mesh-delivery-view/internal/view"
@@ -19,6 +20,14 @@ import (
// Tools are the read tools over a modules state, and build_usage over the build modules' states. // Tools are the read tools over a modules state, and build_usage over the build modules' states.
func Tools(r view.Reader, u view.UsageReader) []stdio.Tool { func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
return []stdio.Tool{ return []stdio.Tool{
{Name: view.StatusTool,
Description: "The mesh's status, derived by the view from the buckets it reads (novox/hq ADR 0564 §10), nothing written as a " +
"summary: every open condition of the condition keeper (mesh-conditions), urgent first then oldest; what is quiet (each " +
"node the watchman says is offline, each key it says stopped moving); each module whose want and have differ on a node " +
"(moving, or stuck with the have's or the watchman's words), as mesh-sdk's delivery motion derives it; and what " +
"mesh-healers did in the last seven days, by outcome. A part that cannot be read says so, never as none; well is true " +
"only when nothing is open, quiet or stuck and every part was read. Read-only.",
Run: func(map[string]any) (any, error) { return r.Status(time.Now()), nil }},
{Name: view.ReachReadTool, {Name: view.ReachReadTool,
Description: "Whether a merge is on the machines: per repository, the reached head (the newest merge every node runs a build " + Description: "Whether a merge is on the machines: per repository, the reached head (the newest merge every node runs a build " +
"containing, as the view derives it from the indexer's modules, the wants and the haves; a merge at or before it is on the " + "containing, as the view derives it from the indexer's modules, the wants and the haves; a merge at or before it is on the " +
@@ -95,6 +104,23 @@ func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
group, _ := args["group"].(string) group, _ := args["group"].(string)
return r.Candidate(group) return r.Candidate(group)
}}, }},
{Name: view.PredefinedReadTool,
Description: "The catalogue's predefined list as the indexer keeps it (novox/hq ADR 0565 §2): its key list, the list whole as " +
"predefined.json holds it with the merge it was read at; a merge older than that never replaces it. mesh-config " +
"publishes the predefined keys from it after its take-over. Read-only.",
Run: func(map[string]any) (any, error) {
return view.ControlRecord(r.PredefinedState, delivery.IndexerPredefinedState, delivery.PredefinedListKey)
}},
{Name: view.ConfigCandidatesReadTool,
Description: "Each open pull request's config declarations as the indexer keeps them (novox/hq ADR 0565 §2): per pull " +
"request (owner.repository.number), its head and per module the config keys its manifests add or change against the " +
"trunk's, an empty set when none; deleted when it merges or closes. Give key for one. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"key": map[string]any{"type": "string", "description": "one pull request's key, owner.repository.number"}}},
Run: func(args map[string]any) (any, error) {
key, _ := args["key"].(string)
return view.ControlRecord(r.ConfigCandidatesState, delivery.IndexerConfigCandidatesState, key)
}},
{Name: view.CandidatesReadTool, {Name: view.CandidatesReadTool,
Description: "Every candidate the indexer holds, by group, each as stored under candidates.<group>; with module, only those with a " + Description: "Every candidate the indexer holds, by group, each as stored under candidates.<group>; with module, only those with a " +
"pull request touching that module. Keys that cannot be read are listed as unreadable. Read-only.", "pull request touching that module. Keys that cannot be read are listed as unreadable. Read-only.",
@@ -119,6 +145,27 @@ func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
limit, _ := args["limit"].(float64) limit, _ := args["limit"].(float64)
return r.BuilderBuilds(module, state, int(limit)) return r.BuilderBuilds(module, state, int(limit))
}}, }},
{Name: view.BuildHistoryTool,
Description: "Every build that ended since a time (default the last 24 hours), oldest first, from the builder's log build-log (kept 30 " +
"days): its module, fingerprint, repository, directory, commit and ref, the build node, when it ended, its outcome (done, failed or " +
"queued), the manifest, the bases it was built against, each artifact made, and seconds, how long it took from its start (absent " +
"when the start is older than since). With module, that module's builds only; more says the log held more: ask again from the " +
"last build's time. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "only this module's builds"},
"since": map[string]any{"type": "string", "description": "RFC 3339 time; default 24 hours ago"}}},
Run: func(args map[string]any) (any, error) {
module, _ := args["module"].(string)
since := time.Now().Add(-view.DefaultHistorySince)
if raw, _ := args["since"].(string); strings.TrimSpace(raw) != "" {
at, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(raw))
if err != nil {
return nil, fmt.Errorf("since %q is not an RFC 3339 time: %w", raw, err)
}
since = at
}
return r.BuildHistory(module, since)
}},
{Name: view.BuilderBuildReadTool, {Name: view.BuilderBuildReadTool,
Description: "One build of the builder as stored. Give the module and its fingerprint (fp2, src2:...), or the module alone for the build " + Description: "One build of the builder as stored. Give the module and its fingerprint (fp2, src2:...), or the module alone for the build " +
"at the fingerprint the indexer holds for it now, joined through modules.<module>; with commit, only if the indexer holds the module at that commit (a module's value names the commit its fingerprint first appeared at, and stays there through merges that leave the module alone). Read-only.", "at the fingerprint the indexer holds for it now, joined through modules.<module>; with commit, only if the indexer holds the module at that commit (a module's value names the commit its fingerprint first appeared at, and stays there through merges that leave the module alone). Read-only.",
@@ -166,6 +213,15 @@ func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
node, module := nodeModuleOf(args) node, module := nodeModuleOf(args)
return r.Haves(node, module) return r.Haves(node, module)
}}, }},
{Name: view.TrustsReadTool,
Description: "What every node trusts a mesh signature from, in one call (novox/hq ADR 0550 §2): each node's record's trusts " +
"(from node-engine_have <node>.node), whether it names the vault's signing key as mesh-vault publishes it in its " +
"state signers and whether it still names the controller's, the nodes that do not name the vault's yet (not_yet), " +
"and every: true once all do, the gate before the controller's lease is handed away. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) {
return r.Trusts()
}},
} }
} }
@@ -223,8 +279,14 @@ func read(state stdio.KeptState) view.StateRead {
func main() { func main() {
r := view.Reader{State: read(stdio.State("mesh-delivery-indexer.modules")), Builds: read(stdio.State("mesh-delivery-builder.builds")), CandidatesState: read(stdio.State("mesh-delivery-indexer.candidates")), r := view.Reader{State: read(stdio.State("mesh-delivery-indexer.modules")), Builds: read(stdio.State("mesh-delivery-builder.builds")), CandidatesState: read(stdio.State("mesh-delivery-indexer.candidates")),
WantsState: read(stdio.State("mesh-delivery-assigner.wants")), DifferencesState: read(stdio.State("mesh-delivery-assigner.differences")), HavesState: read(stdio.State("node-engine.have")), WantsState: read(stdio.State("mesh-delivery-assigner.wants")), DifferencesState: read(stdio.State("mesh-delivery-assigner.differences")), HavesState: read(stdio.State("node-engine.have")),
ReachState: read(stdio.State("reach")), ReachState: read(stdio.State("reach")),
HeardState: read(stdio.State("mesh-delivery-indexer.heard"))} HeardState: read(stdio.State("mesh-delivery-indexer.heard")),
SignersState: read(stdio.State("mesh-vault.signers")),
ConditionsState: read(stdio.State(view.StatusConditionsState)), AttemptsState: read(stdio.State(view.StatusAttemptsState)),
PresenceState: read(stdio.State(view.StatusPresenceState)), StuckState: read(stdio.State(view.StatusStuckState)),
PredefinedState: read(stdio.State(delivery.IndexerModule + "." + delivery.IndexerPredefinedState)),
ConfigCandidatesState: read(stdio.State(delivery.IndexerModule + "." + delivery.IndexerConfigCandidatesState)),
BuildLog: stdio.Log(delivery.BuildLogRead).Since}
u := view.UsageReader{Haves: r.HavesState, Resolved: read(stdio.State("mesh-config.resolved")), Tries: read(stdio.State("mesh-delivery-checker.tries")), u := view.UsageReader{Haves: r.HavesState, Resolved: read(stdio.State("mesh-config.resolved")), Tries: read(stdio.State("mesh-delivery-checker.tries")),
Steps: map[string]view.StateRead{}, Pauses: map[string]view.StateRead{}, Backlog: backlogOf(socket())} Steps: map[string]view.StateRead{}, Pauses: map[string]view.StateRead{}, Backlog: backlogOf(socket())}
for _, lang := range delivery.AskLanguages { for _, lang := range delivery.AskLanguages {
@@ -1,13 +1,18 @@
package main package main
import ( import (
"crypto/ed25519"
"encoding/json" "encoding/json"
"fmt" "fmt"
"reflect"
"strings" "strings"
"sync/atomic" "sync/atomic"
"testing" "testing"
"time" "time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-view/internal/view" "mesh-delivery-view/internal/view"
) )
@@ -173,6 +178,28 @@ func TestBuilderBuildsListsKeysAsStored(t *testing.T) {
t.Errorf("%s filter = %+v, want %s", args, got, want) t.Errorf("%s filter = %+v, want %s", args, got, want)
} }
} }
// 1955/L7 (ADR 0571 clause 1): builder_history answers each end from the builder's log with its duration from its start,
// the start of a later build of the same fingerprint not counted, and narrows by module through the log's key.
t0 := time.Date(2026, 10, 29, 22, 0, 0, 0, time.UTC)
entry := func(event string, at time.Duration) []byte {
e := delivery.BuildLogEntry{Event: event, Module: "netcheck", FP: fp2A, Repository: "novox/mesh-catalog", Directory: "modules/netcheck",
Commit: strings.Repeat("7", 40), Node: "ace", At: t0.Add(at)}
if event == delivery.BuildEndedEvent {
e.Outcome = delivery.BuildDone
}
raw, _ := delivery.EncodeBuildLogEntry(e)
return raw
}
var asked string
r.BuildLog = func(at time.Time, keys string, limit int) (*stdio.LogPage, error) {
asked = keys
return &stdio.LogPage{Entries: []stdio.LogEntry{{Sequence: 1, Value: entry(delivery.BuildStartedEvent, 0)},
{Sequence: 2, Value: entry(delivery.BuildEndedEvent, 90*time.Second)}, {Sequence: 3, Value: entry(delivery.BuildStartedEvent, time.Hour)}}}, nil
}
out, err = run(t, r, view.BuildHistoryTool, map[string]any{"module": "netcheck", "since": t0.Format(time.RFC3339)})
if h, _ := out.(view.HistoryAnswer); err != nil || len(h.Builds) != 1 || h.Builds[0].Seconds != 90 || h.Builds[0].Outcome != delivery.BuildDone || asked != "netcheck" || h.More {
t.Errorf("history = %+v, %v (keys %q), want the one end at 90 s", out, err, asked)
}
} }
// 1723/AC2: builder_build answers one key by module and fp2, and by module alone through the indexer's fp2 (joined at its commit). // 1723/AC2: builder_build answers one key by module and fp2, and by module alone through the indexer's fp2 (joined at its commit).
@@ -438,6 +465,41 @@ func TestAssignerDifferencesAndNodeHavesToolsAnswerAsStored(t *testing.T) {
t.Errorf("haves of ace (the node's own key too) = %v", got) t.Errorf("haves of ace (the node's own key too) = %v", got)
} }
// L3 s6b (ADR 0550 §2): node_trusts answers for every node in one call whether its record names the vault's key the
// vault publishes, and every is true only once all do.
vaultPub, _, _ := ed25519.GenerateKey(nil)
vaultKey, _ := json.Marshal(delivery.NewSignerKey(vaultPub, time.Now()))
vaultFP := delivery.SignerFingerprint(vaultPub)
record := func(node string, trusts ...string) string {
raw, _ := delivery.EncodeHaveNode(delivery.HaveNode{V: delivery.Version, Record: &delivery.NodeRecord{Node: node, Trusts: trusts}})
return string(raw)
}
for _, row := range []struct {
name string
haves fake
every bool
notYet []string
}{
{"one node trusts the controller's alone", fake{"ace.node": record("ace", "controller key sha256:aa", "vault key "+vaultFP),
"g14.node": record("g14", "controller key sha256:aa"), "ace.zsh": `{"fp":"x"}`}, false, []string{"g14"}},
{"every node trusts the vault's key", fake{"ace.node": record("ace", "vault key "+vaultFP),
"g14.node": record("g14", "controller key sha256:aa", "vault key "+vaultFP)}, true, []string{}},
} {
tr := view.Reader{HavesState: row.haves.reader().State, SignersState: fake{"key": string(vaultKey)}.reader().State}
out, err := run(t, tr, view.TrustsReadTool, map[string]any{})
if err != nil {
t.Fatalf("%s: %v", row.name, err)
}
a := out.(view.TrustsAnswer)
if a.Every != row.every || !reflect.DeepEqual(a.NotYet, row.notYet) || a.VaultKey != vaultFP || len(a.Nodes) != 2 {
t.Errorf("%s: %+v", row.name, a)
}
}
if out, err := run(t, view.Reader{HavesState: fake{"ace.node": record("ace", "vault key "+vaultFP)}.reader().State,
SignersState: fake{}.reader().State}, view.TrustsReadTool, map[string]any{}); err != nil || out.(view.TrustsAnswer).Every {
t.Errorf("with no key published: %+v, %v", out, err)
}
// More haves than one call reads (4 nodes, 300 each, 1 ms a Get): a node and a module read that one key and list // More haves than one call reads (4 nodes, 300 each, 1 ms a Get): a node and a module read that one key and list
// nothing; the whole answer reads MaxStateReads values in parallel and says it was cut. // nothing; the whole answer reads MaxStateReads values in parallel and says it was cut.
slow := newSlow(300, time.Millisecond, func(n string, i int) string { return fmt.Sprintf("%s.m%03d", n, i) }) slow := newSlow(300, time.Millisecond, func(n string, i int) string { return fmt.Sprintf("%s.m%03d", n, i) })
@@ -48,7 +48,7 @@ func TestProcessWholeSeesTheBuckets(t *testing.T) { // 1134/AC8
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel() defer cancel()
kvs := map[string]jetstream.KeyValue{} kvs := map[string]jetstream.KeyValue{}
for _, b := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket} { for _, b := range []string{delivery.HaveBucket, delivery.PlacementBucket, delivery.PresenceBucket, delivery.StuckBucket} {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 5}) kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 5})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -68,11 +68,20 @@ func TestProcessWholeSeesTheBuckets(t *testing.T) { // 1134/AC8
} }
return b return b
} }
put(view.AssignmentsBucket, "a.x", []byte(`{"node":"a","module":"x","build":"b1"}`)) // The seat's placement (issue 774, ADR 0549 §3): x and y on a, nothing on b.
put(view.AssignmentsBucket, "a.y", []byte(`{"node":"a","module":"y","build":"b2"}`)) placeValue := func(node string, modules ...string) []byte {
put(view.AssignmentsBucket, "b.x", []byte(`{"node":"b","module":"x","build":"b1","ended":"2026-10-18T11:30:00Z"}`)) p := delivery.Placement{Node: node}
put(view.BuildsBucket, "x.build.b1", []byte(`{"id":"b1","module":"x"}`)) for _, m := range modules {
put(view.BuildsBucket, "x.newest", []byte("b1")) p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
v, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
return v
}
put(delivery.PlacementBucket, "placement.a", placeValue("a", "x", "y"))
put(delivery.PlacementBucket, "placement.b", placeValue("b"))
put(delivery.HaveBucket, "a.x", haveValue("b1", delivery.PhaseChecking, delivery.HealthWaits)) put(delivery.HaveBucket, "a.x", haveValue("b1", delivery.PhaseChecking, delivery.HealthWaits))
put(delivery.HaveBucket, "b.x", haveValue("b1", delivery.PhaseRunning, delivery.HealthOK)) put(delivery.HaveBucket, "b.x", haveValue("b1", delivery.PhaseRunning, delivery.HealthOK))
@@ -150,7 +159,7 @@ func TestProcessWholeShowsWhatTheWatchmanSays(t *testing.T) { // 1168/AC5
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel() defer cancel()
kvs := map[string]jetstream.KeyValue{} kvs := map[string]jetstream.KeyValue{}
for _, b := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket} { for _, b := range []string{delivery.HaveBucket, delivery.PlacementBucket, delivery.PresenceBucket, delivery.StuckBucket} {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 5}) kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 5})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -236,7 +245,7 @@ func TestProcessWholeShowsTheIndexersBucketsOnceTheyExist(t *testing.T) { // 124
} }
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel() defer cancel()
for _, b := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket} { for _, b := range []string{delivery.HaveBucket, delivery.PlacementBucket, delivery.PresenceBucket, delivery.StuckBucket} {
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 5}); err != nil { if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 5}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -1,4 +1,4 @@
// Command mesh-delivery-view is the operator's view of delivery (novox/hq ADR 0429 §3): it watches the have, assignments // Command mesh-delivery-view is the operator's view of delivery (novox/hq ADR 0429 §3): it watches the have, the seat's placement
// and builds buckets on its own bus credential, derives one row per module and node in memory, and serves them on a // and builds buckets on its own bus credential, derives one row per module and node in memory, and serves them on a
// local socket (for mesh-delivery-cli) and on a local web page. It starts, stops or retries nothing; the one thing it writes is the operator's answer to an ask that needs no proof of who // local socket (for mesh-delivery-cli) and on a local web page. It starts, stops or retries nothing; the one thing it writes is the operator's answer to an ask that needs no proof of who
// answered, under `<ask key>.view` in its bucket answers (ADR 0528 clause 6). // answered, under `<ask key>.view` in its bucket answers (ADR 0528 clause 6).
@@ -106,7 +106,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err return err
}) })
model.SetBacklog(view.BacklogOver(js, 5*time.Second)) model.SetBacklog(view.BacklogOver(js, 5*time.Second))
for _, bucket := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket, delivery.CandidatesBucket, delivery.ModulesBucket, delivery.AsksBucket, delivery.AnswersBucket, delivery.BuilderBuildsBucket, delivery.AssignerWantsBucket, delivery.SourceBucket, delivery.ControllerComposedBucket, delivery.AssignerAsksBucket, delivery.ViewAnswersBucket, delivery.TakenBucket} { for _, bucket := range []string{delivery.HaveBucket, delivery.PlacementBucket, delivery.PresenceBucket, delivery.StuckBucket, delivery.CandidatesBucket, delivery.ModulesBucket, delivery.AsksBucket, delivery.AnswersBucket, delivery.BuilderBuildsBucket, delivery.AssignerWantsBucket, delivery.SourceBucket, delivery.AssignerAsksBucket, delivery.ViewAnswersBucket, delivery.TakenBucket} {
go follow(ctx, js, bucket, false, model, logger) go follow(ctx, js, bucket, false, model, logger)
} }
// The modules' and the source's history give each fingerprint's first commit and each head's source revision (ADR 0556). // The modules' and the source's history give each fingerprint's first commit and each head's source revision (ADR 0556).
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-view
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 h1:ycu/sNieTdLpD3bzhZOCXdzcoqXUOxPRmqWV019REpo= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24 h1:s0XXZtR7CFPBbnaDygWnE4/Q827iEFt0YfEzujk4Q9g=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -50,30 +50,6 @@ func modEntry(t *testing.T, name string, m delivery.Module) Entry {
return Entry{Bucket: delivery.ModulesBucket, Key: name, Value: b, At: t0} return Entry{Bucket: delivery.ModulesBucket, Key: name, Value: b, At: t0}
} }
// trunkBuild is a build record as the controller writes it (mesh-sdk go/delivery/testdata/build-record-build-agent.json is
// the captured shape: id, state, repository, ref, module, commit, path, source-fingerprint, source-inputs, at).
func trunkBuild(module, id, ref, state, fp, at string) []Entry {
// an outcome has no state key; a failed one has "failed" (mesh-controller internal/chain/records.go:57-59, :109)
rec := `{"id":"` + id + `",`
switch state {
case "":
case "failed":
rec += `"failed":"the build failed",`
default:
rec += `"state":"` + state + `",`
}
rec += `"repository":"novox/mesh-delivery","ref":"` + ref + `","module":"` + module + `","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/` + module + `","at":"` + at + `"`
if fp != "" {
// the controller writes the inputs whenever it writes the fingerprint (mesh-sdk testdata/build-record-build-agent.json)
rec += `,"source-fingerprint":"` + fp + `","source-inputs":{"Module":"` + module + `","Tree":"t","Bases":null,"Contexts":null,"Toolchains":null,"Unpinned":""}`
}
rec += `}`
return []Entry{
{Bucket: BuildsBucket, Key: module + ".build." + id, Value: []byte(rec), At: t0},
{Bucket: BuildsBucket, Key: module + ".newest", Value: []byte(id), At: t0},
}
}
func fpRow(t *testing.T, s Snapshot, module string) FingerprintRow { func fpRow(t *testing.T, s Snapshot, module string) FingerprintRow {
t.Helper() t.Helper()
for _, f := range s.Fingerprints { for _, f := range s.Fingerprints {
@@ -111,62 +87,27 @@ func TestSnapshotListsTheCandidateGroups(t *testing.T) { // 1241/AC1
} }
} }
func TestFingerprintAgainstTheNewestTrunkBuild(t *testing.T) { // 1241/AC2 // 1955/L7 (ADR 0571 §1): the view reads no build record of the controller's: a module with no fingerprint of the new
// form is said so, never compared with the controller's builds.
func TestAModuleOfTheOldFormIsComparedWithNoBuild(t *testing.T) {
m := New() m := New()
fill(m, trunkBuild("a", "build-1", "main", "", fF, "2026-10-19T10:00:00Z")...)
fill(m, trunkBuild("d", "build-2", "main", "", fF, "2026-10-19T10:00:00Z")...)
fill(m, fill(m,
modEntry(t, "a", delivery.Module{FP: fF, Repository: "novox/mesh-delivery", Commit: hA}), modEntry(t, "a", delivery.Module{FP: fF, Repository: "novox/mesh-delivery", Commit: hA}),
modEntry(t, "d", delivery.Module{FP: fG, Repository: "novox/mesh-delivery", Commit: hA}), modEntry(t, "b", delivery.Module{None: "an artifact compiles a Go program from a build source that git trees do not pin: no fingerprint before release 4", Repository: "novox/mesh-delivery", Commit: hA}))
modEntry(t, "b", delivery.Module{None: "an artifact compiles a Go program from a build source that git trees do not pin: no fingerprint before release 4", Repository: "novox/mesh-delivery", Commit: hA}),
modEntry(t, "e", delivery.Module{FP: fG, Repository: "novox/mesh-delivery", Commit: hA}))
s := m.Snapshot() s := m.Snapshot()
if r := fpRow(t, s, "a"); r.State != FPSame || r.Indexer != fF || r.Build != fF { for _, name := range []string{"a", "b"} {
t.Errorf("a = %+v, want same", r) if r := fpRow(t, s, name); r.State != FPNone || !(strings.Contains(r.Reason, "new form") || strings.Contains(r.Reason, "release 4")) || r.Build != "" {
} t.Errorf("%s = %+v, want none: no build of the builder's is compared", name, r)
if r := fpRow(t, s, "d"); r.State != FPDiffers || r.Indexer != fG || r.Build != fF { }
t.Errorf("d = %+v, want differs with both", r)
}
if r := fpRow(t, s, "b"); r.State != FPNone || !strings.Contains(r.Reason, "release 4") || r.Indexer != "" {
t.Errorf("b = %+v, want none with the reason", r)
}
if r := fpRow(t, s, "e"); r.State != FPNoBuild || r.Indexer != fG || r.Build != "" {
t.Errorf("e = %+v, want no build with the indexer's alone", r)
}
}
func TestOnlyTheNewestSuccessfulTrunkBuildCounts(t *testing.T) { // 1241/AC2 (the newest key is only a hint)
m := New()
fill(m, trunkBuild("a", "build-1", "main", "", fG, "2026-10-19T09:00:00Z")...)
fill(m, trunkBuild("a", "build-2", "main", "", fF, "2026-10-19T10:00:00Z")...)
fill(m, trunkBuild("a", "build-3", "main", "failed", fG, "2026-10-19T11:00:00Z")...) // newer, not built
fill(m, trunkBuild("a", "build-4", "feat/x", "", fG, "2026-10-19T12:00:00Z")...) // newer, not trunk; also the newest key
fill(m, modEntry(t, "a", delivery.Module{FP: fF, Repository: "novox/mesh-delivery", Commit: hA}))
if r := fpRow(t, m.Snapshot(), "a"); r.State != FPSame || r.Build != fF {
t.Errorf("a = %+v, want same with the 10:00 trunk build's fingerprint", r)
}
}
func TestBuildFingerprintIsReadWithTheSDKRecord(t *testing.T) { // 1241/AC3
// The record as the controller's encoder wrote it (mesh-sdk go/delivery/testdata/build-record-build-agent.json), whole,
// with its source-inputs under their Go field names: the view reads source-fingerprint from it.
const rec = `{"id":"build-1792395447088936520","repository":"novox/mesh-controller","ref":"main","module":"build-agent","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/build-agent","manifest-digest":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef","source-fingerprint":"src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7","source-inputs":{"Module":"build-agent","Tree":"11c0b4f90cae0d6e4ef9cc48cbfdf9f6dbe52bca","Bases":["sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"],"Contexts":{"server":"narrow:d51ca744fd11af59d616c858f76c27b3e58e16e11fbdc0c720eb9d7d19d55fac"},"Toolchains":null,"Unpinned":""},"made":[{"name":"server","kind":"image","reference":"registry.example/build-agent@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"}],"at":"2026-10-19T09:38:00Z"}`
const fp = "src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7"
m := New()
fill(m, Entry{Bucket: BuildsBucket, Key: delivery.BuildKey("build-agent", "build-1792395447088936520"), Value: []byte(rec), At: t0},
Entry{Bucket: BuildsBucket, Key: delivery.NewestKey("build-agent"), Value: []byte("build-1792395447088936520"), At: t0},
modEntry(t, "build-agent", delivery.Module{FP: fp, Repository: "novox/mesh-controller", Commit: hA}))
if r := fpRow(t, m.Snapshot(), "build-agent"); r.State != FPSame || r.Build != fp {
t.Errorf("build-agent = %+v, want same with %s", r, fp)
} }
} }
func shown(t *testing.T) *Model { func shown(t *testing.T) *Model {
m := New() m := New()
fill(m, trunkBuild("a", "build-1", "main", "", fF, "2026-10-19T10:00:00Z")...)
fill(m, fill(m,
candidate(t, "feat/x", pullOf("novox/mesh-delivery", 12, hA, "a"), pullOf("novox/mesh-catalog", 300, hB, "c")), candidate(t, "feat/x", pullOf("novox/mesh-delivery", 12, hA, "a"), pullOf("novox/mesh-catalog", 300, hB, "c")),
modEntry(t, "a", delivery.Module{FP: fG, Repository: "novox/mesh-delivery", Commit: hA}), modEntry(t, "a", delivery.Module{FP2: fp2A, Repository: "novox/mesh-delivery", Commit: hA}),
builderKey(t, "a", fp2A, delivery.BuilderBuild{State: delivery.BuildFailed, Error: "the compile failed"}),
modEntry(t, "b", delivery.Module{None: "no tree", Repository: "novox/mesh-delivery", Commit: hA})) modEntry(t, "b", delivery.Module{None: "no tree", Repository: "novox/mesh-delivery", Commit: hA}))
return m return m
} }
@@ -175,14 +116,14 @@ func TestTableAndPagePrintTheGroupsAndTheComparison(t *testing.T) { // 1241/AC6
m := shown(t) m := shown(t)
var out bytes.Buffer var out bytes.Buffer
Table(&out, m.Snapshot()) Table(&out, m.Snapshot())
for _, want := range []string{"CANDIDATES", "feat/x", "novox/mesh-delivery#12", hA[:12], "novox/mesh-catalog#300", "FINGERPRINT", "differs", "indexer " + fG, "build " + fF, "no tree"} { for _, want := range []string{"CANDIDATES", "feat/x", "novox/mesh-delivery#12", hA[:12], "novox/mesh-catalog#300", "FINGERPRINT", "the compile failed", fp2A, "no tree"} {
if !strings.Contains(out.String(), want) { if !strings.Contains(out.String(), want) {
t.Errorf("the table lacks %q:\n%s", want, out.String()) t.Errorf("the table lacks %q:\n%s", want, out.String())
} }
} }
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
Web(m).ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/", nil)) Web(m).ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/", nil))
for _, want := range []string{"CANDIDATES", "feat/x", "novox/mesh-delivery#12", hA[:12], "FINGERPRINTS", "differs", fG, fF, "no tree"} { for _, want := range []string{"CANDIDATES", "feat/x", "novox/mesh-delivery#12", hA[:12], "FINGERPRINTS", "the compile failed", fp2A, "no tree"} {
if !strings.Contains(rec.Body.String(), want) { if !strings.Contains(rec.Body.String(), want) {
t.Errorf("the page lacks %q:\n%s", want, rec.Body.String()) t.Errorf("the page lacks %q:\n%s", want, rec.Body.String())
} }
@@ -198,23 +139,3 @@ func TestUnreadableIndexerValuesAreNamed(t *testing.T) { // never dropped silent
t.Errorf("snapshot = %+v", s) t.Errorf("snapshot = %+v", s)
} }
} }
func TestAPlanAskedBuildWithNoRefIsATrunkBuild(t *testing.T) { // 1281/AC5
// The build record of module mesh-delivery from mesh-catalog, build-1792435539606203626, as `mesh-controller.command
// builds` showed it on 2026-10-19 at about 21:20 local: built on ace at 20:45 local, the repository the clone URL, the
// source fingerprint below, and no ref (a delivery plan asked it after a merge; the listing shows no "at main").
// No reading command shows the whole record: its path is the module's directory in the catalogue, and its
// source-inputs are mailu's live ones (mesh-sdk go/delivery/fingerprint_test.go) with the module's name put in.
const live = "src1:f79e2eedc2689cf6b3591ebfdb81ec65899131122fb2fe3dd9e384091d8a2a6d"
const rec = `{"id":"build-1792435539606203626","repository":"http://novox.internal:20000/novox/mesh-catalog.git",` +
`"module":"mesh-delivery","commit":"9efdd87a","path":"modules/mesh-delivery","at":"2026-10-19T18:45:00Z","source-fingerprint":"` + live + `",` +
`"source-inputs":{"Module":"mesh-delivery","Tree":"cfc6fa8ff5fc1e220132e51ade653b9d3f387b3e","Bases":["sha256:25f3cfeaceca14921366af4d1240b56457ef46273bdb508c7b0e8f469f6fd228"],"Contexts":null,"Toolchains":null,"Unpinned":""}}`
m := New()
fill(m,
Entry{Bucket: BuildsBucket, Key: "mesh-delivery.build.build-1792435539606203626", Value: []byte(rec), At: t0},
Entry{Bucket: BuildsBucket, Key: "mesh-delivery.newest", Value: []byte("build-1792435539606203626"), At: t0},
modEntry(t, "mesh-delivery", delivery.Module{FP: live, Repository: "novox/mesh-catalog", Commit: hA}))
if r := fpRow(t, m.Snapshot(), "mesh-delivery"); r.State != FPSame || r.Build != live {
t.Errorf("mesh-delivery = %+v, want same as the no-ref build's fingerprint", r)
}
}
@@ -12,7 +12,9 @@ import (
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
@@ -44,6 +46,57 @@ type Reader struct {
ReachState StateRead // the view's own reach (ADR 0556) ReachState StateRead // the view's own reach (ADR 0556)
// HeardState is the FORGE messages the indexer handled (issue 1126 note 76); unset reads none. // HeardState is the FORGE messages the indexer handled (issue 1126 note 76); unset reads none.
HeardState StateRead HeardState StateRead
// SignersState is the vault's public signing halves (ADR 0550 §2); unset reads none.
SignersState StateRead
// ConditionsState, AttemptsState, PresenceState and StuckState are what status reads beside the wants and the haves
// (ADR 0564 §10): the condition keeper's open conditions, mesh-healers' attempts, the watchman's presence and stuck keys.
ConditionsState, AttemptsState, PresenceState, StuckState StateRead
// PredefinedState and ConfigCandidatesState are the indexer's control records (ADR 0565 §2, §15); unset reads none.
PredefinedState, ConfigCandidatesState StateRead
// BuildLog reads the builder's log from a time (stdio.Log(delivery.BuildLogRead).Since); unset reads none.
BuildLog LogSince
}
// LogSince is one page of a log from a time on, at most limit entries of keys (stdio.KeptLog.Since).
type LogSince func(at time.Time, keys string, limit int) (*stdio.LogPage, error)
// The read tools of the indexer's control records (ADR 0565 §15): the indexer serves no tool of its own (ADR 0437 §4).
const (
PredefinedReadTool = "indexer_predefined"
ConfigCandidatesReadTool = "indexer_config_candidates"
)
// ControlRecord is every key of one of the indexer's control records as stored, or one key, with how many it holds.
func ControlRecord(s StateRead, name, key string) (any, error) {
if s.Keys == nil || s.Get == nil {
return nil, fmt.Errorf("the indexer's %s cannot be read here", name)
}
if key = strings.TrimSpace(key); key != "" {
raw, err := s.Get(key)
if err != nil {
return nil, err
}
if raw == nil {
return nil, fmt.Errorf("the indexer's %s holds no key %s", name, key)
}
return raw, nil
}
keys, err := s.Keys()
if err != nil {
return nil, err
}
sort.Strings(keys)
values, err := s.values(keys)
if err != nil {
return nil, err
}
return map[string]any{"state": name, "holds": len(keys), "keys": values}, nil
}
// Status is the view's derivation of status at now (status.go).
func (r Reader) Status(now time.Time) Status {
return DeriveStatus(StatusRead{Conditions: r.ConditionsState, Attempts: r.AttemptsState, Presence: r.PresenceState,
Stuck: r.StuckState, Wants: r.WantsState, Haves: r.HavesState}, now)
} }
// ReachReadTool reads the view's reach: per repository, how far each module reached and the reached head (ADR 0556). // ReachReadTool reads the view's reach: per repository, how far each module reached and the reached head (ADR 0556).
@@ -649,3 +702,168 @@ func (r Reader) Differences(node, module string) (KeptAnswer, error) {
func (r Reader) Haves(node, module string) (KeptAnswer, error) { func (r Reader) Haves(node, module string) (KeptAnswer, error) {
return r.kept(r.HavesState, "the haves", node, module) return r.kept(r.HavesState, "the haves", node, module)
} }
// TrustsReadTool reads what every node trusts a mesh signature from, in one call (novox/hq ADR 0550 §2; design 47's L3
// s6b): the gate before the lease write is that every node's record names the vault's key.
const TrustsReadTool = "node_trusts"
// NodeTrusts is what one node's record says it trusts: its lines (delivery.Trusts) and whether they name the vault's key
// the vault publishes and the controller's key.
type NodeTrusts struct {
Trusts []string `json:"trusts"`
TrustsVault bool `json:"trusts_vault"`
TrustsController bool `json:"trusts_controller"`
}
// TrustsAnswer is node_trusts' answer: the vault's key as it publishes it, each node's trusts, the nodes whose record does
// not name the vault's key yet, and whether every node does.
type TrustsAnswer struct {
VaultKey string `json:"vault_key,omitempty"`
Nodes map[string]NodeTrusts `json:"nodes"`
NotYet []string `json:"not_yet"`
Every bool `json:"every"`
Unreadable []string `json:"unreadable,omitempty"`
Note string `json:"note,omitempty"`
}
// Trusts reads every node's record (`<node>.node` of the haves) and the vault's `signers` key, and answers which nodes
// trust the vault's key. Read-only.
func (r Reader) Trusts() (TrustsAnswer, error) {
if r.HavesState.Get == nil || r.HavesState.Keys == nil {
return TrustsAnswer{}, fmt.Errorf("the haves cannot be read here")
}
out := TrustsAnswer{Nodes: map[string]NodeTrusts{}, NotYet: []string{}}
if r.SignersState.Get != nil {
raw, err := r.SignersState.Get(delivery.SignersKeyKey)
if err != nil {
return TrustsAnswer{}, err
}
if len(raw) > 0 {
k, err := delivery.DecodeSignerKey(raw)
if err != nil {
return TrustsAnswer{}, err
}
out.VaultKey = k.Fingerprint
}
}
keys, err := r.HavesState.Keys()
if err != nil {
return TrustsAnswer{}, err
}
var records []string
for _, k := range keys {
if _, _, isNode, err := delivery.ParseHaveKey(k); err == nil && isNode {
records = append(records, k)
}
}
values, _, err := r.HavesState.read(records)
if err != nil {
return TrustsAnswer{}, err
}
for key, raw := range values {
node := strings.TrimSuffix(key, "."+delivery.HaveNodeWord)
h, err := delivery.DecodeHaveNode(raw)
if err != nil {
out.Unreadable = append(out.Unreadable, key)
continue
}
var t NodeTrusts
if h.Record != nil {
t.Trusts = h.Record.Trusts
}
for _, line := range t.Trusts {
t.TrustsController = t.TrustsController || strings.HasPrefix(line, delivery.SignerController+" key ")
}
t.TrustsVault = out.VaultKey != "" && delivery.TrustsKey(t.Trusts, delivery.SignerVault, out.VaultKey)
out.Nodes[node] = t
if !t.TrustsVault {
out.NotYet = append(out.NotYet, node)
}
}
sort.Strings(out.NotYet)
sort.Strings(out.Unreadable)
out.Every = out.VaultKey != "" && len(out.Nodes) > 0 && len(out.NotYet) == 0 && len(out.Unreadable) == 0
if out.VaultKey == "" {
out.Note = "the vault publishes no signing key yet (" + delivery.SignersRead + "), so no node can trust it"
}
return out, nil
}
// BuildHistoryTool is the view's tool of the builds that ended, read from the builder's log (novox/hq ADR 0571 clause 1).
const BuildHistoryTool = "builder_history"
// The bounds of builder_history: how far back it reads when no since is given, and the entries it reads of the log.
const (
DefaultHistorySince = 24 * time.Hour
MaxHistoryEntries = 1000
)
// HistoryRow is one ended build of the builder's log, with how long it took from its start (none when its start is not in
// what was read).
type HistoryRow struct {
delivery.BuildLogEntry
Seconds float64 `json:"seconds,omitempty"`
}
// HistoryAnswer is builder_history's answer: every build that ended since, oldest first, and More when the log held more
// entries than one read takes (ask again from the last build's time).
type HistoryAnswer struct {
Since time.Time `json:"since"`
Builds []HistoryRow `json:"builds"`
More bool `json:"more,omitempty"`
}
// BuildHistory is every build of module (every module when empty) that ended at or after since, oldest first, each with its
// duration from the start the log holds of the same module and fingerprint before it (delivery.BuildDuration). An entry that
// does not read is refused, naming its sequence.
func (r Reader) BuildHistory(module string, since time.Time) (HistoryAnswer, error) {
out := HistoryAnswer{Since: since.UTC(), Builds: []HistoryRow{}}
if r.BuildLog == nil {
return out, fmt.Errorf("the builder's log cannot be read here")
}
keys := ""
if module = strings.TrimSpace(module); module != "" {
k, err := delivery.BuildLogKey(module)
if err != nil {
return out, err
}
keys = k
}
page, err := r.BuildLog(since, keys, MaxHistoryEntries)
if err != nil {
return out, fmt.Errorf("the builder's log %s from %s: %w", delivery.BuildLogRead, since.Format(time.RFC3339), err)
}
if page == nil {
return out, nil
}
var entries []delivery.BuildLogEntry
for _, le := range page.Entries {
e, err := delivery.DecodeBuildLogEntry(le.Value)
if err != nil {
return out, fmt.Errorf("the builder's log %s, entry %d: %w", delivery.BuildLogRead, le.Sequence, err)
}
entries = append(entries, e)
}
starts := map[string][]delivery.BuildLogEntry{}
for _, e := range entries {
if e.Event == delivery.BuildStartedEvent {
starts[e.Module+" "+e.FP] = append(starts[e.Module+" "+e.FP], e)
}
}
for _, e := range delivery.EndedSince(entries, since) {
row := HistoryRow{BuildLogEntry: e}
// the newest start of the same build at or before its end: a fingerprint built again later starts again after it
var start delivery.BuildLogEntry
for _, s := range starts[e.Module+" "+e.FP] {
if !s.At.After(e.At) && s.At.After(start.At) {
start = s
}
}
if d, ok := delivery.BuildDuration(start, e); ok {
row.Seconds = d.Seconds()
}
out.Builds = append(out.Builds, row)
}
out.More = page.Next != 0
return out, nil
}
@@ -0,0 +1,218 @@
package view
import (
"encoding/json"
"fmt"
"sort"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
// `status` is the view's derivation (novox/hq ADR 0564 §10, Limits 7, issue 1952 V1): what is wrong, quiet and on its way,
// read from the buckets it watches, never from a summary someone writes (research 072 §2), so the controller's status, its
// summary and the probe D9 go with its lease. Each part comes from one bucket:
// - Conditions: the condition keeper's open conditions (mesh-conditions.conditions), urgent first, then oldest;
// - Quiet: the watchman's presence of each node that is offline, and its stuck keys not moved;
// - Moving: each module whose want and have differ, as mesh-sdk's delivery.MotionOf derives it (moving or stuck);
// - Heals: what mesh-healers did in the last seven days (mesh-healers.attempts), by outcome.
// A bucket that cannot be read says so in its part's Unread, never as none.
// StatusTool is the view's tool that answers status.
const StatusTool = "view_status"
// The states status reads, as the view's manifest names them.
const (
StatusConditionsState = health.KeeperModule + "." + health.ConditionsState
StatusAttemptsState = health.HealersModule + "." + health.AttemptsState
StatusPresenceState = delivery.WatchmanModule + ".presence"
StatusStuckState = delivery.WatchmanModule + ".stuck"
)
// HealsWithin is how far back status counts the healers' acts.
const HealsWithin = 7 * 24 * time.Hour
// StatusRead is the five buckets status reads, as a tool reads them.
type StatusRead struct {
Conditions, Attempts, Presence, Stuck, Wants, Haves StateRead
}
// Status is the view's answer to status.
type Status struct {
Conditions []StatusCondition `json:"conditions"`
ConditionsUnread string `json:"conditions-unread,omitempty"`
Quiet []StatusQuiet `json:"quiet"`
QuietUnread string `json:"quiet-unread,omitempty"`
Moving []StatusMoving `json:"moving"`
MovingUnread string `json:"moving-unread,omitempty"`
Heals map[string]int `json:"heals-this-week"`
HealsUnread string `json:"heals-unread,omitempty"`
// Well says nothing is open, quiet or stuck; a part unread is not well.
Well bool `json:"well"`
}
// StatusCondition is one open condition in brief.
type StatusCondition struct {
Key string `json:"key"`
Severity string `json:"severity"`
Machine string `json:"machine,omitempty"`
Summary string `json:"summary"`
Raised time.Time `json:"raised"`
Silenced bool `json:"silenced,omitempty"`
}
// StatusQuiet is a node offline, or a key the watchman says stopped moving.
type StatusQuiet struct {
Key string `json:"key"`
What string `json:"what"`
Since time.Time `json:"since,omitzero"`
}
// StatusMoving is a module whose want and have differ on a node.
type StatusMoving struct {
Node string `json:"node"`
Module string `json:"module"`
State string `json:"state"`
Outcome string `json:"outcome,omitempty"`
Words string `json:"words,omitempty"`
FromFP string `json:"from_fp,omitempty"`
ToFP string `json:"to_fp"`
}
func values(s StateRead) (map[string]json.RawMessage, error) {
if s.Keys == nil || s.Get == nil {
return nil, fmt.Errorf("not read here")
}
keys, err := s.Keys()
if err != nil {
return nil, err
}
out := map[string]json.RawMessage{}
for _, k := range keys {
v, err := s.Get(k)
if err != nil {
return nil, fmt.Errorf("%s: %w", k, err)
}
if v != nil {
out[k] = v
}
}
return out, nil
}
// DeriveStatus is status from the buckets r reads, at now.
func DeriveStatus(r StatusRead, now time.Time) Status {
st := Status{Conditions: []StatusCondition{}, Quiet: []StatusQuiet{}, Moving: []StatusMoving{}, Heals: map[string]int{}}
if raw, err := values(r.Conditions); err != nil {
st.ConditionsUnread = "the condition keeper's conditions cannot be read: " + err.Error()
} else {
var open []health.Condition
for _, v := range raw {
var c health.Condition
if json.Unmarshal(v, &c) == nil && c.Key != "" {
open = append(open, c)
}
}
health.Order(open)
for _, c := range open {
st.Conditions = append(st.Conditions, StatusCondition{Key: c.Key, Severity: string(c.Severity), Machine: c.Subject.Machine,
Summary: c.Summary, Raised: c.Raised, Silenced: c.SilencedAt(now)})
}
}
presence, perr := values(r.Presence)
stuck, serr := values(r.Stuck)
switch {
case perr != nil:
st.QuietUnread = "the watchman's presence cannot be read: " + perr.Error()
case serr != nil:
st.QuietUnread = "the watchman's stuck keys cannot be read: " + serr.Error()
default:
for k, v := range presence {
if p, err := delivery.DecodePresence(v); err == nil && !p.Online {
st.Quiet = append(st.Quiet, StatusQuiet{Key: k, What: "offline"})
}
}
for k, v := range stuck {
if s, err := delivery.DecodeStuck(v); err == nil && !s.Moved {
st.Quiet = append(st.Quiet, StatusQuiet{Key: k, What: "stopped moving: it waited for " + s.Waited, Since: s.Since})
}
}
sort.Slice(st.Quiet, func(i, j int) bool { return st.Quiet[i].Key < st.Quiet[j].Key })
}
wants, werr := values(r.Wants)
haves, herr := values(r.Haves)
switch {
case werr != nil:
st.MovingUnread = "the assigner's wants cannot be read: " + werr.Error()
case herr != nil:
st.MovingUnread = "the nodes' haves cannot be read: " + herr.Error()
default:
for wk, v := range wants {
node, module, err := delivery.ParseWantKey(wk)
if err != nil {
continue
}
w, err := delivery.DecodeWant(v)
if err != nil {
continue
}
hk, _ := delivery.HaveKey(node, module)
var h delivery.Have
hv, has := haves[hk]
if has {
if h, err = delivery.DecodeHave(hv); err != nil {
has = false
}
}
words, stopped := "", false
if sk, err := delivery.StuckKey(delivery.AssignerWantsBucket, wk); err == nil && serr == nil {
if s, err := delivery.DecodeStuck(stuck[sk]); err == nil && !s.Moved {
words, stopped = s.Waited, true
}
}
m := delivery.MotionOf(w, true, h, has, words, stopped)
if m.State == "" || m.State == delivery.MotionSettled {
continue
}
row := StatusMoving{Node: node, Module: module, State: m.State, Outcome: m.Outcome, Words: m.Words, ToFP: w.FP}
if has {
row.FromFP = h.FP
}
st.Moving = append(st.Moving, row)
}
sort.Slice(st.Moving, func(i, j int) bool {
if st.Moving[i].Node != st.Moving[j].Node {
return st.Moving[i].Node < st.Moving[j].Node
}
return st.Moving[i].Module < st.Moving[j].Module
})
}
if raw, err := values(r.Attempts); err != nil {
st.HealsUnread = "mesh-healers' attempts cannot be read: " + err.Error()
} else {
for _, v := range raw {
var a health.Attempt
if json.Unmarshal(v, &a) != nil || now.Sub(a.Begun) > HealsWithin {
continue
}
outcome := a.Outcome
if outcome == "" {
outcome = health.OutcomeActing
}
st.Heals[outcome]++
}
}
stuckMoving := false
for _, m := range st.Moving {
stuckMoving = stuckMoving || m.State == delivery.MotionStuck
}
st.Well = len(st.Conditions) == 0 && len(st.Quiet) == 0 && !stuckMoving &&
st.ConditionsUnread == "" && st.QuietUnread == "" && st.MovingUnread == ""
return st
}
+53 -145
View File
@@ -4,6 +4,7 @@ package view
import ( import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"maps"
"slices" "slices"
"sort" "sort"
"strings" "strings"
@@ -102,8 +103,8 @@ type WantRow struct {
Newest string `json:"newest,omitempty"` Newest string `json:"newest,omitempty"`
Rule int `json:"rule,omitempty"` Rule int `json:"rule,omitempty"`
Words string `json:"words,omitempty"` Words string `json:"words,omitempty"`
// Judgement is the controller's judgement of the move to the composition the want names (recreates, moves-data, neither or // Judgement is the node agent's judgement of the move that waits for a person's yes (its have in waiting-yes, ADR 0547
// cannot-judge), and Judged its words; empty when the controller wrote none (ADR 0539 §14). // §3), and Judged its words; empty otherwise. The controller's composed copy is no longer read (ADR 0571 §6).
Judgement string `json:"judgement,omitempty"` Judgement string `json:"judgement,omitempty"`
Judged string `json:"judged,omitempty"` Judged string `json:"judged,omitempty"`
} }
@@ -215,7 +216,7 @@ type UnreadableRow struct {
// Model holds the rows' inputs in memory and nothing else: a restart fills it again from the buckets' last values. // Model holds the rows' inputs in memory and nothing else: a restart fills it again from the buckets' last values.
type Model struct { type Model struct {
mu sync.Mutex mu sync.Mutex
assigns map[string]assignment placed *delivery.Placed // the seat's placement (the assigner's state, ADR 0567, ADR 0549 §3)
builds map[string]buildEntry // by <module>.build.<id> builds map[string]buildEntry // by <module>.build.<id>
groups map[string]groupEntry // the indexer's candidates, by group groups map[string]groupEntry // the indexer's candidates, by group
modules map[string]moduleEntry // the indexer's modules, by module modules map[string]moduleEntry // the indexer's modules, by module
@@ -263,22 +264,20 @@ type stuckEntry struct {
err string err string
} }
// assignment holds only the fields of the controller's Assignment the view reads. // pair is a module the seat's placement puts on a node.
type assignment struct { type pair struct{ Node, Module string }
Node string `json:"node"`
Module string `json:"module"`
Build string `json:"build"`
Commit string `json:"commit"`
Since time.Time `json:"since"`
Ended time.Time `json:"ended"`
State string `json:"state"`
PulledIn bool `json:"pulled-in"`
RequiredBy string `json:"required-by"`
}
// open says whether the value is an assignment that stands for the module on its node, as the controller's // placedPairs is every module the seat's placement puts on a node, by node then module. The caller holds mu.
// Assignment.Stands does. A module pulled in by another stands too. func (m *Model) placedPairs() []pair {
func (a assignment) open() bool { return a.Ended.IsZero() && a.State == "" } byNode := m.placed.ByNode()
var out []pair
for _, node := range slices.Sorted(maps.Keys(byNode)) {
for _, module := range byNode[node] {
out = append(out, pair{Node: node, Module: module})
}
}
return out
}
// build holds only the fields of the controller's Build record the view reads. A record with no state and no failure // build holds only the fields of the controller's Build record the view reads. A record with no state and no failure
// is a done build. // is a done build.
@@ -356,7 +355,7 @@ type nodeEntry struct {
// New is an empty model. // New is an empty model.
func New() *Model { func New() *Model {
return &Model{assigns: map[string]assignment{}, builds: map[string]buildEntry{}, groups: map[string]groupEntry{}, modules: map[string]moduleEntry{}, keys: map[string]keyEntry{}, asks: map[string]askRecord{}, wants: map[string]wantEntry{}, told: map[string]bool{}, initial: map[string]bool{}, composed: map[string]composedEntry{}, aasks: map[string]assignerAskRecord{}, chosen: map[string]delivery.ViewAnswer{}, answers: map[string]string{}, answerOf: map[string]delivery.Answer{}, holds: map[string]heldEntry{}, newest: map[string]string{}, return &Model{placed: delivery.NewPlaced(), builds: map[string]buildEntry{}, groups: map[string]groupEntry{}, modules: map[string]moduleEntry{}, keys: map[string]keyEntry{}, asks: map[string]askRecord{}, wants: map[string]wantEntry{}, told: map[string]bool{}, initial: map[string]bool{}, composed: map[string]composedEntry{}, aasks: map[string]assignerAskRecord{}, chosen: map[string]delivery.ViewAnswer{}, answers: map[string]string{}, answerOf: map[string]delivery.Answer{}, holds: map[string]heldEntry{}, newest: map[string]string{},
haves: map[string]haveEntry{}, nodes: map[string]nodeEntry{}, haves: map[string]haveEntry{}, nodes: map[string]nodeEntry{},
presence: map[string]presenceEntry{}, stuck: map[string]stuckEntry{}, presence: map[string]presenceEntry{}, stuck: map[string]stuckEntry{},
modHist: map[string][]delivery.Module{}, srcHist: map[uint64]map[string]string{}, taken: map[string]delivery.Taken{}, modHist: map[string][]delivery.Module{}, srcHist: map[uint64]map[string]string{}, taken: map[string]delivery.Taken{},
@@ -529,16 +528,9 @@ func (m *Model) apply(e Entry) {
} }
st, err := delivery.DecodeStuck(e.Value) st, err := delivery.DecodeStuck(e.Value)
m.stuck[e.Key] = stuckEntry{value: st, err: errText(err)} m.stuck[e.Key] = stuckEntry{value: st, err: errText(err)}
case AssignmentsBucket: case delivery.PlacementBucket:
if e.Deleted { // A key that does not read places nothing on its node; the seat's verb placement says why.
delete(m.assigns, e.Key) _, _, _ = m.placed.Put(e.Key, e.Value, e.Deleted, e.At)
return
}
var a assignment
if json.Unmarshal(e.Value, &a) != nil || a.Node == "" || a.Module == "" {
return
}
m.assigns[a.Node+"."+a.Module] = a
case delivery.CandidatesBucket: case delivery.CandidatesBucket:
if e.Deleted { if e.Deleted {
delete(m.groups, e.Key) delete(m.groups, e.Key)
@@ -597,13 +589,6 @@ func (m *Model) apply(e Entry) {
} }
w, err := delivery.DecodeWant(e.Value) w, err := delivery.DecodeWant(e.Value)
m.wants[key] = wantEntry{value: w, err: errText(err), at: e.At} m.wants[key] = wantEntry{value: w, err: errText(err), at: e.At}
case delivery.ControllerComposedBucket:
if e.Deleted {
delete(m.composed, e.Key)
return
}
d, err := delivery.DecodeComposedDigest(e.Value)
m.composed[e.Key] = composedEntry{value: d, err: errText(err)}
case delivery.TakenBucket: case delivery.TakenBucket:
if e.Deleted { if e.Deleted {
delete(m.taken, e.Key) delete(m.taken, e.Key)
@@ -651,34 +636,6 @@ func (m *Model) apply(e Entry) {
return return
} }
m.keys[e.Key] = keyEntry{value: b} m.keys[e.Key] = keyEntry{value: b}
case BuildsBucket:
module, rest, ok := strings.Cut(e.Key, ".")
if !ok {
return
}
switch {
case rest == "newest":
if e.Deleted {
delete(m.newest, module)
return
}
m.newest[module] = strings.TrimSpace(string(e.Value))
case strings.HasPrefix(rest, "build."):
if e.Deleted {
delete(m.builds, e.Key)
return
}
var b build
if json.Unmarshal(e.Value, &b) != nil {
return
}
// The source fingerprint is read with the SDK's record, not with the view's own struct.
var rec delivery.BuildRecord
if json.Unmarshal(e.Value, &rec) != nil {
return
}
m.builds[e.Key] = buildEntry{build: b, rec: rec}
}
} }
} }
@@ -714,18 +671,12 @@ func (m *Model) snapshot() Snapshot {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
rows := []Row{} rows := []Row{}
for key, a := range m.assigns { for _, a := range m.placedPairs() {
if !a.open() { r := Row{Node: a.Node, Module: a.Module, Status: StatusAssigned}
continue
}
r := Row{Node: a.Node, Module: a.Module, Status: StatusAssigned, Assigned: a.Build}
if a.PulledIn {
r.RequiredBy = a.RequiredBy
}
var waits []string var waits []string
hk, herr := delivery.HaveKey(a.Node, a.Module) hk, herr := delivery.HaveKey(a.Node, a.Module)
if herr != nil { if herr != nil {
hk = key continue
} }
h, has := m.haves[hk] h, has := m.haves[hk]
// The controller no longer writes the build of an assignment (R6): the row is the want and the have of the node. // The controller no longer writes the build of an assignment (R6): the row is the want and the have of the node.
@@ -745,17 +696,8 @@ func (m *Model) snapshot() Snapshot {
r.Have = h.have.FP r.Have = h.have.FP
} }
r.Since, r.Detail = h.at, h.have.Detail r.Since, r.Detail = h.at, h.have.Detail
if wanted { if wanted && h.have.FP != w.value.FP {
if h.have.FP != w.value.FP { waits = append(waits, "the node has not applied "+w.value.FP)
waits = append(waits, "the node has not applied "+w.value.FP)
}
} else if a.Build != "" && h.have.Build != a.Build {
waits = append(waits, "the node has not applied "+a.Build)
}
}
if id := m.newest[a.Module]; id != "" && a.Build != "" && id != a.Build {
if b, ok := m.builds[a.Module+".build."+id]; ok && b.done() {
waits = append(waits, a.Module+" has a newer build "+id+" not yet assigned")
} }
} }
r.Waits = strings.Join(waits, "; ") r.Waits = strings.Join(waits, "; ")
@@ -766,7 +708,7 @@ func (m *Model) snapshot() Snapshot {
if perr != nil { if perr != nil {
continue continue
} }
if a, ok := m.assigns[node+"."+module]; ok && a.open() { if _, ok := m.placed.Since(node, module); ok {
continue continue
} }
r := Row{Node: node, Module: module, Status: StatusLeftBehind, Since: h.at, Error: h.err} r := Row{Node: node, Module: module, Status: StatusLeftBehind, Since: h.at, Error: h.err}
@@ -892,38 +834,22 @@ func (m *Model) groupRows() []GroupRow {
// successful trunk build (delivery.BuildRecord.IsTrunkBuild: an outcome, not failed, on main or with no ref) that recorded one. The newest key is any state and ref, so it is not // successful trunk build (delivery.BuildRecord.IsTrunkBuild: an outcome, not failed, on main or with no ref) that recorded one. The newest key is any state and ref, so it is not
// used: the records are. // used: the records are.
func (m *Model) fingerprintRows() []FingerprintRow { func (m *Model) fingerprintRows() []FingerprintRow {
newest := map[string]buildEntry{}
for _, b := range m.builds {
if b.rec.Module == "" || !b.rec.IsTrunkBuild() || b.rec.SourceFingerprint == "" {
continue
}
if old, ok := newest[b.rec.Module]; !ok || b.rec.At.After(old.rec.At) || (b.rec.At.Equal(old.rec.At) && b.rec.ID > old.rec.ID) {
newest[b.rec.Module] = b
}
}
out := []FingerprintRow{} out := []FingerprintRow{}
for name, mod := range m.modules { for name, mod := range m.modules {
if mod.err != "" { if mod.err != "" {
continue continue
} }
row := FingerprintRow{Module: name, Repository: mod.value.Repository, Commit: mod.value.Commit} row := FingerprintRow{Module: name, Repository: mod.value.Repository, Commit: mod.value.Commit}
if mod.value.FP2 != "" || mod.value.None2 != "" { if mod.value.FP2 == "" && mod.value.None2 == "" {
// The new form (R4e): the indexer's fp2 against the builder's key of it. // The builder builds the new form alone, and its done keys are the build history the view reads (ADR 0571 §1).
out = append(out, m.newFormRow(name, row, mod.value)) row.State, row.Reason = FPNone, mod.value.None
if row.Reason == "" {
row.Reason = "the indexer holds no fingerprint of the new form for it, so no build of the builder's is compared"
}
out = append(out, row)
continue continue
} }
b, hasBuild := newest[name] out = append(out, m.newFormRow(name, row, mod.value))
switch {
case mod.value.None != "":
row.State, row.Reason = FPNone, mod.value.None
case !hasBuild:
row.State, row.Indexer = FPNoBuild, mod.value.FP
case b.rec.SourceFingerprint == mod.value.FP:
row.State, row.Indexer, row.Build = FPSame, mod.value.FP, b.rec.SourceFingerprint
default:
row.State, row.Indexer, row.Build = FPDiffers, mod.value.FP, b.rec.SourceFingerprint
}
out = append(out, row)
} }
sort.Slice(out, func(i, j int) bool { return out[i].Module < out[j].Module }) sort.Slice(out, func(i, j int) bool { return out[i].Module < out[j].Module })
return out return out
@@ -982,12 +908,6 @@ func (s Snapshot) Counted() int {
return n return n
} }
// The buckets the view reads besides the have bucket (which mesh-sdk's delivery package names).
const (
AssignmentsBucket = "mesh-controller_assignments"
BuildsBucket = "mesh-controller_builds"
)
// The statuses of a row. // The statuses of a row.
const ( const (
StatusAssigned = "assigned" StatusAssigned = "assigned"
@@ -1113,10 +1033,8 @@ func (m *Model) wantRows() []WantRow {
if has && h.err == "" { if has && h.err == "" {
row.Have, row.Phase, row.Health = h.have.FP, string(h.have.Phase), string(h.have.Health) row.Have, row.Phase, row.Health = h.have.FP, string(h.have.Phase), string(h.have.Health)
} }
if ckey, err := delivery.ComposedKey(node, module, v.FP); err == nil && !v.Removed { if has && h.err == "" && h.have.Phase == delivery.PhaseWaitingYes && !v.Removed {
if c, ok := m.composed[ckey]; ok && c.err == "" { row.Judgement, row.Judged = string(h.have.Judgement), h.have.Detail
row.Judgement, row.Judged = string(c.value.Judgement), c.value.Words
}
} }
switch { switch {
case v.Removed: case v.Removed:
@@ -1176,13 +1094,10 @@ func (m *Model) sourceRow() *SourceRow {
waiting = append(waiting, fmt.Sprintf("%s on %s runs %s, not yet running and healthy", module, node, short(w.value.FP))) waiting = append(waiting, fmt.Sprintf("%s on %s runs %s, not yet running and healthy", module, node, short(w.value.FP)))
} }
} }
for key, a := range m.assigns { for _, a := range m.placedPairs() {
if !a.open() {
continue
}
wk, werr := delivery.HaveKey(a.Node, a.Module) wk, werr := delivery.HaveKey(a.Node, a.Module)
if werr != nil { if werr != nil {
wk = key continue
} }
if _, ok := m.wants[wk]; ok { if _, ok := m.wants[wk]; ok {
continue continue
@@ -1212,9 +1127,9 @@ func splitKey(key string) (node, module string) {
// module's want on a node differs from its have (moving), that the have reached the want running and healthy (settled), or that it // module's want on a node differs from its have (moving), that the have reached the want running and healthy (settled), or that it
// is held or failed or stopped moving past the watchman's bound (stuck). Each is said once for a node, a module and a composition. // is held or failed or stopped moving past the watchman's bound (stuck). Each is said once for a node, a module and a composition.
const ( const (
MotionMoving = "moving" MotionMoving = delivery.MotionMoving
MotionSettled = "settled" MotionSettled = delivery.MotionSettled
MotionStuck = "stuck" MotionStuck = delivery.MotionStuck
// MotionBoundSeconds is how long a node may take to apply a want before the watchman says it stuck (watch.StepBound). // MotionBoundSeconds is how long a node may take to apply a want before the watchman says it stuck (watch.StepBound).
MotionBoundSeconds = 1800 MotionBoundSeconds = 1800
) )
@@ -1343,7 +1258,6 @@ func (m *Model) motionOf(hk string) []MotionEvent {
id := func(kind string) string { id := func(kind string) string {
return kind + "." + node + "." + module + "." + strings.TrimPrefix(w.value.Composed, "sha256:") return kind + "." + node + "." + module + "." + strings.TrimPrefix(w.value.Composed, "sha256:")
} }
applied := has && h.have.FP == w.value.FP && (h.have.Composed == "" || h.have.Composed == w.value.Composed)
var out []MotionEvent var out []MotionEvent
say := func(kind string, b MotionBody) { say := func(kind string, b MotionBody) {
if m.told[id(kind)] { if m.told[id(kind)] {
@@ -1352,31 +1266,25 @@ func (m *Model) motionOf(hk string) []MotionEvent {
m.told[id(kind)] = true m.told[id(kind)] = true
out = append(out, MotionEvent{Type: kind, ID: id(kind), Body: b}) out = append(out, MotionEvent{Type: kind, ID: id(kind), Body: b})
} }
switch { // The derivation is mesh-sdk's (delivery.MotionOf, issue 1952 S7), shared with the condition keeper's hold.
case applied && h.have.Phase == delivery.PhaseRunning && h.have.Health == delivery.HealthOK: words, stuck := m.stuckWant(w.value, node, module)
mo := delivery.MotionOf(w.value, true, h.have, has, words, stuck)
if mo.State == delivery.MotionSettled {
if m.told[id(MotionMoving)] { if m.told[id(MotionMoving)] {
b := body b := body
b.Outcome = "running" b.Outcome = mo.Outcome
say(MotionSettled, b) say(MotionSettled, b)
} }
return out return out
case !m.told[id(MotionMoving)]: }
if !m.told[id(MotionMoving)] {
say(MotionMoving, body) say(MotionMoving, body)
} }
stop := body if mo.State == delivery.MotionStuck {
switch { stop := body
case has && h.have.Phase == delivery.PhaseHeld: stop.Outcome, stop.Words = mo.Outcome, mo.Words
stop.Outcome, stop.Words = "stopped", h.have.Detail say(MotionStuck, stop)
case has && (h.have.Health == delivery.HealthFailed || h.have.Phase == delivery.PhaseFailed) && h.have.FP == w.value.FP:
stop.Outcome, stop.Words = "failed", h.have.Detail
default:
if s, ok := m.stuckWant(w.value, node, module); ok {
stop.Outcome, stop.Words = "stopped", s
} else {
return out
}
} }
say(MotionStuck, stop)
return out return out
} }
@@ -1,6 +1,7 @@
package view package view
import ( import (
"encoding/json"
"errors" "errors"
"reflect" "reflect"
"strings" "strings"
@@ -8,6 +9,7 @@ import (
"time" "time"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
) )
var t0 = time.Date(2026, 10, 18, 12, 0, 0, 0, time.UTC) var t0 = time.Date(2026, 10, 18, 12, 0, 0, 0, time.UTC)
@@ -25,17 +27,22 @@ func haveE(t *testing.T, node, module string, v []byte, at time.Time) Entry {
return Entry{Bucket: delivery.HaveBucket, Key: node + "." + module, Value: v, At: at} return Entry{Bucket: delivery.HaveBucket, Key: node + "." + module, Value: v, At: at}
} }
func assign(node, module, build, extra string) Entry { // place is the seat's placement of node, as the assigner keeps it (ADR 0567): the modules placed there.
v := `{"node":"` + node + `","module":"` + module + `","build":"` + build + `","commit":"c","since":"2026-10-18T11:00:00Z"` + extra + `}` func place(t *testing.T, node string, modules ...string) Entry {
return Entry{Bucket: AssignmentsBucket, Key: node + "." + module, Value: []byte(v), At: t0} t.Helper()
} p := delivery.Placement{Node: node}
for _, m := range modules {
func buildRec(module, id, extra string) []Entry { p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
rec := `{"id":"` + id + `","module":"` + module + `","commit":"c"` + extra + `}`
return []Entry{
{Bucket: BuildsBucket, Key: module + ".build." + id, Value: []byte(rec), At: t0},
{Bucket: BuildsBucket, Key: module + ".newest", Value: []byte(id), At: t0},
} }
key, err := delivery.PlacementKey(node)
if err != nil {
t.Fatal(err)
}
v, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
return Entry{Bucket: delivery.PlacementBucket, Key: key, Value: v, At: t0}
} }
func find(t *testing.T, s Snapshot, node, module string) Row { func find(t *testing.T, s Snapshot, node, module string) Row {
@@ -58,14 +65,11 @@ func fill(m *Model, es ...Entry) {
// base: two nodes (a, b), three modules (x, y, z). // base: two nodes (a, b), three modules (x, y, z).
func base(t *testing.T) []Entry { func base(t *testing.T) []Entry {
es := []Entry{ es := []Entry{
assign("a", "x", "b1", ""), assign("a", "y", "b2", ""), assign("b", "x", "b1", ""), assign("b", "z", "b3", ""), place(t, "a", "x", "y"), place(t, "b", "x", "z"),
haveE(t, "a", "x", have(t, "b1", delivery.PhaseRunning, delivery.HealthOK, ""), t0.Add(1*time.Minute)), haveE(t, "a", "x", have(t, "b1", delivery.PhaseRunning, delivery.HealthOK, ""), t0.Add(1*time.Minute)),
haveE(t, "b", "x", have(t, "b1", delivery.PhaseChecking, delivery.HealthWaits, "starting"), t0.Add(2*time.Minute)), haveE(t, "b", "x", have(t, "b1", delivery.PhaseChecking, delivery.HealthWaits, "starting"), t0.Add(2*time.Minute)),
haveE(t, "b", "z", have(t, "b3", delivery.PhaseRunning, delivery.HealthOK, ""), t0.Add(3*time.Minute)), haveE(t, "b", "z", have(t, "b3", delivery.PhaseRunning, delivery.HealthOK, ""), t0.Add(3*time.Minute)),
} }
es = append(es, buildRec("x", "b1", "")...)
es = append(es, buildRec("y", "b2", "")...)
es = append(es, buildRec("z", "b3", "")...)
return es return es
} }
@@ -77,68 +81,35 @@ func TestRowsPerAssignedModuleAndNode(t *testing.T) { // 1134/AC1
t.Fatalf("want 4 rows, got %d counted %d: %+v", len(s.Rows), s.Counted(), s.Rows) t.Fatalf("want 4 rows, got %d counted %d: %+v", len(s.Rows), s.Counted(), s.Rows)
} }
r := find(t, s, "b", "x") r := find(t, s, "b", "x")
want := Row{Node: "b", Module: "x", Status: StatusAssigned, Assigned: "b1", Have: "b1", Phase: "checking", Health: "waits", want := Row{Node: "b", Module: "x", Status: StatusAssigned, Have: "b1", Phase: "checking", Health: "waits",
Since: t0.Add(2 * time.Minute), Detail: "starting"} Since: t0.Add(2 * time.Minute), Detail: "starting"}
if !reflect.DeepEqual(r, want) { if !reflect.DeepEqual(r, want) {
t.Fatalf("row = %+v, want %+v", r, want) t.Fatalf("row = %+v, want %+v", r, want)
} }
y := find(t, s, "a", "y") y := find(t, s, "a", "y")
if y.Waits != "no have yet" || y.Assigned != "b2" || y.Have != "" { if y.Waits != "no have yet" || y.Have != "" {
t.Fatalf("a.y = %+v, want 'no have yet'", y) t.Fatalf("a.y = %+v, want 'no have yet'", y)
} }
} }
func TestWaitsForTheNodeAndForANewerBuild(t *testing.T) { // 1134/AC2 func TestLeftBehindIsWhatThePlacementDoesNotPut(t *testing.T) { // 1134/AC3, issue 774 (the seat's placement)
m := New() m := New()
fill(m, base(t)...) fill(m, base(t)...)
fill(m, haveE(t, "a", "x", have(t, "b0", delivery.PhaseRunning, delivery.HealthOK, ""), t0)) fill(m, place(t, "b", "x")) // z taken off b
if w := find(t, m.Snapshot(), "a", "x").Waits; w != "the node has not applied b1" {
t.Fatalf("waits = %q", w)
}
fill(m, buildRec("z", "b4", "")...)
if w := find(t, m.Snapshot(), "b", "z").Waits; w != "z has a newer build b4 not yet assigned" {
t.Fatalf("waits = %q", w)
}
// a newest build that is not done is not a newer build to assign
fill(m, buildRec("z", "b5", `,"failed":"boom"`)...)
if w := find(t, m.Snapshot(), "b", "z").Waits; w != "" {
t.Fatalf("a failed newest build must not be waited on, waits = %q", w)
}
fill(m, buildRec("z", "b6", `,"state":"asked"`)...)
if w := find(t, m.Snapshot(), "b", "z").Waits; w != "" {
t.Fatalf("an asked newest build must not be waited on, waits = %q", w)
}
}
func TestLeftBehindIsNotRunningAndPulledInIs(t *testing.T) { // 1134/AC3 (both halves)
for name, extra := range map[string]string{"ended": `,"ended":"2026-10-18T11:30:00Z"`, "state": `,"state":"pending"`} {
m := New()
fill(m, base(t)...)
fill(m, assign("b", "z", "b3", extra))
s := m.Snapshot()
r := find(t, s, "b", "z")
if r.Status != StatusLeftBehind {
t.Errorf("%s: status = %q, want %q", name, r.Status, StatusLeftBehind)
}
if s.Counted() != 3 {
t.Errorf("%s: counted = %d, want 3", name, s.Counted())
}
}
m := New()
fill(m, base(t)...)
fill(m, haveE(t, "a", "w", have(t, "b9", delivery.PhaseRunning, delivery.HealthOK, ""), t0)) // no assignment at all
s := m.Snapshot() s := m.Snapshot()
if find(t, s, "a", "w").Status != StatusLeftBehind || s.Counted() != 4 { if r := find(t, s, "b", "z"); r.Status != StatusLeftBehind || s.Counted() != 3 {
t.Fatalf("a have with no assignment: %+v counted %d", find(t, s, "a", "w"), s.Counted()) t.Errorf("status = %q counted %d, want %q and 3", r.Status, s.Counted(), StatusLeftBehind)
} }
// A module pulled in by another stands: shown and counted, with what required it.
m = New() m = New()
fill(m, base(t)...) fill(m, base(t)...)
fill(m, assign("b", "z", "b3", `,"pulled-in":true,"required-by":"x"`)) fill(m, haveE(t, "a", "w", have(t, "b9", delivery.PhaseRunning, delivery.HealthOK, ""), t0)) // placed nowhere
s = m.Snapshot() s = m.Snapshot()
r := find(t, s, "b", "z") if find(t, s, "a", "w").Status != StatusLeftBehind || s.Counted() != 4 {
if r.Status != StatusAssigned || r.RequiredBy != "x" || r.Have != "b3" || s.Counted() != 4 { t.Fatalf("a have with no placement: %+v counted %d", find(t, s, "a", "w"), s.Counted())
t.Fatalf("pulled-in row = %+v counted %d, want shown and counted with required-by x", r, s.Counted()) }
fill(m, Entry{Bucket: delivery.PlacementBucket, Key: "placement.b", Deleted: true})
if s := m.Snapshot(); s.Counted() != 2 || find(t, s, "b", "x").Status != StatusLeftBehind {
t.Fatalf("b's placement deleted: counted %d, %+v", s.Counted(), s.Rows)
} }
} }
@@ -181,18 +152,13 @@ func TestOnlyTheAffectedRowChanges(t *testing.T) { // 1134/AC5
if r := find(t, after, "b", "x"); r.Phase != "running" || r.Health != "ok" || !r.Since.Equal(t0.Add(9*time.Minute)) { if r := find(t, after, "b", "x"); r.Phase != "running" || r.Health != "ok" || !r.Since.Equal(t0.Add(9*time.Minute)) {
t.Fatalf("changed row = %+v", r) t.Fatalf("changed row = %+v", r)
} }
// an assignment change and a build change touch only their rows // a want's change touches only its row
fill(m, assign("a", "y", "b7", "")) fill(m, wantE(t, "a", "y", delivery.Want{FP: fpNew}))
if r := find(t, m.Snapshot(), "a", "y"); r.Assigned != "b7" { if r := find(t, m.Snapshot(), "a", "y"); r.Assigned != fpNew {
t.Fatalf("assignment change not seen: %+v", r) t.Fatalf("want change not seen: %+v", r)
} }
if !reflect.DeepEqual(find(t, m.Snapshot(), "b", "z"), find(t, after, "b", "z")) { if !reflect.DeepEqual(find(t, m.Snapshot(), "b", "z"), find(t, after, "b", "z")) {
t.Fatal("an assignment of another row changed b.z") t.Fatal("a want of another row changed b.z")
}
fill(m, buildRec("x", "b8", "")...)
s := m.Snapshot()
if find(t, s, "a", "x").Waits == "" || find(t, s, "b", "x").Waits == "" || find(t, s, "b", "z").Waits != "" {
t.Fatalf("a build of x must touch x's rows only: %+v", s.Rows)
} }
// a deleted key (a bucket's expiry) removes its value // a deleted key (a bucket's expiry) removes its value
m.Apply(Entry{Bucket: delivery.HaveBucket, Key: "a.x", Deleted: true}) m.Apply(Entry{Bucket: delivery.HaveBucket, Key: "a.x", Deleted: true})
@@ -237,7 +203,13 @@ func stuckE(t *testing.T, bucket, key, waited string, since time.Time, moved boo
return Entry{Bucket: delivery.StuckBucket, Key: bucket + "." + key, Value: b, At: since} return Entry{Bucket: delivery.StuckBucket, Key: bucket + "." + key, Value: b, At: since}
} }
func TestOfflineAndStuckAreListed(t *testing.T) { // 1168/AC2 func TestTheWatchmansKeysAndStatus(t *testing.T) {
t.Run("offline and stuck are listed", offlineAndStuckAreListed)
t.Run("an unknown watchman version is unreadable, not dropped", unknownWatchmanVersionIsUnreadableNotDropped)
t.Run("1952/V1: status derives each part from its bucket, and says a part it cannot read", statusDerivesEachPartFromItsBucket)
}
func offlineAndStuckAreListed(t *testing.T) { // 1168/AC2
m := New() m := New()
fill(m, base(t)...) fill(m, base(t)...)
fill(m, fill(m,
@@ -265,7 +237,7 @@ func TestOfflineAndStuckAreListed(t *testing.T) { // 1168/AC2
} }
} }
func TestUnknownWatchmanVersionIsUnreadableNotDropped(t *testing.T) { // 1168/AC3 func unknownWatchmanVersionIsUnreadableNotDropped(t *testing.T) { // 1168/AC3
m := New() m := New()
fill(m, fill(m,
Entry{Bucket: delivery.PresenceBucket, Key: "c", Value: []byte(`{"v":9,"online":false}`), At: t0}, Entry{Bucket: delivery.PresenceBucket, Key: "c", Value: []byte(`{"v":9,"online":false}`), At: t0},
@@ -292,3 +264,70 @@ func TestUnknownWatchmanVersionIsUnreadableNotDropped(t *testing.T) { // 1168/AC
t.Fatal("a readable value must clear the unreadable mark") t.Fatal("a readable value must clear the unreadable mark")
} }
} }
// 1952/V1 (ADR 0564 §10): status reads the keeper's open conditions, the watchman's presence and stuck keys, the wants against
// the haves and the healers' attempts of the week; nothing writes a summary, and a part it cannot read is said, never well.
func statusDerivesEachPartFromItsBucket(t *testing.T) {
now := t0.Add(time.Hour)
composedC := "sha256:" + strings.Repeat("c", 64)
fp1, fp2, fp3 := delivery.SourcePrefix2+strings.Repeat("1", 64), delivery.SourcePrefix2+strings.Repeat("2", 64), delivery.SourcePrefix2+strings.Repeat("3", 64)
enc := func(v any) string { b, _ := json.Marshal(v); return string(b) }
want := func(fp string) string {
b, _ := delivery.EncodeWant(delivery.Want{FP: fp, Composed: composedC})
return string(b)
}
have := func(fp string, phase delivery.Phase, health delivery.Health, detail string) string {
b, err := delivery.EncodeHave(delivery.Have{FP: fp, Composed: composedC, Phase: phase, Health: health, Detail: detail})
if err != nil {
t.Fatal(err)
}
return string(b)
}
presence := func(on bool) string { b, _ := delivery.EncodePresence(delivery.Presence{Online: on}); return string(b) }
stuckOf := func(bucket, key, waited string, moved bool) string {
b, _ := delivery.EncodeStuck(delivery.Stuck{Bucket: bucket, Key: key, Waited: waited, Since: t0, Moved: moved})
return string(b)
}
stuckWant, _ := delivery.StuckKey(delivery.AssignerWantsBucket, "want.b.web")
fin := t0.Add(time.Minute)
r := StatusRead{
Conditions: states(map[string]string{
"machine.b.silent": enc(health.Condition{Key: "machine.b.silent", Severity: health.Warning, Summary: "b silent", Raised: t0}),
"core.doctor.self-check": enc(health.Condition{Key: "core.doctor.self-check", Severity: health.Urgent, Summary: "no run", Raised: t0.Add(time.Minute)}),
}),
Presence: states(map[string]string{"a": presence(true), "b": presence(false)}),
Stuck: states(map[string]string{stuckWant: stuckOf(delivery.AssignerWantsBucket, "want.b.web", "b to run web", false),
"x.moved": stuckOf("x", "moved", "y", true)}),
Wants: states(map[string]string{"want.a.web": want(fp2), "want.a.mail": want(fp1), "want.b.web": want(fp2), "want.a.db": want(fp3)}),
Haves: states(map[string]string{"a.web": have(fp1, delivery.PhaseRunning, delivery.HealthOK, ""), "a.mail": have(fp1, delivery.PhaseRunning, delivery.HealthOK, ""),
"b.web": have(fp1, delivery.PhaseRunning, delivery.HealthOK, ""), "a.db": have(fp3, delivery.PhaseRunning, delivery.HealthFailed, "the unit failed")}),
Attempts: states(map[string]string{"1": enc(health.Attempt{Outcome: health.OutcomeActed, Begun: t0, Finished: &fin}),
"2": enc(health.Attempt{Outcome: health.OutcomeEscalated, Begun: t0}), "old": enc(health.Attempt{Outcome: health.OutcomeActed, Begun: now.Add(-8 * 24 * time.Hour)})}),
}
st := DeriveStatus(r, now)
if len(st.Conditions) != 2 || st.Conditions[0].Key != "core.doctor.self-check" || st.Conditions[1].Key != "machine.b.silent" {
t.Errorf("conditions = %+v, want the urgent one first", st.Conditions)
}
if len(st.Quiet) != 2 || st.Quiet[0].Key != "b" || st.Quiet[1].Key != stuckWant {
t.Errorf("quiet = %+v", st.Quiet)
}
wantMoving := []StatusMoving{
{Node: "a", Module: "db", State: delivery.MotionStuck, Outcome: delivery.OutcomeFailed, Words: "the unit failed", FromFP: fp3, ToFP: fp3},
{Node: "a", Module: "web", State: delivery.MotionMoving, FromFP: fp1, ToFP: fp2},
{Node: "b", Module: "web", State: delivery.MotionStuck, Outcome: delivery.OutcomeStopped, Words: "b to run web", FromFP: fp1, ToFP: fp2},
}
if !reflect.DeepEqual(st.Moving, wantMoving) {
t.Errorf("moving = %+v\nwant %+v", st.Moving, wantMoving)
}
if !reflect.DeepEqual(st.Heals, map[string]int{health.OutcomeActed: 1, health.OutcomeEscalated: 1}) || st.Well {
t.Errorf("heals = %v, well %v", st.Heals, st.Well)
}
// A part it cannot read is said, and the mesh is not well; with nothing wrong and every part read it is.
if st := DeriveStatus(StatusRead{}, now); st.ConditionsUnread == "" || st.QuietUnread == "" || st.MovingUnread == "" || st.HealsUnread == "" || st.Well {
t.Errorf("nothing read = %+v", st)
}
empty := states(map[string]string{})
if st := DeriveStatus(StatusRead{Conditions: empty, Presence: empty, Stuck: empty, Wants: empty, Haves: empty, Attempts: empty}, now); !st.Well {
t.Errorf("an empty mesh = %+v, want well", st)
}
}
@@ -101,7 +101,7 @@ func TestEveryWantIsARowAgainstItsHave(t *testing.T) {
func TestASourceRevisionIsDerivedFromWantsAndHaves(t *testing.T) { func TestASourceRevisionIsDerivedFromWantsAndHaves(t *testing.T) {
m := New() m := New()
fill(m, sourceE(t, 14355), moduleE(t, "web", fpNew), moduleE(t, "mail", fpNew), fill(m, sourceE(t, 14355), moduleE(t, "web", fpNew), moduleE(t, "mail", fpNew),
assign("ace", "web", "b-1", ""), assign("g14", "web", "b-1", ""), place(t, "ace", "web"), place(t, "g14", "web"),
wantE(t, "ace", "web", delivery.Want{FP: fpNew}), haveFP(t, "ace", "web", fpNew, delivery.HealthOK), wantE(t, "ace", "web", delivery.Want{FP: fpNew}), haveFP(t, "ace", "web", fpNew, delivery.HealthOK),
wantE(t, "g14", "web", delivery.Want{FP: fpNew}), haveFP(t, "g14", "web", fpNew, delivery.HealthOK), wantE(t, "g14", "web", delivery.Want{FP: fpNew}), haveFP(t, "g14", "web", fpNew, delivery.HealthOK),
wantE(t, "ace", "mail", delivery.Want{FP: fpNew}), haveFP(t, "ace", "mail", fpNew, delivery.HealthOK)) wantE(t, "ace", "mail", delivery.Want{FP: fpNew}), haveFP(t, "ace", "mail", fpNew, delivery.HealthOK))
@@ -119,7 +119,7 @@ func TestASourceRevisionIsDerivedFromWantsAndHaves(t *testing.T) {
if s.State != SourceNotDelivered || !strings.Contains(strings.Join(s.Waiting, ";"), "mail") { if s.State != SourceNotDelivered || !strings.Contains(strings.Join(s.Waiting, ";"), "mail") {
t.Errorf("a want that is not the module's fingerprint: %+v", s) t.Errorf("a want that is not the module's fingerprint: %+v", s)
} }
fill(m, wantE(t, "ace", "mail", delivery.Want{FP: fpNew}), assign("novox", "web", "b-1", "")) fill(m, wantE(t, "ace", "mail", delivery.Want{FP: fpNew}), place(t, "novox", "web"))
s = m.Snapshot().Source s = m.Snapshot().Source
if s.State != SourceNotDelivered || !strings.Contains(strings.Join(s.Waiting, ";"), "novox") { if s.State != SourceNotDelivered || !strings.Contains(strings.Join(s.Waiting, ";"), "novox") {
t.Errorf("a node assigned the module with no want: %+v", s) t.Errorf("a node assigned the module with no want: %+v", s)
@@ -143,16 +143,15 @@ func TestTheWantsPageSaysWantsAgainstHaves(t *testing.T) {
// 1906/D7 (ADR 0539 §14): the controller's judgement of the move a want names is shown beside it, with its words. // 1906/D7 (ADR 0539 §14): the controller's judgement of the move a want names is shown beside it, with its words.
func TestTheJudgementOfTheMoveIsShownBesideTheWant(t *testing.T) { func TestTheJudgementOfTheMoveIsShownBesideTheWant(t *testing.T) {
// 1955/L7 (ADR 0571 §6): the judgement is the node agent's, in its have waiting for a yes; the controller's composed copy
// is not read.
m := New() m := New()
key, err := delivery.ComposedKey("ace", "web", fpNew) h, err := delivery.EncodeHave(delivery.Have{FP: fpOld, Phase: delivery.PhaseWaitingYes, Health: delivery.HealthOK,
Judgement: delivery.JudgedCannot, Waits: "sha256:" + strings.Repeat("c", 64), Detail: "the composition it runs is gone"})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
v, err := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("c", 64), Judgement: delivery.JudgedCannot, Words: "the composition it runs is gone"}) fill(m, wantE(t, "ace", "web", delivery.Want{FP: fpNew}), Entry{Bucket: delivery.HaveBucket, Key: "ace.web", Value: h, At: t0})
if err != nil {
t.Fatal(err)
}
fill(m, wantE(t, "ace", "web", delivery.Want{FP: fpNew}), Entry{Bucket: delivery.ControllerComposedBucket, Key: key, Value: v, At: t0})
rows := m.Snapshot().Wants rows := m.Snapshot().Wants
if len(rows) != 1 || rows[0].Judgement != string(delivery.JudgedCannot) || !strings.Contains(rows[0].Judged, "gone") { if len(rows) != 1 || rows[0].Judgement != string(delivery.JudgedCannot) || !strings.Contains(rows[0].Judged, "gone") {
t.Fatalf("rows = %+v", rows) t.Fatalf("rows = %+v", rows)
@@ -177,7 +176,7 @@ func TestTheViewSaysOnceThatASourceRevisionIsDelivered(t *testing.T) {
said = append(said, e) said = append(said, e)
return nil return nil
}) })
fill(m, sourceE(t, 14355), moduleE(t, "web", fpNew), assign("ace", "web", "b-1", ""), fill(m, sourceE(t, 14355), moduleE(t, "web", fpNew), place(t, "ace", "web"),
wantE(t, "ace", "web", delivery.Want{FP: fpNew})) wantE(t, "ace", "web", delivery.Want{FP: fpNew}))
if len(said) != 0 { if len(said) != 0 {
t.Fatalf("said %v before any have", said) t.Fatalf("said %v before any have", said)
@@ -213,7 +212,7 @@ func TestTheViewSaysOnceThatASourceRevisionIsDelivered(t *testing.T) {
// have: the row shows the fingerprint wanted and waits while the node runs another. // have: the row shows the fingerprint wanted and waits while the node runs another.
func TestARowReadsTheWantWhenTheAssignmentNamesNoBuild(t *testing.T) { func TestARowReadsTheWantWhenTheAssignmentNamesNoBuild(t *testing.T) {
m := New() m := New()
fill(m, assign("ace", "web", "", ""), wantE(t, "ace", "web", delivery.Want{FP: fpNew}), haveFP(t, "ace", "web", fpOld, delivery.HealthOK)) fill(m, place(t, "ace", "web"), wantE(t, "ace", "web", delivery.Want{FP: fpNew}), haveFP(t, "ace", "web", fpOld, delivery.HealthOK))
r := find(t, m.Snapshot(), "ace", "web") r := find(t, m.Snapshot(), "ace", "web")
if r.Assigned != fpNew || r.Have != fpOld || !strings.Contains(r.Waits, "has not applied") { if r.Assigned != fpNew || r.Have != fpOld || !strings.Contains(r.Waits, "has not applied") {
t.Fatalf("row = %+v, want it to wait for the want's fingerprint", r) t.Fatalf("row = %+v, want it to wait for the want's fingerprint", r)
@@ -228,10 +227,12 @@ func TestARowReadsTheWantWhenTheAssignmentNamesNoBuild(t *testing.T) {
// and a have nothing is assigned to keeps the module's name. // and a have nothing is assigned to keeps the module's name.
func TestADottedModuleNameIsFoundAndKept(t *testing.T) { func TestADottedModuleNameIsFoundAndKept(t *testing.T) {
m := New() m := New()
fill(m, assign("ace", "novox.be", "", ""), wantE(t, "ace", "novox.be", delivery.Want{FP: fpNew}), haveFP(t, "ace", "novox.be", fpNew, delivery.HealthOK), // The seat's placement refuses a dotted module name (mesh-sdk delivery.Placement), so such a module is never placed:
// its have is a row left behind under its own name.
fill(m, place(t, "ace"), wantE(t, "ace", "novox.be", delivery.Want{FP: fpNew}), haveFP(t, "ace", "novox.be", fpNew, delivery.HealthOK),
haveFP(t, "ace", "old.name", fpNew, delivery.HealthOK)) haveFP(t, "ace", "old.name", fpNew, delivery.HealthOK))
s := m.Snapshot() s := m.Snapshot()
if r := find(t, s, "ace", "novox.be"); r.Have != fpNew || r.Waits != "" { if r := find(t, s, "ace", "novox.be"); r.Have != fpNew || r.Status != StatusLeftBehind {
t.Errorf("the dotted module's row = %+v", r) t.Errorf("the dotted module's row = %+v", r)
} }
if r := find(t, s, "ace", "old.name"); r.Status != StatusLeftBehind { if r := find(t, s, "ace", "old.name"); r.Status != StatusLeftBehind {
+2 -2
View File
@@ -87,7 +87,7 @@ func TestManifestIsTakenAndReadsOnly(t *testing.T) { // 1134/AC7
if m.Module != "mesh-delivery-view" { if m.Module != "mesh-delivery-view" {
t.Errorf("module = %q", m.Module) t.Errorf("module = %q", m.Module)
} }
want := []string{"mesh-build-go.pauses", "mesh-build-go.steps", "mesh-build-image.pauses", "mesh-build-image.steps", "mesh-build-node.pauses", "mesh-build-node.steps", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-delivery-assigner.asks", "mesh-delivery-assigner.differences", "mesh-delivery-assigner.wants", "mesh-delivery-builder.builds", "mesh-delivery-checker.tries", "mesh-delivery-gate.asks", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.heard", "mesh-delivery-indexer.modules", "mesh-delivery-indexer.taken", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-delivery-watchman.stuck", "node-engine.have"} // build_usage reads the build modules' steps and pauses, mesh-config's capacities and the checker's tries; 1168/AC1, 1241/AC4, 1345/AC3, 1297/B8, B9 want := []string{"mesh-build-go.pauses", "mesh-build-go.steps", "mesh-build-image.pauses", "mesh-build-image.steps", "mesh-build-node.pauses", "mesh-build-node.steps", "mesh-conditions.conditions", "mesh-config.resolved", "mesh-delivery-assigner.asks", "mesh-delivery-assigner.differences", "mesh-delivery-assigner.placement", "mesh-delivery-assigner.wants", "mesh-delivery-builder.build-log", "mesh-delivery-builder.builds", "mesh-delivery-checker.tries", "mesh-delivery-gate.asks", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.config-candidates", "mesh-delivery-indexer.heard", "mesh-delivery-indexer.modules", "mesh-delivery-indexer.predefined", "mesh-delivery-indexer.taken", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-delivery-watchman.stuck", "mesh-healers.attempts", "mesh-vault.signers", "node-engine.have"} // view_status reads the keeper's conditions and the healers' attempts (1952/V1); build_usage reads the build modules' steps and pauses, mesh-config's capacities and the checker's tries; 1168/AC1, 1241/AC4, 1345/AC3, 1297/B8, B9
got := slices.Clone(m.Reads) got := slices.Clone(m.Reads)
slices.Sort(got) slices.Sort(got)
if !reflect.DeepEqual(got, want) { if !reflect.DeepEqual(got, want) {
@@ -119,7 +119,7 @@ func TestManifestIsTakenAndReadsOnly(t *testing.T) { // 1134/AC7
} }
// The two read tools of the indexer's modules (issue 1562), carried by a bundle the runtime launches; the indexer // The two read tools of the indexer's modules (issue 1562), carried by a bundle the runtime launches; the indexer
// itself serves no tool (ADR 0437 §4). Carrying them gives the tool runner read access to every bucket the view reads. // itself serves no tool (ADR 0437 §4). Carrying them gives the tool runner read access to every bucket the view reads.
if want := []string{"indexer_module", "indexer_modules", "indexer_candidate", "indexer_candidates", "builder_build", "builder_builds", "assigner_wants", "assigner_want", "assigner_differences", "node_haves", "build_usage", "view_reach", "indexer_module_history", "indexer_heard"}; !reflect.DeepEqual(m.Tools, want) { if want := []string{"view_status", "indexer_module", "indexer_modules", "indexer_candidate", "indexer_candidates", "indexer_predefined", "indexer_config_candidates", "builder_build", "builder_builds", "builder_history", "assigner_wants", "assigner_want", "assigner_differences", "node_haves", "node_trusts", "build_usage", "view_reach", "indexer_module_history", "indexer_heard"}; !reflect.DeepEqual(m.Tools, want) {
t.Errorf("tools = %v, want %v", m.Tools, want) t.Errorf("tools = %v, want %v", m.Tools, want)
} }
for _, a := range m.Build.Artifacts { for _, a := range m.Build.Artifacts {
+12 -3
View File
@@ -4,16 +4,21 @@
"moved": true, "moved": true,
"slug": "view", "slug": "view",
"tools": [ "tools": [
"view_status",
"indexer_module", "indexer_module",
"indexer_modules", "indexer_modules",
"indexer_candidate", "indexer_candidate",
"indexer_candidates", "indexer_candidates",
"indexer_predefined",
"indexer_config_candidates",
"builder_build", "builder_build",
"builder_builds", "builder_builds",
"builder_history",
"assigner_wants", "assigner_wants",
"assigner_want", "assigner_want",
"assigner_differences", "assigner_differences",
"node_haves", "node_haves",
"node_trusts",
"build_usage", "build_usage",
"view_reach", "view_reach",
"indexer_module_history", "indexer_module_history",
@@ -30,8 +35,6 @@
{ "name": "reach", "history": 1 } { "name": "reach", "history": 1 }
], ],
"reads": [ "reads": [
"mesh-controller.assignments",
"mesh-controller.builds",
"mesh-build-go.pauses", "mesh-build-go.pauses",
"mesh-build-go.steps", "mesh-build-go.steps",
"mesh-build-image.pauses", "mesh-build-image.pauses",
@@ -39,21 +42,27 @@
"mesh-build-node.pauses", "mesh-build-node.pauses",
"mesh-build-node.steps", "mesh-build-node.steps",
"mesh-config.resolved", "mesh-config.resolved",
"mesh-controller.composed",
"mesh-delivery-assigner.asks", "mesh-delivery-assigner.asks",
"mesh-delivery-assigner.differences", "mesh-delivery-assigner.differences",
"mesh-delivery-assigner.placement",
"mesh-delivery-assigner.wants", "mesh-delivery-assigner.wants",
"mesh-delivery-builder.build-log",
"mesh-delivery-builder.builds", "mesh-delivery-builder.builds",
"mesh-delivery-checker.tries", "mesh-delivery-checker.tries",
"mesh-delivery-gate.asks", "mesh-delivery-gate.asks",
"mesh-delivery-gate.source", "mesh-delivery-gate.source",
"mesh-delivery-indexer.candidates", "mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.config-candidates",
"mesh-delivery-indexer.heard", "mesh-delivery-indexer.heard",
"mesh-delivery-indexer.modules", "mesh-delivery-indexer.modules",
"mesh-delivery-indexer.predefined",
"mesh-delivery-indexer.taken", "mesh-delivery-indexer.taken",
"mesh-delivery-teller.answers", "mesh-delivery-teller.answers",
"mesh-conditions.conditions",
"mesh-healers.attempts",
"mesh-delivery-watchman.presence", "mesh-delivery-watchman.presence",
"mesh-delivery-watchman.stuck", "mesh-delivery-watchman.stuck",
"mesh-vault.signers",
"node-engine.have" "node-engine.have"
], ],
"own-secrets": { "own-secrets": {

Some files were not shown because too many files have changed in this diff Show More