Compare commits

..
Author SHA1 Message Date
jschoubben 83b962cc6d Pin mesh-sdk 05b3e2e1b94f, mesh-contracts ae5266e77639 and mesh-catalog's buildstep 125929dbc459, the group's final heads (issue 1966)
mesh/delivery ready: it is merged on the Approve
mesh/check failed (12): unit: ..a consumer failed for minutes is announced, naming it, its machine and the…; and 11 more | 83b962cc6d6d on ae31b66a9e33
mesh/delivery-group group feat/registry-milestone rejected: composed together: the checker's verdict at the members' heads (3a038b9f8b4d): unit: ..a consumer f…
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:47:39 +01:00
jschoubben b3b6456ab8 Pin mesh-sdk e5d89c5eedb5, mesh-contracts 347a3053580a and mesh-catalog's buildstep fdc3331465dc, the group's heads after review round 2 (issue 1966)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:26:33 +01:00
jschoubben 4ba4cacc66 G-delivery 7: the indexer's and the assigner's views emitters write view-unsaid
mesh-sdk's views.Emitter writes the owner's finding of a change not said past its bound only when Findings is
set; both owners of views in mesh-delivery now set it to their findings state.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:20:53 +01:00
jschoubben 24f9b6f602 G-delivery 6: gofmt
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:18:39 +01:00
jschoubben 3a3611aa41 G-delivery 6: the builder ends a remove on kept and records the version kept
A seat that cannot remove a tag alone (another tag names its digest) now answers and says kept. The builder ends
that remove there, with no retry, and marks the version kept on its store with the reason and the other tags;
builder_versions shows it, and a later branch end asks it no more. The four seats' kept are consumed.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:18:33 +01:00
jschoubben d0983a8b97 G-delivery 5: the judge refuses a read-only state no process names; grants read it from the manifest
mesh-sdk's state read-only (d420491c) is mapped into the grants' buckets, so the module's account and the tool
runner read it and never write it; the judge refuses a read-only state no process's bus state names, since
nothing would write it.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:16:38 +01:00
jschoubben dafb8e1ce6 G-delivery 5: gofmt
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:13:48 +01:00
jschoubben 276886d8f6 G-delivery 5 (grants half): a read-only state is written by the process naming it alone
The bus module's spend record must not be writable by the tool runner. A state declared read-only gets read on
the module's own account and the tool runner carrying it (direct get, a watch's consumer) and no publish on its
$KV subjects; the process whose bus state names it writes it. The judge rule and the manifest field follow
with mesh-sdk's read-only.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:13:41 +01:00
jschoubben 441b6d2922 G-delivery 3, 4: family-not-heard says it clears at the part's next start; views.Bound is two minutes
Bind runs once at a part's start, so the finding's words no longer promise it clears when the grants part makes
the consumer. The assigner's comment on firstViewBound named five seconds for views.Bound, which is two minutes.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:10:51 +01:00
jschoubben 121c6d9c48 G-delivery 2: sdk/heard's cases are rows of the watchman's findings table
mesh-delivery is over its unit budget, so the new function goes; its three cases run as rows of an existing
table of a user of the package.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:10:16 +01:00
jschoubben e5011e1f83 G-delivery 1: the builder's bounds not watched, or a bound not written, is a finding
A watch of bounds that cannot start lost every bound's end with a log line only: it is now family-not-heard
naming the state. A publish bound that cannot be written would never end: the version is publish-unanswered
at once, saying why.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:09:48 +01:00
jschoubben 4be1d7bae1 Pin mesh-sdk c54648e74346, mesh-contracts ec9529022c44 and mesh-catalog's buildstep ccb7eaff1382, the group's heads (issue 1966)
The placeholder versions of the contracts' seats and clients are gone; the repository's own sdk and buildrun keep their
relative replaces.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 17:44:36 +01:00
jschoubben 3e96e977bc mesh-grants: the bus module's keeper reads its warrant on the account composed from nats' manifest
Review round 1, running F4 (design 25 § 12.7), moved here from the nats test by the lead: in the composed-grants case
of the whole test, the part composes nats and messenger from their manifests (testdata copies of the catalogue's at
the join); the keeper connects on its own process account to a server loading the composed list, reads the router's
record of its ask, and is refused a write to the router's asks and the part's state and another asker's record.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 17:28:21 +01:00
jschoubben bf8f5c5c25 mesh-grants: the bus module's process consumer hears the seat events said to it by name
Its durable consumer filtered only its consumes, composed as module subjects, so the operator channel's
decided for its own asks could not be heard there (design 25 § 12.3: it wakes on the warrant).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 17:11:53 +01:00
jschoubben 4f7ceb61ea mesh-grants: a first view's condition the keeper did not take is said again at the next pass
The raise and clear of first-view-unanswered dropped a failed publish (_ = say). The binding comment
named the bus module's own stream and 30 days; the part hears the seat's events on the shared stream, 7 days.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 17:08:23 +01:00
jschoubben 4dbb6b2aec F-delivery: builder_bounds reads the builder's state bounds
Every state a module keeps has a read tool (the lead's ruling). Each key of bounds now holds what it bounds
and when it ends, and builder_bounds lists them with the time left.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:46:32 +01:00
jschoubben f9adc9d44c F-delivery 8: two claims never one number, on the bus, in the builder's whole-module test
The case ran only on the in-memory store. Eight claims of one name at once on the builder's state versions,
through its own bus adapter, get eight versions; with Create turned into Put they share one.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:43:06 +01:00
jschoubben 71b9737fc7 F-delivery 6: old-form-held is raised at a node's seventh day by an expiring key
The finding was written only at the window's next change. Each node holding the window has a key of the
builder's state bounds whose time to live ends at its seven days; the bus's marker of its end steps the
window, which raises the finding then; a node that no longer holds it loses its key.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:41:35 +01:00
jschoubben 00903e9149 F-delivery 7: the builder's old-form window steps by mesh-sdk's delivery.StepWindow
A read naming no node (an empty node_haves, the placement not read yet) closed the window for good (security
review F3, second of a kind). The window embeds delivery.WindowHeld and steps by the shared rule, named being
the placement's nodes before any is dropped. The builder's expiring keys share one state, bounds (the publish
bound of F-delivery 5 renamed), for the old-form bound next.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:40:44 +01:00
jschoubben 78fc3ffa35 F-delivery 5: the builder raises publish-refused, publish-unanswered and seat-outcome-unread
A holder's refused, and a version no holder answered, raised no finding outside the floor, so a build said done
while its file was not in the registry; a seat's outcome that does not read was only logged (design 51 Health
and conditions). The bound is a key of the new state publish-bounds with its own time to live, armed when a
built version is still numbered; a later published of the name clears both findings.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:38:17 +01:00
jschoubben 0d39254ebf F-delivery 4: gofmt sdk/heard
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:34:06 +01:00
jschoubben d4598f5772 F-delivery 4: a consumer that does not hear its family is a finding until it binds
The indexer's, the assigner's, the builder's and the watchman's consumers that were not there or could not be
heard only said so in the log, so the part stayed healthy while it heard nothing (ADR 0615 §5). sdk/heard is
the one shared helper: the finding family-not-heard names the observer and the family, written when the
consumer does not bind and deleted when it does.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:34:02 +01:00
jschoubben 395a1a0f40 F-delivery 3: the watchman waits for created only for entries not yet said created
The bus module says created once per entry, so every later accepted manifest of the module raised a false
bus-state-overdue five minutes on. The watchman keeps a mark per entry said created, on its pulse state with no
time to live and read back at a restart; a refused or an entry no longer asked drops it.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:29:21 +01:00
jschoubben 2a32721b80 F-delivery 2: the assigner takes the module graph by the indexer's first view (ADR 0631)
The assigner read the indexer's graph state itself, the one reader left after ADR 0615 §1. The indexer serves
graph as a seventh first-view tool to the assigner alone; the assigner asks it, follows views.graph, raises the
first-view condition past the bound while the indexer is down, and its reads of graph goes.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:27:07 +01:00
jschoubben 9674710503 F-delivery 1: the checker judges names against an answered empty packages state; judgeOver judges one digest
An answered packages state with no name handed the judge nil, which judges no name, so a manifest naming an
unknown package passed (DLV-I14, design 32). PackageNames is nil only until the state's first reading ends.
judgeOver takes one module and digest, so dropping the known packages for a digest the trunk does not hold
cannot drop them for another (J7's note).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:22:59 +01:00
jschoubben 0e44467b70 J8b: sdk/forms reads the seat form through catalogue, upstream naming included
catalogue.InRegistrySeat now reads the holder's upstream naming and InArtifactStore derives its store path, so forms'
own reader goes, and the upstream rows run the encode check again.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:58:59 +01:00
jschoubben bf92a37730 J8: sdk/forms names an image and an upstream copy in the seat reference form, each deriving its artifact-store form
ADR 0624: container-image-registry/<module>/<artifact>@<digest> and the holder's container-image-registry/upstream/<host>/<path>@<digest> are form:seat-reference; Older derives artifact-store://... so a key keeps both. The registry form now precedes the artifact-store form, so a node stating nothing keeps the recorded store form of an image.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:55:45 +01:00
jschoubben c47f0af0ab J7 (3): the indexer's verdict on a pull request's candidate does not refuse on a package's name
The checker, which knows the delivery group, judges a change's names; the indexer's state knows only the trunk's packages
and refused a package new in the group. The trunk's manifests are still judged on them, and a merge's manifest in full,
though its candidate was accepted without them (UnknownPackageNames).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:44:36 +01:00
jschoubben 2bb0ccd3c6 J7 (2): the checker hands the judge the known packages in composer.Request.Packages, not in Input
Input is the composition's input; the judge's names now have their own field (mesh-sdk), so packagename.Input and the
judge's inputOf are gone and mesh-judge reads Packages.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:42:03 +01:00
jschoubben e91a3923d0 J7 (1): the assigner, the teller and the view read a kept ask through the operator-channel client
Their three copies of the id and option checks (askjson.go checkedAsk, the teller's routerAskOf, the view's routerAsk)
are deleted; each calls occ.Carried with the id the key and the asking give.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:39:47 +01:00
jschoubben a09f5bde38 Join: gofmt sdk/enginereads (L3 left it unformatted)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:35:30 +01:00
jschoubben fa27137036 Join: a look at a done build key logs no "kept" narration
After the join the builder looks at a done key again whenever a first view is answered late or a package's name is
heard (lead's i), and each look's "version kept" narration was written to the build log after the build's end, with no
start or end around it. A look at a done key builds nothing: a kept version says nothing new there and stays in the
process log only. TestBuilderMakesADoneKeyWithTheDigests failed at about one run in three (kept narrations of
tools-go and tools-typescript after the end); 8 of 8 pass.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:35:22 +01:00
jschoubben 71f4e8d128 Join (lead's i): the builder follows the indexer's packages and numbers a package once its name is known
The indexer's packages state is read through its read tool's first view and its views family (ADR 0615 §1), as the
other four states the builder follows, not by its bucket: the manifest drops the read, invokes indexer_read_packages and
consumes views.packages.

A package built unnumbered because its name was not known at its claim was numbered only at its module's next claim.
Now a series key heard naming a name looks at its module again (Builder.PackageNamed), and a look at a done trunk build
numbers what it left unnumbered with the done key's digests (BuildClaim.Done), so version.numbered carries the digest
for the build module's late publish. Woken by the views family, never a timer.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:32:57 +01:00
jschoubben 5c4016c960 Join item 5: the judge's packages rule takes the known packages by their ecosystem names
Design 32 names a package in a manifest's packages by its name in its own
language, so the judge (DLV-I14) now refuses a name no known package carries
instead of any name that is not a declared <module>/<artifact>. The indexer
hands it its packages state's names, kept before the merge's judge pass; the
checker hands it its followed packages picture plus, for a package new in the
group, the name read at the member's head. Both read a name with one reader,
sdk/packagename.Of, and the judge request carries the names in its Input
(packagename.Input). The series-form check stays.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:30:11 +01:00
jschoubben 12f5e24e7f Join item 7: a key's history through the owners' read tools; the verdict and its event name each series change
History (A2j finding 1, J4): the indexer's and the assigner's read tools are views.ReadTool over views.KeptSource, which
answers `{key, history: true}` now; modules (20), heard (64) and wants (20) keep more than one revision. Rows in the
view's "each tool answers through the owners' read tools as before": indexer_module_history and indexer_heard answer
the owner's kept history, and a want's history (view_tiers' Since) is read through assigner_read_wants (the row it
replaces held that the owner answered no history).
Series (A2j finding 2, J4): the checker keeps each series change as a delivery.SeriesChange {module, artifact, from,
to} and renders the status line from it; delivery.Verdict.Series holds the case's changes (each module's artifact once),
and the verdict event's series the pull request's (meshdeliverychecker.VerdictHolderEventSeriesItem). An artifact new
on the head is no series change (the lead's ruling). Rows: TestTheIndexersJudgeJudgesTheChangesManifests (the verdict's
series; failed first, empty) and TestTheStatusIsPendingWhileTheCheckRunsAndEndsSuccessOrFailure (the event's; failed
with the fill cut).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:17:50 +01:00
jschoubben c8eccf299a Join (lead's iii): a series key's value is mesh-sdk's delivery.SeriesPackage; its readers decode it by the contract
The indexer's SeriesPackage and SeriesVersion are aliases of delivery.SeriesPackage and SeriesVersion (J4), SeriesKey and
IsSeriesKey are delivery.SeriesPackageKey and IsSeriesPackageKey; the indexer's readers (the series read for a user's
fingerprint, indexer_package_users) and the builder's NamesIn decode with delivery.DecodeSeriesPackage (the builder no
longer reads the indexer's value by field names). Found by the contract: a published newest's Contents named no time,
which DecodeSeriesPackage refuses, so the accepted series was left out and its users kept the commit form; the
indexer now stamps the time the contents were read (TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage,
"an accepted series' newest ... pins it", failed with the decode before the stamp).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:14:41 +01:00
jschoubben 7ae4f2bb7f Join item 6: the builder says version.numbered as delivery.VersionNumbered and reads out-of-step by BuildAnswer's field
versions.NumberedBody and NumberedEvent are mesh-sdk's delivery.VersionNumbered and VersionNumberedEvent now (J3), the
contract a build module consumes for BG2's late publish; the body's JSON is unchanged. The lane reads a step's
out-of-step files through delivery.BuildAnswer.OutOfStep, a failed step's result too (as before, A2k's d15d66fc: read
before the step's state; its three rows hold it), in place of a local decode by name. The integration test hears
version.numbered on delivery.VersionNumberedSubject and reads it with DecodeVersionNumbered, as a build module does.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:12:13 +01:00
jschoubben 91b24a5426 Join item 8: versions.WaitsOn dropped (A3i ruling: dead code)
One modules value per module means a trunk claim has no earlier merge of the same name to wait on, so nothing called
it; its subtest "merge order" goes with it. Design 51's merge-order line holds by that (a review check).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:10:58 +01:00
jschoubben 952dedd0ed Join item 2: the readers follow through mesh-sdk's views.ProcessReader; sdk/firstview is gone
mesh-delivery's sdk/firstview moved whole into mesh-sdk go/views (J3): the builder, checker, watchman, teller and view
follow each owner's state with views.ProcessReader.Follow(owner, state, tool, stream), naming the owner's read tool as
their invokes name it and its views family as the owner's manifest's `streams` names it ("indexer-views",
"assigner-views"); a refused Follow stops the start. Their beds serve the owners with viewstest.Serve and grant with
viewstest.Grants by the same names. The view's tools ask the owners' read tools by their literal names. The assigner
names its own views family and the indexer's itself (views.AssignerStream and IndexerStream are gone).
Every views.StatusTools call takes handacts.Recorder{Ask: stdio.Ask}: a first view asked again by hand is a hand act,
recorded through seat:mesh-hand-acts.hand-act, which the builder's, checker's, watchman's, teller's and view's
manifests now invoke (a row in each manifest test, each failed first).
Row (A2j's written-time switch): TestRunNeedsItsCredential "a want stood since the assigner's bus wrote it": the
watchman's want time (createdOr) is the time the owner's bus wrote the key, through ProcessReader's first view (failed
with createdOr ignoring the entry's time). sdk/go.mod: nats.go is indirect now.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:10:36 +01:00
jschoubben 2fe8d01e86 Join item 1: the rewrite map, asks/asker, the vault's types and compose-candidate to mesh-contracts
mesh-sdk holds no go/asks, go/asker, vault answers or compose-candidate types (A1d, J1): each module now names the
operator-channel contract and client, the channel benches (mesh-sdk/go/channel), mesh-vault's contract, and node-engine's
contract and client (a1-progress.md, the A1d and J1 rewrite maps).
- assigner, teller, view: delivery.Ask.Ask and AssignerAsk.Ask are the router's ask as JSON; the refusals mesh-sdk
  dropped (an id other than AskingID(key, asking), no option) are each reader's and writer's: assign.EncodeAsk/DecodeAsk
  (askjson.go), the teller's routerAskOf (teller.go) and the view's routerAsk (view.go); the teller refuses an option
  the asking does not offer and one whose level is not acknowledge (viewanswers.go), the two refusals ViewAnswerFits
  gave. Row: subtest of TestTheHoldAskIsValid (another id, no option, no ask; each written before).
- enrolment, grants: the vault's verbs and answers from meshvault; asks' conditions from the contract and client.
- checker, assigner's judge: compose-candidate from nodeengine/nodeengineclient (nec.Own for "node").
- mesh-grants' user-lists golden: the node-engine's users now subscribe to mesh-delivery-view's settled (mesh-sdk's
  NodeAgentHears, A7's ENG-8 retry).
go.mod: each module requires the mesh-contracts modules it imports (and nodeengineclient, which mesh-sdk imports) at the
zero version, for the group pin.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:05:12 +01:00
jschoubben 5385d6844d Join item 4: the indexer serves indexer_read_seat_definitions, the checker's seventh first view
The checker invokes it (A2j 5700edf) and the indexer said the state on its views family without a read tool for it.
Row in TestManifestDeclarations: every indexer tool a module of this repository invokes is one the indexer serves
(failed first, naming the checker's invoke).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:04:34 +01:00
jschoubben e648955a94 Join item 3: the indexer's empty manifest-accepted takes its id from the event (EventID names module and revision)
delivery.ManifestAccepted.EventID now names, for an event with no digest, the module and the event's revision (J3,
ruling A5(d)), so the indexer drops its own left id. The revision is the modules state's revision of the value that left
(write hands the revision it read), so two leaves of one module are two events; asked again by hand for a module no
longer held it is 0, and the again suffix keeps it unique. Row in TestAMergeThatRemovesAModuleDeletesItsKey (failed
first: Revision 0, id naming the last digest).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:03:35 +01:00
jschoubben 5c4486d512 Join item 2, indexer: its views family and the assigner's by their manifests' names; its ask-again records a hand act
views.IndexerStream and views.AssignerStream are gone from mesh-sdk (J3 lead's (a)): the indexer names its own views
family "indexer-views" and the assigner's "assigner-views", as each manifest's `streams` names them. Its first-view
ask-again records the hand act through seat:mesh-hand-acts.hand-act (views.StatusTools takes a handacts.Recorder;
the manifest already invokes the verb). judge_test.go gofmt'd (c6c6687 left it unformatted).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 15:01:42 +01:00
jschoubben 1cd0d83f41 Join: merge A3's builder and L3 (wip/registry-a3 88a9d19)
L3 (710c4032) arrives with it. Conflicts, each kept from both sides:
- assigner module.json, manifest_test.go, main.go, tools.go, integration_test.go: A2's first-view tools,
  read tools and states beside L3's assigner_engine_reads and engine-reads; the two "consumes" lists
  joined into one (the indexer's views and node-engine.manifest-reads-changed).
- builder module.json and manifest_test.go: A3's tools, states, consumes and invokes beside A2g's first
  views (DLV-M1); one findings state (A3's, history 5) carries both; reads keep A3o's
  mesh-delivery-indexer.packages until the builder follows it (join item).
- builder cmd main.go: rebuilt from A3's with A2g's changes applied (the reader follows modules,
  manifests, candidates and placement; one calls connection; one findings bucket, A3's
  health.FindingsBucket; FollowPlacement on a goroutine once placement's first view is answered;
  buildFromModules in place of the start's settle). Tools main.go: A3's tools then the first-view tools.
- builder run.go: A2k's out-of-step on the end entry, then A3's numbering of a done build.
- view module.json and manifest_test.go: node_manifest_reads and seat:node-engine.manifest-reads beside
  A2j's first views and read tools; one invokes list.
- indexer judge rules_test.go: L3's manifest-reads verb and event, without pull-waits (J1 dropped it).
- sdk/go.mod and builder go.mod: both requires kept.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:58:45 +01:00
jschoubben 84a5094684 Join: merge A5's grants part, keeper and watchman consumers (wip/registry-a5 d53986f)
Conflict: modules/mesh-delivery-watchman/manifest_test.go, both sides asserting m.Consumes,
and both adding a Consumes field to the test's manifest struct (kept once).
Git also auto-merged module.json into two "consumes" keys (A2g's first views and A5's
manifest-accepted and mesh-broker events): joined into one list, and the test asserts all five.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:51:50 +01:00
jschoubben 88a9d19b4f DLV-B2: a package is numbered under its name in its ecosystem, read from the indexer's packages state
The builder numbers, asks the floor of, records and tags a package under the name the indexer's series key names for
its <module>/<artifact> (a Go module path, an npm package name; design 32), so builder_versions answers the holder's
question by the name a registry seat publishes it under. A package the state names nothing for, or a state not readable,
is built unnumbered and said; the rest of the claim is numbered.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:51:27 +01:00
jschoubben d266a4c839 Join: merge A2g's checker, view, watchman, builder, teller and enrolment readers (wip/registry-a2g aafd90b)
No conflicts.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:48:00 +01:00
jschoubben 3caa8291b0 DLV-B9: a bundle answered in the registry seat's form keeps the artifact-store form it derives for each counted engine that does not read it
The builder's forms gain seat-reference for bundles and archives (oci-artifact-registry/<module>/<artifact>@<digest>);
the old-form window writes the older artifact-store reference beside it (catalogue.InArtifactStore), the oldest kept
in the single reference (ADR 0624 §1, §3), so mesh-build-go's seat-form answer no longer strands an engine stating nothing.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:45:00 +01:00
jschoubben 8d7e8d3cb9 DLV-I6 ruling: a package is named in its own ecosystem, and its series key with it
At each trunk merge the indexer reads every package artifact's name from the merge commit (a Go package's go.mod module
path, a TypeScript package's package.json name) and writes it into the series key as name, beside the module and artifact
it is built from, so the builder numbers it under that name before its first version. Series keys, accepted series and
indexer_package_users go by the name (the lead's design limit for design 32, replacing the builder-name fallback).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:44:38 +01:00
jschoubben aafd90b8cf DLV-M5r1: the watchman's want time is the picture's written time once the join's views name it
The lead's ruling on A2h's finding 1: J3 builds each first-view key's written time into mesh-sdk views
(Answer.Written, a Picture entry's Created) and J2 the tool runner's state reads' at. The watchman and
the builder's toolchain gate already read the entry's Created(), so neither changes; this says so where
the fallback stands.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:42:26 +01:00
jschoubben 5700edf955 DLV-M2r1: the checker fails a pull request whose head's seat definition does not parse, naming the seat and where
The lead's ruling on A2i's finding 1 (design 26): the indexer keeps an unparsed definition of a pull
request's head as its entry of the state seat-definitions with `unparsed`, and cannot refuse the manifest.
The checker follows that state as its seventh first view (invokes indexer_read_seat_definitions, consumes
views.seat-definitions; checks start once it is answered too, as M2's six) and fails each pull request
with an entry holding `unparsed`, naming the seat, its module, its directory and the indexer's words; a
state that cannot be read fails it too. Rows: TestTheIndexersJudgeJudgesTheChangesManifests (fails
first without the refusal), the adapter's key match in TestThePublishedVersionsAreTheIndexersPackagesState
(fails first matching every key), the manifest's invokes and consumes; the integration bed serves the
state (without it every check waits and five tests fail).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:40:02 +01:00
jschoubben f7a6118e61 DLV-I19, I20 [i]: the indexer's whole-module test keeps a verdict's and a trunk build's out-of-step files and reads a pull request again by hand
Cases in TestAMeshSDKMergeMovesOnlyTheModulesImportingAChangedPackage: the checker's verdict event keeps a pull request's
files, a trunk build end raises generated-types-out-of-step and a clean one clears it, as indexer_out_of_step reads them;
indexer_seat_definitions_again, asked through takeins, reads a pull request's head and indexer_seat_definition answers it.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:39:32 +01:00
jschoubben d15d66fc47 DLV-I20 (builder): a build's end names the generated files its build module found out of step
The builder reads out-of-step from each lane's step result, a failed step's too, and writes it on the build's end
entry (BuildLogEntry.OutOfStep, J3's mesh-sdk field), where the indexer raises generated-types-out-of-step for a trunk
(the lead's ruling on A2i's finding 2). BuildAnswer names no such field yet, so it is read by name and shape.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:35:59 +01:00
jschoubben 20269818c9 DLV-C5: gofmt of the checker's examiner test
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:32:46 +01:00
jschoubben 30ee47caa6 DLV-C5: the checker asks the seat mesh-config's verb renames for every change that brings or drops a renamed-from
Design 32 (A config value renamed): the checker invokes seat:mesh-config.renames, never the module. A
change bringing renamed-from to a holder that serves no renames (nothing answers the optional verb) is
refused naming it; dropping a renamed-from while its window is open is refused naming the nodes holding it
open; a check that cannot ask fails, saying so. The pull request's trunk and head manifests are read once
for this and for the series changes (manifestPair). Rows in TestTheIndexersJudgeJudgesTheChangesManifests
with the verb faked from the seat's definition (its fixture's answer): not served, served, open on g14,
closed, an ask that fails, no seat to ask; the four refusals failed first. Manifest row for the invokes.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:32:41 +01:00
jschoubben 1d31ef1e85 DLV-I17, I19 rulings: a seat has one trunk definition, and a pull request is read again with its head's manifest
A second repository's trunk defining a seat another trunk holds is refused and kept nowhere, with the finding
seat-defined-twice naming both repositories; indexer_seat_definitions_again for a pull request reads the head's own
generated-from, so a pull request changing that field is read right by hand too (the lead's rulings on A2i 3 and 4).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:31:39 +01:00
jschoubben f9e0e6544c DLV-F5r: the assigner answers its readers' first views before any new-form state, with no binding of the bus module
The case in the assigner's whole-module test serves the tools bundle's first-view tools as the runtime of its own does,
with the callers its manifest writes, while the assigner still waits on the indexer: the grants part, the tool runner and
the indexer each get theirs, every other account (the assigner's own included) is refused (ADR 0621 §1-§3).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:29:10 +01:00
jschoubben 890b8f36e9 DLV-C4: the checker names each change of a package series a pull request makes in its status line
Design 32 (A package, its series): the review of a change sees every series it raises. For each module a
pull request touches, every artifact its trunk manifest and its head's manifest both declare whose series
differs is named as "<module>/<artifact> series <trunk> → <head>" in the pull request's mesh/check line,
never dropped when the line is cut; a manifest that cannot be read is said in the same place. Rows in
TestTheIndexersJudgeJudgesTheChangesManifests (raised: named; kept: not named); the raised row failed
first. gofmt of the integration test's imports (M2).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:28:18 +01:00
jschoubben 8afd34525d DLV-M4, DLV-V1: the view reads the indexer's and the assigner's twelve states through their read tools, never their buckets
ADR 0615 §1, design 25 § 12.2: the manifest drops its twelve reads of the two owners' states and invokes
their twelve read tools. The process follows the six it watches (placement, wants, asks, candidates,
modules, taken) by the first view and then the views families (consumes), through firstview pictures, so
its watch code is unchanged; an owner that is down holds no other watch, and the unanswered first view is
the view's finding (states first-views and findings; tools view_first_views, view_first_view_again).
The tools keep their names, arguments and answers and ask the owners' read tools on each call
(view.OwnerRead; a read of many keys is one paged values read). DLV-V1: a table subtest answers each
tool through the owner's own Read over a fake state and matches what reading the state answered.

The owners' read tools answer no history of a key: indexer_module_history, indexer_heard and view_tiers'
times say so (an error, never none) until the read tool takes a history argument (a request to the join).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:24:17 +01:00
jschoubben 24dd2ec29a DLV-F5r (assigner's case): the indexer's runtime answering after the assigner asked is followed within the bound
A case of the assigner's existing whole-module test: the indexer's first views are held until the
assigner has started, and the wants are written from what they answer (ADR 0621 §3).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:22:31 +01:00
jschoubben b84db41198 DLV-F5r (indexer's case): the assigner's runtime answering after the indexer asked is followed within the bound
A case of the indexer's existing whole-module test: the assigner's first view is held until the
indexer has started, and only the placed node's toolchain counts once it answers (ADR 0621 §3).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:21:13 +01:00
jschoubben 598204b4c4 DLV-I17, I18: the indexer's whole-module test keeps a contracts module's definitions, a pull request's unparsed one, and drops it on close
Research 081 test 21 as a case of the indexer's existing whole-module test (no new test function:
mesh-delivery is over its test budget).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:20:01 +01:00
jschoubben aa647d0eda DLV-I17, I18, I19, I20: the indexer keeps every contracts module's seat definitions on the bus
Design 26 and ADR 0608 §2-§4: the definitions are read at each trunk merge and pull request head
through the forge's get-file tool, parsed by mesh-sdk seatdef and kept in the state seat-definitions
by seat and source; seat-definition-changed is said, kept seven days, only when a digest moved. A
trunk definition that does not parse leaves the trunk's entry and writes seat-definition-unparsed;
a pull request's entries go when it closes or merges. A trunk build end naming out-of-step files
writes generated-types-out-of-step, cleared only by a clean build of the same module; the checker's
verdict keeps a pull request's. Tools indexer_seat_definition, indexer_seat_definitions_again (a hand
act) and indexer_out_of_step.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:18:17 +01:00
jschoubben c6c66873f4 DLV-M6r5: the judge takes a claimed seat's module with tools of its own beside its seat's verbs
The lead's ruling 5: one row in the judge's table, the enrolment's shape (a claim serving the seat's
verbs and the first-view tools in its own tools list); the judge accepts it as it stands.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:14:37 +01:00
jschoubben a6fdb121fe DLV-M2: the checker follows the indexer's candidates, modules, manifests, packages and predefined keys and the assigner's placement by first views
ADR 0615 §1 and ADR 0596: the checker invokes the six read tools, consumes their views families and
reads none of the six buckets; this ends its direct read of the packages state (the lead's ruling 2).
It keeps first-views and findings and announces both as its own. The lead's ruling: an owner that is
down never holds the start; the checks, the tries and the hand asks start once every first view is
answered, the placement is followed into the toolchain gate once its first view is. firstview.Reader
opens a followed state's picture as a process.BucketOpener (the builder uses it too). Tools
checker_first_views and checker_first_view_again.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:13:28 +01:00
jschoubben 28dd86b451 DLV-M1: the builder follows the indexer's modules, manifests and candidates and the assigner's placement by their read tools' first views
ADR 0615 §1: the builder invokes indexer_read_modules, _manifests, _candidates and
assigner_read_placement, consumes their views families and reads none of the four buckets; it keeps
first-views and findings. The lead's ruling: an owner that is down never holds the start; the modules
are built once the modules', manifests' and candidates' first views are answered, the placement is
followed into the toolchain gate once its first view is, and until then the gate passes every report on
as before the placement settled. Tools builder_first_views and builder_first_view_again.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:09:22 +01:00
jschoubben 72d8e2155f DLV-M5: the watchman follows the assigner's wants and the indexer's modules by their read tools' first views, read once answered
ADR 0615 §1: the watchman invokes assigner_read_wants and indexer_read_modules, consumes their views
families and never reads either bucket; it keeps first-views and writes an unanswered first view as a
finding of its own. The lead's ruling: an owner that is down never holds the start; each state is read
on a goroutine once its first view is answered, and what it holds (seeded pulses, stuck keys, the
findings it is the source of) is reconciled then (watch.Awaits, ReadLate), never dropped at the start's
end. Tools watchman_first_views and watchman_first_view_again.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:06:04 +01:00
jschoubben ea6fbd31db DLV-I16: the judge refuses a contracts module claiming a seat, naming the seat
A module whose package carries generated-from holds contracts only (design 26).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:06:03 +01:00
jschoubben 2e5917535c DLV-I15: the judge refuses a change's renamed-from that the trunk's manifest does not allow
A rename moves a module's own value with its trust (design 32): the old name is a value the trunk's manifest of the
same module declares and the change does not, no secret and no key of the predefined list, and the old and new value
have the same trust, declared or by the predefined key they take. Judged for a change over the trunk; each refusal
names the module and the key.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:05:09 +01:00
jschoubben c22f9d25dc DLV-I14: the judge refuses a packages entry naming no declared package or an ill-formed series
Between the manifests (design 32): each name in a module's packages is a package some manifest declares, by the
builder's name <module>/<artifact>, and each accepted series is <major>.<minor>.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:02:57 +01:00
jschoubben 20a9322209 DLV-I13: the judge reads a package series, a package's language and the contracts marker
Design 32's words: a series is <major>.<minor> and only on the four kinds the builder numbers, a package is go or
typescript, generated-from only on a package. A package is published as its source, so it names no system: today's
judge refused mesh-sdk's Go package (no system) as a binary.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:01:38 +01:00
jschoubben 87ba69ed69 DLV-I12: the judge refuses a ledger type whose operations or indexes name what its schema lacks
Left by the bus-state review (ADR 0595 §8): a set-entry or remove names its into and key, an operation's fields and an
index's on name schema fields, and a digest-of names a text field; each refusal names module, entry, type, operation
or index, and the field.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:59:37 +01:00
jschoubben 7372447c68 DLV-I9, I9b, I9c: manifest-accepted for every accepted manifest and a module leaving the trunk, never lost silently
The bus module's orphans need to hear a module that asks nothing now (design 25 § 12.2): every newly accepted manifest
is said, its entries an empty list when it asks for none, and a module whose directory leaves the trunk gets an empty
one; a manifest set aside to wait gets none. A failed publish is published once more; one failing twice writes the
finding accepted-unsaid naming the module, said again at the next judging pass and cleared once said.
indexer_say_accepted says it again by hand from the held verdict, a hand act through mesh-hand-acts; it refuses a
module with no accepted verdict.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:58:02 +01:00
jschoubben 1ce394db97 DLV-M6: the enrolment follows the assigner's placement by its read tool's first view and its views family, never its state
ADR 0615 §1: views.Reader asks mesh-delivery-assigner.assigner_read_placement and hears views.placement;
the holder reads placement from the picture and every other state through the runtime. The manifest drops
the read, invokes the tool, consumes the event, keeps findings and serves enrolment_first_views and
enrolment_first_view_again.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:56:31 +01:00
jschoubben 28c3b27fec DLV-M3: the teller reads the assigner's asks by its read tool's first view and its views family, never its bucket
ADR 0615 §1: sdk/firstview is a process's picture of an owner's state, read as the bucket it
replaces (Get, Keys, Watch; every write refused), asked through <owner>.<short>_read_<state> with
values and kept from the owner's views family; firstviewtest serves an owner in a whole-module test.
The teller declares invokes assigner_read_asks, consumes views.asks, states first-views and findings,
and the tools teller_first_views and teller_first_view_again.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:54:50 +01:00
jschoubben 51829a6b14 DLV-I8: indexer_package_users lists each package's users and the series each accepts
So a user left on an older series is seen (design 32): per package its series with each one's newest published
version, and its trunk users with the series they accept, that series' newest and the version they are pinned to.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:51:43 +01:00
jschoubben eedd3271ce DLV-I6: a user accepting a package's series is fingerprinted against its newest published version, by that version's contents
A series' newest published Go package keeps its module's digests read at the version's commit; each module of another
repository whose series the user's manifest accepts enters the import form and the version line by them, pinned to the
version (Go's form). A series with no published version keeps the commit form; a group member keeps its head.
A publish that moves a series' newest recomputes at once.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:49:27 +01:00
jschoubben 1592abbf4f DLV-I5: the indexer keeps each package's series, its newest numbered and newest published trunk version
A user takes the newest of the series it accepts (ADR 0609 §1), so the
indexer learns versions from the builder's numbered and the registry seats'
published, one key <package>.<series> beside the per-module digests.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:42:33 +01:00
jschoubben ab502a7174 DLV-C3r1: a group's Go package version is keyed by the module path go-package-registry's published names
The seat's contract now carries the Go module path, so the checker no longer
works it out from the artifact's paths, which could disagree with it.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:39:08 +01:00
jschoubben 753ea6a067 DLV-C3r3: name the packed group package the old form, kept while the running checker hands it
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:36:14 +01:00
jschoubben 2a3ce39590 DLV-C2, DLV-C6 (verdict): the checker says each pull request's verdict on its stream asks, with the out-of-step files its steps named
Design 26 has the indexer learn out-of-step generated files from the check's
verdict event, so it runs no generator; the test step's answer carries them.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:35:51 +01:00
jschoubben 8c3916bd7b DLV-C3, DLV-C6 (tag-wanted): the checker's whole-module test hears tag-wanted at the definition's subject and builds on the published
A case of the existing group test (no new integration test): the bed's streams are the manifest's (checks with
preview.written, asks with verdict and tag-wanted at the mesh-delivery-checker definition's subjects) and the shared
events stream with the registry seats' published. A group whose mesh-sdk member has no preview yet is read at its head,
says tag-wanted whose body decodes strictly into the definition's type, is not judged before the Go seat's published,
and then builds mesh-delivery against v0.1.15-feat-x.1. The verdict half of DLV-C6 follows DLV-C2.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:28:44 +01:00
jschoubben 3a64fde7d9 DLV-B10: the builder's three narrations are build.said entries of its build log
Design 51 says the build log tells a version kept and why, a tag ask unanswered after five minutes,
and the old form's window closing; the process log alone was read by nobody. Each numbering decision
is now a build.said of its module and fingerprint, its why one of numbered, kept, waiting, unnumbered;
the tag ask and the window, no one build's, are said under the builder's own key, the ask's or the
window's key standing for the fingerprint. Needs mesh-sdk 7efb173 (build.said). Each failed first.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:26:15 +01:00
jschoubben 28ef5af2df DLV-B9 readers: the indexer and the view read the builder's log kinds through one helper of mesh-delivery's sdk
The two copies of IsBuildEntry move into sdk/buildlog, which both modules already require from the repository's own
sdk module (the lead's ruling 5 on A2c's finding 5). The existing rows in the indexer's registered test and the view's
history test cover it unchanged.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:24:17 +01:00
jschoubben c04fa5ff62 DLV-C3r3, C3r4: a group's check builds its members against the group's tagged versions
Design 51 *Tagged versions* and ADR 0607 §3, §5: once each package's published is heard, a Go member's ask names the
group's tagged version of each package another member publishes (Go's form, v0.1.15-feat-x.2), a package new in the
group included, and a TypeScript member's Node fetch takes {name, version} from the registry instead of a packed head.
The published is matched by seat, head and name, so one head with several Go packages (mesh-contracts) gets each its
own version. With no preview of the merge, a member's manifest is read at its head on the forge, so a package new in
the group is asked for on the first check too (the lead's ruling 4 on A2c's finding 4).

Needs mesh-sdk's PackageVersions.Check to take a Go semantic version (a2d-sdk-tagged-version.patch, for A1).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:23:10 +01:00
jschoubben 321c4e4122 DLV-B4, DLV-B6: the seats' outcomes and a branch's end are cases of the builder's whole-module test
On the real bus the artifact seat's removed (retention) and refused (another digest) move a numbered
version, and the forge bridge's merged asks the Go seat to remove the version tagged for the branch,
reason tag-gone, whose removed turns it removed. Each failed first with its hearing off. No new test
function: the builder holds 16 integration tests.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:23:02 +01:00
jschoubben fb4fc8a602 DLV-B10: design 51's round-2 builder cases are cases of the builder's whole-module test
The bed holds the builder's versions, findings, floor-waits and reasks, EVENTS with the artifact
seat's outcomes, and the floor's three sources served on the bus. Two merges close together number
the second at once; builder_publish_again after published takes a new number; a step bound running
out raises build-failed, and the fingerprint built again keeps its number, said with why, and clears
it. Each case failed first with its behaviour off. The kept version is said in the process log until
the build log takes build.said.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:20:49 +01:00
jschoubben d53986f1f6 DLV-F5r: the grants part started before its owners answer waits, says so, and composes nothing from them
A case of the grants part's whole-module test (ADR 0615 §6, ADR 0621 §3): each first view refused once, the part's log
names the indexer's and the assigner's tools it waits on, no membership is written; once the owners answer it composes
as before.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:12:29 +01:00
jschoubben 6c94ff75e2 DLV-B8: the tag ask's narration is a case of the builder's whole-module test, and two instances together joins the archive table
The builder's integration tests fall from 17 to 16: the two-instance test is a row of the archive table; the bed holds the
tag asks and their key-ttl bounds and the builder's process log.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:11:57 +01:00
jschoubben d2e7754816 DLV-B6: a tag ask is marked numbered with its versions, and one unanswered is narrated when its bound ends on the bus
A check's build of the ask's head writes the ask numbered in tag-asks by compare-and-set. An unanswered ask has a key in the
new key-ttl state tag-ask-bounds (five minutes); the bus's marker of its end, heard on the watch, narrates the ask with what
it waits on (the check's build not claimed, names waiting on their floor) and writes the bound again; no timer (design 51
*Tagged versions*). A restart arms each unanswered ask whose bound ended meanwhile.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:11:57 +01:00
jschoubben 8e60a3e078 DLV-C3 follow: the builder hears tag-wanted on the checker's own stream, mesh.stream.<checker>.event.tag-wanted
The checker's family asks is its own stream, whose subjects are mesh.stream.<module>.event.* (ruled on A2c's finding 1);
its verdicts on the same stream are acknowledged and not taken as tag asks.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:11:57 +01:00
jschoubben e20eb05a9b DLV-J4: the judge refuses keeps on a tools bundle's bus, the bus module's included (ADR 0626 §1, §3)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:08:15 +01:00
jschoubben 273da7535b DLV-R1: a verdict below the picture's revision asks the verdicts' first view again, unless the picture holds it
ADR 0615 §5: a gap in revisions asks the first view once more. The indexer's verdict events carry no mark of when its
verdicts were made, so verdicts made again show only as revisions starting low, which the grants part passed over as
already held. A redelivery of a verdict the picture holds for its manifest is still passed over, so a consumer's backlog
after a restart asks nothing.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:07:57 +01:00
jschoubben b3552eafd6 DLV-C3: tag-wanted is the mesh-delivery-checker definition's holder event, branch included
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:07:35 +01:00
jschoubben 606e2b9212 DLV-B9 readers: the indexer and the view pass over build-log entries that are no build's start or end
The builder writes design 51's old-form close into its build log; the indexer's Builds.Put refused it
(no repository) and the view's history stopped on it. Both read the entry's event first and skip any other
kind, whatever its form.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:06:57 +01:00
jschoubben 8a78e00bb0 DLV-G1, DLV-W1: the grants part and the watchman hear mesh-broker's events on the seat's subject; the keeper is granted them
The lead's ruling of 13:0x (registry milestone): mesh-broker's events are mesh.seat.mesh-broker.event.<event> on the
shared events stream like every seat's. The grants part and the watchman consume mesh-broker.<event> and no longer hear
the bus module's own stream, which is its history for its tool. The keeper's process account is composed the seat's
event subjects, and the acknowledgement of a snapshot's chunks of each entry it keeps, which ADR 0626 §1 grants and the
composition lacked; a row holds it to no subscription of a restricted stream's subjects and no consumer of one.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:05:27 +01:00
jschoubben 5865d3c632 DLV-C3: a group's check says tag-wanted for each package its members declare and builds no member before its published
The checker's family asks (stream asks, seven days, in the old streams section) carries tag-wanted
{repository, branch, head, module, package, pull}; the tag is left to the builder, which makes it from the
branch. Each package's published is heard from the two package registry seats on the checker's durable
consumer of the shared events stream; none within the ten-minute bound fails the check naming the package,
before any member is built. A lone pull request and a try ask nothing (design 51 *Tagged versions*).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:03:42 +01:00
jschoubben 5fbd84091e DLV-B6: a check's build of a head a kept tag ask names numbers its packages tagged at its claim
The builder reads tag-asks for the candidate's repository and head (versions.TagFor), numbers only its packages under
the tag below the next version, keeps each with its repository for the branch's end, and says numbered with the head;
the same fingerprint claimed again keeps its number (design 51 *Tagged versions*).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 13:00:17 +01:00
jschoubben 0dc0cf9c9a DLV-B9: the old reference form's window over sdk/forms, kept in old-form, read and dropped by builder_old_form
The window opens while a counted node-engine reads, of some kind, none of the newest form sdk/forms knows, and closes
when none does: said once as version.old-form-closed, and a node holding it seven days is old-form-held about it. It
steps on a statement's change, the placement, a hand drop and each build key written, never on a timer. A retired node
dropped by hand leaves the count of the forms a build key keeps (design 51). internal/versions' own Forms goes: which
forms a key keeps is sdk/forms'.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:55:45 +01:00
jschoubben 20a4a79993 DLV-G2: a row that no tool runner, no tools bundle of the bus module and no account invoking every tool holds the bus's create, update, snapshot or delete subjects of a module's bus state (ADR 0626 §4, How it is checked)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:50:18 +01:00
jschoubben 5c8d59b6c7 Merge lead-47's L3 mesh-delivery head 710c4032 into the registry builder
The builder keeps both: the registry's versions, floor, tag asks and findings, and ADR 0618's engine-reads state,
its tool builder_engine_reads, the node-engine seat's manifest-reads verb and manifest-reads-changed event.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:48:42 +01:00
jschoubben cb1cf7fd73 DLV-R1: the grants part follows the verdicts on the indexer's manifest-accepted and manifest-refused at the revision each carries, not on its views family (ADR 0622 §2)
The next revision changes the verdicts' picture with no ask; one already held is passed over; an
event carrying no revision, past the next, or naming no manifest above the picture asks the first
view once more, said in the log. The manifest consumes the two events in place of views.verdicts.
Needs mesh-sdk 3ff694e (the events' Revision) and the indexer setting it (A2b).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:48:30 +01:00
jschoubben 0811861ede DLV-I7: the requirements state and indexer_requirements, what consumers' go.sum files require of each mesh Go module
Filled at each merge from the go.sum the indexer reads (the lead's ruling on I7), read once per consumer per process
even where its hash is unchanged; the builder's floor asks {package} and reads {requirements: [...]}, highest last.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:48:02 +01:00
jschoubben 27e3bb266f DLV-I11: the manifest test counts indexer_bus_state
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:45:04 +01:00
jschoubben b5a88fae9d DLV-I11: indexer_bus_state, the bus module's keeper's first view of every trunk module's accepted bus state
Answers {revision, modules:[{module, digest, revision, bus-state}]} from the indexer's own states, at the verdicts
state's mark, as A5's keeper reads it.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:44:46 +01:00
jschoubben e612fbebe7 DLV-B3: a name waiting on its floor is the builder's finding floor-silent, cleared once all three sources answer
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:44:18 +01:00
jschoubben 627cb0c0c9 DLV-G2: the bus module's keeper is the process whose bus declares keeps, and only it is composed the create, update, snapshot and delete of every module's bus state (ADR 0626 §1, §3)
A process of the bus module without keeps, or keeps on any other module, is composed nothing
of it. mesh-sdk's ProcessBus.Keeps is asked of A1 (a5-sdk-keeps.patch).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:44:17 +01:00
jschoubben 2178db5a1d DLV-B7: a failed trunk build is the builder's finding build-failed, cleared by a later done build
Read from the builder's own build log entries at their end; a candidate's
build is the check's to say.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:43:51 +01:00
jschoubben 710c4032d2 The checker hands the composer the node-engine's input encoded again from the seat contract's object
mesh-sdk's compose-candidate answer is now the node-engine seat's generated type, whose input is a JSON object; the
composer's request takes bytes, so the checker encodes it again, its numbers kept as written (json.Number).

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 12:43:11 +01:00
jschoubben bbe6ac6ef5 DLV-B6: a branch's end removes its tagged versions, acknowledged once every seat said removed; the checker's tag-wanted is kept in tag-asks
The builder hears the forge bridge's stream on a durable consumer: merged, a
pull's closed and branch-deleted ask remove of each tagged version still
held, and the bus gives the message again each minute until all are removed;
past 15 minutes its finding tag-removal-not-done names the seats. The builder
gains the state findings (ADR 0564 §2). Every forge message also wakes a floor
whose bridge_tags was silent.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:42:57 +01:00
jschoubben f271a70659 DLV-F4r: the grants part takes the views follower from mesh-sdk (go/views); the held-seat rows use a seat whose verbs its holder declares
mesh-broker's verbs are all optional now and the composition grants its reading verbs to its
holder, so taking one from a membership loses no holder: the rows that prove a lost last holder
is held back name a seat the catalogue defines without verbs, claimed by the bus module beside
mesh-broker.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:41:36 +01:00
jschoubben 7fd07d0236 DLV-R1: the judge's seat-set test reads mesh-contracts' seat list: fifty-nine seats, the registry seats beside the artifact store, pull-waits and renames
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:41:30 +01:00
jschoubben a7d490b0e5 DLV-J4: the judge allows the bus module's keeper its own process account, and keeps on it alone (ADR 0626 §1, §3)
The new nats manifest has a keeper process with a bus account of its own; the judge refused it under ADR 0235 §8.
Needs mesh-sdk's ProcessBus to carry keeps (requested of A1); until then the strict decoder refuses the key.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:40:49 +01:00
jschoubben a70b9a0951 DLV-B4: the builder hears the four registry seats' published, refused and removed, and keeps its floor waits in the state floor-waits
Each outcome is a seat event <seat>.<event> on the shared events stream (the
lead's ruling of 12:4x), heard on the builder's events consumer through its
consumes; the subjects are mesh-contracts'. A restarted builder takes its
floor waits in again, so its silent sources' events still wake them;
builder_floor_waits reads them.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:38:15 +01:00
jschoubben 3a789b7a6b DLV-O4: manifest-accepted and manifest-refused carry the verdicts state's revision of the verdict they say (ADR 0622 §2)
The grants part follows verdicts by these events after its first view, so each must carry the revision it is at.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:37:33 +01:00
jschoubben 72da51a10a The node-engine seat's manifest-reads answer and event come from mesh-contracts' nodeengine contract, judged by their reader (ADR 0618 §3, §4, issue 774)
The contract carries no checks, so sdk/enginereads judges the layout's version, the node, each capability, the reason and the times. The local replaces of mesh-sdk are gone: the lead pins mesh-sdk and mesh-contracts at the join.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 12:36:48 +01:00
jschoubben b109d09b84 DLV-F3r: the indexer and the assigner take the views follower from mesh-sdk go/views, and mesh-delivery's sdk/views goes
One copy of the reader's rule for every reader, kept where shared code lives (mesh-sdk), as the lead ruled.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:36:00 +01:00
jschoubben d78161f579 DLV-T1: the watchman's three integration tests are one whole-module test with two subtests, so mesh-delivery's integration count falls by two (test budget)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben 2cb19c6415 DLV-F4r, DLV-R1: the grants part reads placement, wants, manifests, modules and verdicts by their owners' first views, then their views families; an answer empty at revision 0 drops no login (ADR 0615 §4 to §6, ADR 0621, ADR 0622)
Each first view is asked at the start until answered, past two minutes raising
first-view-unanswered naming the owner and the tool; an empty answer at revision 0 reads as
a failed read, so the user list stays in force, and first-view-empty names the operator's
step until an answer above 0. Each owner's views event applies one change with no ask; a gap
asks once more. The reads of the five buckets leave the manifest, and the tools ask the
first views too. Tools grants_first_views and grants_first_view_again (a hand act).

The verdicts are followed on the indexer's views family (views.verdicts), not on
manifest-accepted and manifest-refused as ADR 0622 §2 words it: those carry no revision.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben 56891abe20 TLS-5 grants: the tool runner's consumes of the owners' views get their durable consumers and its findings state its grant, as any manifest's (A7's request; ADR 0615 §5)
Nothing in the composition changes for it: the consumers on module streams are made since
DLV-G1. The row proves it on the tool runner's own manifest.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben 5b68106662 DLV-G2: the bus module's process serves nothing, as every process account (ADR 0436 §3): it hears its module's consumes on a consumer of its own, says its stream's events and raises its conditions; the seat's reading verbs stay its module's tools'
The keeper is asked by a person through its own state requests (CAT-N8); the acting verbs
are served by nobody on the bus and denied to every principal whose grant reaches them. The
bus module's tools bundle may read its module's own stream back (nats_history).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben 745c30e068 DLV-G1, DLV-W1: read the bus module's events by the mesh-broker contract's field names, on its own stream or the seat's subject
The seat's definition (mesh-contracts meshbroker) names local-name and storage-type and puts
the events on mesh.seat.mesh-broker.event.<event>; design 25 § 12.3 keeps them on the bus
module's own stream. Both are read until that is ruled.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben fc150986b7 DLV-W1: the watchman says an accepted entry of bus state the bus module said nothing of in five minutes, and a refused one, until created or no longer asked (ADR 0595 §4, design 25 § 12.2)
It consumes the indexer's manifest-accepted and the bus module's created and refused on its
durable consumers. The five-minute bound is a row of its signals table (a pulse key whose
expiry is the finding, read back at a restart), not a new state: the prepare's state
overdue would have kept the same keys with no expiry to act on.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben 8f6562368c DLV-G2: the bus module's process gets its own account: it defines every module's bus state by its names and an orphan's it created, serves mesh-broker's eight verbs, and no other account holds the acting four (ADR 0595 §4, §5)
The grants are composed for the process of the module holding mesh-broker, from every
manifest's bus-state entries and the streams of the bindings kept; the writers table names
it beside the grants part as a definer of streams; the acting verbs are denied to every
principal whose grant reaches them, and neither the holder's own account nor the tool runner
serves the seat's verbs.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben 8975c37a0a ENG-11 grants: a node's engine hears the status view's settled, on which it retries a failed pull from a store's holder (design 51)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben 7cd5bfb31b DLV-G1: the grants part writes the bus module's binding into the module's memberships on created, rewrites it on recreated, removes it on dropped, kept in memberships (ADR 0595 §1, §7)
The part hears the bus module's own stream on its durable consumer there. That consumer did
not exist: since the controller ended (ADR 0590) nobody created a module's consumer on a
module stream it hears (AccountConsumers left them to the controller), so it is created now
with every account consumer, and a pass that created consumers wakes one more pass so the
part's own hearing opens at once. Bindings are kept per module under bound.<module>.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben e33cf8459b DLV-G3: the grants part raises no history-full; fill is the bus module's, and the old key is cleared once at start (ADR 0595 §6, design 25 § 12.3)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:34:50 +01:00
jschoubben 6b3a679151 DLV-B5: builder_publish_again, builder_floor_again, builder_tag, builder_tag_asks and builder_untag, with the state tag-asks
A holder that failed leaves a version to publish again by hand: said numbered again while the build is held, built
again under the same number when its file is gone, numbered anew after a published; refused for other bytes. The
floor is asked again by hand; a branch's tagged versions are removed through each seat's remove, per repository.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:30:20 +01:00
jschoubben e41666d846 mesh-delivery-view: node_manifest_reads lists what each node's engine reads by asking the node-engine seat's verb of every node at once, each bounded, naming a node not served or away with its record's reads (ADR 0618 §3, issue 774)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 12:26:48 +01:00
jschoubben ad77815d88 mesh-delivery-builder: a done key keeps each reference form a counted node's engine reads, the oldest in its single reference, and names its needs; what each counted node reads is asked through the node-engine seat's verb and kept in the state engine-reads (ADR 0618 §1, §2, §4, ADR 0624 §1, §3, issue 774)
A counted node is one the placement names; a node with no statement reads as the baseline's first forms, and no node record is read in its place. builder_engine_reads lists, asks again and drops.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 12:24:21 +01:00
jschoubben b873b9af4b DLV-I10: indexer_config_readers; and indexer_read_findings with its views event
indexer_config_readers {module} answers {readers: [{module, key, kind,
readers}]} for every value key and own secret any kept manifest of the
module ever declared: the module where its trunk's manifest declares it,
each open pull request whose head declares it, none for a key no longer
read (design 32 *A value no longer read*). It reads one watched copy of
the modules, manifests and config-candidates states. Nodes is left out:
the indexer cannot say which nodes' builds read a key. The indexer's
findings are said on its views family and read by indexer_read_findings,
for the condition keeper (the lead's ruling).

Issue 1966.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben d662c63db5 DLV-J1, DLV-J2, DLV-J3: the foundation is the nine parts, and indexer_old_sections counts the reads left on the owners' state
The judge's foundation holds exactly the nine parts of ADR 0619 §1 and §3,
left out of the old sections' count, and refuses a foundation manifest
asking bus state in the new form. The indexer also counts, across the
accepted manifests, the reads entries still declared on the indexer's and
the assigner's state, by owner and by entry; indexer_old_sections answers
both, so design 25 § 12.7's live row (0 on each owner) has its tool.

Issue 1966.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben 3e241b3034 gofmt the assigner
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben 51eedd93a1 DLV-F2r, DLV-F3r (assigner): the assigner reads the indexer's manifests and modules through its first views
The assigner no longer reads the indexer's manifests and modules buckets
(ADR 0615 §1): it asks mesh-delivery-indexer.manifests and .modules at its
start, its first pass waiting for both answers as it waits for every watch,
then follows indexer-views on its durable consumer, asking once more on a
gap; past the bound its finding names the indexer and the tool. Its status
is in its state first-views, read by assigner_first_views and asked again
by hand with assigner_first_view_again. The whole-module tests run against
an indexer owner built from the views emitter and first-view answer.

Issue 1966.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben 7e77074fec sdk/views: gofmt
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben ccfca79c90 DLV-F1r, DLV-F3r (indexer): the indexer reads the placement through the assigner's first view
The indexer no longer reads the assigner's placement bucket (ADR 0615 §1):
it asks mesh-delivery-assigner.placement at its start, waiting at most the
two-minute bound so a normal start computes the fingerprints once, then
follows assigner-views on its durable consumer, asking once more on a gap.
Past the bound it raises its finding naming the assigner and placement,
cleared by the answer. Its status is kept in its state first-views, read
by indexer_first_views; indexer_first_view_again asks again by hand.
sdk/views gains the event's write time, the status hook, and the status
state and tools a reader's bundle serves.

Issue 1966.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben 1d7124e388 DLV-FB1 (indexer's side): a FORGE message of a kind the indexer does not take is acknowledged and passed over
The forge bridge now says a branch's end as branch-deleted (design 51); the
indexer terminated it as unreadable. It acknowledges it, writes it to heard
as ignored, and the two handle tests are one table (mesh-delivery is over
its unit budget).

Issue 1966.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben 9d79379c5f DLV-O5: the assigner announces its marks state; its tests' placement stream takes only the seat's four events
Issue 1966.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben 249a9333aa DLV-O2, DLV-O3, DLV-O5: the assigner answers its first views and says its views family
The assigner's bundle runs in a runtime of its own on its own account and
serves the first-view tools placement, seat-holders and wants, each answering
only the callers its manifest writes (ADR 0621 §1-§2), and the six read tools
of design 25 § 12.2. The process says each change of those six states on the
family assigner-views beside placement in its old streams section (ADR 0619
§2, ADR 0615 §5); a waiter on put never wakes on it.

Issue 1966.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben 5d7fff91ce DLV-O1, DLV-O3, DLV-O4: the indexer answers its first views and says its views family
The indexer's tools bundle runs in a runtime of its own on its own account
(runs-as) and serves the first-view tools manifests, modules and verdicts,
each answering only the callers its manifest writes (ADR 0621 §1-§2, ADR
0622 §1), and the nine read tools of design 25 § 12.2. The process says each
change of its ten read states on the family indexer-views in its old streams
section (7 days, 256 MiB), with a mark per state the first view answers its
revision from, so a reader that takes the answer misses no event after it
(ADR 0615 §4-§5). The shared shape lives in sdk/views: event, answer, mark,
the emitter, the tools and the reader's follower.

Issue 1966.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:22:26 +01:00
jschoubben 5aa7bc7d49 mesh-delivery-assigner: what each node's engine reads comes from the node-engine seat's verb, asked at its start, on manifest-reads-changed and when a node is back online; a build's reference forms hold a want only when the node reads none (ADR 0618 §1, §3, §4, ADR 0624 §2, issue 774)
Where the engine does not answer the verb, the node's record is read once in its place, said in the journal (the fallback until every counted engine serves the verb). Kept in the state engine-reads, which assigner_engine_reads lists and asks again.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 12:19:34 +01:00
jschoubben e68a47886f sdk: the reference forms a build keeps and holds by, and the keeper of what each node's engine reads, shared by the assigner, the builder and the view (ADR 0618, ADR 0624, issue 774)
Logic only mesh-delivery's parts share, so it is mesh-delivery's and not mesh-sdk's (ADR 0593 §1, ADR 0608 §5). Tested through the assigner's and the builder's rows (the test budget). mesh-sdk is pinned by a local replace to the group's head while developing.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 12:19:34 +01:00
jschoubben a8bcd2abb9 DLV-B3b, DLV-B4 logic: a floor wakes on manifest-accepted, condition-cleared and the next source revision; the seats' outcomes move a version
The builder's one consumer of EVENTS now consumes the indexer's manifest-accepted and
the condition keeper's condition-cleared; a heard source revision stands for the
forge's merge. Numberer.Heard marks a version published, refused or removed and wakes
a floor whose seat was silent; its consumer of each <seat>.events waits on the seat
family's stream form (asked of A1).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:18:00 +01:00
jschoubben 688f6c21fa DLV-B2, DLV-B3 wiring: number each trunk build before its claim, record its digest, ask the floor as the builder's own account
A package whose floor is not known waits and nothing is claimed; an image, bundle or
archive is built unnumbered. The state versions is bound when the bus holds it, every
new number is said as version.numbered, and the manifest invokes the floor's three
sources (design 51 The versions, The floor; ADR 0606, ADR 0617 §1).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:11:53 +01:00
jschoubben ea193186c5 mesh-delivery-indexer: its judge knows the node-engine's manifest-reads and manifest-reads-changed (ADR 0618 §3, §4, issue 774)
mesh-sdk is pinned by a local replace to the group's head while developing.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 12:06:58 +01:00
jschoubben 4d94b0b9e4 mesh-grants: a node seat's event that names its holder's machine is heard from every machine, by the one rule its holder is granted it by (ADR 0618 §4, issue 774)
A watch of the node-engine's manifest-reads-changed subscribed the flat subject, which no engine says, so its consumer would hear nothing. mesh-sdk is pinned by a local replace to the group's head while developing.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 12:05:58 +01:00
jschoubben bd13302ecf DLV-FB1: the forge bridge says a branch's end as branch-deleted and answers bridge_tags
The bridge consumes the forge module's forge.delete and writes a branch-deleted message on FORGE for a deleted
branch (never the trunk, never a tag), so the builder removes the branch's tagged versions (design 51 *The forge
bridge says a branch's end*). Only the forge module's account may say forge.delete: the bus refuses it from
another (a case of the bridge's whole-module test). bridge_tags, served by a new tools bundle, answers a
package's git tags by prefix through the forge module's gitea_api, for the builder's floor.

One unit test merged into another, so the tags test adds no function (test budget).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 11:44:53 +01:00
jschoubben 2274155766 DLV-B3, DLV-B3b, DLV-B9: the floor's three sources and what wakes a silent one; the old reference form's window
The floor asks the seat's versions, bridge_tags and indexer_requirements; a silent source is named in the
floor finding and woken by an event of that source (the holder's published, the forge bridge's next message, the
indexer's next manifest-accepted) or its module's condition clearing, or by hand (builder_floor_again), never by
a timer (design 51 *The floor*, ADR 0617 §1). The old form's window (ADR 0624 §1-§3, ADR 0618 §2): each counted
engine gets the newest form it reads among those written, the new form only once every present engine states
it, the oldest kept in the single reference; a node not serving manifest-reads counts as the baseline; a node
dropped by hand counts no more; old-form-closed is said once, and a node holding it seven days is a finding.

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 11:41:05 +01:00
jschoubben 64b918c64c DLV-B1, DLV-B2: the builder's state versions, numbering under compare-and-set per fingerprint or source revision, and builder_versions
The builder keeps one key per name, tag and version (design 51 *The versions*, ADR 0606), in its old state
section (ADR 0619), history one, 64 MiB with the 80 % mark. A package is numbered per source revision and keeps a
version held published, publishing nothing; one numbered but never stored is published again under it. An image,
bundle or archive keeps its fingerprint's number. An unknown floor builds an image, bundle or archive unnumbered and
makes a package wait (ADR 0617 §1). Merge order and tagged versions below the next version are the package's
helpers. builder_versions answers a name's versions and the holder's question (ADR 0605 §4).

Four claim tests merged into two, so the new table test adds no function (test budget).

Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 11:39:17 +01:00
mesh-tools ae31b66a9e Merge pull request 'mesh-grants and the teller: the removed controller's leftover consumers are removable, and the teller pulls its durable consumer (issue 1126)' (#150) from fix/consumer-remove-any into main 2026-11-04 10:36:44 +00:00
jschoubben fc300216ec mesh-delivery-teller: pull the durable consumer for the router's warrants, so an answer sent while it is down is not lost (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ae31b66a9e33
mesh/delivery-group group fix/consumer-remove-any merged: every member is merged
mesh/check being carried to fc300216ecf7 on ae31b66a9e33 | from dd90a26251b9, mesh-catalog 7a7db0166ad2
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 10:57:33 +01:00
jschoubben c4bba04e8f mesh-grants: the forge's ended take-in consumer is a removable leftover; the retired build-agent seat's subject is no writer (issue 1126)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 10:57:33 +01:00
jschoubben 1c6ac37fc1 mesh-grants: the controller's take-in and config-follow consumers on three buckets are removable leftovers (issue 1126)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 10:53:44 +01:00
mesh-tools dd90a26251 Merge pull request 'L3 final: verb, reader-capability and bus-state edges; one unreadable manifest holds only its logins; the indexer holds mesh-catalog (ADR 0614, ADR 0604, issue 774)' (#149) from feat/l3-final into main 2026-11-04 00:23:49 +00:00
jschoubben 70ade15c36 A refusal an older judge made never stands: the indexer judges again a pull's manifests and, at its start, every other judge's held verdict; indexer_judge takes a digest (issue 774)
mesh/delivery-group group feat/l3-final merged: every member is merged
mesh/delivery merged: source.main moved to novox/mesh-delivery@dd90a26251b9
mesh/check checking 70ade15c36eb on dd90a26251b9 | novox/mesh-delivery#149 in group feat/l3-final
The indexer judged only the manifests its modules state holds, so a digest refused by an older judge (node-tools
48c729f6 and mesh-provisioner-client a382b9df, refused for words then unknown) kept that refusal once it was back on
the trunk or in a candidate. Each such digest is now judged alone, in the catalogue the trunk holds, writing its
verdict and nothing else; an accepted verdict stays accepted under any later judge (ADR 0579 §1), so only a refused or
waiting verdict by another judge, or none, is judged again.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 00:45:59 +01:00
jschoubben cbbc10871f Pin mesh-sdk 85ba0a9a70d8, the group's head of feat/l3-final (issue 774)
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/l3-final rejected: composed together: the checker's verdict at the members' heads (278aa5f0ccbe): unit: mesh-provisioner-client/…
mesh/check failed (2): unit: mesh-provisioner-client/cmd/mesh-provisioner-client.TestNatsANodeComingBackFil…; and 1 more | cbbc10871f2e on 007c982ae670
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:45:00 +01:00
jschoubben 084bfb9f3e Merge branch 'feat/l3-final-c' into feat/l3-final 2026-11-03 23:43:51 +01:00
jschoubben 9d0b3cafc5 A modules value with no verdict yet holds only its module's users, and one module's value that does not read is a gap of that module, never of the whole list (ADR 0614 §8, issue 774)
The indexer writes a changed manifest's value before it judges it; the part now reads the indexer's verdicts.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:43:26 +01:00
jschoubben 62f7375522 Merge branch 'feat/l3-final-b' into feat/l3-final 2026-11-03 23:43:11 +01:00
jschoubben 2d49c660c0 view_tiers names bus state by the assigner's held want alone, never by a provisioner key no one makes (ADR 0614 §5; issue 774)
bus-state is offered with no credential of a consumer's own, so the provisioner keys nothing for it; the view showed
every module asking bus state as waiting for that key for good.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:42:37 +01:00
jschoubben 9d2816d001 The module offering bus-state never waits on bus state itself, and an offline node's build offers nothing (ADR 0614 §5; issue 774)
A bus module that both offers and asks bus state waited on itself for good at its first such build; it now waits on
nobody for it, as the graph and catalogue.Awaited leave a module's own provision out.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:42:18 +01:00
jschoubben 7006380599 Merge branch 'feat/l3-final-c' into feat/l3-final 2026-11-03 23:30:41 +01:00
jschoubben 46c735473f A want asking bus state waits until a build offering bus-state runs healthy on a node present (ADR 0614 §5; issue 774)
bus-state is answered with no credential of a consumer's own, so ADR 0603 §5 counts the provider's own have: the
assigner reads, per pass, what the running healthy builds on online nodes offer (their manifests through the builder's
builds) and holds a consumer with a bus-state entry under the provisions' rule until one offers it, handed or not and
with no provisioner key. The words name bus state, and the delivery view names the edge from them.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:28:41 +01:00
jschoubben 702b19fdb1 view_tiers names each wait by its edge, a verb, a reader capability or bus state, and every verb edge left out (ADR 0614 §2-§5; issue 774)
The words of a want held on a verb or a reader capability are written and read in one place, the module graph's
package: the assigner writes them, the view names the edge from them. A bus-state wait is named bus state, and each
module's verb edges the graph leaves out on a cycle are listed beside its waits.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:26:25 +01:00
jschoubben 0279c5c288 The check's cycle refusal meets no verb edge on a cycle and no package edge, and still refuses provisions and bases (ADR 0614 §3, §6; issue 774)
Rows in the judge's table: two modules invoking each other, or a verb closing a cycle with a provision, are not
refused; a cycle of provisions and one of bases are. The refusal's words name what a part waits on now.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:26:08 +01:00
jschoubben 1d9f9cfa2c The check names a changed build module that invokes a verb no running build serves as one it cannot run (ADR 0614 §7; issue 774)
What the running builds serve is read when a changed build module is met: the node-engine's haves, the builder's
build of each, its manifest. The modules it builds are composed from their manifests, as for any changed build
module, and the composition's warnings name the verb.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:26:08 +01:00
jschoubben c45bbfc391 A want waits until each verb it invokes is served by a running build, read from the module graph (ADR 0614 §2, §3; issue 774)
The assigner reads the indexer's module graph (a new reads line): of what a module invokes it waits only on the edges
the graph keeps and another module serves, so a left-out cycle holds nothing; a verb counts as served once a running
healthy build's manifest serves it. The hold is under the provisions' rule, its words naming the verb, since no reader
knows a rule number of its own yet. Without the graph no want waits on a verb.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:26:08 +01:00
jschoubben b4bfc14556 The indexer writes each module's verb edges into the module graph, the ones on a cycle left out over the whole graph (ADR 0614 §2, §3; issue 774)
The assigner reads which edges are left out from the graph, so one place decides the cycle; the whole graph is
judged, a module past the graph's cap included, before the first GraphMost are written.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:25:35 +01:00
jschoubben df05b1a5f7 The module graph carries verb and bus-state edges, and leaves out every verb edge on a cycle (ADR 0614 §2, §3, §5, §6; issue 774)
A graph node now names what its manifest invokes and serves, and the verb edges the graph leaves out; a bus-state
entry is a requirement of bus-state. Before and Cycle count the kept verb edges beside bases and awaited provisions,
never a package or toolchain edge. LeaveOut leaves out every verb edge whose ends reach each other, so the builder's
remove and a holder's builder_versions hold nothing and the check never refuses them.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:25:35 +01:00
jschoubben 509d277b32 The assigner holds a node's want while that node's engine does not state a reader capability the manifest needs, and moves it once it does (ADR 0614 §4, issue 774)
The rules read Module.Needs from the indexer's modules value and each node's stated reads from its record in
`<node>.node` of the haves; an engine stating none is read as the baseline, so the node-engine's own new build, whose
manifest uses only the baseline's words, is not held. The hold is rule 7 (no rule of its own is numbered in this
release, ADR 0614 F2) with words naming the node's engine and each capability, which the want tools and view_tiers
already show.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:24:39 +01:00
jschoubben 477b3bfea6 The indexer's judge names, with each verdict, the reader capabilities a manifest's own words, forms and seat names need, in its module value (ADR 0614 §4, issue 774)
Module.Needs is catalogue.ManifestNeeds of the manifest held: written by the merge that takes a changed manifest in, so
no reader takes it as needing nothing before the judge's pass, named again by the judge with the verdict, and carried
over by a write that leaves the digest, as the verdict is. The module value is the one the assigner and the checker
already read.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:22:08 +01:00
jschoubben ffdeb9771a The indexer claims the seat mesh-catalog and its tools bundle serves the four verbs from its states (ADR 0604 §2, issue 774)
CatalogTools moves into the index package so its row sits in the existing catalog test.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:11:21 +01:00
jschoubben b83085adac An unreadable manifest holds only its module's users: the list advances for every other principal, and the condition and grants_user_list name the module unreadable (ADR 0614 §8, issue 774)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:10:13 +01:00
jschoubben 390e50d7d1 Words back: the tool runner is what says tool-runner again; sdk/toolrunner goes (issue 774, issue 1585)
Reverts e3ce4b0: node-tools' manifest says tool-runner again, so the judge and
mesh-grants ask catalogue.IsToolRunner and the kept RuntimeModule is not counted.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:09:20 +01:00
mesh-tools 007c982ae6 Merge pull request 'L3 recovery: count mesh-sdk's kept RuntimeModule as the tool runner while node-tools' trunk manifest leaves tool-runner out (issue 774, issue 1585)' (#148) from fix/l3-words-after-judge into main 2026-11-03 21:09:09 +00:00
jschoubben e3ce4b095c Count mesh-sdk's kept RuntimeModule as the tool runner while node-tools' trunk manifest leaves tool-runner out
mesh/delivery merged: source.main moved to novox/mesh-delivery@007c982ae670
mesh/delivery-group group fix/l3-words-after-judge merged: every member is merged
mesh/check being carried to e3ce4b095ce9 on 007c982ae670 | from 4154c3040cee, mesh-catalog e6de42630355, mesh-tools 01ee2ef39488
The running indexer refuses the tool-runner word, so node-tools' trunk manifest takes it out again (mesh-tools
fix/l3-words-after-judge); trunk's judge then refuses node-tools twice (its waits, its runtime bundle) and mesh-grants
would compose its node users as plain modules. One helper, sdk/toolrunner.Is, counts a manifest as the tool runner
when it says tool-runner or its module is the SDK's deprecated catalogue.RuntimeModule; the judge's waits and
undelivered-bundle rules and mesh-grants' declared records ask it. It goes once tool-runner is back on node-tools'
trunk manifest (issue 774, issue 1585 note 42).

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 21:36:26 +01:00
mesh-tools 4154c3040c Merge pull request 'L3: a source revision ordered by the module graph, the check waits on its candidate's builds, and an emptied repository never drops a module another trunk holds (ADR 0603, ADR 0604, issues 1367, 774)' (#147) from feat/l3b-bus into main 2026-11-03 20:12:55 +00:00
jschoubben 945906ac97 The assigner waits for a requirement nothing offers, and for a provision the provisioner keys whoever is placed yet; a requirement something offers (a module's own name, a module the mesh provides itself) is left to Awaited as before (ADR 0603 §5)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4154c3040cee
mesh/delivery-group group feat/l3b-bus merged: every member is merged
mesh/check checking 945906ac976c on 4154c3040cee | novox/mesh-delivery#147 in group feat/l3b-bus
2026-11-03 19:45:15 +01:00
jschoubben 1c66d58bce The assigner waits for a provision whose provider is not known yet: an unknown provider has provided nothing, so a consumer never moves ahead of a provider the indexer has not taken in (ADR 0603 §5); the integration test's write order is unordered again 2026-11-03 19:36:18 +01:00
jschoubben 34ef185ba5 The assigner's integration test puts the provider's module before the consumer's, so the consumer's requirement always has a provider to be judged against (the group check's flaky failure, 5 of 40 runs) 2026-11-03 19:33:43 +01:00
jschoubben f12823681c Merge the indexer's integration test: a field its judge does not know waits while the running build is unknown (ADR 0601 §3; the group check's failure)
mesh/delivery-group group feat/l3b-bus rejected: composed together: the checker's verdict at the members' heads (62ca67fc6b58): integration: mesh-delivery-assi…
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: mesh-delivery-assigner/cmd/mesh-delivery-assigner.TestWantsAreWrittenAndNoDifferenceIs:… | f12823681c06 on c4470f42dba9
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 18:51:14 +01:00
jschoubben b96a4f1d99 Pin mesh-sdk b7abe203a929, the group's head of feat/l3b-bus: it keeps, deprecated, the exported names the running parts still use
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 18:50:12 +01:00
jschoubben 70ac4ff79a The indexer's integration test expects a field from later to wait, not be refused, while no engine says which build runs (ADR 0601 §3)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 18:41:10 +01:00
jschoubben 51deed51e3 Merge ADR 0603 §10: the check waits on the builder's builds of its candidate before it composes, asks itself again when a failed build is done again, and names what it composes from manifests (the operator, 2026-11-03: build §10 in L3)
mesh/delivery-group group feat/l3b-bus checking: 4 of 4 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 51deed51e3b4 on c4470f42dba9 | novox/mesh-delivery#147 in group feat/l3b-bus
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:56:03 +01:00
jschoubben 3842a464ed A check failed on a build asks itself again once the build is done again, and a failed check never waits on builds (ADR 0603 §10)
A failed candidate build no longer needs mesh-delivery.recheck, an exception verb (ADR 0421 §12): the checker hears the
builder's keys the failed verdict names and, once each is written done (builder_build_again), retakes the check of the
same heads as it does after a failure of the nodes, at its start too. When the tests have already failed the check
composes at once and names the builds it did not wait on, so its red status is never held back by a build.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:55:41 +01:00
jschoubben 877b09e92b checker_status names the builds a running check waits on (ADR 0603 §10)
Worked out from the builder's builds state at each read, stored nowhere: of each running check, the candidate's builds
that are neither done nor failed.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:51:09 +01:00
jschoubben 453cdfa607 The check waits on the builder's builds of its candidate before it composes, so it composes with them (ADR 0603 §10)
The builder already builds every candidate fingerprint into its own key. The check now watches exactly those keys on the
bus and composes once each is done or failed, within the new setting build-wait-bound (1800 s). A failed build fails the
check, naming it; a build not ended within the bound is named and the check composes without it. A module built by a
build module the change alters (mesh-build-go or mesh-build-node), one standing on a base the change alters, and one the
trunk holds no record of are not waited on, and the composition's warnings name each with why.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:51:09 +01:00
jschoubben 8f7945a109 Merge round 2 finding C and the check's true words: the new rows fold into existing tests, the check names what it composes from manifests (ADR 0603 §10; test budget)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:36:49 +01:00
jschoubben 2773885600 Merge round 2 findings A and B: only the newest manifest's provisions hold a want, and a wait keeps its first start while it stands (ADR 0603 §8, §9)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:36:49 +01:00
jschoubben 45bd63427f A wait the watchman mirrors keeps its earliest start while the same mark stands, so a key written again with new words does not start the bound again (ADR 0603 §8) 2026-11-03 17:26:45 +01:00
jschoubben effa12c4b4 Fold the three new tests into existing ones (test budget); the base note says what the check does
The check builds no artifacts and no changed base before its dependents; build_change.go now says only that they are composed from their manifests.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:26:31 +01:00
jschoubben 9da14440e8 The assigner waits only on the keys of provisions the newest manifest still requires: an older edge's asked or retiring key holds nothing (ADR 0603 §9) 2026-11-03 17:26:30 +01:00
jschoubben fd638b99d5 Merge D9, D13b, D15 and review round 1 findings 3 and 4: the indexer runs without its graph state, a package inside a module's directory has its publisher (ADR 0603 §3, §4, §10; ADR 0604 §1)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:28 +01:00
jschoubben 0d1edef846 Merge D11, D12, D14: delivered over the nodes present, the watchman's waits with a bound, the newest provider edge (ADR 0603 §7, §8, §9; L3 review round 1, finding 2)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:28 +01:00
jschoubben 5c4d770aa1 The check names the modules built with a changed build module or base; rows for D9 and the assigner's integration test (ADR 0603 §3, §4, §10)
D13b: the composition's warnings, and so the check's line, name the changed modules built with the Go build module the change alters and those standing on a changed base. D9: builder and verdict rows. D15: the assigner's integration test gets a source revision changing a provider, its new consumer and a build module.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:07 +01:00
jschoubben b4855f21c3 Test rows: a missing graph state fails no recompute (review finding 3); a recipe change builds a Go module again (D9)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:07 +01:00
jschoubben 1387b10e78 Take a missing graph state as no graph yet and a Publisher's directory as the nearest module above it
The indexer's recompute and merge failed on listing a graph bucket not yet on the bus, and a package published from inside a module's directory had no publisher, so the dependents of a build module missed what imports its packages (review findings 3 and 4 of L3).

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:01 +01:00
jschoubben a1814c6571 Assigner row: the newest source revision's provider edge replaces the older one, and only it is waited on (ADR 0603 §9) 2026-11-03 17:19:05 +01:00
jschoubben 92b4307922 The watchman mirrors a want held by a rule, a provision asked and a build queued behind a package as waits with a bound, and raises a finding past it (ADR 0603 §8) 2026-11-03 17:18:36 +01:00
jschoubben 2c37f48f9d A source revision is delivered over the nodes present: the view leaves out a node the watchman says is away and names it, and the node counts once it runs its wants (ADR 0603 §7) 2026-11-03 17:16:30 +01:00
jschoubben 4a736858b6 Pin mesh-sdk cf5812101b0b, the group's head of feat/l3b-bus with the module graph taken out 2026-11-03 16:56:37 +01:00
jschoubben 3411b91571 The module graph is mesh-delivery's own: a Go module sdk/ with package graph, used by path by the indexer, the view and the checker, and the seat mesh-catalog's answers built from mesh-sdk's contract types (the operator, 2026-11-03: no graph-related code in mesh-sdk, only contracts; research 081 D11, D13) 2026-11-03 16:56:37 +01:00
jschoubben 0ce16ad773 Pin mesh-sdk 18b3eac47c8e, the group's head of feat/l3b-bus rebased onto main 2341e8e9 2026-11-03 16:46:40 +01:00
jschoubben 875d42fe3e On B1: the ledger test names its tool runner by its manifest's tool-runner, and ShelfWaiting takes B1's old-section counts (rebase onto main c4470f42) 2026-11-03 16:46:40 +01:00
jschoubben aacdba44e3 The judge's copy of the mesh-catalog seat is the SDK's entry: four optional verbs, so the old module's claim is not outgrown (ADR 0604 §1; F3) 2026-11-03 16:45:06 +01:00
jschoubben 39b3ce3c73 builder_builds_since names the operator's command nox builds as its reader, not mesh-catalog (ADR 0604) 2026-11-03 16:45:06 +01:00
jschoubben e1d0c9fe03 A merge that leaves a repository no module never deletes a module another trunk holds, and indexer_take_in takes a repository in (issue 1367)
dropRemoved hands the module over from the other trunk, read from the forge at the head the source names, so no wait in memory is needed. indexer_take_in with a repository takes in a head with no module.json as a merge that leaves none, and answers by name for a head with modules.
2026-11-03 16:45:06 +01:00
jschoubben 6bc16d74db Write and take in a merge that leaves no module.json, so a module-less repository's reach is not held for ever and a removed last module is dropped (issue 1126)
The forge bridge wrote no merged message for such a merge, so taken.<repo> never moved
and the view held 'the indexer has not yet taken in <head>' (hq #490-#492). The bridge
now writes it with no trees; the indexer drops every module of the repository on it,
after the same order check as any merged message. The view needed no change: with the
taken key moving it reaches the merge, which its new row proves.

Needs mesh-sdk fix/1126-no-module-reach (a merged message may carry no trees).
2026-11-03 16:45:06 +01:00
jschoubben 9f49bc92ad The seat mesh-catalog's four verbs answered from the indexer's modules, manifests and graph, with the old catalog's field names (captured answers as fixtures); served once the indexer claims the seat, in the release after L3's (ADR 0604 §1, §2) 2026-11-03 16:45:06 +01:00
jschoubben 4f6e1021c6 view_tiers: a source revision's tiers worked out from the module graph at each call, and per module what it waits on now, its build, each node's want and have and each provision it awaits, the nodes offline named away (ADR 0603 §6) 2026-11-03 16:45:06 +01:00
jschoubben 027c445bae Pin mesh-sdk 182d8757dba9, the group's head of feat/l3b-bus with the graph's ordering edges (ADR 0603 §1, §6) 2026-11-03 16:45:06 +01:00
jschoubben 7ba2668aa8 The check refuses a change whose module graph has a cycle and names it, worked out from the catalogue as the change leaves it; a cycle the trunk already has among the same modules is said, not the change's failure (ADR 0603 §2) 2026-11-03 16:45:06 +01:00
jschoubben a901b840a0 Pin mesh-sdk acad60547c84, and the indexer's graph nodes take their bases and provision edges from the contract's GraphEdgesOf (ADR 0603 §2) 2026-11-03 16:45:06 +01:00
jschoubben f70faed5b6 The indexer keeps the module graph as its state graph: each module's build and provision edges written at every recompute when they change, at most 2000 modules, a module's key deleted once the modules state no longer holds it (ADR 0603 §2) 2026-11-03 16:45:06 +01:00
jschoubben 5ee3b4c15f Pin mesh-sdk 25b371bc8397, the group's head of feat/l3b-bus with the module graph and the seat mesh-catalog's contract (ADR 0603 §2, ADR 0604 §1) 2026-11-03 16:44:47 +01:00
jschoubben 2f387a3f4c Pin mesh-sdk e7ce86ff62b5, the group's head of feat/l3b-bus (review round 2) 2026-11-03 16:44:47 +01:00
jschoubben 22aedbe97f A recompute with no merge writes the fingerprint's source revision as not known, so no release's Approve answers a build it never carried, though the source revision is still that release's (ADR 0601 §1; review round 2) 2026-11-03 16:44:47 +01:00
jschoubben 478f0110e9 Pin mesh-sdk 825483043549, the group's head of feat/l3b-bus (review round 1) 2026-11-03 16:44:47 +01:00
jschoubben 6d24922172 The node-engine's witness ping is granted by the tool runner's name its manifest gives, any service's instance on its own node while none says so, never a name in mesh-grants (ADR 0601 §4; review round 1, finding 4) 2026-11-03 16:44:47 +01:00
jschoubben 0fa00d5e6c The watchman raises what a node keeps of a module as it runs, for a manifest that does not yet say it, from the module's have, and clears it once the have says none (ADR 0601 §4; review round 1, finding 3) 2026-11-03 16:44:47 +01:00
jschoubben a41d2851c7 A release answers only the build whose fingerprint it moved: the indexer stamps the source revision at which a fingerprint last changed, a recompute with no merge the revision of its moment, and the assigner reads that stamp, never the Src a recompute keeps (ADR 0601 §1; review round 1, finding 1) 2026-11-03 16:44:47 +01:00
jschoubben ea5be10532 Pin mesh-sdk 09ec82cb3551, the group's head of feat/l3b-bus (ADR 0601) 2026-11-03 16:44:47 +01:00
jschoubben 6b9d211f72 The checker decides a node the sandbox did not judge by the contract's field (composer.Machine.NotJudged, verdict not-judged) and names it with the composer's words, never by a phrase in its reason (ADR 0593 §9; the prepare's seams finding on "not judged") 2026-11-03 16:44:47 +01:00
jschoubben b342c67b19 The grants part knows the tool runner by what its manifest says, never by its name, and keeps a node's tool runner user as the list in force holds it while no manifest there says it yet, said in the held condition (ADR 0601 §4; the prepare's seams finding on mesh-grants's own name) 2026-11-03 16:44:47 +01:00
jschoubben daf178c4ca A release's Approve answers the record ask of a move of the build alone to the build it carries: the assigner reads the release by the build's source revision from the gate's source and merges, and names it in the want; a move that changes a setting or a secret too is asked as before (ADR 0601 §1) 2026-11-03 16:44:37 +01:00
jschoubben a9aa6d29ea The assigner moves the bus module by its rules as any module, with no bar and no snapshot of its own, and a person may choose its policy as any module's: its node backs its streams up before it applies (ADR 0601 §1, §2) 2026-11-03 16:44:37 +01:00
jschoubben ecf96be8ce The judge's row for a field it does not know names a field mesh-sdk does not know now: core-credentials is one of its manifest words (ADR 0601 §4) 2026-11-03 16:44:37 +01:00
jschoubben ab3cb1ca9e The judge knows the tool runner by what its manifest says and keeps the kept streams itself, and this repository's check that no manifest names the ended controller reads them as text: mesh-sdk names no module (ADR 0601 §4) 2026-11-03 16:44:37 +01:00
jschoubben 054f089614 The bus has the upgrade policy its manifest says, as any module: the judge and the assigner rows read it from the manifest (ADR 0601 §1) 2026-11-03 16:44:37 +01:00
jschoubben 6301e49c73 Review round 2: a held module's object reads come from its running manifest; one with none keeps the list in force, said; indexer_judge asks the engine for the running build again (issue 774) 2026-11-03 16:44:37 +01:00
jschoubben 78fe1ce960 Review round 1: the indexer learns its build from the node-engine's seat; a held module's seats and hearings come from its running manifest; indexer_judge; one judge pass at a time (issue 774) 2026-11-03 16:44:37 +01:00
jschoubben 6645ac58c2 A manifest with a field the judge does not know waits instead of being refused; a held module keeps its bus users; tools list and redo the verdicts (ADR 0578 amended, issue 774) 2026-11-03 16:44:19 +01:00
mesh-tools c4470f42db Merge pull request 'Bus state by template: the judge's rules, manifest-accepted, the old-section count and the ledger account (issue 1966, B1)' (#146) from feat/bus-state into main 2026-11-03 15:24:36 +00:00
jschoubben 4bb790bedf Pin the group's mesh-sdk head f8bd12e in place of the local replace (issue 1966)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c4470f42dba9
mesh/delivery-group group feat/bus-state merged: every member is merged
mesh/check pass 4bb790bedffe on 8f83608503da | unit 6460 run, 262 kept, 27 skipped | integration 6639 run, 264 kept, 35 skipped | over bound: unit, in…
2026-11-03 15:35:28 +01:00
jschoubben aa98729457 The zero rule takes the count of the accepted set the last pass held, and is not applied when it cannot be read (issue 1966 D4) 2026-11-03 15:28:58 +01:00
jschoubben 2fdb0516cd The checker's judge counts the old sections from the trunk case as the indexer does, so a change adding a section at 0 is refused at the merge check (issue 1966 D4) 2026-11-03 15:14:28 +01:00
jschoubben 28a6858863 The ledger account's subjects come from mesh-sdk's one naming of them (issue 1966 D5) 2026-11-03 15:13:38 +01:00
jschoubben 1ab0274547 The wanted accounts include the ledger kind, held in the tool runner's ledgers, in step with mesh-sdk's buscompose (issue 1966 D5b) 2026-11-03 13:39:21 +01:00
jschoubben 07af86b153 The grants part composes a ledger account per assignment of a module asking for a ledger, publishing only on its ledger histories and snapshots (issue 1966 D5) 2026-11-03 13:38:20 +01:00
jschoubben dd2e7c8e64 The indexer counts the modules on each old section, a tool reads it, and the judge refuses a section at 0 outside the foundation (issue 1966 D4) 2026-11-03 13:34:54 +01:00
jschoubben 756a3813e2 The indexer says manifest-accepted, with the bus-state entries whole, for a newly accepted verdict (issue 1966 D3) 2026-11-03 13:32:41 +01:00
jschoubben 9a508701ff One local name is one form, and the old state says what it accepts: 0 is taken as 1 (issue 1966 D2) 2026-11-03 13:31:42 +01:00
jschoubben 7ee7248fad The judge refuses bus state in requires that breaks a template's bounds, naming the entry and the field (issue 1966 D1) 2026-11-03 13:30:42 +01:00
mesh-tools 8f83608503 Merge pull request 'The checker waits its compose answer bound, not the SDK's 30 s, for the machines' compositions' (#144) from fix/1126-compose-answer-bound into main 2026-11-03 00:40:26 +00:00
jschoubben bedda86a22 Fold the call-bound check into the answer-bound settings test (test budget)
mesh/delivery merged: source.main moved to novox/mesh-delivery@8f83608503da
mesh/check being carried to bedda86a22c5 on 8f83608503da | from a62276bcafad
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-03 01:26:26 +01:00
jschoubben 0e26fd8ed5 The checker's calls to node-engines wait the compose-answer-bound, not the SDK's 30 s (issue 1126)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-03 01:13:36 +01:00
mesh-tools a62276bcaf Merge pull request 'Tests no longer race the wall clock; the checker invokes compose-candidate again' (#143) from fix/timing-tests into main 2026-11-02 22:17:53 +00:00
jschoubben 34a607a1f5 Give the checker back its compose-candidate invoke, now that the new indexer runs
mesh/delivery merged: source.main moved to novox/mesh-delivery@a62276bcafad
mesh/check being carried to 34a607a1f5cf on a62276bcafad | from 15d28ddf5bd3
The invoke was left out (mesh-delivery #142) only while the indexer built before the verb ran; the new indexer and the
node-engine's compose-candidate run on every machine since 22:57, so a check composes on the machines again instead of
passing with a warning (ADR 0593 §7, §9).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 23:04:44 +01:00
jschoubben f73b0bf6f6 Tests wait on events or the clock they are given, never on milliseconds a loaded machine outlasts
The check of 2026-11-02 22:35 on a loaded shanks failed two buildrun tests on wall-clock bounds of 60 ms, the
second failure of this kind tonight (the first: mesh-sdk's busretry tests, f02f142).

Quick fix: give the two buildrun asker tests bounds of seconds.
Thorough fix (chosen, because the kind recurs across every module of the repository): every test that raced a
wall clock now waits on an event, steps a clock it is given, or has a bound only a failing test reaches.
- buildrun asker tests: the steps' own times stand for the clock (bound an hour, starts and phases written in the
  past), no goroutine paces phases with sleeps.
- builder: the decline bound reads Builder.Now; the decline test steps it; the stream-wait test waits for five
  check reads instead of 100 ms.
- checker: the lost-ask try bounds an untaken ask at 200 ms only where that is what ends it; the program says when a
  merge turn changes, and the integration test waits on that line instead of 300 ms.
- teller: the bed makes the teller's own ends bucket (a 10 s stall at start the tests raced by 0.5 s), waits for its
  warrant subscription instead of 500 ms pauses, and the package-level now is restored with t.Cleanup.
- watchman: the recount, run in its own goroutine, is waited for instead of 10 ms.
- polls of 2-5 s become 10 s (unit) and 20 s (integration).

Under a throttled cgroup (half a core shared with two busy loops) beside the node-engine's suite, the buildrun
tests failed 5 of 20 runs before and passed 30 of 30 after; no test added.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 23:04:44 +01:00
mesh-tools 15d28ddf5b Merge pull request 'mesh-delivery-checker: leave compose-candidate out of its invokes until the new indexer runs' (#142) from fix/checker-invoke-later into main 2026-11-02 21:55:29 +00:00
jschoubben 22f3658a36 Leave compose-candidate out of the checker's invokes until the new indexer runs
mesh/delivery merged: source.main moved to novox/mesh-delivery@15d28ddf5bd3
mesh/check checking 22f3658a36a6 on 15d28ddf5bd3
The running indexer, built before the verb existed, refuses a manifest invoking it, and one refused manifest holds the
grants part's whole list: no new grant is written, so the new indexer never moves (issue 1126, 2026-11-02). Quick fix,
chosen because the thorough one (the grants part keeps only the refused module's grants) cannot be delivered while the
list is held; a follow-up adds the invoke back. Until then a check's compose-candidate call is refused on the bus and the
node is named silent: the check passes with that warning (engines.go, ADR 0593 §9).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 22:22:23 +01:00
mesh-tools 073e3fab65 Merge pull request 'sdk1: the indexer fingerprints by imported content; the checker asks the node-engines to compose' (#141) from feat/sdk1 into main 2026-11-02 20:42:16 +00:00
jschoubben 2b5a7fe831 Pack the checker's settings template as its files archive, or the indexer's judge refuses the manifest
mesh/delivery merged: source.main moved to novox/mesh-delivery@073e3fab65db
mesh/delivery-group group feat/sdk1 merged: every member is merged
mesh/check checking 2b5a7fe831c1 on 511e4b741497 | novox/mesh-delivery#141 in group feat/sdk1
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 21:19:38 +01:00
jschoubben 3b3b096be8 Place mesh-networking beside a node-engine check, so route-proxy's proofs of a composition are judged once this release runs
mesh/delivery-group group feat/sdk1 rejected: composed together: the checker's verdict at the members' heads (5fc952f974ca): mesh-build-go
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: mesh-build-go: the indexer's judge at the group's mesh-delivery head 3b3b096be8cdf8f49246ed3961ece3eb5c349d62 refuses the manifest …
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:40:01 +01:00
jschoubben 6313f3ea10 Place the node-engine beside a catalogue check, so mesh-build-go's composer test is judged once this release runs
The composer command moved to the node-engine (ADR 0593 §7); a catalogue check had only mesh-sdk beside it. Pin the build runner to the catalogue head whose composer test skips aloud until then.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:36:19 +01:00
jschoubben 712d136fe6 Pin the build runner to the release's new mesh-catalog head
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/sdk1 checking: 6 of 7 member(s) ready
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:23:41 +01:00
jschoubben 09ac178888 Pin mesh-sdk at f02f142, the sdk1 head whose tests no longer leak package globals, so the group's check builds against it
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:14:55 +01:00
jschoubben d4600efd7d Pin the build runner to the release's mesh-catalog head, computed by go
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/sdk1 checking: 6 of 7 member(s) ready
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
The placeholder was written in local time and go refused it as a pseudo-version.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:41:56 +01:00
jschoubben 0edd0bca74 mesh-delivery-builder: fold the step-choice and step-run tests into two tables, within the test budget
Review finding 5: the repository's unit tests rose from 810 to 814 (over the 500 budget); eleven test
functions of buildrun become two, so the count is 805.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:38:09 +01:00
jschoubben 2fb1dc3214 mesh-delivery-checker: a placement that cannot be read leaves the composition not made, said
Review finding 4: an unreadable placement gave no nodes, so nobody was asked and the check passed. The
placement read now answers its error, and the verdict says the composition was not made and why.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:37:22 +01:00
jschoubben e5ce35cd00 mesh-delivery-checker: a composer at the node-engine's head that cannot run fails the composition
Review finding 3 (ADR 0593 §9): only a node that does not answer may let a check pass with a warning; a
sandbox composer that cannot be built or run was filed with the silent nodes. It now fails that node's
composition, naming the head.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:36:42 +01:00
jschoubben e31da597bb mesh-delivery-indexer: the version line takes a directory-replaced sibling by its content, so it never goes stale
Review finding 2 (ADR 0593 §2, §3): after the release no build has reported imports, and a module that
replaces a mesh Go module of its own repository by a directory had no go.sum line and no version for it,
so a change of buildstep or buildrun alone moved no importer. The indexer now reads go.mod's directory
replaces, publishes those modules, and puts their content digest on the version line; builds are pinned
to the other repositories' versions alone.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:35:58 +01:00
jschoubben 33675f548a Pin mesh-sdk at the sdk1 group's head 1c7356c, so the group's check builds every module against the narrowed SDK it was written for
The build runner's pin stays at its placeholder until mesh-catalog's final head is pushed.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:33:10 +01:00
jschoubben 008a85b73c mesh-delivery-indexer: the judge's tests ask nothing of the composition's offers, overflow or left-out reasons
ADR 0593 §7: OffersIssuedOutside, Resolution and UnknownFieldReason left mesh-sdk with the composition,
so the judge's tests keep its own reading and refusals and those proofs go to the node-engine's tests.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:23:17 +01:00
jschoubben 6fb743f521 mesh-delivery-indexer: the judge's tests judge manifests and compose nothing
ADR 0593 §7: the catalogue's composition (Resolution.Compose and Declaration, LeftOut, Resolve,
JudgeSettings, ApplySettings, HeldTemplateProblem) leaves mesh-sdk for the node-engine, so the judge's
tests keep only its verdicts and refusals; each proof that a manifest composes goes to the node-engine's
tests, and the helpers only those proofs used go with them.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:08:41 +01:00
jschoubben c0209b3d8e Builder's fetch test takes the named tree as data, not computed
The named tree is the indexer's computation (ADR 0593 §2): the builder reads it
from the module's value, so its test gives one as the indexer would write it.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:00:57 +01:00
jschoubben c12804f4d9 mesh-delivery-indexer: its integration test no longer reads verdicts through mesh-sdk's composition reader
ADR 0593 §7: compose and frombus left mesh-sdk for the node-engine, so how a composer holds a refused
module is the engine's to test; the indexer's test keeps proving the verdicts and refusals it writes.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:00:16 +01:00
jschoubben fbb50f8fcd mesh-delivery-checker: run check asks in its tests with the Go build module's real runner
ADR 0593 §1: the step runner left mesh-sdk for mesh-build-go's buildstep package, so the examiner's unit
tests and the module's integration test require it by the group's version (as the builder does) rather
than an imitation of its plan and kept-verdict rules.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:58:18 +01:00
jschoubben 0ce3409141 mesh-delivery-checker: make a check's composition from every placed node's engine answer
ADR 0593 §7, §9: the checker asks compose-candidate of each placed node in parallel within its own setting
compose-answer-bound (60 s); a refusal fails the check naming node and module, a silent, unready or
unreachable node passes with "not composed on that node: no answer". A group with a node-engine member is
composed by the composer at that head on the Go build seat over each node's input. The wait on the
assigner's compositions goes, and the checker invokes the one engine verb it needs.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:20 +01:00
jschoubben 46e53a81c2 mesh-delivery-checker: build mesh packages at the versions the indexer publishes instead of computing them
ADR 0593 §2: only the indexer computes a package's version; the checker follows its packages state and
hands those commits to a Go check, so LatestPackages leaves repoModules and mesh-sdk's
PackageVersionsOf has no reader here.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:20 +01:00
jschoubben 7cf47c04eb mesh-delivery-assigner: compose nothing; judge proposals with its own judge and compare the published compositions
ADR 0593 §7, §8: the check's composition, its compose asks to the Go build seat and the state
compositions go (the checker asks the engines); a proposal is judged without frombus; mesh-wants.upgrade
compare reads every node's composed-<node> through mesh-host.composed. The assigner no longer imports
compose, composer, frombus or buildrun.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:20 +01:00
jschoubben 8855735341 mesh-delivery-assigner: the placement judge is the assigner's own, judging a proposal from placement alone
ADR 0593 §1, §8: compose/judge moves back from mesh-sdk; every node answers a proposal, what a placement
changes is read from its entries, and a node silent past the answer bound (60 s) is named and the
proposal decided on the answers it has.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:20 +01:00
jschoubben c62a501bf0 mesh-grants: a node reads the indexer's candidates and verdicts whole, as its engine reads a key through a short consumer
ADR 0593 §7: compose-candidate's read of one key times out without the consumer grants, so both buckets
are granted as the builder's builds are; the node grant now equals mesh-host's golden node.json.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 0b03b9c7c5 mesh-grants: create a module's object buckets and history streams, grant their readers read-only, and warn at 75 percent
ADR 0592 §5, §6, §9, §25 (the grants part's share by ADR 0593 §10): an object bucket and a stream that
keeps its history are created create-only with their caps and no age limit, refusing new data when full;
the owner alone writes its bucket and a module its grant names reads it; the part raises
mesh-grants.history-full once any passes 75 percent. Each engine also reads the indexer's manifest
verdicts by one key for compose-candidate.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben d283efdc9f mesh-grants: the machine's runtime hears a node's compositions bucket made later for the bundles it carries
ADR 0593 §6: a bundle such as the provisioner reads mesh-host.composed on the runtime's connection, so the
runtime account gets the stream listing and the creation advisory beside the read of each bucket.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 74304052b4 mesh-grants: every node's engine writes its own compositions and the named readers read them all
ADR 0593 §6, §7: the grants part creates composed-<node> per node, grants each engine the writes of its
own bucket and a direct get of the indexer's candidates for compose-candidate, and grants
mesh-host.composed read-only, with the stream listing and creation advisory a per-node read needs, to
the provisioner, the assigner and the view alone; memberships issue it per node.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 2a91caddaf mesh-delivery-checker: place no sibling for the controller's repository, which this group empties
ADR 0593 §11: the controller's repository holds nothing that builds once route-proxy moves, so its
sibling rule goes; its tests fold into the sibling test, which now uses mesh-sdk's rule.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 78fedabeb1 mesh-grants: own the bus's composer, so a change to who may say what rebuilds only the grants part
ADR 0593 §1: buscompose's engine moves from mesh-sdk into mesh-grants/internal/buscompose, its golden
user lists unchanged; the controller's raise and assert code is left behind, and the merged switch
test runs inside the part's whole integration test.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 5c20b9ef15 mesh-delivery-checker: the line test holds ADR 0591's case of a first failure longer than the forge's limit
The first failure's words are shortened; "failed (N)" and "and N-1 more" stay whole within the limit.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben fa85de8ef6 mesh-delivery-checker: write a member's preview whenever its fetch merged, and do not wait on a composition that cannot come
Review of 650a1d0: a member that only failed its tests got no preview, so the indexer held no manifest
of it, the assigner's composition only said it waits, and the check waited for ever. The preview is now
written whenever every build module's fetch merged; a member with no preview of a merge (a conflict, a
fetch that did not answer) makes the composition not run, said with the members it names, unawaited.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 290e1696b2 mesh-delivery-checker: a check runs every member, level and the composition to its end and names every failure
The operator, 2026-11-02: "A failed check shows only its first fault". Group feat/1126-removal's
check failed four times, each verdict naming one fault, so three faults took three runs of 20-40 minutes.

The checker no longer returns at a member's first failure: every member is asked, every level's
failures and every member's refused step or conflict are joined in judging order, the manifests are
judged and the composition is read also when the tests failed, and a failed composition is added after
them. The status line says how many failed and names the first, cut so "; and N more" stays whole.
checker_verdicts and checker_pull answer the whole list (failed). Needs mesh-sdk's buildstep change
(every level runs after a failure) pinned at the group's head.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben b7551d2efb Indexer's tests compute with its own fingerprint package
So mesh-sdk can drop SourceFingerprint2, NamedTree and NoPackageVersion (SDK1)
without breaking them.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:41:22 +01:00
jschoubben 2e91f0b061 Fold the release's new tests into existing ones to stay within the test budget
mesh-delivery is over its budget (810 unit, 73 integration on main), so this
branch adds none net: its new cases are subtests, the three work-queue process
tests and the consumes test become one, the history stream bounds join the
stream values table, and the go/-only mesh-sdk merge check joins the one new
integration test. buildrun's 20 unit and 1 integration tests moved in from
mesh-sdk with it.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:40:57 +01:00
jschoubben 61b2b718e6 View reads the indexer's published packages with indexer_packages
A module's fingerprint takes the digests of the packages its last build
imported, so the published packages say why a merge moved a module or not;
the state needs a read tool like every other (ADR 0593 §2; D2).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:38:58 +01:00
jschoubben 2e23b0b736 Prove against the bus that a mesh-sdk merge moves only importers of a changed package
The indexer whole: two catalogue modules report their imports at their builds'
end, mesh-sdk merges a change of buildstep alone, the packages key names the
merge with a digest per package, the buildstep importer moves and the other
keeps its fingerprint though the version line would have moved it (ADR 0593
§2-§4; D11).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:37:37 +01:00
jschoubben b81e05387b Judge says its rules are its own
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:35:17 +01:00
jschoubben 525d7374b2 Judge object buckets and history streams as ADR 0592 decides
The indexer's judge takes a module's own object bucket (a plain local name,
once, no name its states take, none of the mesh's derived prefixes, 1 to 8192
MiB with no default, a grant only by a trusted value) and a stream that keeps
its history (no max age, deliveries or duplicates), counts such a stream among
the bus's predefined forms, gives a bucket's name one owner on the bus, and
refuses a read of a bucket its owner grants nobody. The seat table expects
the node-engine's compose-candidate verb (ADR 0593 §7; D5).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:34:54 +01:00
jschoubben 4c4380b94b Builder's integration test pulls asks with mesh-build-go's runner
The pulling code leaves mesh-sdk for mesh-build-go's own package (ADR 0593 §1),
so the bed imports it by its published version and tests the builder against
the real runner, not an imitation. The required version is a placeholder at
mesh-catalog 9c57f7af: the group pins the catalogue's head before its push.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:33:31 +01:00
jschoubben f6fa75e189 Builder's unit tests run an ask with a local runner, not mesh-sdk's
The build runner leaves mesh-sdk for mesh-build-go (ADR 0593 §1), another
repository the builder does not import; the tests answer asks as the builder
needs them answered.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:31:07 +01:00
jschoubben 604f1528d6 Builder owns the step runner; a build's imports reach its done key and log
buildrun leaves mesh-sdk for the builder's own Go module (ADR 0593 §1); the
checker, assigner, indexer and watchman require it by a replace to the sibling
directory. Issue 1930's bound was already on mesh-sdk's main and moves with it.
The builder copies the imports a build module answers to the done key and the
log's end entry, where the indexer keeps them (§2; D7).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:29:56 +01:00
jschoubben 2aa43ec249 Bridge reads no mesh package from a go.sum
Only the indexer computes what a fingerprint takes (ADR 0593 §2): the merged and
pull messages carry no Packages, and the kept inputs no longer hold them (D6).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:26:53 +01:00
jschoubben fba98ee80f Indexer computes fingerprints alone, over the packages a module imports
The fingerprint, named tree and package versions move from mesh-sdk into the
indexer's internal/fingerprint (ADR 0593 §2). The indexer publishes each reached
mesh Go module's package digests from the forge's listing in its new packages
state, keeps the imports a done build reports, and writes the import form only
when an input differs from the build that reported them (§4), so the form change
costs no rebuild and a package merge moves only its importers (§3). Candidates
take the same form against the group's member heads (D1-D4).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:25:57 +01:00
mesh-tools 511e4b7414 Merge pull request 'The grants part and the indexer without the controller; consumers far behind raise a condition (issue 1126 group B)' (#138) from feat/1126-removal into main 2026-11-02 13:41:16 +00:00
mesh-tools 4dd7dcfc14 Merge pull request 'A group check places its own catalogue head beside the other members (issue 1126)' (#139) from fix/1126-group-catalogue-beside into main 2026-11-02 12:33:04 +00:00
jschoubben d26781b403 Place a group's catalogue head beside its other members' tests (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4dd7dcfc1411
mesh/check being carried to d26781b403df on 4dd7dcfc1411 | from be88162cde31
A group's check judges each member against the group's heads (ADR 0482 §2,
ADR 0573 §4), but the checker placed mesh-catalog at its trunk beside every
other member even when the group held a catalogue pull request, so
mesh-delivery's judge tests in group feat/1126-removal judged the new judge
against catalogue main and failed. The catalogue now follows the rule of
every other sibling: the member's head and ref in a group, the trunk alone.
A catalogue change's own trunk copy (ADR 0529 §2) is placed separately and
is unchanged.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 13:12:54 +01:00
jschoubben e2cb398186 Pin mesh-sdk to the group head of issue 1126
mesh/delivery merged: source.main moved to novox/mesh-delivery@511e4b741497
mesh/delivery-group group feat/1126-removal merged: every member is merged
mesh/check checking e2cb3981866e on 4dd7dcfc1411 | novox/mesh-delivery#138 in group feat/1126-removal
Every member builds against mesh-sdk abf5eac, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:27:33 +01:00
jschoubben 6a4d88265d Say mesh-grants.consumer-behind again only when the consumers behind change, not their counts
The counts change every pass, so a consumer that stays behind was said to the operator again at each pass. The
condition names the consumers alone; how far each is stays in the last pass.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:23:23 +01:00
jschoubben 655692c7ba Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk f569924, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:20:14 +01:00
jschoubben 8f9766e46c Let the grants part say an account consumer far behind as mesh-grants.consumer-behind
The self-check's bus-consumer probe judged consumers the controller defined, and with the controller ended it judges none
(issue 1126). The grants part owns the account consumers, so it says when one stays more than 1000 messages behind its
stream's head at two passes running, the probe's bound and confirmation, and clears it once none is.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:03:29 +01:00
jschoubben 83c1b3ce38 Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk 729a86b, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:56:55 +01:00
jschoubben 000a24d261 Match the judge's copy of mesh-wants' upgrade to mesh-sdk (issue 1126)
mesh-sdk now says what upgrade's compare answers without the controller:
each node's own composition and the assigner's findings. The judge keeps
a copy of the seat, which must read the same.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:49:17 +01:00
jschoubben 3228d911cb Let a module move between repositories by the merges that move it (issue 1126, route-proxy)
route-proxy leaves the controller's repository, which is archived, and nothing registered a move since the controller
went: the indexer left a module named by another repository's merge as it was, forever. Now that merge waits, and the
holding repository's merge that drops the module hands it over in place, so its key never goes missing between the two;
merged the other way round, the key is deleted and written new. A trunk that still holds the module keeps it (ADR 0420
clause 7, ADR 0501 clause 3).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:48:34 +01:00
jschoubben 7e358b0f1d Recognise the control node by the bus module, not the controller (issue 1126)
The watchman held a node a control node, urgent after half an hour
silent, while its have named mesh-controller; the controller ended by
its unassignment (ADR 0590), so no node was one any more and the bus
node's silence stayed a warning. The control node is now the node whose
have holds the bus module, nats, which the watchman already hears by
that name.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:47:53 +01:00
jschoubben 8d4c13427a Build against mesh-sdk without the controller (issue 1126)
mesh-sdk's group B member renames the grant to Grant, drops the
controller principal, the signer and the lease fields from the user
list key, and the keeper's and config's take-over verbs. The grants
part, the assigner and the view use the new names; the judge's copies of
the health and config seats follow; the tests' fixtures lose the
controller's user, and the refused principal is one removing what the
closed set does not name, the controller's mirror being gone.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:46:12 +01:00
jschoubben 2049b1a370 Say the assigner's conditions as findings alone, the keeper being the one source (issue 1126, B8)
ADR 0590 clause 3 ends the choice of a condition source by the keeper's
take-over key, which mesh-sdk no longer defines. The assigner watched
that key and said every condition straight to the router while it named
the controller. It now writes each as a finding the keeper carries, and
says one straight only where it has no findings bucket at all.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:46:12 +01:00
jschoubben 43d4f511c5 Drop the lease write's findings, the controller's artifacts-gone and the issuing gate (issue 1126, B9)
ADR 0590 supersedes the lease write: mesh-grants says no lease event any
more, so the watchman's lease-altered and removal-failed findings and
their row had nothing to hear. The ended controller's consumers are
removed as hand acts, so their deletion is no consumer lost; only the
one on DEAD_LETTER_NOTICES is still read, since its removal hands the
notices to the watchman. The builder hears artifacts-gone from the
artifact sweep seat alone, and mesh-enrolment's tokens no longer report
an issuing gate that nothing can turn off.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:35:55 +01:00
jschoubben f7f3518a04 Take the controller out of the judge, the checker's siblings and the assigner (issue 1126, B8)
The controller ended by its unassignment (ADR 0590) and mesh-sdk drops
its seat, the grants seat and placement's adopt and hand-back. The
indexer's judge still carried the controller's seat entry, the old
removal words and the two placement verbs, and counted 58 seats; the
checker still placed mesh-controller beside the node-engine's and the
catalogue's tests and failed a node-engine check without its trunk; the
assigner still asked a compose of the running controller, served adopt
and hand-back as stubs, and kept a policy-split finding that could no
longer arise.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:32:56 +01:00
jschoubben d4aa1fe11b Hear the keeper alone and stop reading the controller's lease in mesh-grants (issue 1126, B7b)
ADR 0590 §3 ends the choice of a condition source by the keeper's
take-over key: the controller is unassigned and the keeper is the one
source. The part still read the take-over key, fell back to the
controller's condition events when it was absent, and read and watched
the controller's lease, which nothing writes any more. It now hears the
keeper's events alone and reads neither.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:27:19 +01:00
jschoubben 14046f9b14 Remove the lease write, take-over and their tools from mesh-grants (issue 1126, B7a)
The controller ended by its unassignment (ADR 0590), which supersedes the
lease write and its hand-back whole and makes the grants part's list the
only one the bus node follows. The lease write, its check, the removal
marker, the take-over and hand-back asks, their five tools, the seat
claim and the lease and takeover states had nothing left to act on; the
membership notice on its subject had no reader. A whole-module test on
the bus replaces the lease write's: with no controller at all, the part
publishes a membership in its state alone, makes the module's consumer
and writes the key the bus node follows.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:25:59 +01:00
mesh-tools be88162cde Merge pull request 'Without the controller: mesh-grants makes consumers, enrolment issues, the watchman keeps dead letters, and the controller cannot be placed again (issue 1126)' (#137) from fix/1126-gates-delivery into main 2026-11-02 09:51:14 +00:00
jschoubben a192052d82 Refuse any placement of the controller (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@be88162cde31
mesh/check being carried to a192052d82fe on be88162cde31 | from 0490043990e2
The grants part, the enrolment seat and the watchman now act without
reading the lease, so a controller placed again would write beside them
with nothing saying so. The assigner refuses every proposal, seed or
joining node's write that would place mesh-controller, naming why.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:38:22 +01:00
jschoubben 1dae2de835 Issue enrolments and keep dead letters without the lease (issue 1126, A2)
The controller ended by its unassignment, so its lease key is absent for
good and mesh-sdk reads that as not handed away: mesh-enrolment's issue
stayed off and the watchman never took DEAD_LETTER_NOTICES, whose notices
age out after a week. Both now act without reading the lease; the
watchman still waits, and says so, while the controller's consumer on the
notices stands. Neither manifest reads the lease any more.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:35:40 +01:00
mesh-tools 0490043990 Merge pull request 'view_reach answers one module's delivery reach' (#136) from fix/view-reach-filter into main 2026-11-02 09:33:51 +00:00
jschoubben e094e362f4 Run mesh-grants' handed behaviour without the lease (issue 1126, A1)
The controller ended by its unassignment and its lease key expired, so
every gate on "the lease is handed away" stayed shut for good: no account
consumer was created or matched, grant-missing was never raised, the
enrolment users kept the controller's subject, memberships were still
said on their subject and the sealed key never said LeaseHanded. The
handed branch now always runs, and a hand-back is gone.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:33:14 +01:00
jschoubben 8850a5673e view_reach takes a module, so one module's reach is asked without reading the whole repository
mesh/delivery merged: source.main moved to novox/mesh-delivery@0490043990e2
mesh/check being carried to 8850a5673e61 on 0490043990e2 | from cc9c681fa348
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:24:54 +01:00
mesh-tools cc9c681fa3 Merge pull request 'The lease write stops the controller on its placed node and reads the operator's answer from the right bucket (issue 774)' (#135) from fix/774-lease-write-node into main 2026-11-02 08:52:47 +00:00
jschoubben 8f87dbe94d mesh-delivery-teller: read asks from the router's bucket the grant names
mesh/delivery merged: source.main moved to novox/mesh-delivery@cc9c681fa348
mesh/check being carried to 8f87dbe94d76 on cc9c681fa348 | from f1c33439292d
The teller read the router's record from KV_asks, a bucket no composed grant
names, so every read was denied and its operator answers could not be read
back. It now reads messenger_asks, as mesh-grants does. The integration bed
grants the real subject and reads a record on the teller's credential, so it
no longer agrees with the mistake. (issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 09:44:26 +01:00
jschoubben 7cee3842b8 mesh-grants: stop the controller on its placed node and read asks from the router's bucket
The lease write addressed node-service-manager at the lease holder's host, a
hostname no node carries ("novox-be" on novox), so the operator's yes timed out
at its first step and failed without a log line. The node now comes from
placement (exactly one node placing mesh-controller, else refused by name), a
failed act is logged beside its condition, and the asks record is read from
messenger_asks, the bucket the composed grant names, instead of asks, which no
grant allows. The fixtures give the lease a host unlike the node, so the tests
no longer agree with the mistake. (issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 09:42:19 +01:00
mesh-tools f1c3343929 Merge pull request 'mesh-enrolment holds the enrolment seat; placement takes add-node and remove-node; the grants part composes enrolling and rotating nodes (issue 774)' (#134) from feat/l3-enrol into main 2026-11-02 07:56:39 +00:00
jschoubben 63c8687166 Let a bus module's tools bundle declare what it invokes (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@f1c33439292d
mesh/delivery-group group feat/l3-enrol merged: every member is merged
mesh/check failed (2 tests): integration: mesh-mail-reader/cmd/mesh-mail-reader.TestAcceptanceOneMatteringMailSendsExactlyOneTellNamingIt: timed out w…
The judge refused nats's manifest for its invokes (seat:mesh-hand-acts.hand-act,
which nats_bucket_expiry asks to record its hand act), and with one manifest
refused the grants part holds the whole bus user list, freezing every build
that adds bus permissions. The bundle asks through the node's tool runner,
which buscompose grants every tool and seat verb, so the invokes are granted:
the same condition that already lets the bundle declare emits (ADR 0357).
ADR 0235's rule stands for a bus module with no bundle.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 02:21:05 +01:00
jschoubben b13f65674b Pin mesh-sdk at dff0ce2, the enrolment group's rebased head, so every member builds the same contract (issue 774)
mesh/delivery-group group feat/l3-enrol checking: 4 of 4 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking b13f65674b35 on 9125fcc71206 | novox/mesh-delivery#134 in group feat/l3-enrol
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:48:53 +01:00
jschoubben e6ee939bd4 The indexer's closed seat set counts fifty-eight with mesh-enrolment (ADR 0587 §1)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:48:53 +01:00
jschoubben bde60130a8 The assigner's add-node test writes no lease: since the removal group the assigner reads none and its bed has no lease bucket
The enrolment's issue waits for the lease write itself (ADR 0587 §4); the assigner takes an add-node on its warrant at any time (ADR 0567 §8). (issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:48:53 +01:00
jschoubben 795724f5ca mesh-enrolment: catch up only on an ask answered yes, and drop the address key first at confirm, decline and refused add
ADR 0589 version 29, clause 6. At start and after a catch-up an address key is an expiry never heard only when its
enrolment key is gone and the router records its ask answered yes; one declined or not answered yet is no expiry, and one
not answered stays until its ask ends. A confirm, a decline and a refused add delete the address key before their other
steps, so one stopped midway never leaves a key the catch-up would take for an expiry and unplace a node that joined. A
refused add (mesh-placement's refusal of the proposal, not a call that did not reach it) now ends the enrolment as refused;
a call that failed still waits for again. Tests: rows in the end table (after a confirm stopped before its enrolment key
went, after a decline, after a refused add, a declined ask, an ask not answered: nothing unplaced) and the integration test's
holder-was-down case reading the yes; no new function.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben 44fee9eb0e mesh-enrolment: end an enrolment that expired unseen from its address key, taking the node out of placement
ADR 0589 clause 6: a placed but unconfirmed node is always taken out again, on the warrant of its ask. The address key now
names the ask and the issue's time (delivery.EnrolmentAddress, mesh-sdk c7130e3), written just before the ask is raised and
with no expiry of its own, so it outlives the enrolment key; the holder deletes it at every end. An expiry, seen or not, ends
the enrolment in this order, each step safe to repeat: remove-node on the ask's warrant (only when placement holds the
node), enrolment-expired naming the rebuilt issued event (delivery.EnrolmentIssuedID), the vault's drop of the token, and
the address key last. Reconcile finds every address key that names an ask and whose enrolment key is gone (expired while
the holder was down, before its watch caught up, or an end that stopped) and ends it; it runs once the watch has caught up,
on every expiry, and again with a growing wait while a step fails. An address reserved by an issue under way (no ask yet) is
left alone. Tests: rows in the end table (expired unseen; the holder down with a drop that fails once; a reservation under
way) and the integration test's expiry case (the address key goes; a holder that was down unplaces on its pass); no new
function.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben cbcbb9644b mesh-enrolment: keep the tunnel key in delivery.Enrolment's own field
mesh-sdk 22c03af carries the machine's tunnel key on the enrolment (OverlayKey, overlay-key, required at every stage), so the
holder's wrapper that kept it beside the contract is gone; the wire is unchanged. Test fixtures give the seeded enrolments a
real tunnel key, which the contract now requires.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben d1a256b429 mesh-grants: give the enrolment fixture its tunnel key, which an enrolment now requires
mesh-sdk 22c03af (feat/l3-enrol) requires overlay-key on every enrolment; the fixture of the enrolment-user subtest carries one.

(issue 774, review round 1b)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben 40bbc7e0d3 mesh-enrolment: take the tunnel key at issue, place the node on the yes, take it off again at an unconfirmed end
The operator's decision of 2026-11-02 01:05, on mesh-sdk 4224665:
- issue takes overlay_key, the machine's tunnel public key as `nox-mesh-host key` prints it, and refuses a missing one. The
  token's first tunnel is whole: key, address, range (the hub's tunnel range in placement, the one the controller's token
  carries today; a hub with none is a finding, never a guess), hub key and hub endpoint.
- On the operator's yes the node is proposed into placement at once (add-node with its tunnel key alone, on the warrant), and
  only then is the token made, so the hub peers it before redeem. Redeem writes no placement: it refuses a tunnel key other
  than the one issued for, or a node placement does not hold yet, and the certificate's call carries all four keys presented.
- A revoke, a refusal or an expiry before confirm takes the node off placement (remove-node, same warrant) before the end is
  said, and the end brings the token's drop.
- The assigner applies a remove-node: the node's key deleted at once, with no ask, placement-changed carrying the warrant.
- The tunnel key is kept as one more member of the node key's JSON, beside delivery.Enrolment, until the contract carries it.
Tests: rows in the existing tables and both integration tests (the token's Missing() is empty; the node is placed before
redeem and leaves placement when its enrolment expires); no function added, mesh-delivery stays at 816.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben 70f13a3775 mesh-enrolment: place before the vault is asked, read the grants pass whole, name the expired enrolment's issued event, carry the first tunnel
- redeem writes the node into placement before it asks the vault for the account and the certificate of the serving key
  presented (the vault reads the node's address from placement); a retry by the same keys still finishes (blocker 3).
- loadedVia reads mesh-grants.user-list's pass as the part's whole Result, the list state under user_list; the fixture
  testdata/grants-pass.json was written by mesh-grants' own Result, not by hand (blocker 4).
- enrolment-expired is delivery.EnrolmentExpiry with IssuedID, through its codec; a key that expired before its content was
  seen names no issued event, so nothing is said and its token is dropped at once (blocker 6).
- issue builds the joining node's first tunnel from placement (the hub, its endpoint, the node's address) and the hub's
  record (its tunnel key) and passes it to enrolment-token; no hub, or a hub with no key or endpoint, is a finding of issue
  (ADR 0587 §3, insight of 2026-11-02).
- The assigner takes an add-node only from mesh-enrolment's own bus user (stdio.Callers), never person.mesh-enrolment.
- The watchman raises machine.<node>.node-account-fallback from a node's health report's fall back and clears it when the
  report no longer carries one (ADR 0583 §6, blocker 7).
- mesh-enrolment's ten unit test functions are two table tests now (blocker 8): the repository is back at 816.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:44 +01:00
jschoubben 189e64c486 mesh-enrolment: its integration test, whole on a real bus
The holder's code over a real nats-server bucket with a time to live per key: issue, the operator's yes, redeem as enrol.ace,
the node confirming as the login its key of node-logins names (the stand-in vault writes it at node-account), node-enrolled
and placement-changed said on their subjects, the user enrol.ace composed while the key is live and gone with it, the token
dropped only after the end; and a token never redeemed expires on the bus, is heard, says enrolment-expired and is dropped.
Found by it: a bucket with a time to live per key must keep one revision per key, or the bus expires nothing (nats-server
2.11.17), so the state enrolments declares no history. Paid for: the integration test is paid by merging the builder's
archive-only and root-module integration tests into one table test; the module's ten unit tests by merging the assigner's
three small composition tests into one table (net 2 of 10 paid; the rest is for the lead, the repository being over budget).

(issue 774, 774/S60)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:44 +01:00
jschoubben 1a1d3a1ecc Compose both of a rotating node's logins from the vault's node-logins, so its engine can test-dial the new one
ADR 0583 §1 (insight 2026-11-01) and §2: the grants part reads the vault's node-logins as its named reader and
composes a node user per login, both while the node rotates and the new one alone after rotation-done; each node's
engine reads its own node's key alone (composed by mesh-sdk while the vault serves rotate-node). The part follows the
state by watch, so a rotation begun or ended composes again at once. A second login is judged its node's when the
list is compared (Placed), and a hash changing by a rotation node-logins names is not counted as a writer changing
several hashes at once (sealed.go). The state not on the bus yet is a standing gap: every node logs in as node.<n>,
as today. Tested as a subtest of TestGrants, no new test function. Built against the group's mesh-sdk head 5000774
through a go.work outside the worktree; the pin is the lead's.

(issue 774, 774/S53)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:44 +01:00
jschoubben 288364f28d Compose each enrol.<node> user from the seat mesh-enrolment's keys, so a token issued by the seat can be redeemed
ADR 0587 §4: the grants part composes enrol.<node> from a key at live and the vault's hash, granted only the seat's
redeem and its own inbox; §5: when the key ends the user goes. The part reads mesh-enrolment.enrolments as its named
reader and follows it by watch: a key reaching live composes the user, and its end (by the holder, or by the bus at
its expiry) is a change of the key that takes it out, so no consume of the seat's ending events is added beside the
watch (the subtask named one; the watch already hears every end, the bus's expiry included). From the lease write the
user loses the controller's enrolment subject (§6) and the standing gap of the controller's tokens is dropped; before
it the gap stands. Tested as a subtest of TestGrants, no new test function. Built against the group's mesh-sdk head
5000774 through a go.work outside the worktree; the pin is the lead's.

(issue 774, 774/S52)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:32 +01:00
jschoubben 447bcf3a6b The assigner takes an add-node from mesh-enrolment and writes the joining node with no ask
propose takes the enrolment's warrant {node, address, ask, warrant} and refuses an add-node from any caller but
mesh-enrolment, with no warrant or with a warrant naming another node or address (delivery.AddNodeRefusal); the joining node
counts as having a record for this one change, as ADR 0587 §7 changes ADR 0567 §2. The assigner's process then writes
placement.<node> at once, with no node to answer and no ask raised, and says placement-changed carrying the enrolment's
warrant; the hand-act record of that warrant is the enrolment's, made when it was given. The holder of mesh-enrolment calls
propose with the change, the keys and the warrant read from the router's record of the ask. Tests: rows appended to the
seat's verbs test and a case to the whole-seat integration test; none added, so none to pay for (the budget is paid in
774/S60).

(issue 774, 774/S59)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:12 +01:00
jschoubben 07d26adcd3 mesh-enrolment: read the group's mesh-sdk contract instead of its own copies of the shapes
The seat, the enrolment state, redeem and its proof, the answer the vault signs, and the vault's two verbs' answers are now
mesh-sdk's (catalogue and delivery), which the vault, the grants part and the engine read too; the proof is the contract's
RedeemProof, the answer carries the login, account, certificate and the vault's signing key (read from its state signers),
and the bucket's values are written through the contract's codecs. The expiry event names the issued event by message id as
well as by the sequence, which a bundle cannot learn. The module builds against the group's mesh-sdk head through a go.work
outside the tree; the lead re-pins go.mod before the push. Tests merged into the same functions; none added.

(issue 774, 774/S54-S58)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben 02214cf633 mesh-enrolment: revoke, the expiry marker and the drop of the token
revoke ends an enrolment at any stage without an Approve and records it as an act done by hand (the vault retires an
account minted for a redeem under way); the bus's expiry marker of a key at live says enrolment-expired with the issued
event; the durable consumer of the seat's own ending events asks the vault to drop the token and its hash, acknowledging
only once the vault answers, never before an end. No event or key holds a token (ADR 0587 §5, §8). Run starts the watch,
the consumer and the operator's answers. Tests added: two table-style tests.

(issue 774, 774/S58)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben f904ff425e mesh-enrolment: confirm closes an enrolment on the node's own call
Only the node dialled as node.<node> confirms (the runtime's proof of the caller is the confirmation, ADR 0587 §6); the
holder then says node-enrolled with the key's content and deletes the address key (while it still names the node) and the
node's key. A stop between the event and the deletes is finished by calling confirm again; the event keeps one message.
release now answers what it could not delete. Tests added: one table-style test.

(issue 774, 774/S57)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben 89370b6f5c mesh-enrolment: redeem, the joining machine's one call
redeem answers only enrol.<node> as the runtime proves it, checks the proof against the identity key presented, spends the
token once by compare-and-set (the keys lose their expiry), asks the vault for the node's account and certificate, writes
the node into placement on the issue's warrant, answers once the bus node loaded node.<node>, signed by the vault; the
same keys called again finish what is not done and answer the same (ADR 0587 §5-§7). The proof is mesh-host's EnrolProof
with the node's name in the token's place, since the seat never holds the secret. Tests added: two table-style tests.

(issue 774, 774/S56)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben 4edd0e526c mesh-enrolment: issue, the operator's ask and the warrant, and again
issue refuses at once naming every finding (a bad name, a name a node record, a placement key or an open enrolment
holds, an expiry past a day, any call before the lease write), reserves the lowest free address by creating its key,
writes the enrolment at asked and asks the operator (ADR 0587 §4). On the yes the warrant is recorded, the vault
makes the token and both keys go live with the bus's expiry; a no or an expired ask takes both keys and says
enrolment-refused. Tests added: three table-style tests; none deleted here (paid with 774/S59's merges).

(issue 774, 774/S55)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben 5d5ee69775 Add the module mesh-enrolment: the seat's skeleton, its bucket of open enrolments and the tokens verb
The holder of the seat mesh-enrolment (ADR 0587 §1, §2, §8) runs as a bundle on its own account, keeps the bucket
enrolments with a time to live per key, and answers tokens: every open enrolment or one node's with stage, expiry,
ask and revision, the bucket's fill, whether issuing is on and why, and for one node at live the token sealed by
the vault and relayed, kept nowhere. Tests added: two (one table-style test of tokens, the manifest test); the
repository is over its budget, paid by merging tests in the assigner's seat tests in 774/S59's commit.

(issue 774, 774/S54)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
mesh-tools 9125fcc712 Merge pull request 'mesh-grants writes the controller's lease away; the assigner and the grants part stop reading the controller (issue 774)' (#133) from feat/l3-removal into main 2026-11-02 00:35:38 +00:00
jschoubben 17ef1d577d Pin mesh-sdk at c21d8fd, the removal group's head, so every member builds the same contract (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@9125fcc71206
mesh/delivery-group group feat/l3-removal merged: every member is merged
mesh/check failed: mesh-build-go: the check of novox/mesh-catalog#590: the ask mvyhumdm860c7f's step on shanks did not end within 30m0s of its start |…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:53:42 +01:00
jschoubben da27cd1593 Name the lease write's earlier steps by their issues, not milestone labels
(issue 774, 774/S14)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:51:06 +01:00
jschoubben aa6a4e7a59 The indexer's closed seat set counts fifty-seven with mesh-grants
ADR 0582 clause 1 adds the seat mesh-grants to mesh-sdk's set; the indexer's guard of the closed set counts it.

(issue 774, 774/S1)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:51:06 +01:00
jschoubben 8bf45fbafe Integration test: mesh-grants writes the lease and hands it back on a real bus
ADR 0582 "How it is checked": against an embedded bus, a stand-in controller holds the lease in a bucket with an
expiry (2 s standing in for 15 s, through grants.LeaseExpiry, so the wait past it is seconds), with stand-in
node-service-manager and bus-tool seats (the expiry tool acting on the real bucket's stream, as nats-tools'
SetBucketExpiry does). On the warrant the key holder names mesh-grants, the expiry is gone, the controller's
create of the key is refused, and the lease still reads handed away past the old expiry; on the hand-back's
warrant the expiry is back and the controller takes the lease again. mesh-grants requires nats-server for it,
at mesh-sdk's version.

Budget: TestTheLeaseWriteOnTheBus (new, integration) is paid by merging the checker's
TestTheCheckerAwaitsTheAssignersComposition into TestTheCheckersLineSaysTheAssignersComposition as its first case.

(issue 774, 774/S23)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:40:00 +01:00
jschoubben e18be5cabe The watchman raises lease-altered and removal-failed as findings, and clears them when the lease and the removal read right
ADR 0582 clauses 4 and 6: mesh-grants says lease-altered and removal-failed "as a finding the watchman raises".
The watchman hears the seat mesh-grants' three events on core subjects (mesh-sdk WatchmanHears, its grant and
its subscriptions from one list, rather than a consume and a durable consumer), raises lease-altered naming
what changed and clears it at the next lease-handed, and raises removal-failed naming what failed and clears it
once what it names reads gone: the controller's consumer of DEAD_LETTER_NOTICES by its deletion advisory
(heard already), the stream ASSIGNMENTS by its own. The finding DEAD_LETTER_NOTICES.controller stays as it is.

Test: a row in TestTheWatchmansFindings; no new function.

(issue 774, 774/S22)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:37:54 +01:00
jschoubben a78fa3db9d mesh-grants reads placement alone and hears no controller event; this repository's manifests depend on the controller only by the lease
ADR 0549 §5, ADR 0564 §9, ADR 0571 §10: the part no longer reads mesh-controller.assignments (the union with
placement goes; placement not read is a failed read that places nothing) nor consumes the controller's
condition-raised and condition-cleared, the keeper's being the one source. ADR 0571 §13's test for this
repository replaces the guard of ADR 0549 §6 and its list ../assignments-readers, now empty and deleted: at
phase write no module.json here depends on mesh-controller beyond the lease and the three kept to the write
(catalogue.ManifestRefusals).

Tests: in TestGrants the fixture places through the seat's placement (assign, unassign), the subtest of the
stale assignments becomes 774/S29/AC1 (placement alone gives the fixture's three memberships with no read,
consume or call of the controller; the fixture is the suite's own, whose assignments named the same modules,
not a copy of live memberships), and the assignments-readers subtest becomes 774/S24/AC1. No new test function.

(issue 774, 774/S29, 774/S24)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:35:59 +01:00
jschoubben b81d7475e8 mesh-grants judges the controller's dependents with mesh-sdk's judge, and its keep list holds the three kept to the write
ADR 0571 §13: the judge moved to mesh-sdk (catalogue.Dependencies, Judge, KeepList; issue 774, 774/S4) so the
manifest tests of three repositories and grants_lease_check give one judgement; mesh-grants re-exports it by
its names. The keep list there holds the three kept to the write alone (774/S30), superseding the keep of
pacman's read at 4a84370. grants_lease_check no longer counts the controller's standing assignments, which
nothing reads any more.

Tests: TestTheLeaseCheckJudgesEveryDependencyOnTheController keeps the tool's own rows (what it read, the
lease, the streams); its dependency rows moved to mesh-sdk's TestTheLeaseWritesCheckJudgesEveryDependencyOnTheController
(774/S24's replacement of the moved rows), and TestTheKeepListNamesItsRecordsAndThreeToTheWrite went with them.

(issue 774, 774/S30, 774/S24)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:35:59 +01:00
jschoubben 7ec6767418 mesh-grants holds the seat mesh-grants and writes the controller's lease on the operator's warrant
ADR 0582: lease-write is refused, naming each check, until the lease check at phase write and every condition
of clause 2 read green (design 47's earlier issues resolved, every node trusting the vault's key, mesh-config's
compare empty from the indexer, the upgrade compare empty, the keeper's take-over done, the vault minting with
nothing held back, the bus module listing its expiry and removal tools). It asks the operator once; the yes
binds the act, the holder, the ask and the checks' digest, and the checks are read again on the warrant (one
that reads otherwise refuses it, named). Then, each step read back: node-service-manager.stop on the
controller's node, the list composed with Records.LeaseCut and waited on until the bus node's have says it
loaded it, the expiry removed through nats_bucket_expiry, the key holder written (with the ask and the time),
lease-handed said, the act recorded through mesh-hand-acts.warranted, and removal-due written with a day to
live. lease-altered is said once when the key or the expiry differs; lease-hand-back reverses it for a day
and is refused once the removal ran; removal-due's marker alone starts the removal of the controller's
consumer and the stream ASSIGNMENTS, recorded as hand acts, and a failure says removal-failed and writes the
key again with a day (grants_removal_due is the recovery for a missed marker). From the write memberships are
written to the state alone (ADR 0571 clause 10), the user-list key says lease-handed, and the user list's
hand-back answers gone (ADR 0547 §11). Every step waits on an event, never a timer.

Bent rule: S14-S21 are one commit, not eight: the steps are one state machine in leasewrite.go, and splitting
it by hunks would review the same file eight times for nothing it protects.

Tests: TestTheLeaseWrite (new, one function with a subtest per subtask), paid by the deletion of
TestTheKeepListNamesItsRecordsAndThreeToTheWrite (moved to mesh-sdk, S4/S30) and the assigner's two deletions
(S25). Fixtures shaped from each writer's code: systemd's Manager.Status, nats-tools' BucketExpirySet,
ConsumerRemoved and StreamRemoved, mesh-issues' Brief, mesh-delivery-view's TrustsAnswer, mesh-config's
Comparison, the assigner's upgrade compare, mesh-vault's vault_status.

(issue 774, 774/S14, 774/S15, 774/S16, 774/S17, 774/S18, 774/S19, 774/S20, 774/S21)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:31:35 +01:00
jschoubben 30ec4329e0 The assigner's integration tests place their modules and name the node's own composition; a placed node's backup answer stays in its want
The integration tests put placement (not assignments) and the node's own composition (not the controller's) on the bus; the
three that tested the controller's judgements of a move (waiting for a composition, a record move, a user-list revision) go,
their unit counterparts standing. Run locally against the embedded bus, all of this module's integration tests pass.
Found by them: the answer to a failed backup was dropped from a placed node's want at every pass, because the carry-over
compared the want's digest, which a placed node's want does not carry; fixed in wants.go, with a case in the existing unit test.
Tests: three integration tests deleted, none added; one case added to an existing unit test.

(issue 774, 774/S26, 774/S27)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:30:51 +01:00
jschoubben 3bae0d8fa2 Placement's adopt and hand-back are answered as gone, naming their records, and change nothing (ADR 0547 §11, ADR 0567 §11)
The controller's placement-export is no longer called, its answer reader, the seed and the hand-back flow of the holder are
removed, and a seed or hand-back proposal a previous process left open ends at its next pass.
Tests: the adopt, hand-back and seed cases of the seat's verb test and of Propose are replaced by one gone-case; the export
reader's table and the seed test go.

(issue 774, 774/S28)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:21:46 +01:00
jschoubben 7bd18306d4 The assigner reads no composition of the controller: upgrade compare and rule 7 read the digest a node's own have names (ADR 0571 §8)
mesh-controller.composed leaves the reads and the watches; upgrade compare lists the node composition digests of the haves
instead of comparing with the controller's policies, which the controller no longer writes.

(issue 774, 774/S27)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:20:13 +01:00
jschoubben 1302eb9e1f The assigner reads no assignment and no lease: the placement alone names where modules run (ADR 0567 §8, ADR 0549 §5)
Every handed-away branch is the only one: world.assigned is the placement, the pair comparison and its conditions go, the
lease and assignments buckets are no longer opened, watched or declared, and a proposal of any kind is taken.
Tests: the pair-comparison test and the controller-composition record-ask test are replaced by one placement test; the
rest are edited to place their modules instead of assigning them.

(issue 774, 774/S26)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:18:50 +01:00
jschoubben f2ed0938ce The assigner calls no verb of the controller after a placement is written: placement is the one source (ADR 0567 §7, §8)
Drops callAll, the refused-call conditions and the pin reads; the conditions a previous process raised are cleared once.
Tests: deletes four (the refused-call condition, its clearing, the contract of the calls, the condition's shape) and adds one.

(issue 774, 774/S25)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:13:50 +01:00
jschoubben 4a84370b2f mesh-grants: the lease check keeps pacman's read of declared-packages until the write's delivery
pacman still falls back to the controller's declared-packages for a node whose engine has
not written its own packages (ADR 0565 §6), so the check named that read as ended and would
refuse the lease write. It is now a kept read, dropped with the others at the write's own
delivery (issue 774, design 47 L3).

Quick fix chosen over removing pacman's fallback: it matches the record as it stands, and
the read goes with the other kept reads at the write's delivery.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 23:27:29 +01:00
mesh-tools 7d1b8864cc Merge pull request 'mesh-grants pins the mesh-sdk that grants each engine a read of its own declared-packages key (ADR 0565 §6, issue 1953)' (#132) from feat/node-packages into main 2026-11-01 17:02:47 +00:00
jschoubben b18c62f671 Pin the grants part to the group's mesh-sdk head b2a27ed5, so each engine may read its node's key of the packages mirror (ADR 0565 clause 6, issue 1953)
mesh/delivery merged: source.main moved to novox/mesh-delivery@7d1b8864cc08
mesh/delivery-group group feat/node-packages merged: every member is merged
mesh/check checking b18c62f671e1 on cc38ca3f8dc8 | novox/mesh-delivery#132 in group feat/node-packages
The node seat's verb packages compares the engine's key with the controller's mirror until the lease write; the grants part composes the engine's read of its own node's mirror key only from this mesh-sdk.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 16:24:04 +01:00
mesh-tools cc38ca3f8d Merge pull request 'Agree a toolchain by strict majority; a node on another version sits out (ADR 0584, issue 1578)' (#131) from fix/toolchain-majority into main 2026-11-01 15:04:20 +00:00
jschoubben d739516056 Name the agreed toolchain in the checker's check asks; pin mesh-sdk 67354f8 (ADR 0584, issue 1578)
mesh/delivery merged: source.main moved to novox/mesh-delivery@cc38ca3f8dc8
mesh/delivery-group group fix/toolchain-majority merged: every member is merged
mesh/check pass d739516056dd on 55a506975b01 | unit 7056 run, 243 kept, 16 skipped | integration 7194 run, 275 kept, 25 skipped | over bound: unit, in…
A check ask named no toolchain, so a node on the minority version could run it and KeepPasses record the pass as run on the agreed one. The check ask now carries the agreed toolchain and a node off it declines (mesh-sdk), for the checker's decline loop to put again. Pins the builder, indexer and checker.

Known limit: the checker puts a declined check ask again at once, with no pause; the declining puller skips that kind until its next pull wait, which paces a lone minority node.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:38:16 +01:00
jschoubben a6c225eb45 Pause before a declined ask is put again with a context's bound, not a timer (ADR 0437 §8, issue 1578)
The builder's manifest test refuses time.After. The previous commit's pause used it.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:36:52 +01:00
jschoubben 5085b681e9 Count no decline as an asking; bound declines by time and name the nodes (ADR 0584, issue 1578)
A build node off the agreed toolchain declines the ask and, idle, takes the ask put again at once, so three declines used the three askings and failed the build. A decline now uses no asking: the ask is put again after a short pause, and past a bound from the first decline the key fails naming the declining nodes and their reasons. A person reads which node sits out in the key's step text, which the view's builder_builds answers as stored.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:36:23 +01:00
jschoubben 9e407f8f76 Build with the toolchain a majority of build nodes report; say which nodes sit out (ADR 0584, issue 1578)
One build node on another version refused every build of the language. The builder and the checker now follow mesh-sdk's majority agreement; the key's step names a node that sits out ("ace sits out: typescript 26.12.1, agreed 26.13.0"), and a build ask a minority node declined is put again for a node on the agreed version. Check steps compare no toolchain, so clause 2 already holds for them. Pins builder, indexer and checker to mesh-sdk 71be15f.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:26:55 +01:00
jschoubben 8545e7fc9f Cite ADR 0584 where the builder waits on a toolchain the nodes do not agree on (issue 1578)
Comment only; the majority rule itself is in mesh-sdk's AgreedToolchain.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:22:04 +01:00
mesh-tools 55a506975b Merge pull request 'The assigner, builder and grants read off the controller; upgrade compare by watch (design 47 L5, L7)' (#130) from feat/l5-l7-main into main 2026-11-01 13:53:33 +00:00
jschoubben b7dfe4c637 builder: the held fake in its integration tests answers on the artifact sweep's seat, which the builder now asks instead of the controller
mesh/delivery merged: source.main moved to novox/mesh-delivery@55a506975b01
mesh/delivery-group group feat/l5-l7-main merged: every member is merged
mesh/check checking b7dfe4c6375a on 58748c2be161 | novox/mesh-delivery#130 in group feat/l5-l7-main
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 10:58:15 +01:00
jschoubben c4b05a3e93 Pin group A's mesh-sdk head on main
mesh/delivery-group group feat/l5-l7-main rejected: composed together: the checker's verdict at the members' heads (e32e2c9c4fea): integration: mesh-delivery-b…
mesh/delivery closed: a newer head of the same pull request
mesh/check failed (2 tests): integration: mesh-delivery-builder/cmd/mesh-delivery-builder.TestADoneKeyWhoseBlobIsGoneIsBuiltAgainAtStart: builds.ca-tr…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:57:47 +01:00
jschoubben 38f2401f17 mesh-grants and the builder pin mesh-sdk at 8311ecb
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:28 +01:00
jschoubben c8b041ab37 mesh-wants upgrade reads through mesh-sdk's watched reader, keeping no mirror of its own
One helper serves every part that reads whole states (ADR 0580 clause 3): the assigner's own watched mirror goes, its
compare reads the large states from stdio.WatchedStates and each manifest once per digest. mesh-sdk pinned at 8311ecb.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:22 +01:00
jschoubben f52f47e1ff mesh-grants and the assigner read the grants part's key alone, root-free is each engine's word, and the hand-act log is the seat's alone (ADR 0571 clauses 9 and 11, ADR 0565 clause 11, issues 1953 and 1955)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:22 +01:00
jschoubben 213ebe7cfd The builder asks the artifact sweep's seat for held and no longer the controller (ADR 0571 clause 2, issue 1955)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:22 +01:00
jschoubben 90e278cb97 mesh-wants upgrade: read the large states by one watch each, started once, not a get per key, so compare answers in time (issue 1953 L5f)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:22 +01:00
mesh-tools 58748c2be1 Merge pull request 'The indexer judges every manifest once and says its verdict; the checker judges a change with the same judge (ADR 0578, 0579)' (#129) from feat/judge-once into main 2026-11-01 07:54:35 +00:00
jschoubben 8e9cc58301 Fall back to a captured copy only for mesh-host, which the checker places beside no mesh-delivery check (issue 1585)
mesh/delivery merged: source.main moved to novox/mesh-delivery@58748c2be161
mesh/delivery-group group feat/judge-once merged: every member is merged
mesh/check checking 8e9cc5830119 on b1f262d7d06e | novox/mesh-delivery#129 in group feat/judge-once
Review of fb96216: a missing mesh-sdk or catalogue in a merge check is the checker's fault and fails, never judged
against a stale copy. The captured placement guard is refreshed to mesh-host's main 233bb23.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 04:49:23 +01:00
jschoubben fb96216e46 Read a sibling the checker does not place beside mesh-delivery from its captured copy, said (issue 1585)
The judge's tests moved from mesh-sdk, where mesh-host was beside them; the checker places only mesh-sdk and the
catalogue beside mesh-delivery (besideRules), so the node-engine's path rules failed as not beside. A sibling the run
holds beside is read there; one it does not is read from testdata/beside and the log says which; only a sibling
with neither fails.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 04:47:22 +01:00
jschoubben 5059fef3e6 Expect the mesh packages a merged message carries in the forge bridge's whole test (ADR 0573 §2)
mesh/delivery-group group feat/judge-once rejected: composed together: the checker's verdict at the members' heads (253f35815198): unit: mesh-delivery-indexer/…
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: mesh-delivery-indexer/judge.TestTheResolvedPathRulesAreTheNodeEnginesOwn: mesh-host is not beside this check in MESH_CHECK_BE…
Issue 1585: the bridge names the mesh packages each module's go.sum names (builder-1585b's fold); its integration test
still wanted none. All nine modules' integration suites pass locally (the lead allowed a whole local run this round).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 04:20:05 +01:00
jschoubben 564a04b45c Pin the group's mesh-sdk head after the gofmt fix
mesh/delivery-group group feat/judge-once checking: 6 of 6 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 564a04b45c42 on b1f262d7d06e | novox/mesh-delivery#129 in group feat/judge-once
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 03:26:58 +01:00
427 changed files with 50065 additions and 10395 deletions
-7
View File
@@ -1,7 +0,0 @@
# The modules of this repository that still read the controller's assignments (mesh-controller.assignments), each with
# why. mesh-grants' test (TestGrants, "assignments readers") refuses any other module.json that reads them, and an entry
# here whose module no longer does: the list only shrinks, and it is empty in the first change after the controller's
# lease is written to a holder that is no controller (novox/hq ADR 0549 §3, §6; issue 774). Each module named here
# uses the assignments only while the controller holds the lease (buscompose.LeaseHandedAway), placement alone after it.
mesh-delivery-assigner unioned with the placement until the lease write (assign.Placed with handedAway); placement alone after
mesh-grants unioned with the placement until the lease write (ReadInputs); placement alone after
@@ -1,161 +0,0 @@
package main
import (
"context"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-assigner/internal/assign"
)
// AssignerModule is the asker the assigner's compose asks name.
const AssignerModule = "mesh-delivery-assigner"
// composeAsks are the compose asks the assigner put on the Go build seat's check stream (novox/hq ADR 0482 clause 4,
// issues 1622 and 1623), by what each composes: one ask per key, put once and followed by a watch of its step key until
// it ends, which wakes a pass. No timer: an ask that waits in the stream waits (its backlog is the view's and the
// watchman's to show). A restart forgets them and asks again; the build module keeps a composer built for a commit, so a
// second ask builds nothing.
type composeAsks struct {
ctx context.Context
asker buildrun.Asker
wake func()
logf func(string, ...any)
now func() time.Time
mu sync.Mutex
asks map[string]*composeAsk
}
type composeAsk struct {
id string
cancel context.CancelFunc
ran *assign.Ran
}
func newComposeAsks(ctx context.Context, asker buildrun.Asker, wake func(), logf func(string, ...any)) *composeAsks {
return &composeAsks{ctx: ctx, asker: asker, wake: wake, logf: logf, now: time.Now, asks: map[string]*composeAsk{}}
}
// get is the end of the ask that composes key, or nil and its id while it runs. The first get of a key puts the ask.
// A job its layout refuses ends at once, failed with why: never silent.
func (a *composeAsks) get(key string, job delivery.ComposeJob) (*assign.Ran, string) {
a.mu.Lock()
defer a.mu.Unlock()
if k, ok := a.asks[key]; ok {
return k.ran, k.id
}
ask := delivery.StepAsk{ID: buildstep.NewID(), Kind: delivery.AskCheck, Language: delivery.AskGo, Asker: AssignerModule,
At: a.now(), Compose: &job}
ctx, cancel := context.WithCancel(a.ctx)
k := &composeAsk{id: ask.ID, cancel: cancel}
a.asks[key] = k
if err := a.asker.Put(ctx, ask); err != nil {
a.logf("the compose ask %s for %s could not be put: %v", ask.ID, key, err)
k.ran = &assign.Ran{State: delivery.StepFailed, Reason: "the compose ask could not be put on the Go build seat: " + err.Error()}
cancel()
return k.ran, k.id
}
go func() {
step, err := a.asker.Wait(ctx, ask)
if ctx.Err() != nil {
return
}
ran := assign.RanOf(step)
if err != nil {
ran = assign.Ran{State: delivery.StepFailed, Reason: "the compose ask's step could not be followed: " + err.Error()}
}
a.mu.Lock()
k.ran = &ran
a.mu.Unlock()
a.wake()
}()
return nil, k.id
}
// sweep forgets, and stops following, every ask whose key no pass asked for this time: a candidate gone or moved on.
func (a *composeAsks) sweep(seen map[string]bool) {
a.mu.Lock()
defer a.mu.Unlock()
for key, k := range a.asks {
if !seen[key] {
k.cancel()
delete(a.asks, key)
}
}
}
// composeAtHead is the composition of a group with a mesh-sdk member: the composer at its head and at each reader's
// mesh-sdk, asked of the Go build seat, and the verdict once every ask ended; until then the verdict waits, naming the
// asks. The keys it used are marked in seen.
func (p *passer) composeAtHead(ctx context.Context, m assign.Mesh, ch assign.Change, seen map[string]bool, began time.Time) delivery.CompositionVerdict {
waiting := func(words string) delivery.CompositionVerdict {
return delivery.CompositionVerdict{Group: ch.Group, Heads: ch.Heads, State: delivery.CompositionWaits, Words: words}
}
head, err := assign.HeadJob(m, ch)
if err != nil {
return delivery.CompositionVerdict{Group: ch.Group, Heads: ch.Heads, State: delivery.CompositionFail, SDK: "mesh-sdk " + ch.SDKHead,
Words: "the compose ask at the group's mesh-sdk head cannot be made: " + err.Error()}
}
var pending []string
key := "head." + ch.Group + "." + ch.Heads + "." + ch.SDKHead
seen[key] = true
headRan, id := p.asks.get(key, head)
if headRan == nil {
pending = append(pending, "mesh-sdk "+ch.SDKHead+" (ask "+id+")")
}
var readers []assign.Reader
var notJudged []string
for _, module := range sortedNames(assign.ReaderModules) {
repository := assign.ReaderModules[module]
var commits []string
member := false
if h := ch.Members[repository]; h != "" {
commits, member = []string{h}, true
} else {
commits = p.w.runningCommits(module)
}
if len(commits) == 0 {
notJudged = append(notJudged, "not judged whether "+module+" reads the group's manifests: the builder's log holds no done "+
"trunk build of it, and the group holds no member of "+repository)
continue
}
for _, commit := range commits {
job, err := assign.ReaderJob(ch, module, repository, commit)
if err != nil {
readers = append(readers, assign.Reader{Module: module, Commit: commit, Member: member,
Ran: assign.Ran{State: delivery.StepFailed, Reason: err.Error()}})
continue
}
rkey := "reader." + ch.Group + "." + ch.Heads + "." + module + "." + commit
seen[rkey] = true
ran, rid := p.asks.get(rkey, job)
if ran == nil {
pending = append(pending, module+" at "+commit+" (ask "+rid+")")
continue
}
readers = append(readers, assign.Reader{Module: module, Commit: commit, Member: member, Ran: *ran})
}
}
if len(pending) > 0 {
return waiting("composing on the Go build seat with the mesh-sdk of " + strings.Join(pending, ", "))
}
v := assign.FromComposer(m, ch, *headRan, readers, time.Since(began))
v.Warnings = append(v.Warnings, notJudged...)
return v
}
func sortedNames(m map[string]string) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -1,180 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// holder is a test holder of the Go build seat: it takes every ask from the seat's check worker, as mesh-build-go does,
// and answers a compose ask by running mesh-sdk's composer over the request filled from the indexer's manifests (the
// linked composer standing for the one the build module builds at the asked commit), writing its step on its node's key.
type holder struct {
mu sync.Mutex
asks []delivery.StepAsk
}
func (h *holder) taken() []delivery.StepAsk {
h.mu.Lock()
defer h.mu.Unlock()
return append([]delivery.StepAsk(nil), h.asks...)
}
func (b *bed) hold(t *testing.T) *holder {
t.Helper()
seat, _ := delivery.AskStream(delivery.AskGo)
worker, _ := delivery.AskWorker(delivery.AskGo, delivery.AskCheck)
subject, _ := delivery.AskSubject(delivery.AskGo, delivery.AskCheck)
cons, err := b.js.CreateOrUpdateConsumer(b.ctx, seat, jetstream.ConsumerConfig{Durable: worker, FilterSubject: subject, AckPolicy: jetstream.AckExplicitPolicy})
if err != nil {
t.Fatal(err)
}
steps := b.buckets[delivery.StepsBucket(goBuildModule)]
h := &holder{}
cc, err := cons.Consume(func(m jetstream.Msg) {
a, err := delivery.DecodeStepAsk(m.Data())
if err != nil || a.Compose == nil {
t.Errorf("the seat was put an ask that is not a compose: %v %s", err, m.Data())
m.Ack()
return
}
h.mu.Lock()
h.asks = append(h.asks, a)
h.mu.Unlock()
req, err := composer.Decode(a.Compose.Request)
if err == nil {
err = composer.Fill(&req, func(digest string) ([]byte, error) {
key, _ := delivery.ManifestsKey(digest)
e, err := b.buckets[delivery.ManifestsBucket].Get(b.ctx, key)
if err != nil {
return nil, err
}
man, err := delivery.DecodeManifest(e.Value())
return []byte(man.JSON), err
})
}
step := delivery.Step{ID: a.ID, Tool: "compose", Node: "ace", State: delivery.StepPassed, Started: time.Now().UTC(), Ended: time.Now().UTC()}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
} else if v, err := composer.Run(req); err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
} else {
vr, _ := json.Marshal(v)
step.Result, _ = json.Marshal(delivery.ComposeAnswer{Built: true, Key: "composer-0123456789abcdef", Verdict: vr})
}
raw, err := delivery.EncodeStep(step)
if err != nil {
t.Errorf("the step cannot be written: %v", err)
}
key, _ := delivery.AskStepKey("ace", a.ID)
if _, err := steps.Put(b.ctx, key, raw); err != nil {
t.Errorf("the step %s: %v", key, err)
}
m.Ack()
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(cc.Stop)
return h
}
// 1622/AC1-AC3 and 1623 on the bus: started whole, the assigner composes a group holding a mesh-sdk pull request by
// asking the Go build seat for the composer at the member's head, and for the composer at the mesh-sdk the running
// controller pins; it waits naming the asks while they run, writes the verdict from their answers naming the mesh-sdk the
// head's composer was built with, and asks nothing more on a later pass of the same heads.
func TestAGroupWithAnSDKMemberIsComposedByTheComposerAtItsHead(t *testing.T) {
b := startBed(t, "")
trunk := b.manifest(t, "web", `{"module":"web","version":"1"}`)
changed := b.manifest(t, "web", `{"module":"web","version":"2"}`)
mod, err := delivery.EncodeModule(delivery.Module{FP2: delivery.SourcePrefix2 + strings.Repeat("a", 64), Manifest: trunk, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ModulesBucket, "web", mod)
b.putJSON(t, provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"build-1"}`))
running := strings.Repeat("3", 40)
b.putJSON(t, provider.AssignmentsBucket, "novox.mesh-controller", []byte(`{"node":"novox","module":"mesh-controller","build":"c-1"}`))
b.built(t, "mesh-controller", "novox/mesh-controller", running)
sdkHead := strings.Repeat("4", 40)
heads := []delivery.Head{{Repository: "novox/mesh-sdk", Number: 80, Head: sdkHead}, {Repository: "novox/web", Number: 7, Head: strings.Repeat("2", 40)}}
hash, _ := delivery.HeadsHash(heads)
pulls := []delivery.CandidatePull{{Head: heads[0]}, {Head: heads[1],
Modules: []string{"web"}, Fps: map[string]delivery.CandidateModule{"web": {FP2: delivery.SourcePrefix2 + strings.Repeat("b", 64), Manifest: changed}}}}
cand, err := delivery.EncodeCandidate(delivery.Candidate{Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.CandidatesKey("feat/x")
b.putJSON(t, delivery.CandidatesBucket, ckey, cand)
b.start(t)
key, _ := delivery.CompositionKey("feat/x", hash)
// No holder takes the asks yet: the composition waits, naming them.
waits := b.verdict(t, key, func(v delivery.CompositionVerdict) bool { return v.State == delivery.CompositionWaits })
if !strings.Contains(waits.Words, "mesh-sdk "+sdkHead) || !strings.Contains(waits.Words, "mesh-controller at "+running) {
t.Fatalf("the waiting composition says %q, want the head's and the running controller's asks", waits.Words)
}
h := b.hold(t)
v := b.verdict(t, key, func(v delivery.CompositionVerdict) bool { return v.State != delivery.CompositionWaits })
if v.State != delivery.CompositionPass || len(v.Machines) != 1 || v.Machines[0] != "g14" || v.SDK == "" {
t.Fatalf("composition = %+v, want a pass on g14", v)
}
asks := h.taken()
if len(asks) != 2 {
t.Fatalf("the seat was asked %d composes, want two (the head, the running controller)", len(asks))
}
byRepo := map[string]delivery.StepAsk{}
for _, a := range asks {
byRepo[a.Compose.Fetch["repository"].(string)] = a
if a.Asker != AssignerModule || a.Kind != delivery.AskCheck || a.Language != delivery.AskGo {
t.Errorf("ask = %+v", a)
}
}
if a := byRepo["novox/mesh-sdk"]; a.Compose == nil || a.Compose.Fetch["commit"] != sdkHead || a.Compose.Dir != "go" {
t.Errorf("the head's ask = %+v", a.Compose)
}
if a := byRepo["novox/mesh-controller"]; a.Compose == nil || a.Compose.Fetch["commit"] != running || a.Compose.Dir != delivery.RootPath {
t.Errorf("the running controller's ask = %+v", a.Compose)
}
// A later pass of the same heads (woken by any change) asks nothing more.
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
time.Sleep(500 * time.Millisecond)
if n := len(h.taken()); n != 2 {
t.Errorf("a later pass of the same heads asked again: %d asks", n)
}
}
// verdict waits for the composition under key to satisfy ok, and answers it.
func (b *bed) verdict(t *testing.T, key string, ok func(delivery.CompositionVerdict) bool) delivery.CompositionVerdict {
t.Helper()
var last delivery.CompositionVerdict
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
e, err := b.buckets[delivery.AssignerCompositionsBucket].Get(b.ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
continue
}
if err != nil {
t.Fatal(err)
}
if last, err = delivery.DecodeCompositionVerdict(e.Value()); err != nil {
t.Fatal(err)
}
if ok(last) {
return last
}
}
t.Fatalf("no composition under %s as wanted after 10 s; last %+v", key, last)
return last
}
@@ -0,0 +1,73 @@
package main
import (
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Every node's published compositions, read only to compare (novox/hq ADR 0593 §8): each node's engine writes its own
// `composed-<node>` (delivery.ComposedRead, one bucket per node), and `mesh-wants.upgrade` with compare names, for every
// node, the composition its own have runs beside the newest it published of each module and of the node, so a person sees
// where they part. The assigner composes nothing and never writes these.
// publishedRow is one composition a node published: the module ("node" for its own), the fingerprint, the digest, or why
// it was refused or cannot be read.
type publishedRow struct {
Key string `json:"key"`
Module string `json:"module"`
FP string `json:"fp,omitempty"`
Digest string `json:"digest,omitempty"`
Refused string `json:"refused,omitempty"`
Unread string `json:"unread,omitempty"`
}
// published is, by node, every composition it published, sorted by key, from every key of the per-node state (a key
// names its node first): each value decoded and digested here, as the node-engine digests it (delivery.ComposedDigestOf).
// A value or key that does not read is named with why, never left out.
func published(values map[string][]byte) map[string][]publishedRow {
byNode := map[string]map[string][]byte{}
for key, raw := range values {
node, _, _ := strings.Cut(key, ".")
if byNode[node] == nil {
byNode[node] = map[string][]byte{}
}
byNode[node][key] = raw
}
out := map[string][]publishedRow{}
for node, keys := range byNode {
var rows []publishedRow
for key, raw := range keys {
row := publishedRow{Key: key}
_, module, fp, isNode, err := delivery.ParseComposedKey(key)
switch {
case err != nil:
row.Unread = err.Error()
case isNode:
row.Module = "node"
default:
row.Module, row.FP = module, fp
}
if err == nil {
c, derr := delivery.DecodeComposed(raw)
switch {
case derr != nil:
row.Unread = derr.Error()
case c.Refused != "":
row.Refused = c.Refused
default:
if d, err := delivery.ComposedDigestOf(c); err != nil {
row.Unread = err.Error()
} else {
row.Digest = d
}
}
}
rows = append(rows, row)
}
sort.Slice(rows, func(i, j int) bool { return rows[i].Key < rows[j].Key })
out[node] = rows
}
return out
}
File diff suppressed because it is too large Load Diff
@@ -15,24 +15,23 @@ import (
"os"
"os/signal"
"slices"
"sync/atomic"
"syscall"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-sdk/go/provisions"
"git.novox.be/novox/mesh-delivery/sdk/enginereads"
"git.novox.be/novox/mesh-sdk/go/views"
"mesh-delivery-assigner/internal/assign"
)
@@ -57,41 +56,33 @@ func main() {
// readBuckets are the buckets of other parts the assigner reads, each named in its manifest's reads (the manifest test
// holds the two lists equal), so the controller's broker grants exactly these.
var readBuckets = []string{
buscompose.ResolvedBucket,
provider.AssignmentsBucket,
delivery.PartGrantsBucket,
buscompose.LeaseBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
delivery.CandidatesBucket,
delivery.ManifestsBucket,
delivery.ModulesBucket,
delivery.PresenceBucket,
delivery.HaveBucket,
delivery.StepsBucket(goBuildModule),
takeoverBucket,
provisions.ProvisionsBucket,
keptBucket,
delivery.SourceBucket,
delivery.MergesBucket,
}
// readLogs are the logs of other parts the assigner reads, each named in its manifest's reads: the builder's log, whose
// newest done trunk build of a module names the commit a reader of the manifests runs at (novox/hq ADR 0571 clause 1).
var readLogs = []string{delivery.BuildLogRead}
// takeoverBucket is the condition keeper's take-over key's bucket (novox/hq ADR 0564 §9): whether the assigner's said
// conditions go straight to the router or only as findings (issue 1952 WM6).
var takeoverBucket = health.Bucket(health.KeeperModule, health.TakeoverState)
// goBuildModule is the Go build module, which takes the assigner's compose asks on its seat's check stream and writes
// their steps in its steps state (novox/hq ADR 0482 clause 4).
const goBuildModule = "mesh-build-go"
// readFamilies are the states kept one bucket per node the assigner reads, each named in its manifest's reads: every
// node's compositions, which mesh-wants.upgrade compares (novox/hq ADR 0593 §8).
var readFamilies = []string{delivery.ComposedRead}
// ownBuckets are the assigner's own states, as its manifest declares them: the seat mesh-placement's three among them
// (ADR 0567 §1), named after the assigner as every module state is.
var ownBuckets = []string{delivery.AssignerAsksBucket, delivery.AssignerCompositionsBucket, delivery.AssignerDifferencesBucket,
var ownBuckets = []string{delivery.AssignerAsksBucket, delivery.AssignerDifferencesBucket,
delivery.AssignerPutsBucket, delivery.AssignerWantsBucket, delivery.PlacementBucket, delivery.PlacementProposalsBucket,
delivery.PlacementCandidatesBucket, delivery.PlacementPulsesBucket, assign.FindingsBucket, assign.PoliciesBucket, delivery.SeatHoldersBucket}
delivery.PlacementCandidatesBucket, delivery.PlacementPulsesBucket, assign.FindingsBucket, assign.PoliciesBucket, delivery.SeatHoldersBucket,
engineReadsBucket}
// engineReadsState is the assigner's state of what each node's engine states it reads (ADR 0618 §3), one key `<node>`, as
// the verb manifest-reads last answered it or the fallback read it: what assigner_engine_reads lists and asks again.
const engineReadsState = "engine-reads"
// engineReadsBucket is that state's bucket.
const engineReadsBucket = delivery.AssignerModule + "_" + engineReadsState
// seatBuckets are the seat's states, new with it: until the controller has made them from the manifest and the bus grants
// them, the assigner starts without them and binds each once the bus says it was created (issue 1790's kind).
@@ -99,7 +90,7 @@ var seatBuckets = []string{delivery.PlacementBucket, delivery.PlacementProposals
delivery.PlacementPulsesBucket}
// keptStates is what the assigner keeps and who may read it (ADR 0495 §2-§3): its wants by the watchman and the view
// (and the node agent from release 6), its differences by the view, its compositions by the checker, its asks by the view
// (and the node agent from release 6), its differences by the view, its asks by the view
// and the teller, its puts by whoever follows one (ADR 0546), and the seat mesh-placement's placement, proposals and
// candidates by the node agents, the grants part, mesh-config and the view (ADR 0567 §9), and its placement by
// the builder, the checker and the indexer, through mesh-sdk's toolchain gate (issue 774, ADR 0549 §3).
@@ -109,13 +100,15 @@ var keptStates = []announce.State{
{Name: delivery.PlacementCandidatesState, Readable: true},
{Name: delivery.PlacementPulsesState, Readable: true},
{Name: "asks", Readable: true},
{Name: "compositions", Readable: true},
{Name: "differences", Readable: true},
{Name: "puts", Readable: true},
{Name: "wants", Readable: true},
{Name: health.FindingsState, Readable: true},
{Name: delivery.AssignerPoliciesState, Readable: true},
{Name: delivery.SeatHoldersState, Readable: true},
{Name: views.MarksState, Readable: true}, // what its views family said each state up to (views.go)
{Name: views.StatusState, Readable: true}, // its first views of the indexer's states (views.go)
{Name: engineReadsState, Readable: true},
}
// buckets is every bucket the assigner opened, by name.
@@ -127,40 +120,14 @@ type buckets map[string]jetstream.KeyValue
// without it and takes no put further until it starts again (ADR 0546).
func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any)) (buckets, error) {
out := buckets{}
// The controller's lease first (ADR 0549 §5, ADR 0561 §3): once it is handed away the seat's placement alone names
// where modules run, and the controller's assignments are no bucket the assigner needs. Not granted or not there, the
// controller holds it, as before.
away, why := false, ""
if kv, err := js.KeyValue(ctx, buscompose.LeaseBucket); err != nil {
logf("the bucket %s cannot be opened (%v): the controller is taken to hold its lease, so the assignments are read beside the placement", buscompose.LeaseBucket, err)
} else {
out[buscompose.LeaseBucket] = kv
e, gerr := kv.Get(ctx, buscompose.ControllerLeaseKey)
var raw []byte
if gerr == nil {
raw = e.Value()
}
if errors.Is(gerr, jetstream.ErrKeyNotFound) {
gerr = nil
}
away, why = buscompose.LeaseHandedAway(raw, raw != nil, gerr)
}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
if name == buscompose.LeaseBucket {
continue
}
kv, err := js.KeyValue(ctx, name)
if name == provider.AssignmentsBucket && err != nil && away {
logf("the bucket %s cannot be opened (%v), and none is needed: %s, so the seat's placement alone is read", name, err, why)
continue
}
if name == delivery.AssignerPutsBucket && err != nil {
logf("the bucket %s cannot be opened (%v): no put is taken further until the assigner starts again", name, err)
continue
}
if slices.Contains(seatBuckets, name) && err != nil {
// The seat's states come in the same delivery as this assigner (ADR 0567 §1): without placement the controller's
// assignments alone name where modules run, as before any placement was written (ADR 0549 §5).
// The seat's states come in the same delivery as this assigner (ADR 0567 §1): without placement no module is placed.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
@@ -170,11 +137,6 @@ func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == keptBucket && err != nil {
// mesh-config's held values (ADR 0565 §5): without them a composition lays none, said in the journal.
logf("the bucket %s cannot be opened (%v): no held value is laid on a composition until it is bound", name, err)
continue
}
if name == provisions.ProvisionsBucket && err != nil {
// The seat mesh-provisioner's state, with its own delivery (ADR 0570 §4): bound once the bus says it was made;
// until then no want waits for a provision, as before.
@@ -187,16 +149,27 @@ func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if (name == assign.FindingsBucket || name == takeoverBucket) && err != nil {
// New with health leaving the controller (issue 1952 WM6): without them the conditions are said straight to the
// router and kept as no finding, as before.
if name == assign.FindingsBucket && err != nil {
// New with health leaving the controller (issue 1952 WM6): without it the conditions are said straight to the
// router and kept as no finding.
logf("the bucket %s cannot be opened (%v): the assigner's conditions are said straight to the router until it starts again", name, err)
continue
}
if sourceOfGrants(name) && err != nil {
// The grants part's grants and the controller's lease choose where grants are read from (issue 774): without
// them the controller's grants are read, as before the lease is handed away.
logf("the bucket %s cannot be opened (%v): the controller's grants are read until the assigner starts again", name, err)
if (name == delivery.SourceBucket || name == delivery.MergesBucket) && err != nil {
// The gate's source and merges (ADR 0601 §1): without them no release is known to carry a build, so every move of
// a record module is asked of a person, as before.
logf("the bucket %s cannot be opened (%v): no release's Approve answers a record ask until it is bound", name, err)
continue
}
if name == engineReadsBucket && err != nil {
// New with ADR 0618 §3: without it the statements the verb answers are kept in memory only, and
// assigner_engine_reads lists nothing.
logf("the bucket %s cannot be opened (%v): what each node's engine reads is kept in memory only until it starts again", name, err)
continue
}
if name == delivery.PartGrantsBucket && err != nil {
// The grants part's grants (ADR 0571 clause 9): without them no module counts as granted until it is bound.
logf("the bucket %s cannot be opened (%v): no grant is read until it is bound once the bus says it was made", name, err)
continue
}
if err != nil {
@@ -211,17 +184,11 @@ func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any
// states and those new with a delivery of their own, bound once made, and the two that choose the grants' source.
func startsWithout(name string) bool {
switch name {
case delivery.SeatHoldersBucket, keptBucket, provisions.ProvisionsBucket, assign.PoliciesBucket, assign.FindingsBucket, takeoverBucket,
delivery.AssignerPutsBucket:
case delivery.SeatHoldersBucket, provisions.ProvisionsBucket, assign.PoliciesBucket, assign.FindingsBucket,
delivery.AssignerPutsBucket, delivery.SourceBucket, delivery.MergesBucket, engineReadsBucket:
return true
}
return slices.Contains(seatBuckets, name) || sourceOfGrants(name)
}
// sourceOfGrants says the bucket is the grants part's grants or the controller's lease: neither stops the assigner when it
// is not there.
func sourceOfGrants(name string) bool {
return name == delivery.PartGrantsBucket || name == buscompose.LeaseBucket
return slices.Contains(seatBuckets, name) || name == delivery.PartGrantsBucket
}
// run is the program: it returns when ctx ends or the bus cannot be used.
@@ -253,16 +220,25 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
thisNode = account.Node
logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs))
findings := findingsOf(ctx, kvs, logger.Printf)
buildLog := func(handle func(stdio.LogEntry) error) error {
return said.ReadLog(delivery.BuilderModule, delivery.BuilderLog).Watch("", stdio.LogFromAll, handle)
}
return follow(ctx, js, conn, kvs, findings.Say(sayVia(ctx, js)), findings, emitting{said}, buildLog, logger)
// Each change of the states its readers read is said on its views family (ADR 0615 §5, ADR 0619 §2); a watch of it
// that ends stops the program, as a watch of what a pass reads does.
viewErrs := make(chan error, len(viewedStates))
startViews(ctx, js, conn, account, kvs, findings, viewErrs, logger.Printf)
go func() {
select {
case <-ctx.Done():
case err := <-viewErrs:
logger.Printf("%v: the assigner stops, and is started again", err)
conn.Close()
}
}()
iv := &indexerViews{js: js, account: account, call: said.Call, findings: findings, logf: logger.Printf,
status: process.AwaitBucket(ctx, js, conn, firstViewsBucket, "assigner_first_views reads nothing and assigner_first_view_again is not heard", logger.Printf)}
return follow(ctx, js, conn, kvs, findings.Say(sayVia(ctx, js)), findings, emitting{said}, iv, logger)
}
// findingsOf is where the assigner's said conditions go as findings (issue 1952 WM6): its findings bucket, and the
// condition keeper's take-over key, watched, so a condition is said straight to the router only until the take-over. A
// bucket it cannot open is said: without its findings the conditions are said straight, as before; without the take-over
// key they are said straight too, the controller being followed.
// findingsOf is where the assigner's said conditions go as findings (issue 1952 WM6): its findings bucket, which the
// condition keeper carries (novox/hq ADR 0590 §3). Without it the conditions are said straight to the router.
func findingsOf(ctx context.Context, kvs buckets, logf func(string, ...any)) assign.Findings {
var f assign.Findings
if kv := kvs[assign.FindingsBucket]; kv != nil {
@@ -276,46 +252,6 @@ func findingsOf(ctx context.Context, kvs buckets, logf func(string, ...any)) ass
return keys, err
}
}
var keeper atomic.Bool
f.ToKeeper = keeper.Load
takeover := takeoverBucket
kv := kvs[takeover]
if kv == nil {
return f
}
w, err := kv.Watch(ctx, health.TakeoverKey)
if err != nil {
process.NotWatched(logf, takeover, err, "the controller is followed: the conditions are said straight to the router")
return f
}
go func() {
defer w.Stop()
for {
select {
case <-ctx.Done():
return
case e, ok := <-w.Updates():
if !ok {
logf("the watch of %s ended: the last source it named is followed until the assigner starts again", takeover)
return
}
if e == nil {
continue
}
var raw []byte
if e.Operation() == jetstream.KeyValuePut {
raw = e.Value()
}
t, err := health.DecodeTakeover(raw, raw != nil)
if err != nil {
logf("the take-over key cannot be read (%v): the controller is followed", err)
}
if was := keeper.Swap(t.Follow == health.FollowKeeper); was != keeper.Load() {
logf("the assigner's conditions are said by %s from now (the take-over key follows %s)", t.Seat(), t.Follow)
}
}
}
}()
return f
}
@@ -330,15 +266,9 @@ func (e emitting) Emit(eventType string, body any) error { return e.Conn.Emit(ev
// watched are the buckets whose every value a pass reads; the manifests are read by digest when a pass needs one, and
// watched only for keys, so a manifest that arrives wakes a composition that waits for it.
var watched = []string{
buscompose.ResolvedBucket,
provider.AssignmentsBucket,
delivery.PartGrantsBucket,
buscompose.LeaseBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
delivery.CandidatesBucket,
delivery.ModulesBucket,
delivery.PlacementBucket,
delivery.PlacementProposalsBucket,
delivery.PlacementCandidatesBucket,
@@ -346,18 +276,19 @@ var watched = []string{
delivery.PresenceBucket,
delivery.HaveBucket,
delivery.AssignerAsksBucket,
delivery.AssignerCompositionsBucket,
delivery.AssignerDifferencesBucket,
delivery.AssignerWantsBucket,
assign.PoliciesBucket,
provisions.ProvisionsBucket,
keptBucket,
delivery.SeatHoldersBucket,
delivery.SourceBucket,
delivery.MergesBucket,
engineReadsBucket,
}
// follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after
// each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends.
func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, kvs buckets, say func(asks.Condition) error, findings assign.Findings, bus caller, buildLog logWatch, logger *log.Logger) error {
func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, kvs buckets, say func(oc.Condition) error, findings assign.Findings, bus caller, iv *indexerViews, logger *log.Logger) error {
w := newWorld()
dirty := make(chan struct{}, 1)
wake := func() {
@@ -366,24 +297,18 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
default:
}
}
// The builder's log (ADR 0571 clause 1): every entry it keeps before the first pass, then each new one as it is appended;
// a module's newer done trunk build makes a pass, as a composition waiting on its commit is judged then. Where it cannot be
// read, a composition names no running reader of the manifests until the assigner starts again, and it is said.
if buildLog != nil {
var first atomic.Bool
if err := buildLog(func(le stdio.LogEntry) error {
if w.putBuilt(le.Value) && first.Load() {
wake()
}
return nil
}); err != nil {
logger.Printf("%s is not read (%v): no composition is judged against a running reader of the manifests until the assigner starts again",
delivery.BuildLogRead, err)
}
first.Store(true)
}
errs := make(chan error, len(watched)+2)
settled := make(chan struct{}, len(watched)+2)
// What each node's engine reads (ADR 0618 §3, §4): asked through the seat's verb, kept in the state engine-reads, read
// from the node's record where the engine does not answer (the fallback, said in the journal).
reads := &enginereads.Keeper{Fallback: w.recordReads, Changed: wake, Logf: logger.Printf}
if bus != nil {
reads.Call = bus.CallOn
}
if kv := kvs[engineReadsBucket]; kv != nil {
reads.Put = func(node string, v []byte) error { _, err := kv.Put(ctx, node, v); return err }
reads.Delete = func(node string) error { return kv.Delete(ctx, node) }
}
watch := func(name string, opts ...jetstream.WatchOpt) error {
kw, err := kvs[name].WatchAll(ctx, opts...)
if err != nil {
@@ -417,12 +342,11 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
if e.Operation() == jetstream.KeyValuePut {
value = e.Value()
}
before := w.handedAway()
w.putEntry(name, e.Key(), value, e.Revision(), e.Created())
if name == buscompose.LeaseBucket && w.handedAway() == before {
// The controller renews its lease every few seconds: only the lease's being handed away makes a pass.
if name == engineReadsBucket {
reads.Heard(e.Key(), value, value == nil) // it wakes a pass when a statement changed
continue
}
w.putEntry(name, e.Key(), value, e.Revision(), e.Created())
wake()
}
}
@@ -458,30 +382,33 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
return
}
if bus != nil {
p.acts = newPutActs(bus, wake, logger.Printf, w.handedAway)
p.acts = newPutActs(bus, wake, logger.Printf)
}
logger.Printf("the bucket %s is open: puts are taken further", delivery.AssignerPutsBucket)
}
started := 0
for _, name := range watched {
if kvs[name] == nil && (startsWithout(name) || name == provider.AssignmentsBucket) {
if kvs[name] == nil && startsWithout(name) {
continue // not on the bus at this start (open said so)
}
started++
var opts []jetstream.WatchOpt
if name == delivery.HaveBucket {
opts = append(opts, jetstream.IncludeHistory()) // what each node ran healthy before, for a set-back
if name == delivery.HaveBucket || name == delivery.SourceBucket {
// what each node ran healthy before, for a set-back; and the release behind each source revision (ADR 0601 §1)
opts = append(opts, jetstream.IncludeHistory())
}
if err := watch(name, opts...); err != nil {
return err
}
}
if err := watch(delivery.ManifestsBucket, jetstream.MetaOnly()); err != nil {
return err
// The indexer's manifests, modules and module graph, through its first views and views family, never its buckets
// (DLV-F2r, ADR 0631): the first pass waits for the three answers, as it waits for every watch to hand what stands.
if iv != nil {
started += iv.start(ctx, w, wake, settled)
}
// The seat's states arrive with the same delivery as this assigner (ADR 0567 §1): one not on the bus at this start is
// bound once the bus says it was created, and watched from then on, with no restart (issue 1790's kind). Until then the
// controller's assignments alone name where modules run (ADR 0549 §5).
// bound once the bus says it was created, and watched from then on, with no restart (issue 1790's kind). Until then no
// module is placed.
type appeared struct {
name string
kv jetstream.KeyValue
@@ -494,7 +421,7 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
}
}
// And the two that choose where grants are read from, with the same delivery as the grants part that writes one.
for _, name := range []string{delivery.PartGrantsBucket, buscompose.LeaseBucket, assign.PoliciesBucket, provisions.ProvisionsBucket, keptBucket, delivery.SeatHoldersBucket} {
for _, name := range []string{delivery.PartGrantsBucket, assign.PoliciesBucket, provisions.ProvisionsBucket, delivery.SeatHoldersBucket} {
if kvs[name] == nil && js != nil {
name := name
process.AwaitBucket(ctx, js, nc, name, "until then the bucket is not read",
@@ -509,7 +436,7 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
logger.Printf("the bucket %s is open and watched", a.name)
return watch(a.name)
}
for range started + 1 {
for range started {
select {
case <-ctx.Done():
return nil
@@ -518,13 +445,19 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
case <-settled:
}
}
p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, findings: findings, said: map[string]bool{}, answeredAt: map[string]uint64{}}
p.asks = newComposeAsks(ctx, buildrun.Asker{Publisher: buildrun.JetStreamAsks{JS: js}, Waiter: buildrun.JetStreamAsks{JS: js}}, wake, logger.Printf)
p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, findings: findings, said: map[string]bool{}, answeredAt: map[string]uint64{},
reads: reads}
// The node-engine seat's manifest-reads-changed, for one node (ADR 0618 §4): that node is asked again, with no timer.
// Heard on the assigner's durable consumer on EVENTS, which the grants part makes from the manifest's consumes; where it
// is not there yet, a node is asked again only when it comes back online.
if js != nil {
hearReadsChanged(ctx, js, reads, findings, logger.Printf)
}
if bus != nil && kvs[delivery.AssignerPutsBucket] != nil {
p.acts = newPutActs(bus, wake, logger.Printf, w.handedAway)
p.acts = newPutActs(bus, wake, logger.Printf)
}
if bus != nil {
p.seat = newSeatActs(bus, wake, logger.Printf, w.handedAway)
p.seat = newSeatActs(bus, wake, logger.Printf)
}
if js != nil {
p.publish = streamPublisher(ctx, js, thisNode)
@@ -570,13 +503,13 @@ var PutsReopenEvery = time.Minute
// sayVia publishes a condition on the operator-channel seat's one subject for conditions, under the assigner's own name
// (ADR 0437 clause 9).
func sayVia(ctx context.Context, js jetstream.JetStream) func(asks.Condition) error {
return func(c asks.Condition) error {
subject, err := asks.ConditionSubject(delivery.AssignerModule)
func sayVia(ctx context.Context, js jetstream.JetStream) func(oc.Condition) error {
return func(c oc.Condition) error {
subject, err := occ.ConditionSubject(delivery.AssignerModule)
if err != nil {
return err
}
data, err := asks.EncodeCondition(c)
data, err := occ.EncodeCondition(c)
if err != nil {
return err
}
@@ -584,3 +517,39 @@ func sayVia(ctx context.Context, js jetstream.JetStream) func(asks.Condition) er
return err
}
}
// hearReadsChanged hears the node-engine seat's event manifest-reads-changed on the assigner's durable consumer on EVENTS,
// and asks the node it names again. Any other event the consumer hands is acknowledged and left.
func hearReadsChanged(ctx context.Context, js jetstream.JetStream, reads *enginereads.Keeper, findings assign.Findings, logf func(string, ...any)) {
account := process.Account{Node: thisNode, Module: delivery.AssignerModule}
w := notHeardWriter(findings, logf)
cons, err := account.EventConsumer(ctx, js, process.EventsStream)
if err != nil {
w.NotHeard(process.EventsStream, enginereads.Event+" is not heard until the assigner starts again (a node is asked again when it comes back online)",
fmt.Errorf("the consumer %s on %s does not exist yet: %w", account.EventConsumerName(), process.EventsStream, err))
return
}
cc, err := process.HearEvents(cons, func(m jetstream.Msg) {
if err := m.Ack(); err != nil {
logf("acknowledging %s: %v", m.Subject(), err)
}
node, ours, err := enginereads.NodeOf(m.Subject(), m.Data())
switch {
case !ours:
case err != nil:
logf("%s holds what is not %s, and %s is asked again all the same: %v", m.Subject(), enginereads.Event, node, err)
reads.AskNode(node, enginereads.Event+" that does not read")
default:
reads.AskNode(node, enginereads.Event)
}
}, logf)
if err != nil {
w.NotHeard(process.EventsStream, enginereads.Event+" is not heard until the assigner starts again", err)
return
}
w.Heard(process.EventsStream)
go func() {
<-ctx.Done()
cc.Stop()
}()
}
@@ -18,16 +18,20 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-sdk/go/process"
)
// 1297/B2a: the buckets the program opens are exactly the manifest's reads and states, as the broker names them
// (`<module>.<state>` read as the bucket `<module>_<state>`), so it is granted what it reads and reads nothing else.
func TestOpensWhatItsManifestNames(t *testing.T) {
// Review round 2: the assigner's views emitter writes its finding view-unsaid (Emitter.Findings set) when it keeps findings.
findings := assign.Findings{Put: func(string, []byte) error { return nil }, Delete: func(string) error { return nil }}
if viewsEmitter(nil, process.Account{}, nil, findings, nil, t.Logf).Findings == nil {
t.Error("the assigner's views emitter writes no view-unsaid finding")
}
// 1297/B2a: without a credential the program does not start, and says why.
if err := run(context.Background(), func(string) string { return "" }, io.Discard); err == nil || !strings.Contains(err.Error(), "MESH_BROKER_FILE") {
t.Errorf("run without a credential = %v", err)
@@ -42,14 +46,14 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
}
var reads []string
for _, r := range m.Reads {
if slices.Contains(readLogs, r) {
continue // a log, read through its own reader (ADR 0571 clause 1), not a bucket
if slices.Contains(readFamilies, r) {
continue // a state kept one bucket per node, read by its prefix (ADR 0593 §6), not one bucket
}
reads = append(reads, strings.Replace(r, ".", "_", 1))
}
for _, l := range readLogs {
for _, l := range readFamilies {
if !slices.Contains(m.Reads, l) {
t.Errorf("the program reads the log %s, the manifest does not name it", l)
t.Errorf("the program reads the per-node state %s, the manifest does not name it", l)
}
}
var states []string
@@ -59,7 +63,7 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
for _, pair := range []struct {
what string
code, want []string
}{{"reads", readBuckets, reads}, {"states", ownBuckets, states}} {
}{{"reads", readBuckets, reads}, {"states", append(slices.Clone(ownBuckets), marksBucket, firstViewsBucket), states}} { // the marks are bound by the views family (views.go)
got, want := slices.Sorted(slices.Values(pair.code)), slices.Sorted(slices.Values(pair.want))
if !reflect.DeepEqual(got, want) {
t.Errorf("the program's %s are %v, the manifest's %v", pair.what, got, want)
@@ -69,33 +73,26 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
for _, s := range keptStates {
kept = append(kept, m.Module+"_"+s.Name)
}
if got, want := slices.Sorted(slices.Values(kept)), slices.Sorted(slices.Values(ownBuckets)); !reflect.DeepEqual(got, want) {
if got, want := slices.Sorted(slices.Values(kept)), slices.Sorted(slices.Values(append(slices.Clone(ownBuckets), marksBucket, firstViewsBucket))); !reflect.DeepEqual(got, want) {
t.Errorf("the program announces %v, keeps %v", got, want)
}
// ADR 0546: whatever keeps the puts' bucket from opening (not there, not granted, no answer in time), the assigner opens
// the rest and starts, writing wants; any other bucket it cannot open stops the start, named.
// Issue 774 (ADR 0549 §5): the controller's lease not granted, the assigner starts taking it as the controller's; the
// assignments refused while the controller holds it stop the start, and once it is handed away they are not needed.
handed := []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`)
for _, c := range []struct {
missing string
err error
starts bool
lease []byte
said string
}{
{delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true, nil, "no put is taken further"},
{delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true, nil, "no put is taken further"},
{delivery.AssignerPutsBucket, context.DeadlineExceeded, true, nil, "no put is taken further"},
{delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false, nil, ""},
// Issue 774: the grants part's grants and the controller's lease choose the grants' source; neither stops the start.
{delivery.PartGrantsBucket, jetstream.ErrBucketNotFound, true, nil, "the controller's grants are read"},
{buscompose.LeaseBucket, errors.New("nats: permissions violation"), true, nil, "taken to hold its lease"},
{provider.AssignmentsBucket, errors.New("nats: permissions violation"), false, []byte(`{"instance":"novox/1"}`), ""},
{provider.AssignmentsBucket, errors.New("nats: permissions violation"), true, handed, "the seat's placement alone is read"},
{delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true, "no put is taken further"},
{delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true, "no put is taken further"},
{delivery.AssignerPutsBucket, context.DeadlineExceeded, true, "no put is taken further"},
{delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false, ""},
// Issue 774: the grants part's grants not there do not stop the start.
{delivery.PartGrantsBucket, jetstream.ErrBucketNotFound, true, "no grant is read"},
} {
var logged string
kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err, lease: c.lease}, func(f string, a ...any) { logged += fmt.Sprintf(f, a...) })
kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err}, func(f string, a ...any) { logged += fmt.Sprintf(f, a...) })
switch {
case c.starts && (err != nil || kvs[c.missing] != nil || len(kvs) != len(readBuckets)+len(ownBuckets)-1 || !strings.Contains(logged, c.said)):
t.Errorf("%s refused with %v: the start answered %v, %d buckets, logged %q", c.missing, c.err, err, len(kvs), logged)
@@ -112,15 +109,17 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
kvs[name] = &watchKV{memKV: &memKV{values: map[string][]byte{}}}
}
ckey, _ := delivery.ComposedNodeKey("ace")
cd, err := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("d", 64), Judgement: delivery.JudgedUnchanged})
placed, err := delivery.PlacementValues(delivery.Placement{Node: "ace"})
if err != nil {
t.Fatal(err)
}
kvs[delivery.ControllerComposedBucket].(*watchKV).entries = []kvEntry{{key: ckey, value: cd}}
for key, raw := range placed {
kv := kvs[delivery.PlacementBucket].(*watchKV)
kv.entries = append(kv.entries, kvEntry{key: key, value: raw})
}
kvs[delivery.AssignerPutsBucket].(*watchKV).closed = true
kvs[delivery.AssignerWantsBucket].(*watchKV).wrote = wrote
go follow(ctx, nil, nil, kvs, func(asks.Condition) error { return nil }, assign.Findings{}, nil, nil, log.New(io.Discard, "", 0))
go follow(ctx, nil, nil, kvs, func(oc.Condition) error { return nil }, assign.Findings{}, nil, nil, log.New(io.Discard, "", 0))
select {
case key := <-wrote:
if key != "want.ace.node" {
@@ -136,18 +135,13 @@ type refusingJS struct {
jetstream.JetStream
missing string
err error
lease []byte // the controller's lease, when it holds one
}
func (r refusingJS) KeyValue(_ context.Context, name string) (jetstream.KeyValue, error) {
if name == r.missing {
return nil, r.err
}
kv := &memKV{values: map[string][]byte{}}
if name == buscompose.LeaseBucket && r.lease != nil {
kv.values[buscompose.ControllerLeaseKey] = r.lease
}
return kv, nil
return &memKV{values: map[string][]byte{}}, nil
}
// watchKV is a memKV whose watch hands its entries and then the marker that all stands was handed, or ends at once when
@@ -10,11 +10,12 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
"git.novox.be/novox/mesh-sdk/go/sdkversion"
"git.novox.be/novox/mesh-delivery/sdk/enginereads"
"mesh-delivery-assigner/internal/assign"
)
@@ -28,12 +29,10 @@ type passer struct {
sdk string
// say publishes a condition to the operator; said is, by condition key and state, what this process has said, so a
// pass says a condition once and a restart says it again (the router keeps one per key).
say func(asks.Condition) error
say func(oc.Condition) error
said map[string]bool
// findings is the assigner's findings bucket (its said conditions and policy-split, ADR 0564 §2, ADR 0565 §8); splits the
// modules whose policy-split finding stands, nil until the first pass, which deletes any a previous process left.
// findings is the assigner's findings bucket (its said conditions, ADR 0564 §2).
findings assign.Findings
splits map[string]bool
// failedProvisions are the failed provisions whose finding this process wrote (ADR 0570 §4).
failedProvisions map[string]bool
// missing are the required provisions with no key this process pulsed, and missingSaid those whose finding it wrote.
@@ -42,7 +41,6 @@ type passer struct {
unplaced map[string]bool
// answeredAt is, by the id of a backup ask, how many values the have had when its answer was written into the want.
answeredAt map[string]uint64
asks *composeAsks
// now is the clock; nil is time.Now.
now func() time.Time
// neither is, by node and module, each module on neither list whose condition this process raised (issue 1945). Nil until the
@@ -59,12 +57,10 @@ type passer struct {
// nodeFailed is, by node and module, the words of the condition last raised for a failure that is the node's own (ADR 0557
// §1, issue 1496): raised once per words, cleared once when it ends.
nodeFailed map[string]string
// callSaid is, by the key of a refused controller call's condition, what is raised (issue 774): by this process, or by
// a previous one and taken over from the findings.
callSaid map[string]*callSaid
// callsSwept says the first pass took over the refused calls' conditions a previous process raised (sweepCalls).
callsSwept bool
// The seat mesh-placement's holder (seat.go, ADR 0567): its calls of the controller (nil: none is made), how it says
// callsCleared says the first pass cleared the conditions of refused controller calls a previous process raised
// (clearStaleCalls): the assigner makes no such call now (issue 774).
callsCleared bool
// The seat mesh-placement's holder (seat.go, ADR 0567): its acts (nil: none is made), how it says
// the seat's events (nil: none is said), the proposals it said, and what Intended computed by proposal and base.
seat *seatActs
publish publisher
@@ -74,6 +70,40 @@ type passer struct {
// saidWaits is the last said of the seat's wait for its states (sayWaits).
saidWaits string
intended map[string]map[string]map[string]bool
// reads keeps what each node's engine states it reads, asked through the node-engine seat's verb (ADR 0618 §3); nil reads
// each node's record, as the rules did before the verb. online is, by node, whether the last pass saw it online.
reads *enginereads.Keeper
online map[string]bool
}
// engineReads sets what the rules read of each node's engine (ADR 0618 §3, §4): the statement the verb manifest-reads
// answered, or its record where the verb was not answered (the fallback) or not asked yet. A node is asked when the
// assigner first hears of it, and again when it comes back online, since an engine built before the event says none.
func (p *passer) engineReads(n assign.Nodes) {
if p.reads == nil {
return
}
known := p.w.knownNodes()
first := p.online == nil
for _, node := range known {
switch {
case first:
p.reads.AskNode(node, "the assigner started")
case !p.reads.Known(node):
p.reads.AskNode(node, "first heard of")
case !first && n.Online[node] && !p.online[node]:
p.reads.AskNode(node, "back online")
}
}
p.online = maps.Clone(n.Online)
if p.online == nil {
p.online = map[string]bool{}
}
for node, reads := range p.reads.Reads(known) {
if p.reads.Known(node) {
n.Reads[node] = reads
}
}
}
func (p *passer) clock() time.Time {
@@ -85,15 +115,13 @@ func (p *passer) clock() time.Time {
func (p *passer) pass(ctx context.Context) {
p.placementSeat(ctx)
p.compositions(ctx)
p.wants(ctx)
p.puts(ctx)
p.nodeWants(ctx)
p.backups(ctx)
}
// nodeWants writes each node's own want, `want.<node>.node`, naming the digest of the node's own composition, once the
// controller has composed it and again whenever that digest changes (ADR 0539 §6).
// nodeWants writes each placed node's own want, `want.<node>.node`: the node composes its own (ADR 0547 §2, ADR 0571 §6).
func (p *passer) nodeWants(ctx context.Context) {
written := p.w.read(delivery.AssignerWantsBucket)
placed := p.w.placedNodes()
@@ -109,28 +137,14 @@ func (p *passer) nodeWants(ctx context.Context) {
want := delivery.NodeWant{Self: true, Words: "the node composes its own composition (ADR 0547 §2)"}
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeNodeWant(want) })
}
for node, d := range p.w.nodeComposed() {
if placed[node] {
continue
}
key, err := delivery.WantNodeKey(node)
if err != nil {
continue
}
if before, err := delivery.DecodeNodeWant(written[key]); err == nil && before.Composed == d.Digest {
continue
}
want := delivery.NodeWant{Composed: d.Digest, Words: "the node's own composition"}
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeNodeWant(want) })
}
}
// wants writes, for every module the trunk holds a fingerprint of, each node's want by the six rules (ADR 0429 §6), and
// the hold of a fingerprint whose health check failed. The assigner compares its wants with nothing the controller
// sends: the differences bucket is kept read-only with its history (ADR 0539 §8).
func (p *passer) wants(ctx context.Context) {
get := kvGetter(p.kvs[delivery.ManifestsBucket])
assigned, _ := p.w.assigned()
get := p.w.getter(delivery.ManifestsBucket) // the indexer's manifests, as its first view and views family say them
assigned := p.w.assigned()
trunk := p.w.trunk()
manifests := map[string]catalogue.Manifest{}
for name, mod := range trunk {
@@ -142,6 +156,7 @@ func (p *passer) wants(ctx context.Context) {
}
}
nodes := p.w.nodes(manifests, assigned)
p.engineReads(nodes)
nodesOf := map[string][]string{}
for node, modules := range assigned {
for _, m := range modules {
@@ -155,10 +170,7 @@ func (p *passer) wants(ctx context.Context) {
defer func() { p.provisionsMissing(ctx, missing) }()
defer p.unplacedFindings()
defer p.seatHolders(ctx)
defer p.policySplits(manifests, chosen)
differ := p.withCalls(p.w.placementDiffers())
p.sayCalls(differ)
frozen := p.sayPlacement(differ)
p.clearStaleCalls()
placed := p.w.placedNodes()
puts := p.putsByModule()
p.staleAsks(ctx, trunk)
@@ -178,26 +190,22 @@ func (p *passer) wants(ctx context.Context) {
stale := map[string]neitherOne{}
left := map[string]string{}
defer func() { p.sayNeither(neither, stale, left) }()
// Where grants are read from, once for the pass: the grants part's while the lease is handed away (issue 774).
// Where grants are read from, once for the pass: the grants part's (issue 774, ADR 0571 clause 9).
grants := p.w.grantsBucket()
// The verb edges, once for the pass (ADR 0614 §2, §3).
moduleGraph := p.moduleGraph()
served := p.served(ctx, get, moduleGraph, nodes)
offering := p.offering(ctx, get, manifests, nodes) // bus state (ADR 0614 §5)
for name := range names {
fp := trunk[name].FP2
name := name
m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Chosen: chosenOf(chosen, name), Provisions: provisionsOf(name), Answering: answering(manifests, nodes), Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name],
m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Needs: trunk[name].Needs, Chosen: chosenOf(chosen, name), Provisions: provisionsOf(name), Answering: answering(manifests, nodes), Offered: offered(manifests), Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name], FormNeeds: p.w.buildNeeds(name, fp),
Self: func(node string) bool { return placed[node] },
Composed: func(node, fp string) (delivery.ComposedDigest, bool) {
if placed[node] && p.w.handedAway() {
return p.w.ownComposed(node) // from the lease write: the digest its own have names (ADR 0571 §6)
}
if placed[node] {
// The node composes its own (ADR 0547 §2, §4): no composition of the controller's is waited for, the want
// names the build alone, and the node judges the move itself.
return delivery.ComposedDigest{Judgement: delivery.JudgedNeither, Words: "the node composes its own"}, true
}
return p.w.composed(node, name, fp)
return p.w.composed(node, name, fp) // a placed node's own: the digest its own have names (ADR 0571 §6)
},
GrantedAt: func(fp string) bool {
key, err := delivery.ControllerGrantKey(name, fp)
key, err := delivery.PartGrantKey(name, fp)
return err == nil && p.w.has(grants, key)
},
Restored: func(node string) bool {
@@ -207,16 +215,17 @@ func (p *passer) wants(ctx context.Context) {
Allowed: func(node, digest string) bool {
key, err := delivery.RecordAskKey(node, name, digest)
return err == nil && p.answeredWith(key, assign.OptionYes)
}}
},
Release: func(fp string) (string, bool) { return p.w.release(name, fp) }}
if fp != "" {
held, _ := delivery.HeldKey(name, fp)
grant, _ := delivery.ControllerGrantKey(name, fp)
grant, _ := delivery.PartGrantKey(name, fp)
hold, _ := delivery.HoldAskKey(name, fp)
m.Released = p.answeredWith(hold, "release")
m.Kept = p.answeredWith(hold, "keep")
m.Built, m.Held, m.Granted = p.w.built(name, fp), p.w.has(delivery.AssignerWantsBucket, held) && !m.Released, p.w.has(grants, grant)
if raw := p.w.value(grants, grant); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil {
if g, err := delivery.DecodeGrant(raw); err == nil {
m.GrantedOn = func(node string) bool { return assign.GrantNamesNode(g, node, name) }
}
}
@@ -224,6 +233,10 @@ func (p *passer) wants(ctx context.Context) {
p.clearHeld(name, fp)
}
}
if served != nil {
m.Invokes, m.Served = keptInvokes(moduleGraph, name), served
}
m.Offering = offering
last := map[string]string{}
for _, node := range m.Nodes {
last[node] = p.w.lastHealthy(node, name, fp)
@@ -267,9 +280,6 @@ func (p *passer) wants(ctx context.Context) {
p.recordAsk(ctx, name, fp, need)
}
for node, want := range out.Wants {
if frozen[node+"."+name] {
continue // one source alone names it (ADR 0549 §5): no want, and the one written before stays
}
key, err := delivery.WantKey(node, name)
if err != nil {
p.logf("no want of %s on %s is written: %v", name, node, err)
@@ -511,57 +521,6 @@ func (p *passer) sayNeither(now, stale map[string]neitherOne, left map[string]st
}
}
// compositions writes the composition check's verdict of every group the indexer holds a candidate for, under the
// checker's key of that check: waits, with why, while the indexer holds what it composes from only in part.
func (p *passer) compositions(ctx context.Context) {
if p.sdk == "" {
p.sdk = sdkversion.Version()
}
written := p.w.read(delivery.AssignerCompositionsBucket)
trunk := p.w.trunk()
var mesh *assign.Mesh
seen := map[string]bool{}
if p.asks != nil {
defer p.asks.sweep(seen)
}
for key, value := range p.w.read(delivery.CandidatesBucket) {
c, err := delivery.DecodeCandidate(value)
if err != nil {
p.logf("the candidate %s cannot be read, so it is not composed: %v", key, err)
continue
}
out, err := delivery.CompositionKey(c.Group, c.HeadsHash)
if err != nil {
p.logf("the candidate %s names no composition key: %v", key, err)
continue
}
began := time.Now()
ch, waits := p.w.change(ctx, kvGetter(p.kvs[delivery.ManifestsBucket]), c, trunk)
var v delivery.CompositionVerdict
if waits != "" {
v = delivery.CompositionVerdict{Group: c.Group, Heads: c.HeadsHash, State: delivery.CompositionWaits, Words: waits}
} else {
if mesh == nil {
m := p.w.mesh(ctx, kvGetter(p.kvs[delivery.ManifestsBucket]))
mesh = &m
}
if ch.SDKHead != "" && p.asks != nil {
// A group with a mesh-sdk member is composed by the composer built at its head (ADR 0482 clauses 3 to 5).
v = p.composeAtHead(ctx, *mesh, ch, seen, began)
} else {
v = assign.Compose(*mesh, ch, p.sdk, time.Since(began))
}
}
if before, err := delivery.DecodeCompositionVerdict(written[out]); err == nil {
before.Took, before.V = v.Took, 0
if delivery.SameCompositionVerdict(before, v) {
continue
}
}
p.put(ctx, delivery.AssignerCompositionsBucket, out, func() ([]byte, error) { return delivery.EncodeCompositionVerdict(v) })
}
}
// put writes one key of the assigner's own bucket; a write that fails is said and made again at the next pass.
func (p *passer) put(ctx context.Context, bucket, key string, encode func() ([]byte, error)) {
value, err := encode()
@@ -595,7 +554,7 @@ func (p *passer) holdAsk(ctx context.Context, module, fp string, fresh *delivery
asking := int(now.Unix())
h := p.heldHold(module, fp, fresh)
a := assign.HoldAsk(key, module, fp, h.Node, h.Words, asking, now)
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return assign.EncodeAsk(key, a) })
}
// lift takes away a hold the assigner lifted by itself, since every node that runs the module runs the held fingerprint healthy
@@ -778,7 +737,7 @@ func (p *passer) restoreAsk(ctx context.Context, module, node, fp string) {
}
words := p.heldHold(module, fp, nil).Words
a := assign.RestoreAsk(key, node, module, fp, words, time.Now())
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return assign.EncodeAsk(key, a) })
}
// answeredWith says whether the teller wrote an answer to the ask at key as it stands now and the option chosen is option. The
@@ -790,12 +749,12 @@ func (p *passer) answeredWith(key, option string) bool {
if !ok || a.Option != option {
return false
}
ask, err := delivery.DecodeAssignerAsk(key, p.w.value(delivery.AssignerAsksBucket, key))
_, ask, err := assign.DecodeAsk(key, p.w.value(delivery.AssignerAsksBucket, key))
if err != nil {
return false
}
id, _, _ := strings.Cut(a.Warrant, "@")
return id == ask.Ask.ID
return id == ask.ID
}
// clearHeld clears, once, the condition of a hold the operator released.
@@ -820,7 +779,7 @@ func (p *passer) recordAsk(ctx context.Context, module, fp string, need assign.R
return
}
a := assign.RecordAsk(key, need.Node, module, fp, need, time.Now())
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return assign.EncodeAsk(key, a) })
}
// backups sees to the modules held on a node because the backup before an apply failed (the have's phase held, ADR 0539 §16):
@@ -851,7 +810,7 @@ func (p *passer) backups(ctx context.Context) {
if err != nil {
continue
}
current, there := delivery.DecodeAssignerAsk(akey, p.w.read(delivery.AssignerAsksBucket)[akey])
current, currentAsk, there := assign.DecodeAsk(akey, p.w.read(delivery.AssignerAsksBucket)[akey])
if there != nil || assign.BackupAskDigest(current) != want.Composed {
after := -1
if there == nil {
@@ -861,7 +820,7 @@ func (p *passer) backups(ctx context.Context) {
continue
}
answer, answered := p.w.answer(akey)
if router, _, _ := strings.Cut(answer.Warrant, "@"); !answered || router != current.Ask.ID ||
if router, _, _ := strings.Cut(answer.Warrant, "@"); !answered || router != currentAsk.ID ||
(answer.Option != delivery.BackupAgain && answer.Option != delivery.BackupSkip) {
continue // not answered yet: the ask stands
}
@@ -872,7 +831,7 @@ func (p *passer) backups(ctx context.Context) {
if wkey, err := delivery.WantKey(node, module); err == nil {
w := want
p.put(ctx, delivery.AssignerWantsBucket, wkey, func() ([]byte, error) { return delivery.EncodeWant(w) })
p.answeredAt[current.Ask.ID] = haveChanges
p.answeredAt[currentAsk.ID] = haveChanges
}
continue
}
@@ -880,8 +839,8 @@ func (p *passer) backups(ctx context.Context) {
continue
}
// The answer is in the want. The node retried and the backup failed again when the have took another value since.
if seen, known := p.answeredAt[current.Ask.ID]; !known {
p.answeredAt[current.Ask.ID] = haveChanges // after a restart: wait for the next failure
if seen, known := p.answeredAt[currentAsk.ID]; !known {
p.answeredAt[currentAsk.ID] = haveChanges // after a restart: wait for the next failure
} else if haveChanges > seen {
p.backupAsk(ctx, akey, node, module, want, have.Detail, assign.NextBackupAsking(current.Asking))
}
@@ -891,7 +850,7 @@ func (p *passer) backups(ctx context.Context) {
// backupAsk writes the backup ask at its asking and raises the condition of the failed backup, once.
func (p *passer) backupAsk(ctx context.Context, key, node, module string, want delivery.Want, words string, asking int) {
a := assign.BackupAsk(key, node, module, want.FP, want.Composed, words, asking, time.Now())
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return assign.EncodeAsk(key, a) })
c := assign.BackupCondition(node, module, want.FP, words, true)
if p.say == nil || p.said[c.Key] {
return
@@ -903,46 +862,12 @@ func (p *passer) backupAsk(ctx context.Context, key, node, module string, want d
p.said[c.Key] = true
}
// sayPlacement raises, once, the condition of every pair the controller's assignments and the catalogue's placement do not
// both name, clears it once they agree, and answers those pairs by node and module (ADR 0549 §5): the pass writes no want
// for them and keeps the one written before. A say that fails is made again at the next pass.
func (p *passer) sayPlacement(differ []assign.Pair) map[string]bool {
frozen := map[string]bool{}
now := map[string]bool{}
for _, d := range differ {
frozen[d.Node+"."+d.Module] = true
c := assign.PlacementCondition(d, true)
now[c.Key] = true
if p.said[c.Key+".raised"] {
continue
}
p.logf("%s on %s is named by %s alone: no want is written for it there until both sources name it (ADR 0549 §5)", d.Module, d.Node, d.Only)
if err := p.say(c); err != nil {
p.logf("the placement condition of %s on %s could not be said, and is said at the next pass: %v", d.Module, d.Node, err)
continue
}
p.said[c.Key+".raised"] = true
}
prefix := delivery.AssignerModule + ".placement."
for key := range p.said {
c, raised := strings.CutSuffix(key, ".raised")
if !raised || !strings.HasPrefix(c, prefix) || now[c] {
continue
}
node, module, _ := strings.Cut(strings.TrimPrefix(c, prefix), ".")
if err := p.say(assign.PlacementCondition(assign.Pair{Node: node, Module: module}, false)); err != nil {
p.logf("the placement condition of %s on %s could not be cleared, and is cleared at the next pass: %v", module, node, err)
continue
}
delete(p.said, key)
}
return frozen
}
// yeses writes a placed node's wants naming the build alone, and sees to the moves a node that composes its own judged a
// person must allow (ADR 0547 §3): for a have of phase
// waiting-yes it raises the record ask once, naming the composition waiting and the node's judgement, and on a yes writes
// the want again naming it; a yes once written is kept while the want stays at its build, so the node is not asked again.
// the want again naming it; a yes once written is kept while the want stays at its build, so the node is not asked again. A
// move of the build alone (the have says so) to a build an approved release carries is given its yes by that Approve, the
// want naming the release, with no ask (ADR 0601 §1).
func (p *passer) yeses(ctx context.Context, module string, m assign.Module, before map[string]delivery.Want, out *assign.Outcome) {
for _, node := range m.Nodes {
if m.Self == nil || !m.Self(node) {
@@ -959,25 +884,37 @@ func (p *passer) yeses(ctx context.Context, module string, m assign.Module, befo
if w.Removed {
continue
}
yes := ""
yes, release := "", ""
if had && was.FP == w.FP {
yes = was.Yes
yes, release = was.Yes, was.Release
}
key, err := delivery.HaveKey(node, module)
if err != nil {
continue
}
if h, err := delivery.DecodeHave(p.w.value(delivery.HaveBucket, key)); err == nil && h.Phase == delivery.PhaseWaitingYes && h.Waits != yes {
if m.Allowed != nil && m.Allowed(node, h.Waits) {
yes = h.Waits
} else {
switch {
case m.Allowed != nil && m.Allowed(node, h.Waits):
yes, release = h.Waits, ""
case h.BuildAlone && h.FP != w.FP && m.Release != nil:
// A move of the build alone, to the build an approved release carries: that Approve is the yes (ADR 0601 §1),
// named in the want. A move that changes a setting or a secret too, or one the node could not tell, is asked.
if r, ok := m.Release(w.FP); ok {
yes, release = h.Waits, r
break
}
p.recordAsk(ctx, module, w.FP, assign.RecordNeed{Node: node, Digest: h.Waits, Judgement: h.Judgement, Words: h.Detail})
default:
p.recordAsk(ctx, module, w.FP, assign.RecordNeed{Node: node, Digest: h.Waits, Judgement: h.Judgement, Words: h.Detail})
}
}
// A want carried over from before the node composed its own may still name the controller's composition; a placed
// node's want names the build alone (ADR 0547 §2).
if yes != w.Yes || w.Composed != "" || !w.Self {
w.Yes, w.Composed, w.Self = yes, "", true
if yes == "" {
release = ""
}
if yes != w.Yes || release != w.Release || w.Composed != "" || !w.Self {
w.Yes, w.Release, w.Composed, w.Self = yes, release, "", true
out.Wants[node] = w
}
}
@@ -991,58 +928,6 @@ func chosenOf(chosen map[string]catalogue.PersonPolicy, module string) *catalogu
return nil
}
// policySplits writes the finding policy-split for each module whose policy at the assigner differs from one the controller
// writes beside a composition of it, and deletes it once they agree (ADR 0565 §8): written when a split begins, never again
// while it stands. The first pass after a start deletes every one a previous process left that no longer splits.
func (p *passer) policySplits(manifests map[string]catalogue.Manifest, chosen map[string]catalogue.PersonPolicy) {
if p.findings.Put == nil {
return
}
now := time.Now
if p.now != nil {
now = p.now
}
split := map[string]bool{}
theirs := map[string][]string{}
if !p.w.handedAway() { // the controller writes no policy from the lease write (ADR 0571 §6)
theirs = assign.ControllerPolicies(p.w.read(delivery.ControllerComposedBucket))
}
for _, s := range assign.Splits(manifests, chosen, theirs) {
split[s.Module] = true
if p.splits[s.Module] {
continue
}
value, err := assign.PolicySplitFinding(s, now())
if err == nil {
err = p.findings.Put(assign.PolicySplitKey(s.Module), value)
}
if err != nil {
p.logf("the policy split of %s is not written: %v", s.Module, err)
delete(split, s.Module)
}
}
gone := p.splits
if gone == nil && p.findings.Keys != nil {
gone = map[string]bool{}
if keys, err := p.findings.Keys(); err == nil {
for _, k := range keys {
if m, ok := strings.CutSuffix(strings.TrimPrefix(k, "module."), "."+assign.KindPolicySplit); ok && strings.HasPrefix(k, "module.") {
gone[m] = true
}
}
}
}
for m := range gone {
if !split[m] {
if err := p.findings.Delete(assign.PolicySplitKey(m)); err != nil {
p.logf("the policy split of %s is not deleted: %v", m, err)
split[m] = true // tried again at the next pass
}
}
}
p.splits = split
}
// provisionsOf answers, per module, the provisions it consumes on a node, by key (ADR 0570 §4), from the seat
// mesh-provisioner's state; nil while the state is not read (the seat not delivered yet), so no want waits on it. A failed
// provision is also said as the assigner's finding `provider.<key>.provision-failed` (a waiting one is the watchman's
@@ -1246,6 +1131,26 @@ func (p *passer) seatHolders(ctx context.Context) {
}
}
// offered says whether some manifest of the trunk, or a module the mesh provides itself, offers a provision under its name
// (a module offers its own name too, catalogue.Manifest.Offers): what a requirement nobody offers is told from.
func offered(manifests map[string]catalogue.Manifest) func(string) bool {
names := map[string]bool{}
add := func(m catalogue.Manifest) {
for _, o := range m.Offers() {
names[o] = true
}
}
for _, m := range manifests {
add(m)
}
for _, module := range delivery.ProvidedModules() {
if m, _, err := delivery.ProvidedManifest(module); err == nil {
add(m)
}
}
return func(provision string) bool { return names[provision] }
}
// answering is, by provision, the manifests of the modules placed on some node that provide it: what a want's
// provisions are judged by (catalogue.Awaited, issue 774). A provider in the catalogue placed nowhere answers nobody.
func answering(manifests map[string]catalogue.Manifest, n assign.Nodes) func(string) []catalogue.Manifest {
File diff suppressed because it is too large Load Diff
@@ -10,7 +10,7 @@ import (
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
@@ -55,8 +55,8 @@ type putActs struct {
snapshot func(ctx context.Context, node, module string) (string, error)
}
func newPutActs(bus caller, wake func(), logf func(string, ...any), leaseAway func() bool) *putActs {
a := &putActs{rec: handacts.Recorder{Ask: bus.Call, LeaseAway: leaseAway}, bus: bus, wake: wake, logf: logf, snapping: map[string]bool{}, snapped: map[string]string{}, snapNo: map[string]string{}, warranted: map[string]bool{}, warrantNo: map[string]string{},
func newPutActs(bus caller, wake func(), logf func(string, ...any)) *putActs {
a := &putActs{rec: handacts.Recorder{Ask: bus.Call}, bus: bus, wake: wake, logf: logf, snapping: map[string]bool{}, snapped: map[string]string{}, snapNo: map[string]string{}, warranted: map[string]bool{}, warrantNo: map[string]string{},
said: map[string]string{}}
a.snapshot = a.snapshotTheBus
return a
@@ -97,9 +97,9 @@ func (p *passer) puts(ctx context.Context) {
if len(all) == 0 {
return
}
assigned, _ := p.w.assigned()
assigned := p.w.assigned()
trunk := p.w.trunk()
get := kvGetter(p.kvs[delivery.ManifestsBucket])
get := p.w.getter(delivery.ManifestsBucket) // the indexer's manifests, as its first view and views family say them
wanted := p.w.wantsOf()
modules := make([]string, 0, len(all))
for m := range all {
@@ -195,14 +195,15 @@ func (p *passer) putWorld(node, module string, put delivery.Put, manifest catalo
w.Recorded = policy == catalogue.PolicyRecord
if akey, err := delivery.RecordAskKey(node, module, d.Digest); err == nil && put.Ask == akey {
// Only this put's own ask counts: an ask (and its answer) of an earlier put under the same key is another asking.
if a, err := delivery.DecodeAssignerAsk(akey, p.w.value(delivery.AssignerAsksBucket, akey)); err == nil && a.Asking == assign.PutAsking(put) {
if a, ask, err := assign.DecodeAsk(akey, p.w.value(delivery.AssignerAsksBucket, akey)); err == nil && a.Asking == assign.PutAsking(put) {
switch {
case p.answeredWith(akey, assign.OptionYes):
w.Answer = assign.OptionYes
case p.answeredWith(akey, assign.OptionNo):
w.Answer = assign.OptionNo
default:
w.AskExpired = !a.Ask.Expires.IsZero() && w.Now.After(a.Ask.Expires)
expires := assign.ExpiresAt(ask)
w.AskExpired = !expires.IsZero() && w.Now.After(expires)
}
}
}
@@ -226,9 +227,9 @@ func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest,
out = &d
}
granted := false
if gkey, err := delivery.ControllerGrantKey(module, fp); err == nil {
if gkey, err := delivery.PartGrantKey(module, fp); err == nil {
if raw := p.w.value(p.w.grantsBucket(), gkey); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil {
if g, err := delivery.DecodeGrant(raw); err == nil {
granted = g.NamesNode(node, module)
}
}
@@ -242,13 +243,13 @@ func (p *passer) actOn(ctx context.Context, node, module, id string, w assign.Pu
// The put's own ask, written again over one an earlier put left under the key (its asking differs), so the teller
// raises it anew under its own id.
akey := next.Put.Ask
current, err := delivery.DecodeAssignerAsk(akey, p.w.value(delivery.AssignerAsksBucket, akey))
current, _, err := assign.DecodeAsk(akey, p.w.value(delivery.AssignerAsksBucket, akey))
if err != nil || current.Asking != assign.PutAsking(next.Put) {
a, err := assign.PutAsk(akey, node, module, next.Put, *w.Composed, w.Bus, w.Now)
if err != nil {
p.logf("the ask about putting %s on %s is not written: %v", module, node, err)
} else {
p.put(ctx, delivery.AssignerAsksBucket, akey, func() ([]byte, error) { return delivery.EncodeAssignerAsk(akey, a) })
p.put(ctx, delivery.AssignerAsksBucket, akey, func() ([]byte, error) { return assign.EncodeAsk(akey, a) })
}
}
}
@@ -257,15 +258,15 @@ func (p *passer) actOn(ctx context.Context, node, module, id string, w assign.Pu
}
}
// performs checks that the act about to be performed is the one the yes of the ask at key bound (asks.Option.Performs).
func (p *passer) performs(key string, act asks.Act) error {
a, err := delivery.DecodeAssignerAsk(key, p.w.value(delivery.AssignerAsksBucket, key))
// performs checks that the act about to be performed is the one the yes of the ask at key bound (occ.Performs).
func (p *passer) performs(key string, act occ.Act) error {
_, ask, err := assign.DecodeAsk(key, p.w.value(delivery.AssignerAsksBucket, key))
if err != nil {
return fmt.Errorf("the ask %s cannot be read: %w", key, err)
}
for _, o := range a.Ask.Options {
for _, o := range ask.Options {
if o.ID == assign.OptionYes {
return o.Performs(act)
return occ.Performs(o, act)
}
}
return fmt.Errorf("the ask %s offers no yes", key)
@@ -10,22 +10,18 @@ import (
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/compose/judge"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-assigner/internal/assign"
"mesh-delivery-assigner/internal/judge"
)
// The seat mesh-placement's holder in the assigner's process (novox/hq ADR 0567 §2-§8): each pass follows every open
@@ -59,80 +55,23 @@ func streamPublisher(ctx context.Context, js jetstream.JetStream, node string) p
}
}
// seatActs keeps what the seat's calls of the controller came to while this process runs: the export read again at a
// seed's warrant, and each call that keeps the controller's assignments in step (§8).
// seatActs keeps the seat's acts while this process runs: the warrant recorded in the hand-act log. The seat calls no verb of
// the controller: placement is the one source (ADR 0567 §8, issue 774).
type seatActs struct {
mu sync.Mutex
bus caller
wake func()
logf func(string, ...any)
export map[string]exportCheck
// calls is, by pair (node and module, or node and provision for a pin), what the last call for it came to.
calls map[string]string
// outcomes is, by the key of a call's condition, the last call made for its pair once answered or refused: refused
// "" when the controller answered it (issue 774: a refused call is a condition, never only a journal line).
outcomes map[string]callOutcome
// pins is, by node, the controller's pins as its placement-export last answered them (pinsOf).
pins map[string]pinRead
bus caller
wake func()
logf func(string, ...any)
// rec records the seat's acts in the hand-act log (ADR 0565 §11).
rec handacts.Recorder
}
// callOutcome is what one call of the controller came to: the call, and the controller's refusal ("" when it answered).
type callOutcome struct {
call assign.ControllerCall
refused string
}
// exportCheck is the export read again for one proposal: running, its answer, or why it was not read.
type exportCheck struct {
running bool
export map[string]delivery.Placement
err string
}
func newSeatActs(bus caller, wake func(), logf func(string, ...any), leaseAway func() bool) *seatActs {
return &seatActs{bus: bus, wake: wake, logf: logf, export: map[string]exportCheck{}, calls: map[string]string{},
outcomes: map[string]callOutcome{}, pins: map[string]pinRead{}, rec: handacts.Recorder{Ask: bus.Call, LeaseAway: leaseAway}}
}
// exportOf is the export read again for a proposal, asked once in the background.
func (a *seatActs) exportOf(id string) exportCheck {
a.mu.Lock()
defer a.mu.Unlock()
if c, ok := a.export[id]; ok {
return c
}
a.export[id] = exportCheck{running: true}
go func() {
defer a.wake()
raw, err := a.bus.Call(exportCall, map[string]any{})
c := exportCheck{}
if err != nil {
c.err = "the controller's placement-export did not answer: " + err.Error()
} else if c.export, err = assign.ExportOf(raw); err != nil {
c.err = err.Error()
}
a.mu.Lock()
a.export[id] = c
a.mu.Unlock()
}()
return exportCheck{running: true}
}
// forget drops what is kept of a proposal's export, so a later warrant reads it again.
func (a *seatActs) forget(id string) {
if a == nil {
return
}
a.mu.Lock()
delete(a.export, id)
a.mu.Unlock()
func newSeatActs(bus caller, wake func(), logf func(string, ...any)) *seatActs {
return &seatActs{bus: bus, wake: wake, logf: logf, rec: handacts.Recorder{Ask: bus.Call}}
}
// record records once, in the hand-act log, that placement was written on the operator's warrant (§5): the router's
// record of the teller's ask says who chose and how (ADR 0274). The log is the seat mesh-hand-acts' from its delivery, and
// the controller's while that seat has no holder and the controller holds its lease (ADR 0565 §11, handacts.Recorder).
// the controller's while that seat has no holder (ADR 0565 §11, handacts.Recorder).
func (a *seatActs) record(warrant string) {
if a == nil || warrant == "" {
return
@@ -147,257 +86,35 @@ func (a *seatActs) record(warrant string) {
}()
}
// handAct records a hand-back in the hand-act log (§7), as record does.
func (a *seatActs) handAct(prop delivery.PlacementProposal) {
go func() {
if taken, err := a.rec.Record(handacts.Request{What: "mesh-placement hand-back of " + strings.Join(prop.Nodes(), ", "),
Why: prop.Why, Cause: "asked by " + prop.By + ", approved by the operator"}); err != nil {
a.logf("the hand-back %s is not recorded in the hand-act log: %v", prop.ID, err)
} else {
a.logf("the hand-back %s is recorded in the log of %s", prop.ID, taken.Log)
}
}()
}
// callAll makes the controller's calls in order in the background, each said in the journal and kept by its pair.
func (a *seatActs) callAll(calls []assign.ControllerCall) {
if len(calls) == 0 {
// clearStaleCalls clears, once, the conditions of the refused controller calls a previous process raised (findings of the
// kinds controller-call and controller-pin): the assigner calls no verb of the controller any more (ADR 0567 §8, issue 774),
// so nothing would clear them. It is tried again at the next pass while the findings cannot be read.
func (p *passer) clearStaleCalls() {
if p.callsCleared || p.say == nil || p.findings.Keys == nil {
return
}
a.mu.Lock()
for _, c := range calls {
a.calls[c.Node+"."+c.Module] = c.Key() + " is asked"
}
a.mu.Unlock()
go func() {
defer a.wake()
for _, c := range calls {
words, refused := c.Key()+" answered", ""
if _, err := a.bus.Call("seat:"+catalogue.ControllerSeatName+"."+c.Verb, c.Args); err != nil {
refused = err.Error()
words = c.Key() + " refused: " + refused
}
a.logf("keeping the controller in step with placement: %s", words)
a.mu.Lock()
a.calls[c.Node+"."+c.Module] = words
a.outcomes[assign.CallConditionKey(c)] = callOutcome{call: c, refused: refused}
a.mu.Unlock()
}
}()
}
// outcomesNow is a copy of every call's last outcome, by the key of its condition.
func (a *seatActs) outcomesNow() map[string]callOutcome {
if a == nil {
return nil
}
a.mu.Lock()
defer a.mu.Unlock()
return maps.Clone(a.outcomes)
}
// pinRead is the controller's pins of one node as its placement-export answered them, for the generation of the
// controller's compositions it was asked at: a pin made at the controller changes a composition, which the assigner
// watches, so a read is asked again only after such a change, never on a timer.
type pinRead struct {
gen uint64
running bool
known bool
pins map[string]delivery.Provider
}
// pinsOf is the controller's pins of node as last read, and whether a read answered; a read is asked in the background
// when none was made at gen, and wakes a pass when it answers.
func (a *seatActs) pinsOf(node string, gen uint64) (map[string]delivery.Provider, bool) {
if a == nil {
return nil, false
}
a.mu.Lock()
defer a.mu.Unlock()
r, ok := a.pins[node]
if !ok || (r.gen != gen && !r.running) {
r.gen, r.running = gen, true
a.pins[node] = r
go func() {
defer a.wake()
raw, err := a.bus.Call(exportCall, map[string]any{"node": node})
var pins map[string]delivery.Provider
var known bool
if err == nil {
var export map[string]delivery.Placement
if export, err = assign.ExportOf(raw); err == nil {
pins, known = export[node].Pins, true
}
}
if err != nil {
a.logf("the controller's pins of %s were not read, so a refused pin's condition there waits: %v", node, err)
}
a.mu.Lock()
now := a.pins[node]
now.running = false
if known {
now.pins, now.known = pins, true
}
a.pins[node] = now
a.mu.Unlock()
}()
}
return r.pins, r.known
}
// sayCalls makes a refused call of the controller loud (issue 774): the condition of each pair whose last call was refused
// is raised once per refusal, naming the verb, the node, the module and the controller's words. It is cleared when a later
// call for the pair is answered, or once the pair, seen differing since the refusal, agrees between the two sources (a
// person made the call by hand): for a module's place the controller's assignments and the placement, for a pin the
// controller's pins (its placement-export, read again after its compositions change) and the placement's. The first pass
// after a start takes over what a previous process raised (its findings) and clears each whose pair agrees; until the
// findings are read it is tried again at every pass. A say that fails is made again at the next pass.
func (p *passer) sayCalls(differ []assign.Pair) {
if p.say == nil {
return
}
if p.callSaid == nil {
p.callSaid = map[string]*callSaid{}
}
differs := map[string]bool{}
for _, d := range differ {
differs[d.Node+"."+d.Module] = true
}
outcomes := p.seat.outcomesNow()
if !p.callsSwept {
p.callsSwept = p.sweepCalls(outcomes)
}
for key, o := range outcomes {
said := p.callSaid[key]
switch {
case o.refused != "" && (said == nil || said.adopted || said.refused != o.refused):
if err := p.say(assign.CallRefusedCondition(o.call, o.refused, true, "")); err != nil {
p.logf("that %s was refused could not be said, and is said at the next pass: %v", o.call.Key(), err)
continue
}
p.callSaid[key] = &callSaid{call: o.call, refused: o.refused}
case o.refused == "" && said != nil:
if err := p.say(assign.CallRefusedCondition(o.call, "", false, o.call.Key()+" was answered.")); err != nil {
p.logf("the condition of %s could not be cleared, and is cleared at the next pass: %v", o.call.Key(), err)
continue
}
delete(p.callSaid, key)
}
}
var placements map[string]delivery.Placement
for key, said := range p.callSaid {
c := said.call
var apart bool
if c.CallKind() == assign.KindControllerCall {
apart = differs[c.Node+"."+c.Module]
} else {
theirs, known := p.seat.pinsOf(c.Node, p.w.bucketGen(delivery.ControllerComposedBucket))
if !known {
continue
}
if placements == nil {
placements, _ = p.seatPlacements()
}
ours, has := placements[c.Node].Pins[c.Module]
to, had := theirs[c.Module]
apart = has != had || ours != to
}
if apart {
said.differed = true
continue
}
if !said.differed {
continue // not seen differing since the refusal: the world may not hold the write yet
}
if err := p.say(assign.CallRefusedCondition(c, "", false, agreeWords(c))); err != nil {
p.logf("the condition of %s could not be cleared, and is cleared at the next pass: %v", c.Key(), err)
continue
}
if o, ok := outcomes[key]; ok {
p.seat.forgetOutcome(key, o)
}
delete(p.callSaid, key)
}
}
// sweepCalls takes over the refused calls' conditions a previous process raised, from the findings: each is followed as
// one seen differing, so it clears at the first pass its pair agrees. It answers whether the findings were read.
func (p *passer) sweepCalls(outcomes map[string]callOutcome) bool {
if p.findings.Keys == nil {
return true
}
keys, err := p.findings.Keys()
if err != nil {
p.logf("the findings were not read, so the refused calls a previous process raised are taken over at the next pass: %v", err)
return false
p.logf("the findings were not read, so the refused controller calls a previous process raised are cleared at the next pass: %v", err)
return
}
p.callsCleared = true
for _, k := range keys {
rest, ok := strings.CutPrefix(k, "module.")
if !ok {
continue
}
for _, kind := range []string{assign.KindControllerCall, assign.KindControllerPin} {
pair, ok := strings.CutSuffix(rest, "."+kind)
node, module, two := strings.Cut(pair, ".")
if !ok || !two {
continue
if pair, ok := strings.CutSuffix(rest, "."+kind); ok {
if node, module, two := strings.Cut(pair, "."); two {
if err := p.say(assign.CallClearedCondition(kind, node, module)); err != nil {
p.callsCleared = false
p.logf("the condition of the old %s call for %s on %s could not be cleared, and is cleared at the next pass: %v", kind, module, node, err)
}
}
}
verb := "assign"
if kind == assign.KindControllerPin {
verb = "pin"
}
c := assign.ControllerCall{Verb: verb, Node: node, Module: module}
key := assign.CallConditionKey(c)
if _, made := outcomes[key]; made || p.callSaid[key] != nil {
continue
}
p.callSaid[key] = &callSaid{call: c, differed: true, adopted: true}
}
}
return true
}
func agreeWords(c assign.ControllerCall) string {
if c.CallKind() == assign.KindControllerPin {
return "the controller's pins and the placement agree on " + c.Module + " for " + c.Node + "."
}
return "the controller's assignments and the placement agree on " + c.Module + " on " + c.Node + "."
}
// callSaid is a raised call's condition: its call, the refusal it said, whether its pair was seen differing since, and
// whether it was taken over from a previous process.
type callSaid struct {
call assign.ControllerCall
refused string
differed bool
adopted bool
}
// forgetOutcome drops a refused call's outcome once its condition is cleared, unless a later call replaced it.
func (a *seatActs) forgetOutcome(key string, o callOutcome) {
a.mu.Lock()
defer a.mu.Unlock()
if now, ok := a.outcomes[key]; ok && now.refused == o.refused && now.call.Verb == o.call.Verb {
delete(a.outcomes, key)
}
}
// callOf is what the last call for a pair came to, "" when none was made.
func (a *seatActs) callOf(node, module string) string {
if a == nil {
return ""
}
a.mu.Lock()
defer a.mu.Unlock()
return a.calls[node+"."+module]
}
// withCalls names, on each pair the two sources do not both name, the call the assigner made for it (§8).
func (p *passer) withCalls(pairs []assign.Pair) []assign.Pair {
for i := range pairs {
pairs[i].Call = p.seat.callOf(pairs[i].Node, pairs[i].Module)
}
return pairs
}
// sayWaits raises, once, that the seat waits for the states not yet bound, and clears it once, when all are.
@@ -407,7 +124,7 @@ func (p *passer) sayWaits(unbound []string) {
if p.say == nil || p.said[c.Key] && p.saidWaits == state {
return
}
if c.State == asks.StateCleared && p.saidWaits == "" {
if c.State == occ.StateCleared && p.saidWaits == "" {
return // nothing was raised by this process
}
if err := p.say(c); err != nil {
@@ -526,11 +243,16 @@ func (p *passer) ask(ctx context.Context, prop delivery.PlacementProposal, akey,
p.logf("the ask about the proposal %s is not written: %v", prop.ID, err)
return prop, false
}
p.put(ctx, delivery.AssignerAsksBucket, akey, func() ([]byte, error) { return delivery.EncodeAssignerAsk(akey, a) })
if pk, err := delivery.PulseAskKey(prop.ID); err == nil {
p.pulse(ctx, pk, a.Ask.Expires.Sub(p.clock()))
placed, err := assign.AskOf(a)
if err != nil {
p.logf("the ask about the proposal %s is not written: %v", prop.ID, err)
return prop, false
}
prop.State, prop.Ask = delivery.ProposalAsked, a.Ask.ID
p.put(ctx, delivery.AssignerAsksBucket, akey, func() ([]byte, error) { return assign.EncodeAsk(akey, a) })
if pk, err := delivery.PulseAskKey(prop.ID); err == nil {
p.pulse(ctx, pk, assign.ExpiresAt(placed).Sub(p.clock()))
}
prop.State, prop.Ask = delivery.ProposalAsked, placed.ID
return prop, true
}
@@ -544,17 +266,22 @@ func (p *passer) stepProposal(ctx context.Context, prop delivery.PlacementPropos
p.endProposal(ctx, prop, prop.Withdrawn)
return
}
if prop.Seed || prop.HandBack {
// Adopt and hand-back are gone since the lease write (ADR 0547 §11, ADR 0567 §11): a proposal one left open ends.
p.endProposal(ctx, prop, "adopt and hand-back are gone since the lease write: placement is the mesh's alone")
return
}
if prop.Warrant != nil {
p.addNode(ctx, prop, revs)
return
}
akey, _ := delivery.PlacementAskKey(prop.ID)
again := !prop.Again.IsZero() && (prop.Verdict == nil || prop.Again.After(prop.Verdict.At))
if prop.State == delivery.ProposalAsked {
askPulse, _ := delivery.PulseAskKey(prop.ID)
switch {
case p.answeredWith(akey, assign.OptionYes):
if prop.HandBack {
p.handBack(ctx, prop, akey, revs)
} else {
p.warranted(ctx, prop, rev, akey, current, revs)
}
p.warranted(ctx, prop, rev, akey, current, revs)
return
case p.answeredWith(akey, assign.OptionNo):
p.endProposal(ctx, prop, "declined by the operator")
@@ -564,19 +291,6 @@ func (p *passer) stepProposal(ctx context.Context, prop delivery.PlacementPropos
return
}
}
if prop.HandBack {
// The operator's alone (§7): no node answers it; its ask is raised at once, and again on again.
if prop.State != delivery.ProposalAsked || again {
brief := fmt.Sprintf("%s (asked by %s): every node's placement key is deleted, %s, and each node returns to the "+
"controller's compositions; the history is kept", prop.Why, prop.By, strings.Join(prop.Nodes(), ", "))
now := p.clock().UTC()
prop.Verdict = &delivery.PlacementVerdict{State: delivery.VerdictPass, At: now}
if prop, ok := p.ask(ctx, prop, akey, brief); ok {
p.writeProposal(ctx, prop, rev)
}
}
return
}
if moved(prop, revs) {
p.layAgain(ctx, prop, rev, current, revs, "a key it was laid over moved since")
return
@@ -643,67 +357,35 @@ func moved(prop delivery.PlacementProposal, revs map[string]uint64) bool {
return false
}
// intended is, by proposal and the revisions it was laid over, what Intended computed, kept while this process runs.
// intendedOf is, by proposal and the revisions it was laid over, what Intended computed, kept while this process runs.
func (p *passer) intendedOf(prop delivery.PlacementProposal, current map[string]delivery.Placement, cands map[string]delivery.PlacementCandidate) map[string]map[string]bool {
key := fmt.Sprint(prop.ID, prop.Base)
if got, ok := p.intended[key]; ok {
return got
}
got, _, _ := p.compute(prop, current, cands)
got, _ := p.compute(prop, current, cands)
p.intended[key] = got
return got
}
// compute composes every node with the current placement and with the proposal laid over it (judge.Intended).
func (p *passer) compute(prop delivery.PlacementProposal, current map[string]delivery.Placement, cands map[string]delivery.PlacementCandidate) (map[string]map[string]bool, []string, []string) {
// compute is what the proposal changes on each node by its placement, and the nodes that answer it: every node with a
// placement or a record (judge.Intended, ADR 0593 §8). The holder composes nothing: each node says what its compositions
// change in its answer.
func (p *passer) compute(prop delivery.PlacementProposal, current map[string]delivery.Placement, cands map[string]delivery.PlacementCandidate) (map[string]map[string]bool, []string) {
laid := map[string]*delivery.Placement{}
for _, node := range prop.Nodes() {
if c, ok := cands[node]; ok && !c.Reached {
laid[node] = c.Placement
}
}
get := kvGetter(p.kvs[delivery.ManifestsBucket])
r := worldReader{w: p.w, manifests: get}
// A read that fails is said, never dropped silently (novox/hq ADR 0578 §3); a refused manifest no longer fails it, and
// its module is held, named in the input's Refused.
var now, with *compose.Input
if in, err := frombus.Input(context.Background(), r, frombus.Mesh{}, frombus.Scope{Every: true}); err == nil {
now = &in
} else if p.logf != nil {
p.logf("proposal %s: the current composition input cannot be read, so what it changes is judged without it: %v", prop.ID, err)
}
if in, err := frombus.Input(context.Background(), r, frombus.Mesh{}, frombus.Scope{Every: true, Placements: laid}); err == nil {
with = &in
} else if p.logf != nil {
p.logf("proposal %s: the composition input with the proposal cannot be read, so what it changes is judged without it: %v", prop.ID, err)
}
if now != nil && len(now.Refused) > 0 && p.logf != nil {
var held []string
for _, m := range slices.Sorted(maps.Keys(now.Refused)) {
held = append(held, m+" ("+now.Refused[m]+")")
}
p.logf("proposal %s: held as refused by the indexer's judge: %s", prop.ID, strings.Join(held, "; "))
}
nodes := map[string]bool{}
for n := range current {
nodes[n] = true
}
if records, err := r.Records(context.Background()); err == nil {
for n := range records {
nodes[n] = true
}
for n := range p.w.records() {
nodes[n] = true
}
names := make([]string, 0, len(nodes))
for n := range nodes {
names = append(names, n)
}
fps := map[string]string{}
for name, m := range p.w.trunk() {
if m.FP2 != "" {
fps[name] = m.FP2
}
}
return judge.Intended(now, with, current, laid, names, fps)
return judge.Intended(current, laid, slices.Sorted(maps.Keys(nodes)))
}
// mustAnswer finds the nodes the proposal must hear from (§3), writes a reached candidate for each whose key it does not
@@ -714,7 +396,7 @@ func (p *passer) mustAnswer(ctx context.Context, prop delivery.PlacementProposal
return // the verbs write the candidates before the proposal: wait to hear them
}
}
intended, must, notes := p.compute(prop, current, cands)
intended, must := p.compute(prop, current, cands)
p.intended[fmt.Sprint(prop.ID, prop.Base)] = intended
for _, node := range must {
if _, ok := cands[node]; ok {
@@ -736,9 +418,6 @@ func (p *passer) mustAnswer(ctx context.Context, prop delivery.PlacementProposal
}
}
prop.Must = must
if len(notes) > 0 {
prop.Words = strings.TrimPrefix(prop.Words+"; "+strings.Join(notes, "; "), "; ")
}
p.writeProposal(ctx, prop, rev)
}
@@ -747,8 +426,7 @@ func (p *passer) mustAnswer(ctx context.Context, prop delivery.PlacementProposal
func (p *passer) layAgain(ctx context.Context, prop delivery.PlacementProposal, rev uint64, current map[string]delivery.Placement, revs map[string]uint64, why string) {
held, recorded := p.lays(ctx)
again, laid, findings := assign.Propose(assign.ProposeIn{ID: prop.ID, By: prop.By, Why: prop.Why, At: prop.At, Changes: prop.Changes,
Current: current, Revisions: revs, Held: held, Recorded: recorded,
Seed: prop.Seed, ExportDigest: prop.ExportDigest, Leased: p.w.handedAway()})
Current: current, Revisions: revs, Held: held, Recorded: recorded})
if len(findings) > 0 {
p.endProposal(ctx, prop, why+", and laid over the keys now it is refused: "+strings.Join(findings, "; "))
return
@@ -761,7 +439,6 @@ func (p *passer) layAgain(ctx context.Context, prop delivery.PlacementProposal,
p.put(ctx, delivery.PlacementCandidatesBucket, key, func() ([]byte, error) { return delivery.EncodePlacementCandidate(c) })
}
again.Written, again.Words = p.clock().UTC(), why+": laid again over the keys as they stand, to be judged and asked again"
p.seat.forget(prop.ID)
p.writeProposal(ctx, again, rev)
}
@@ -774,39 +451,18 @@ func (p *passer) lays(ctx context.Context) (func(string) bool, func(string) bool
held[name] = true
}
}
records, _ := worldReader{w: p.w}.Records(ctx)
records := p.w.records()
return func(m string) bool { return held[m] }, func(n string) bool { _, ok := records[n]; return ok }
}
// warranted writes an approved proposal (§5): for the seed, after the export is read again and found unchanged (§7);
// each changed key by compare-and-set at the revision it was laid over; then the warrant recorded, the candidates and the
// proposal deleted, placement-changed said, and the controller kept in step (§8).
// warranted writes an approved proposal (§5): each changed key by compare-and-set at the revision it was laid over; then the warrant recorded, the candidates and the
// proposal deleted, and placement-changed said.
func (p *passer) warranted(ctx context.Context, prop delivery.PlacementProposal, rev uint64, akey string, current map[string]delivery.Placement, revs map[string]uint64) {
if err := p.performs(akey, assign.PlacementAct(prop)); err != nil {
p.endProposal(ctx, prop, "its warrant does not bind these changes, so nothing is written: "+err.Error())
return
}
answer, _ := p.w.answer(akey)
if prop.Seed {
if p.seat == nil {
return
}
c := p.seat.exportOf(prop.ID)
switch {
case c.running:
return // its answer wakes the next pass
case c.err != "":
if words := "the export cannot be read again at the warrant: " + c.err; prop.Words != words {
prop.Words = words
p.writeProposal(ctx, prop, rev)
}
return
case delivery.ExportDigest(c.export) != prop.ExportDigest:
p.endProposal(ctx, prop, "the controller's export changed since the seed was proposed: the new export is proposed as the seed")
p.proposeSeed(ctx, prop, c.export)
return
}
}
if moved(prop, revs) {
p.layAgain(ctx, prop, rev, current, revs, "a key moved since it was approved, so its warrant is spent")
return
@@ -831,7 +487,7 @@ func (p *passer) warranted(ctx context.Context, prop delivery.PlacementProposal,
// Another writer came first (a hand-back): what is written stands and is followed through; the rest is laid
// again over the keys as this write left them.
p.logf("placement %s is not written at revision %d (%v): the rest of the proposal %s is laid again", w.Key, w.Base, err, prop.ID)
p.followWrite(prop, answer, changes, current, writtenLaid)
p.followWrite(prop, answer, changes)
if rest, ok := assign.Rest(prop, writtenLaid); ok {
p.layAgain(ctx, rest, rev, after, afterRevs, "a key moved while it was written, after "+strings.Join(sortedNodes(writtenLaid), ", ")+" were")
} else {
@@ -857,22 +513,83 @@ func (p *passer) warranted(ctx context.Context, prop delivery.PlacementProposal,
p.deletePulses(ctx, prop.ID)
_ = p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID)
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
p.followWrite(prop, answer, changes, current, writtenLaid)
p.followWrite(prop, answer, changes)
}
// followWrite follows the keys a warrant wrote: the warrant recorded, placement-changed said, and, until the lease
// write, the controller's assignments kept in step for them (§8). The seed calls no controller verb: it is what the
// controller holds already (§9).
func (p *passer) followWrite(prop delivery.PlacementProposal, answer delivery.Answer, changes []delivery.PlacementChange,
before map[string]delivery.Placement, written map[string]*delivery.Placement) {
// addNode writes a joining node on the warrant of the enrolment that issued its token (ADR 0587 §7): the operator's Approve
// was the issue's, so no node is waited on and no ask is raised. The key is created at the revision it was laid over (none: a
// node with a key is no joining node); the enrolment recorded the warrant in the hand-act log when it was given, so this does
// not record it again; placement-changed carries the warrant. A key written meanwhile ends the proposal, saying so.
func (p *passer) addNode(ctx context.Context, prop delivery.PlacementProposal, revs map[string]uint64) {
current, _ := p.seatPlacements()
if moved(prop, revs) {
p.endProposal(ctx, prop, "a key it was laid over was written since: the enrolment proposes the node again")
return
}
laid, findings := delivery.LayChanges(current, prop.Changes)
findings = append(findings, assign.ControllerRefusals(prop.Changes, laid)...)
node := prop.Changes[0].Node
if prop.Changes[0].Op == delivery.OpRemoveNode {
p.removeNode(ctx, prop, current, findings)
return
}
value, ok := laid[node]
if len(findings) > 0 || !ok {
p.endProposal(ctx, prop, "the node cannot be added: "+strings.Join(findings, "; "))
return
}
rev, err := p.writeNode(ctx, node, &value)
if err != nil {
p.endProposal(ctx, prop, "the node's key was not written: "+err.Error())
return
}
c := delivery.ChangeOf(delivery.Placement{Node: node}, value)
c.Before, c.After = prop.Base[node], rev
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
_ = p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID)
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
w := prop.Warrant
p.sayChanged(prop, delivery.Answer{Option: assign.OptionYes, Warrant: w.Warrant,
Words: fmt.Sprintf("the operator approved the machine %s at %s through the enrolment's ask %s", w.Node, w.Address, w.Ask)},
[]delivery.PlacementChange{c})
}
// removeNode deletes the key of a joining node whose enrolment ended unconfirmed, on that enrolment's warrant (the operator's
// decision of 2026-11-02 01:05): no node answers and no ask is raised, as for its add-node; placement-changed carries the warrant.
// A node that runs modules or holds seats is no joining node and stays (LayChanges' finding ends the proposal).
func (p *passer) removeNode(ctx context.Context, prop delivery.PlacementProposal, current map[string]delivery.Placement, findings []string) {
node := prop.Changes[0].Node
before, ok := current[node]
if len(findings) > 0 || !ok {
p.endProposal(ctx, prop, "the node cannot be taken off: "+strings.Join(findings, "; "))
return
}
rev, err := p.writeNode(ctx, node, nil)
if err != nil {
p.endProposal(ctx, prop, "the node's key was not deleted: "+err.Error())
return
}
c := delivery.ChangeOf(before, delivery.Placement{Node: node})
c.Before, c.After = prop.Base[node], rev
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
_ = p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID)
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
w := prop.Warrant
p.sayChanged(prop, delivery.Answer{Option: assign.OptionYes, Warrant: w.Warrant,
Words: fmt.Sprintf("the machine %s left placement: its enrolment (ask %s) ended before it confirmed", w.Node, w.Ask)},
[]delivery.PlacementChange{c})
}
// followWrite follows the keys a warrant wrote: the warrant recorded and placement-changed said. No verb of the
// controller is called: the placement is the one source of where modules run (ADR 0567 §8, issue 774).
func (p *passer) followWrite(prop delivery.PlacementProposal, answer delivery.Answer, changes []delivery.PlacementChange) {
if len(changes) == 0 {
return
}
p.seat.record(answer.Warrant)
p.sayChanged(prop, answer, changes)
if p.seat != nil && !p.w.handedAway() && !prop.Seed {
p.seat.callAll(assign.ControllerCalls(before, written))
}
}
func sortedNodes(m map[string]*delivery.Placement) []string {
@@ -884,44 +601,6 @@ func sortedNodes(m map[string]*delivery.Placement) []string {
return out
}
// handBack deletes every key on the operator's warrant (§7) at the revision the hand-back read, records it in the
// controller's hand-act log and says placement-changed; every node returns to the controller's compositions. A key moved
// since ends it, and the operator hands back again.
func (p *passer) handBack(ctx context.Context, prop delivery.PlacementProposal, akey string, revs map[string]uint64) {
if err := p.performs(akey, assign.PlacementAct(prop)); err != nil {
p.endProposal(ctx, prop, "its warrant does not bind this hand-back, so nothing is deleted: "+err.Error())
return
}
answer, _ := p.w.answer(akey)
current, held := p.seatPlacements()
var changes []delivery.PlacementChange
for _, node := range prop.Nodes() {
key, _ := delivery.PlacementNodeKey(node)
err := error(nil)
if held[node] != prop.Base[node] {
err = fmt.Errorf("its placement is at revision %d, the hand-back read %d", held[node], prop.Base[node])
} else {
_, err = p.writeNode(ctx, node, nil)
}
if err != nil {
p.sayChanged(prop, answer, changes)
p.endProposal(ctx, prop, fmt.Sprintf("%s moved since the hand-back was asked (%v): hand back again", key, err))
return
}
c := delivery.ChangeOf(current[node], delivery.Placement{Node: node})
c.Before = prop.Base[node]
changes = append(changes, c)
}
if p.seat != nil {
p.seat.handAct(prop)
}
p.seat.record(answer.Warrant)
_ = p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID)
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
p.deletePulses(ctx, prop.ID)
p.sayChanged(prop, answer, changes)
}
// sayChanged says placement-changed for the keys written.
func (p *passer) sayChanged(prop delivery.PlacementProposal, answer delivery.Answer, changes []delivery.PlacementChange) {
if len(changes) == 0 {
@@ -938,26 +617,6 @@ func (p *passer) sayChanged(prop delivery.PlacementProposal, answer delivery.Ans
Warrant: &delivery.PlacementWarrant{Warrant: answer.Warrant, Words: answer.Words, At: at}}, prop.ID+".changed")
}
// proposeSeed proposes an export as the seed again, by the same proposer and why.
func (p *passer) proposeSeed(ctx context.Context, was delivery.PlacementProposal, export map[string]delivery.Placement) {
held, recorded := p.lays(ctx)
now := p.clock()
seed, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: was.By, Why: was.Why, At: now,
Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{},
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export), Leased: p.w.handedAway()})
if len(findings) > 0 {
p.logf("the new export is not proposed as the seed: %s", strings.Join(findings, "; "))
return
}
for _, node := range seed.Nodes() {
key, _ := delivery.PlacementCandidateKey(seed.ID, node)
c := delivery.PlacementCandidate{Proposal: seed.ID, Placement: laid[node]}
p.put(ctx, delivery.PlacementCandidatesBucket, key, func() ([]byte, error) { return delivery.EncodePlacementCandidate(c) })
}
seed.Written = now.UTC()
p.writeProposal(ctx, seed, 0)
}
// endProposal ends a proposal: its ask, its candidates and its key deleted, and placement-withdrawn said carrying it
// whole with why it ended.
func (p *passer) endProposal(ctx context.Context, prop delivery.PlacementProposal, why string) {
@@ -973,7 +632,6 @@ func (p *passer) endProposal(ctx context.Context, prop delivery.PlacementProposa
return
}
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
p.seat.forget(prop.ID)
p.logf("the proposal %s ends: %s", prop.ID, why)
p.sayEvent(delivery.PlacementWithdrawnEvent, delivery.PlacementEvent{Proposal: prop, Ended: why}, prop.ID+".withdrawn")
}
@@ -10,7 +10,6 @@ import (
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
@@ -207,7 +206,7 @@ func (s *seatTools) doPropose(args map[string]any, by string) (any, error) {
}
for _, c := range changes {
if c.Op == delivery.OpSeed {
return nil, errors.New("the seed is adopt's alone: propose takes place, take-off, pin, unpin, hold, unhold, decide and config")
return nil, errors.New("the seed is gone with adopt: propose takes place, take-off, pin, unpin, hold, unhold, decide and config")
}
}
current, revs, err := s.placements()
@@ -218,9 +217,16 @@ func (s *seatTools) doPropose(args map[string]any, by string) (any, error) {
if err != nil {
return nil, err
}
warrant, err := warrantOf(args["warrant"])
if err != nil {
return nil, err
}
now := s.now()
p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now,
Changes: changes, Current: current, Revisions: revs, Held: held, Recorded: recorded, Leased: s.leased()})
Changes: changes, Current: current, Revisions: revs, Held: held, Recorded: recorded, Warrant: warrant})
if refusal := delivery.AddNodeRefusal(p, moduleOfCaller(by)); refusal != "" {
findings = append(findings, refusal)
}
if len(findings) > 0 {
return nil, fmt.Errorf("the proposal is refused: %s", strings.Join(findings, "; "))
}
@@ -230,6 +236,42 @@ func (s *seatTools) doPropose(args map[string]any, by string) (any, error) {
return proposed{Proposal: p, Follow: followProposal(p.ID)}, nil
}
// warrantOf reads propose's warrant, an enrolment's for an add-node: an object, given as one or as its text; none is none.
func warrantOf(v any) (*delivery.AddNodeWarrant, error) {
var raw []byte
switch w := v.(type) {
case nil:
return nil, nil
case string:
if strings.TrimSpace(w) == "" {
return nil, nil
}
raw = []byte(w)
default:
var err error
if raw, err = json.Marshal(w); err != nil {
return nil, err
}
}
var out delivery.AddNodeWarrant
dec := json.NewDecoder(strings.NewReader(string(raw)))
dec.DisallowUnknownFields()
if err := dec.Decode(&out); err != nil {
return nil, fmt.Errorf("the warrant is no enrolment's warrant {node, address, ask, warrant}: %w", err)
}
return &out, nil
}
// moduleOfCaller is the module a caller is, as the runtime proves it, for AddNodeRefusal: mesh-enrolment when the proven
// caller is that module's (its bus user `<node>.mesh-enrolment`, never a person's `person.mesh-enrolment`), else the caller
// as proven, which the refusal names.
func moduleOfCaller(by string) string {
if (stdio.Callers{delivery.EnrolmentModule}).Allows(by) {
return delivery.EnrolmentModule
}
return by
}
// doWithdraw is the verb withdraw (§2): the proposal is marked, and the assigner deletes it and its candidates and says
// placement-withdrawn carrying it whole.
func (s *seatTools) doWithdraw(args map[string]any, by string) (any, error) {
@@ -270,94 +312,6 @@ func (s *seatTools) doAgain(args map[string]any) (any, error) {
return map[string]any{"proposal": p.ID, "again": p.Again, "follow": followProposal(p.ID)}, nil
}
// exportCall is the controller's verb the seed is read from, by request and reply on its seat (§7).
const exportCall = "seat:" + catalogue.ControllerSeatName + ".placement-export"
// doAdopt is the verb adopt (§7): the controller's export proposed whole as the seed.
func (s *seatTools) doAdopt(args map[string]any, by string) (any, error) {
current, _, err := s.placements()
if err != nil {
return nil, err
}
if len(current) > 0 {
return nil, fmt.Errorf("placement is taken over already (%d keys): propose a change instead, or hand it back first", len(current))
}
if ids, err := s.st.keys(delivery.PlacementProposalsState); err == nil {
for _, id := range ids {
if p, err := s.proposal(id); err == nil && p.Seed {
return nil, fmt.Errorf("the seed %s is open: withdraw it first, or answer its ask", id)
}
}
}
raw, err := s.ask(exportCall, map[string]any{})
if err != nil {
return nil, fmt.Errorf("the controller's placement-export did not answer: %w", err)
}
export, err := assign.ExportOf(raw)
if err != nil {
return nil, err
}
held, recorded, err := s.lays()
if err != nil {
return nil, err
}
now := s.now()
p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now,
Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{},
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export), Leased: s.leased()})
if len(findings) > 0 {
return nil, fmt.Errorf("the seed is refused: %s", strings.Join(findings, "; "))
}
if err := s.write(p, laid); err != nil {
return nil, err
}
return proposed{Proposal: p, Follow: followProposal(p.ID)}, nil
}
// leased says the controller's lease is handed away (mesh-sdk buscompose.LeaseHandedAway, ADR 0561 §3; issue 774): the lease
// write of ADR 0547 that ends keeping the controller in step (ADR 0567 §8). A lease not read, a controller's, expired or
// unreadable, is not handed away.
func (s *seatTools) leased() bool {
raw, _, found, err := s.st.get(buscompose.ControllerLease, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, err)
return away
}
// doHandBack is the verb hand-back (§7): the operator's alone, so it is asked as a proposal is. It writes a hand-back
// proposal naming every key at its revision; the assigner raises its authorising ask and, on its warrant, deletes the keys,
// records the hand act and says placement-changed; every node returns to the controller's compositions.
func (s *seatTools) doHandBack(args map[string]any, by string) (any, error) {
if s.leased() {
return nil, errors.New("placement is the mesh's alone since the lease write: there is no controller to hand it back to")
}
why := arg(args, "why")
if why == "" {
return nil, errors.New("hand-back says why: it is recorded with the hand act")
}
if by == "" {
return nil, errors.New("the runtime names no caller: a hand-back is made by someone")
}
_, revs, err := s.placements()
if err != nil {
return nil, err
}
if len(revs) == 0 {
return nil, errors.New("there is no placement key to hand back")
}
now := s.now()
p := delivery.PlacementProposal{ID: delivery.NewProposalID(now), By: by, Why: why, At: now.UTC(), Written: now.UTC(),
HandBack: true, Base: revs, State: delivery.ProposalAnswering}
raw, err := delivery.EncodePlacementProposal(p)
if err != nil {
return nil, err
}
if err := s.st.put(delivery.PlacementProposalsState, p.ID, raw); err != nil {
return nil, fmt.Errorf("the hand-back is not written: %w", err)
}
return proposed{Proposal: p, Follow: "the operator is asked to approve it; on the warrant every key is deleted and each " +
"node returns to the controller's compositions"}, nil
}
// placementAnswer is what the verb placement answers.
type placementAnswer struct {
Keys map[string]keyed `json:"keys,omitempty"`
@@ -531,8 +485,6 @@ func (s *seatTools) tools() []stdio.Tool {
catalogue.PlacementVerbPropose: func(a map[string]any) (any, error) { return s.doPropose(a, stdio.CallerOf(a)) },
catalogue.PlacementVerbWithdraw: func(a map[string]any) (any, error) { return s.doWithdraw(a, stdio.CallerOf(a)) },
catalogue.PlacementVerbAgain: s.doAgain,
catalogue.PlacementVerbAdopt: func(a map[string]any) (any, error) { return s.doAdopt(a, stdio.CallerOf(a)) },
catalogue.PlacementVerbHandBack: func(a map[string]any) (any, error) { return s.doHandBack(a, stdio.CallerOf(a)) },
}
var out []stdio.Tool
for _, v := range seat.Serves {
@@ -13,7 +13,6 @@ import (
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
@@ -36,9 +35,8 @@ func (s memStore) history(state, key string) ([]stdio.StateRevision, error) {
return []stdio.StateRevision{{Key: key, Value: v, Revision: uint64(len(v)), Op: "put"}}, nil
}
// ADR 0567 §2, §7, §10: the seat's seven verbs over the states. propose lays a proposal and writes it with its candidates,
// or refuses naming every finding and writes nothing; withdraw and again mark it for the holder; adopt proposes the
// controller's export whole as the seed, once; hand-back records the hand act and deletes every key; placement and
// ADR 0567 §2, §7, §10: the seat's five verbs over the states. propose lays a proposal and writes it with its candidates,
// or refuses naming every finding and writes nothing; withdraw and again mark it for the holder; placement and
// proposals read every state the seat keeps, and history names the stream for what the state no longer keeps.
func TestTheSeatsVerbs(t *testing.T) {
anchor := delivery.Placement{V: delivery.Version, Node: "anchor", Modules: []delivery.PlacedModule{{Module: "web"}}}
@@ -53,32 +51,22 @@ func TestTheSeatsVerbs(t *testing.T) {
raw, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: n}})
st[haveState].values[n+".node"] = raw
}
export, _ := json.Marshal(map[string]any{"anchor": anchor})
var asked []string
s := newSeatTools(st, func(key string, body any) (json.RawMessage, error) {
asked = append(asked, key)
if key == exportCall {
out, _ := json.Marshal(map[string]any{"ok": true, "output": string(export)})
return out, nil
}
return json.RawMessage(`{"ok":true}`), nil
})
s.now = func() time.Time { return time.Date(2026, 10, 28, 20, 0, 0, 0, time.UTC) }
// adopt: the export proposed whole as the seed, once.
got, err := s.doAdopt(map[string]any{"why": "take over"}, "operator@laptop")
raw, _ := delivery.EncodePlacement(anchor)
st[delivery.PlacementState].values["placement.anchor"] = raw
// withdraw and again mark the proposal; the holder ends or judges it.
got, err := s.doPropose(map[string]any{"why": "hold", "changes": `[{"op":"hold","node":"anchor","module":"web","claim":"node-shell","scope":"node"}]`}, "op")
if err != nil {
t.Fatal(err)
}
seed := got.(proposed).Proposal
if !seed.Seed || seed.ExportDigest != delivery.ExportDigest(map[string]delivery.Placement{"anchor": anchor}) || !slices.Equal(asked, []string{exportCall}) ||
len(st[delivery.PlacementCandidatesState].values) != 1 || st[delivery.PlacementProposalsState].values[seed.ID] == nil {
t.Fatalf("adopt = %+v, asked %v", seed, asked)
}
if _, err := s.doAdopt(map[string]any{"why": "again"}, "op"); err == nil || !strings.Contains(err.Error(), "is open") {
t.Errorf("a second adopt with a seed open: %v", err)
}
// withdraw and again mark the proposal; the holder ends or judges it.
if _, err := s.doWithdraw(map[string]any{"id": seed.ID, "why": "not yet"}, "op"); err != nil {
t.Fatal(err)
}
@@ -95,10 +83,8 @@ func TestTheSeatsVerbs(t *testing.T) {
st[delivery.PlacementCandidatesState].values = map[string][]byte{}
// propose: refused whole, naming every finding; a valid move laid over the keys at their revisions.
raw, _ := delivery.EncodePlacement(anchor)
st[delivery.PlacementState].values["placement.anchor"] = raw
if _, err := s.doPropose(map[string]any{"why": "x", "changes": `[{"op":"place","node":"anchor","module":"ghost"},{"op":"hold","node":"anchor","module":"web","claim":"x"}]`}, "op"); err == nil ||
!strings.Contains(err.Error(), "ghost") || !strings.Contains(err.Error(), "lease write") || len(st[delivery.PlacementProposalsState].values) != 0 {
!strings.Contains(err.Error(), "ghost") || len(st[delivery.PlacementProposalsState].values) != 0 {
t.Errorf("a refused proposal: %v", err)
}
if _, err := s.doPropose(map[string]any{"why": "x", "changes": `[{"op":"swap","node":"anchor"}]`}, "op"); err == nil {
@@ -141,37 +127,68 @@ func TestTheSeatsVerbs(t *testing.T) {
}
}
for _, stream := range raw0.Streams {
if !reads["stream "+stream.Name] {
// The views family is no seat's stream: the readers of the assigner's states follow it (ADR 0615 §5).
if stream.Name != viewsStream && !reads["stream "+stream.Name] {
t.Errorf("no verb reads the seat's stream %s", stream.Name)
}
}
// hand-back: the operator's alone, written as a hand-back proposal naming every key at its revision, for the assigner
// to ask; nothing is deleted by the verb.
got, err = s.doHandBack(map[string]any{"why": "back"}, "op")
if hb := got.(proposed).Proposal; err != nil || !hb.HandBack || hb.Base["anchor"] != uint64(len(raw)) || len(st[delivery.PlacementState].values) != 1 {
t.Errorf("hand-back = %+v, %v", got, err)
}
// Issue 774 (ADR 0567 §8): once the controller's lease is handed away a change its verbs cannot take is proposed, and
// there is no controller to hand placement back to.
st[buscompose.ControllerLease] = &memKV{values: map[string][]byte{buscompose.ControllerLeaseKey: []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`)}}
// Issue 774 (ADR 0567 §8): every kind of change is proposed, and there is no controller to hand placement back to.
if _, err := s.doPropose(map[string]any{"why": "hold", "changes": `[{"op":"hold","node":"anchor","module":"web","claim":"node-shell","scope":"node"}]`}, "op"); err != nil {
t.Errorf("a hold after the lease write: %v", err)
t.Errorf("a hold: %v", err)
}
if _, err := s.doHandBack(map[string]any{"why": "back"}, "op"); err == nil || !strings.Contains(err.Error(), "since the lease write") {
t.Errorf("a hand-back after the lease write: %v", err)
// 774/S59/AC1 (ADR 0587 §7): an add-node from mesh-enrolment whose warrant names this node and this address is proposed, with
// the node a joining one with no record; from another caller, naming another node or address, or with no warrant it is refused.
addNode := func(by, warrant string) error {
_, err := s.doPropose(map[string]any{"why": "enrolment of ace", "warrant": warrant, "changes": []any{map[string]any{"op": "add-node", "node": "ace",
"decided": map[string]any{"address": "10.9.0.2"}, "keys": map[string]any{"overlay": "dHVubmVs"}}}}, by)
return err
}
if tools := s.tools(); len(tools) != 7 || tools[2].Name != "mesh-placement.propose" || tools[2].Run == nil || tools[2].Input == nil {
good := `{"node":"ace","address":"10.9.0.2","ask":"enrol-ace-1","warrant":"enrol-ace-1@2026-10-28T20:00:00Z"}`
for _, c := range []struct{ by, warrant, want string }{
{"novox.mesh-grants", good, "mesh-enrolment's alone"},
{"person.mesh-enrolment", good, "mesh-enrolment's alone"},
{"novox.mesh-enrolment", `{"node":"g14","address":"10.9.0.2","ask":"a","warrant":"a@t"}`, "carries the warrant of g14"},
{"novox.mesh-enrolment", `{"node":"ace","address":"10.9.0.9","ask":"a","warrant":"a@t"}`, "carries the warrant of ace at 10.9.0.9"},
{"novox.mesh-enrolment", "", "carries no warrant"},
} {
if err := addNode(c.by, c.warrant); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("an add-node by %s with warrant %q: %v, want a refusal saying %q", c.by, c.warrant, err, c.want)
}
}
if err := addNode("novox.mesh-enrolment", good); err != nil {
t.Errorf("an add-node from mesh-enrolment: %v", err)
}
// A remove-node (the operator's decision of 2026-11-02 01:05) takes a joining node that never confirmed off placement: from
// mesh-enrolment, on the warrant naming that node, of a node running nothing, it is proposed with the node's key to delete;
// of a node running modules, or with another node's warrant, it is refused.
joiner, _ := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: "joiner", Decided: delivery.NodeDecided{Address: "10.9.0.7"}, Modules: []delivery.PlacedModule{}})
st[delivery.PlacementState].values["placement.joiner"] = joiner
removeNode := func(node, warrant string) (any, error) {
return s.doPropose(map[string]any{"why": "enrolment ended", "warrant": warrant,
"changes": []any{map[string]any{"op": "remove-node", "node": node}}}, "novox.mesh-enrolment")
}
for _, c := range []struct{ node, warrant, want string }{
{"anchor", `{"node":"anchor","address":"10.9.0.1","ask":"a","warrant":"a@t"}`, "runs modules or holds seats"},
{"joiner", `{"node":"g14","address":"10.9.0.7","ask":"a","warrant":"a@t"}`, "carries the warrant of g14"},
} {
if _, err := removeNode(c.node, c.warrant); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("a remove-node of %s: %v, want a refusal saying %q", c.node, err, c.want)
}
}
if got, err := removeNode("joiner", `{"node":"joiner","address":"10.9.0.7","ask":"a","warrant":"a@t"}`); err != nil {
t.Errorf("a remove-node from mesh-enrolment: %v", err)
} else if rm := got.(proposed).Proposal; !slices.Equal(rm.Nodes(), []string{"joiner"}) || !slices.Equal(rm.RemovedNodes(), []string{"joiner"}) {
t.Errorf("the remove-node proposed %+v", rm)
}
// 1126/B8: the seat serves its five verbs; adopt and hand-back went with the controller (ADR 0590).
if tools := s.tools(); len(tools) != 5 || tools[2].Name != "mesh-placement.propose" || tools[2].Run == nil || tools[2].Input == nil {
t.Errorf("the seat's verbs as served: %+v", tools)
}
t.Run("1953/D4-D6: the seat mesh-wants' upgrade and bus", wantsVerbs)
t.Run("1955/L7 (ADR 0571 §6): rule 7 reads the controller's composition until the lease write, the node's own after; an unplaced node is said", func(t *testing.T) {
t.Run("1955/L7 (ADR 0571 §6): rule 7 reads the node's own composition; an unplaced node is said", func(t *testing.T) {
w := newWorld()
fp := delivery.SourcePrefix2 + strings.Repeat("2", 64)
ckey, _ := delivery.ComposedKey("ace", "web", fp)
cv, _ := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("c", 64), Judgement: delivery.JudgedNeither,
Policy: catalogue.PolicyRoll, PolicyFrom: catalogue.FromDefault})
w.putEntry(delivery.ControllerComposedBucket, ckey, cv, 1, time.Now())
pl, _ := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: "ace", Modules: []delivery.PlacedModule{{Module: "web"}}})
pkey, _ := delivery.LegacyPlacementKey("ace")
w.putEntry(delivery.PlacementBucket, pkey, pl, 1, time.Now())
@@ -180,15 +197,11 @@ func TestTheSeatsVerbs(t *testing.T) {
hk, _ := delivery.HaveNodeKey(node)
w.putEntry(delivery.HaveBucket, hk, hn, 1, time.Now())
}
if d, ok := w.composed("ace", "web", fp); !ok || d.Digest != "sha256:"+strings.Repeat("c", 64) || len(w.unplacedNodes()) != 0 {
t.Fatalf("before the lease write: %+v %v, unplaced %v", d, ok, w.unplacedNodes())
}
w.putEntry(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(`{"handed-to":"mesh-vault","handed-by":"operator"}`), 2, time.Now())
if d, ok := w.composed("ace", "web", fp); !ok || d.Digest != "sha256:"+strings.Repeat("a", 64) {
t.Errorf("after the lease write, ace: %+v %v", d, ok)
t.Errorf("ace: %+v %v", d, ok)
}
if _, ok := w.composed("g14", "web", fp); ok || !slices.Equal(w.unplacedNodes(), []string{"g14"}) {
t.Errorf("after the lease write, g14 unplaced: composed %v, unplaced %v", ok, w.unplacedNodes())
t.Errorf("g14 unplaced: composed %v, unplaced %v", ok, w.unplacedNodes())
}
findings := map[string][]byte{}
p := &passer{w: w, logf: func(string, ...any) {}, findings: assign.Findings{
@@ -351,14 +364,20 @@ func TestTheSeatsVerbs(t *testing.T) {
// wantsVerbs (novox/hq ADR 0565 §7, §8, §9): upgrade answers each module's policy and its source; a person's choice is
// written with who and why, recorded as a hand act in the seat's log and said as policy-changed, and default takes it back;
// the bus is refused; compare lists the modules the controller writes another policy for; the process writes policy-split
// for a split and deletes it once they agree; bus names the node behind the indexer's newest.
// the bus is refused; compare lists each node's composition and the findings; bus names the node behind the indexer's newest.
func wantsVerbs(t *testing.T) {
fpOld, fpNew := delivery.SourcePrefix2+strings.Repeat("1", 64), delivery.SourcePrefix2+strings.Repeat("2", 64)
st := memStore{}
for _, state := range []string{modulesState, manifestsState, composedState, policiesState, haveState, putsState, "findings"} {
for _, state := range []string{modulesState, manifestsState, policiesState, haveState, putsState, "findings", delivery.ComposedRead} {
st[state] = &memKV{values: map[string][]byte{}}
}
// ADR 0593 §6, §8: what each node published, every node's bucket as one state: anchor's own composition and its web at
// a fingerprint, and laptop's web refused there.
webKey, _ := delivery.ComposedKey("anchor", "web", fpNew)
refusedKey, _ := delivery.ComposedKey("laptop", "web", fpNew)
st[delivery.ComposedRead].values["anchor.node"], _ = delivery.EncodeComposed(delivery.Composed{Node: "anchor"})
st[delivery.ComposedRead].values[webKey], _ = delivery.EncodeComposed(delivery.Composed{Node: "anchor", Module: "web", FP: fpNew})
st[delivery.ComposedRead].values[refusedKey], _ = delivery.EncodeComposed(delivery.Composed{Node: "laptop", Module: "web", FP: fpNew, Refused: "a required value is unset"})
manifest := func(name, extra string) {
raw := `{"module":"` + name + `","version":"1"` + extra + `}`
digest := delivery.ManifestDigest([]byte(raw))
@@ -367,17 +386,11 @@ func wantsVerbs(t *testing.T) {
}
manifest("web", "")
manifest("photos", `,"upgrade":{"policy":"record","why":"a person checks the albums"}`)
manifest("nats", `,"provides":[{"name":"mesh-bus","scope":"mesh"}]`)
manifest("nats", `,"provides":[{"name":"mesh-bus","scope":"mesh"}],"upgrade":{"policy":"record","why":"its streams"}`)
for node, fp := range map[string]string{"anchor": fpOld, "laptop": fpNew} {
st[haveState].values[node+".nats"], _ = delivery.EncodeHave(delivery.Have{FP: fp, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
}
composed := func(module, policy string) {
key, _ := delivery.ComposedKey("anchor", module, fpNew)
st[composedState].values[key], _ = json.Marshal(delivery.ComposedDigest{V: delivery.Version, Digest: "sha256:" + strings.Repeat("c", 64),
Judgement: delivery.JudgedNeither, Policy: policy, PolicyFrom: catalogue.FromPerson, PolicyWhy: "chosen at the controller"})
}
composed("web", catalogue.PolicyRecord)
composed("photos", catalogue.PolicyRecord)
st[haveState].values["anchor.node"], _ = delivery.EncodeHaveNode(delivery.HaveNode{Composed: "sha256:" + strings.Repeat("c", 64)})
var asked []string
var said []string
w := newWantsTools(st, func(key string, body any) (json.RawMessage, error) {
@@ -394,13 +407,20 @@ func wantsVerbs(t *testing.T) {
if p := policyOf("photos"); p.Policy != catalogue.PolicyRecord || p.From != catalogue.FromModule {
t.Errorf("photos = %+v", p)
}
if p := policyOf("nats"); p.Policy != catalogue.PolicyRecord || p.From != catalogue.FromBus {
if p := policyOf("nats"); p.Policy != catalogue.PolicyRecord || p.From != catalogue.FromModule { // the policy its manifest says (issue 774)
t.Errorf("nats = %+v", p)
}
// compare: web rolls at the assigner and records at the controller.
// compare: each node's own composition is the digest its have names, and the assigner's findings; nothing splits, the
// controller being gone (1126/B8).
out, _ := w.doUpgrade(map[string]any{"compare": "true"}, "op")
if splits := out.(map[string]any)["splits"].([]assign.Split); len(splits) != 1 || splits[0].Module != "web" || out.(map[string]any)["findings"] == nil {
t.Errorf("splits = %+v", splits)
if got := out.(map[string]any); got["splits"] != nil || got["findings"] == nil ||
got["composed"].(map[string]string)["anchor"] != "sha256:"+strings.Repeat("c", 64) || len(got["composed"].(map[string]string)) != 1 {
t.Errorf("compare = %+v", got)
}
pub, _ := out.(map[string]any)["published"].(map[string][]publishedRow)
if a, l := pub["anchor"], pub["laptop"]; len(a) != 2 || a[0].Module != "node" || a[1].Module != "web" || a[1].FP != fpNew ||
!strings.HasPrefix(a[1].Digest, "sha256:") || len(l) != 1 || l[0].Refused != "a required value is unset" || l[0].Digest != "" {
t.Errorf("published = %+v (unread %v)", pub, out.(map[string]any)["published-unread"])
}
// Read by watch (issue 1953 L5f): the same answer, each watched state started once, and no get per key of the
// large states (the modules, their manifests, the composed record).
@@ -422,14 +442,14 @@ func wantsVerbs(t *testing.T) {
if werr != nil {
t.Fatal(werr)
}
if ws := wout.(map[string]any)["splits"].([]assign.Split); len(ws) != 1 || ws[0].Module != "web" {
t.Errorf("splits by watch = %+v", ws)
if got := wout.(map[string]any); got["composed"].(map[string]string)["anchor"] != "sha256:"+strings.Repeat("c", 64) {
t.Errorf("compare by watch = %+v", got)
}
}
if n := watches[manifestsState+" by key"]; n == 0 || n > len(st[manifestsState].values) {
t.Errorf("the manifests were read by key %d times, want each once", n)
}
for _, state := range []string{modulesState, composedState, health.FindingsState} {
for _, state := range []string{modulesState, haveState, health.FindingsState} {
if watches[state] != 1 {
t.Errorf("%s was watched %d times, want once", state, watches[state])
}
@@ -443,29 +463,7 @@ func wantsVerbs(t *testing.T) {
if watches[policiesState] != 0 {
t.Errorf("the seat's own small policies state was watched: %v", watches)
}
// The process says the split as its finding, and deletes it once they agree.
findings := map[string][]byte{}
p := &passer{w: newWorld(), logf: func(string, ...any) {}, findings: assign.Findings{
Put: func(k string, v []byte) error { findings[k] = v; return nil },
Delete: func(k string) error { delete(findings, k); return nil },
Keys: func() ([]string, error) { return []string{"module.gone.policy-split"}, nil }}}
for k, v := range st[composedState].values {
p.w.putEntry(delivery.ControllerComposedBucket, k, v, 1, time.Now())
}
man := map[string]catalogue.Manifest{}
for _, name := range []string{"web", "photos"} {
raw, _, _, _ := st.get(modulesState, name)
mod, _ := delivery.DecodeModule(raw)
mraw, _, _, _ := st.get(manifestsState, mod.Manifest)
kept, _ := delivery.DecodeManifest(mraw)
man[name], _ = catalogue.Decode([]byte(kept.JSON))
}
findings["module.gone.policy-split"] = []byte("{}")
p.policySplits(man, nil)
if _, ok := findings[assign.PolicySplitKey("web")]; !ok || len(findings) != 1 {
t.Errorf("findings = %v, want web's split alone (a stale one deleted)", findings)
}
// a person's record for web: written, recorded, said; the split is gone.
// a person's record for web: written, recorded, said.
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "record", "why": "the operator checks it"}, "person.jochen"); err != nil {
t.Fatal(err)
}
@@ -474,12 +472,12 @@ func wantsVerbs(t *testing.T) {
len(said) != 1 || !strings.HasPrefix(said[0], "policy-changed policy-changed.web.") {
t.Errorf("chosen %+v, asked %v, said %v", c, asked, said)
}
p.policySplits(man, chosen)
if len(findings) != 0 {
t.Errorf("findings after the choice = %v", findings)
// A person may choose the bus's policy as any module's (ADR 0601 §1: no part treats the module that provides the bus apart).
if _, err := w.doUpgrade(map[string]any{"module": "nats", "policy": "roll", "why": "x"}, "op"); err != nil {
t.Errorf("a person's policy for the bus was refused: %v", err)
}
if _, err := w.doUpgrade(map[string]any{"module": "nats", "policy": "roll", "why": "x"}, "op"); err == nil {
t.Error("the bus was given a policy")
if _, err := w.doUpgrade(map[string]any{"module": "nats", "policy": "default", "why": "back to the module"}, "op"); err != nil {
t.Fatal(err)
}
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "roll"}, "op"); err == nil {
t.Error("a policy without why was set")
@@ -14,6 +14,8 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-delivery/sdk/enginereads"
)
// The assigner's tools (novox/hq ADR 0546), served through the tool runner from its bundle: a process never serves (ADR
@@ -21,6 +23,9 @@ import (
// every step as the event put. They are the module's own tools; the seat mesh-placement's verbs are served beside them
// (seat_tools.go, ADR 0567 §10).
// engineReadsTool lists what each node's engine reads, as the assigner uses it, and asks one node's engine again.
const engineReadsTool = "assigner_engine_reads"
// putsState is the assigner's state of puts, as the runtime names a module's own state.
const putsState = "puts"
@@ -358,6 +363,9 @@ func serveTools() error {
})
wants.watched = stdio.NewWatchedStates() // the large states read whole from one watched copy (ADR 0580 clause 3)
tools = append(tools, wants.tools()...)
tools = append(tools, viewTools(os.Getenv, keptSource, time.Now)...)
// What each node's engine reads, as the assigner uses it (ADR 0618 §3): its own state engine-reads, asked again through it.
tools = append(tools, enginereads.Tool(engineReadsTool, "the assigner", enginereads.KeptState(stdio.State(engineReadsState)), false, time.Now))
return stdio.Serve("", tools)
}
@@ -0,0 +1,128 @@
package main
import (
"context"
"slices"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-delivery/sdk/graph"
"mesh-delivery-assigner/internal/assign"
)
// The verb edges a want waits on (novox/hq ADR 0614 §2, §3): the module graph the indexer writes names what each module
// invokes and serves and which of its verb edges it leaves out on a cycle (one place decides the cycle); a want waits for
// each kept edge until a build serving its verb runs healthy, read from the haves, the builder's builds and the indexer's
// manifests, as a provider is counted (ADR 0603 §5).
// moduleGraph is the module graph as the indexer's first view and views family last said it (ADR 0631); a key that
// does not decode is left out, so its module waits on no verb.
func (p *passer) moduleGraph() graph.Graph {
g := graph.Graph{}
for key, raw := range p.w.read(graph.Bucket) {
module, err := graph.ParseKey(key)
if err != nil {
continue
}
if n, err := graph.Decode(raw); err == nil {
g[module] = n
}
}
return g
}
// keptInvokes are the verbs of module the graph keeps as edges: each it invokes and does not leave out, that a module
// other than itself serves. A verb no module of the graph serves is no edge, so it holds nothing; nil when the graph holds
// no node of module.
func keptInvokes(g graph.Graph, module string) []string {
n, ok := g[module]
if !ok {
return nil
}
var out []string
for _, v := range n.Invokes {
if slices.Contains(n.LeftOut, v) {
continue
}
if slices.ContainsFunc(g.Servers(v), func(s string) bool { return s != module }) {
out = append(out, v)
}
}
return out
}
// served is, for this pass, whether a build serving a verb runs healthy on some node (graph.Served): every running healthy
// have's build (the builder's builds key of its module and fingerprint) names its manifest, and the verbs that manifest
// serves are served. A healthy build whose manifest cannot be read counts what the graph says its module serves, said in
// the log, so a build read late never holds its callers for good; nil when the graph holds nothing, and no want waits on
// a verb.
func (p *passer) served(ctx context.Context, get getter, g graph.Graph, n assign.Nodes) func(string) bool {
if len(g) == 0 {
return nil
}
served, unread := graph.Served(n.Haves, func(module, fp string) ([]string, bool) { return p.servesOf(ctx, get, module, fp) },
func(module string) []string { return g[module].Serves })
if len(unread) > 0 {
p.logf("the manifest of the running build of %v cannot be read, so what the module graph says they serve is counted as served", unread)
}
return served
}
// offering is, for this pass, whether a build whose manifest offers a provision runs healthy on a node present (ADR 0603
// §5, ADR 0614 §5): every running healthy have on an online node, its build's manifest read as servesOf reads it. A
// healthy build whose manifest cannot be read counts what its module's trunk manifest offers, so a build read late never
// holds a consumer for good.
func (p *passer) offering(ctx context.Context, get getter, trunk map[string]catalogue.Manifest, n assign.Nodes) func(string) bool {
offered := map[string]bool{}
for node, haves := range n.Haves {
if !n.Online[node] {
continue
}
for module, h := range haves {
if h.Phase != delivery.PhaseRunning || h.Health != delivery.HealthOK {
continue
}
m, ok := p.manifestOf(ctx, get, module, h.FP)
if !ok {
m = trunk[module]
}
for _, o := range m.Offers() {
offered[o] = true
}
}
}
return func(provision string) bool { return offered[provision] }
}
// servesOf are the verbs the manifest of module's build at fp serves (graph.ServesOf), and whether it could be read.
func (p *passer) servesOf(ctx context.Context, get getter, module, fp string) ([]string, bool) {
m, ok := p.manifestOf(ctx, get, module, fp)
if !ok {
return nil, false
}
return graph.ServesOf(m), true
}
// manifestOf is the manifest of module's build at fp (the builder's build names its digest in the indexer's manifests),
// and whether it could be read.
func (p *passer) manifestOf(ctx context.Context, get getter, module, fp string) (catalogue.Manifest, bool) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return catalogue.Manifest{}, false
}
raw := p.w.value(delivery.BuilderBuildsBucket, key)
if raw == nil {
return catalogue.Manifest{}, false
}
b, err := delivery.DecodeBuilderBuild(raw)
if err != nil || b.Manifest == "" {
return catalogue.Manifest{}, false
}
got := p.w.manifest(ctx, get, b.Manifest)
if got.err != "" {
return catalogue.Manifest{}, false
}
return got.manifest, true
}
@@ -0,0 +1,235 @@
package main
import (
"context"
"encoding/json"
"fmt"
"slices"
"time"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/views"
"git.novox.be/novox/mesh-delivery/sdk/graph"
"git.novox.be/novox/mesh-delivery/sdk/heard"
"mesh-delivery-assigner/internal/assign"
)
// viewsStream is the assigner's views family on its old streams section, beside placement (ADR 0619 §2, ADR 0615 §5),
// as its manifest's `streams` names it; indexerViewsStream is the indexer's, as the indexer's manifest names it.
const (
viewsStream = "assigner-views"
indexerViewsStream = "indexer-views"
)
// marksBucket is the assigner's state of what its views family said each state up to.
var marksBucket = delivery.AssignerModule + "_" + views.MarksState
// viewedTool is one tool over one state of the assigner's.
type viewedTool struct{ tool, state, words string }
// firstViewTools are the assigner's first-view tools (ADR 0621 §1): each answers only its callers list
// (views.CallersEnv), which the manifest writes.
var firstViewTools = []viewedTool{
{"placement", delivery.PlacementState, "where each module is placed: the seat mesh-placement's placement, by module."},
{"seat-holders", delivery.SeatHoldersState, "who holds which seat on which node."},
{"wants", "wants", "what each node should run: want.<node>.<module>, its fingerprint and the rule that gave it."},
}
// readTools are the assigner's ordinary read tools, one per state its readers read (design 25 § 12.2).
var readTools = []viewedTool{
{"assigner_read_placement", delivery.PlacementState, "where each module is placed, by module."},
{"assigner_read_seat_holders", delivery.SeatHoldersState, "who holds which seat on which node."},
{"assigner_read_wants", "wants", "what each node should run: want.<node>.<module>."},
{"assigner_read_findings", health.FindingsState, "the assigner's findings, which the condition keeper raises and clears."},
{"assigner_read_asks", "asks", "the asks the assigner put to a person, and their answers."},
{"assigner_read_differences", "differences", "every difference between a want and what a node runs."},
}
// viewedStates are the states the views family says, each by its bucket: the six its readers read.
var viewedStates = map[string]string{
delivery.PlacementState: delivery.PlacementBucket,
delivery.SeatHoldersState: delivery.SeatHoldersBucket,
"wants": delivery.AssignerWantsBucket,
health.FindingsState: assign.FindingsBucket,
"asks": delivery.AssignerAsksBucket,
"differences": delivery.AssignerDifferencesBucket,
}
// viewTools are the assigner's first-view and read tools, reading only its own states, so they answer as soon as its
// runtime of its own is connected (ADR 0621 §3).
func viewTools(getenv func(string) string, state func(string) views.Source, now func() time.Time) []stdio.Tool {
marks := state(views.MarksState)
var out []stdio.Tool
for _, t := range firstViewTools {
out = append(out, views.FirstViewTool(delivery.AssignerModule, t.tool, t.state, t.words, views.ParseCallers(getenv(views.CallersEnv(t.tool))), state(t.state), marks, now))
}
for _, t := range readTools {
out = append(out, views.ReadTool(delivery.AssignerModule, t.tool, t.state, t.words, state(t.state), marks, now))
}
// Its own first views, of the indexer's manifests, modules and graph (DLV-F3r, ADR 0631 §2): read, and asked again by hand.
return append(out, views.StatusTools(delivery.AssignerModule, "assigner_first_views", "assigner_first_view_again", stdio.State(views.StatusState), handacts.Recorder{Ask: stdio.Ask}, now)...)
}
func keptSource(name string) views.Source { return views.KeptSource{State: stdio.State(name)} }
// startViews says each change of the assigner's viewed states on its views family, once its marks state is bound,
// each state as soon as its bucket is (kvs, or bound once it appears). A watch that ends is sent on errs.
func startViews(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, account process.Account, kvs buckets, findings assign.Findings, errs chan<- error, logf func(string, ...any)) {
tooLarge := func(state, key string, size int) {
if findings.Put == nil {
return
}
k, v, err := views.TooLargeFinding(delivery.AssignerModule, state, key, size, time.Now())
if err == nil {
err = findings.Put(k, v)
}
if err != nil {
logf("views: the finding of %s.%s too large is not written: %v", state, key, err)
}
}
process.AwaitBucket(ctx, js, nc, marksBucket, "no change is said on the views family, so every reader waits on its first view", logf).Ready(func(mk jetstream.KeyValue) {
em := viewsEmitter(js, account, mk, findings, tooLarge, logf)
for _, state := range []string{"asks", "differences", health.FindingsState, delivery.PlacementState, delivery.SeatHoldersState, "wants"} {
state := state
start := func(kv jetstream.KeyValue) {
if err := em.Start(ctx, state, kv, errs); err != nil {
logf("%v: its readers ask its first view at each change until the assigner starts again", err)
}
}
if kv := kvs[viewedStates[state]]; kv != nil {
start(kv)
continue
}
process.AwaitBucket(ctx, js, nc, viewedStates[state], "its changes are not said on the views family", logf).Ready(start)
}
})
}
// viewsEmitter is the assigner's emitter of its views family: a change not said past views.Bound is its finding
// view-unsaid, in its findings (review round 2); with no findings state it is said in the log only.
func viewsEmitter(js jetstream.JetStream, account process.Account, mk jetstream.KeyValue, findings assign.Findings, tooLarge func(string, string, int), logf func(string, ...any)) views.Emitter {
em := views.Emitter{Owner: delivery.AssignerModule, Subject: account.StreamSubject, Publish: views.JetStreamPublish(js), Marks: mk, TooLarge: tooLarge, Logf: logf}
if findings.Put != nil && findings.Delete != nil {
em.Findings = assignerFindings{findings}
}
return em
}
// assignerFindings is the assigner's findings as views.Emitter's Findings.
type assignerFindings struct{ f assign.Findings }
func (a assignerFindings) Put(key string, value []byte) error { return a.f.Put(key, value) }
func (a assignerFindings) Delete(key string) error { return a.f.Delete(key) }
// firstViewsBucket is the assigner's own state of its first views (views.StatusState).
var firstViewsBucket = delivery.AssignerModule + "_" + views.StatusState
// indexerViews are the assigner's first views of the indexer's manifests, modules and module graph (ADR 0615 §1,
// ADR 0621 §2, ADR 0631; DLV-F2r): asked at the start through the indexer's tools, then followed on indexer-views, asking once more on a gap;
// past the bound its finding names the indexer and the tool, cleared by the answer.
type indexerViews struct {
js jetstream.JetStream
account process.Account
call func(string, any) (json.RawMessage, error)
findings assign.Findings
status *process.AwaitedBucket
logf func(string, ...any)
}
// indexerFirstViews are the indexer's states the assigner takes by first view and the views family, each by the name of
// the indexer's bucket the world keeps it under: manifests and modules (ADR 0621 §2), and the module graph, whose verb
// edges a want waits on (ADR 0614 §2, §3; ADR 0631 §1).
var indexerFirstViews = map[string]string{"manifests": delivery.ManifestsBucket, "modules": delivery.ModulesBucket, graph.State: graph.Bucket}
// firstViewBound and firstViewWait are each first view's bound before its condition is raised, and its first wait
// between two failed asks; zero is the views package's (views.Bound, two minutes, ADR 0615 §6; and a first wait of five
// seconds). A test shortens them.
var firstViewBound, firstViewWait time.Duration
// start asks the three first views, each answer taken into w under the indexer's bucket names and settled once (a pass waits
// for it), and hears the views family; it answers how many settles to wait for.
func (iv *indexerViews) start(ctx context.Context, w *world, wake func(), settled chan<- struct{}) int {
finding := func(s views.Status, raised bool) {
if iv.findings.Put == nil {
return
}
key, value, err := views.UnansweredFinding(delivery.AssignerModule, s, time.Now())
if err == nil && raised {
err = iv.findings.Put(key, value)
} else if err == nil {
err = iv.findings.Delete(key)
}
if err != nil {
iv.logf("views: the finding of the first view %s is not written: %v", s.Tool, err)
}
}
followers := map[string]*views.Follower{}
var all []*views.Follower
for state, bucket := range indexerFirstViews {
bucket := bucket
tool := delivery.IndexerModule + "." + state
f := &views.Follower{Owner: delivery.IndexerModule, Tool: tool, State: state, Ask: views.AskVia(iv.call, tool), Logf: iv.logf,
Bound: firstViewBound, Wait: firstViewWait,
Raise: func(s views.Status) { finding(s, true) }, Clear: func(s views.Status) { finding(s, false) },
Changed: func(key string, value json.RawMessage, deleted bool, at time.Time) {
if deleted {
value = nil
}
w.putEntry(bucket, key, value, 0, at)
wake()
}}
followers[state] = f
all = append(all, f)
go func() {
if f.Start(ctx) == nil {
settled <- struct{}{}
}
}()
}
if iv.status != nil {
iv.status.Ready(func(kv jetstream.KeyValue) {
for _, f := range all {
f.Said = views.KeepStatus(ctx, kv, iv.logf)
}
if err := views.HearAgain(ctx, kv, all, iv.logf); err != nil {
iv.logf("views: assigner_first_view_again is not heard (%v) until the assigner starts again", err)
}
})
}
// Not heard, the indexer's views family is the finding family-not-heard until it binds (ADR 0615 §5).
stream := catalogue.StreamBusName(indexerViewsStream)
_ = notHeardWriter(iv.findings, iv.logf).Bind(stream, "the indexer's manifests, modules and graph are the first views' alone until the assigner starts again", func() error {
cons, err := iv.account.EventConsumer(ctx, iv.js, stream)
if err != nil {
return fmt.Errorf("the consumer %s on %s does not exist yet: %w", iv.account.EventConsumerName(), stream, err)
}
_, err = process.HearEvents(cons, views.Hear(ctx, followers, iv.logf), iv.logf)
return err
})
return len(all)
}
// notHeardWriter writes the assigner's finding of a family it does not hear into its findings (heard.Writer); with no
// findings state it is said in the log alone.
func notHeardWriter(findings assign.Findings, logf func(string, ...any)) heard.Writer {
w := heard.Writer{Observer: delivery.AssignerModule, Logf: logf, Put: findings.Put}
if findings.Delete != nil {
w.Delete = func(key string) error {
if keys, err := findings.Keys(); err == nil && !slices.Contains(keys, key) {
return nil // none stands
}
return findings.Delete(key)
}
}
return w
}
@@ -9,7 +9,6 @@ import (
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
@@ -20,15 +19,14 @@ import (
// The seat mesh-wants' verbs (novox/hq ADR 0565 §7, §8, §9, §15; issue 1953 D4-D6), served through the tool runner beside the
// put tools and mesh-placement's verbs. `upgrade` reads and sets a person's upgrade policy, kept in the state `policies`
// (each set recorded as a hand act and said as policy-changed), and with compare lists every module whose policy differs from
// the one the controller writes beside its compositions; `bus` says what a bus upgrade would do. The assigner's process reads
// (each set recorded as a hand act and said as policy-changed), and with compare lists the digest of the composition each node's
// own have names beside every composition each node published (ADR 0593 §8, composed.go); `bus` says what a bus upgrade would do. The assigner's process reads
// `policies` when it judges a move and writes the finding policy-split while a split stands.
// The states the verbs read, as the runtime names them.
const (
policiesState = delivery.AssignerPoliciesState
manifestsState = delivery.IndexerModule + ".manifests"
composedState = "mesh-controller.composed"
// PolicyChangedEvent is the seat's event of a person's policy set or taken back.
PolicyChangedEvent = "policy-changed"
// busStepCause is the cause the bus's step is recorded under in the hand-act log.
@@ -56,13 +54,6 @@ func newWantsTools(st seatStore, ask func(string, any) (json.RawMessage, error),
return &wantsTools{st: st, ask: ask, emit: emit, now: time.Now}
}
// leased says the controller's lease is handed away.
func (s *wantsTools) leased() bool {
raw, _, found, err := s.st.get(buscompose.ControllerLease, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, err)
return away
}
// values reads every key of a state: from the watched copy when there is one, else key by key.
func (s *wantsTools) values(state string) (map[string][]byte, error) {
if s.watched != nil {
@@ -142,7 +133,7 @@ type policyRow struct {
}
// doUpgrade is the verb upgrade: read every module's policy or one's; with policy and why set a person's choice, or take it
// back with default; with compare list the splits.
// back with default; with compare each node's composition and the assigner's findings.
func (s *wantsTools) doUpgrade(args map[string]any, by string) (any, error) {
manifests, _, err := s.manifests()
if err != nil {
@@ -154,16 +145,26 @@ func (s *wantsTools) doUpgrade(args map[string]any, by string) (any, error) {
}
chosen, unreadable := assign.DecodePolicies(raw)
if arg(args, "compare") == "true" {
composed, err := s.values(composedState)
haves, err := s.values(haveState)
if err != nil {
return nil, err
}
splits := assign.Splits(manifests, chosen, assign.ControllerPolicies(composed))
if splits == nil {
splits = []assign.Split{}
composed := map[string]string{}
for key, raw := range haves {
if node, _, isNode, err := delivery.ParseHaveKey(key); err == nil && isNode {
if h, err := delivery.DecodeHaveNode(raw); err == nil && h.Composed != "" {
composed[node] = h.Composed
}
}
}
out := map[string]any{"splits": splits, "empty": len(splits) == 0}
// The assigner's findings, its said conditions and its policy splits as the condition keeper reads them (§8, §15).
out := map[string]any{"composed": composed}
// And what every node published (ADR 0593 §8), read only to compare.
if values, err := s.values(delivery.ComposedRead); err != nil {
out["published-unread"] = err.Error()
} else {
out["published"] = published(values)
}
// The assigner's findings, its said conditions as the condition keeper reads them (§15).
if found, err := s.values(health.FindingsState); err != nil {
out["findings-unread"] = err.Error()
} else {
@@ -196,18 +197,14 @@ func (s *wantsTools) doUpgrade(args map[string]any, by string) (any, error) {
}
// setPolicy writes a person's choice of a module's policy, or deletes it for default, records the act in the hand-act log
// and says policy-changed. The bus is never rolled: a choice for it is refused.
// and says policy-changed. The bus's choice is a person's as any module's (ADR 0601 §1).
func (s *wantsTools) setPolicy(module, policy, why, by string, manifests map[string]catalogue.Manifest) (any, error) {
if module == "" {
return nil, errors.New("a policy is set for one module: name it")
}
m, ok := manifests[module]
if !ok {
if _, ok := manifests[module]; !ok {
return nil, fmt.Errorf("the indexer holds no manifest of %s", module)
}
if catalogue.ProvidesBus(m) {
return nil, fmt.Errorf("%s is the bus, which is never rolled whoever says so (novox/hq ADR 0236): its step is the put on the operator's warrant", module)
}
if why == "" {
return nil, errors.New("a person's policy says why")
}
@@ -229,7 +226,7 @@ func (s *wantsTools) setPolicy(module, policy, why, by string, manifests map[str
return nil, fmt.Errorf("the choice for %s cannot be written: %w", module, err)
}
}
rec := handacts.Recorder{Ask: s.ask, LeaseAway: s.leased}
rec := handacts.Recorder{Ask: s.ask}
taken, rerr := rec.Record(handacts.Request{What: what, Why: why, Cause: "upgrade-policy", Verb: "mesh-wants.upgrade"})
recorded := taken.Log
if rerr != nil {
@@ -2,24 +2,17 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
"mesh-delivery-assigner/internal/assign"
)
@@ -28,18 +21,6 @@ import (
// for (mesh-sdk delivery.CandidateNoneNoModule, which the indexer writes): the change removes the module.
const noModuleWords = delivery.CandidateNoneNoModule
// assignment holds only the fields of the controller's Assignment the assigner reads (the view's own, mesh-delivery-view
// internal/view/view.go): an assignment stands while it has not ended and has no state.
type assignment struct {
Node string `json:"node"`
Module string `json:"module"`
Build string `json:"build"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
func (a assignment) open() bool { return a.Ended.IsZero() && a.State == "" }
// world is what the assigner watched, as it stands now: every value by bucket and key, and the manifests read by
// digest. The watches write it; a pass reads it.
type world struct {
@@ -57,6 +38,9 @@ type world struct {
// placement is the seat's state placement assembled node by node from its keys in either form (mesh-sdk
// delivery.PlacementKeys; research 072 row 37): the one reader of its keys in the assigner.
placement *delivery.PlacementKeys
// sourceGroups is, by revision of the gate's source.main, the release (group) that moved it, from every revision its
// history keeps (watched with history): a build's source revision names the release that carries it (ADR 0601 §1).
sourceGroups map[uint64]string
}
// entryMeta is a value's revision in its bucket and when it was written.
@@ -74,7 +58,8 @@ type parsed struct {
func newWorld() *world {
return &world{values: map[string]map[string][]byte{}, manifests: map[string]parsed{}, healthy: map[string][]string{}, changes: map[string]uint64{},
meta: map[string]entryMeta{}, trunkBuilt: map[string]delivery.BuildLogEntry{}, placement: delivery.NewPlacementKeys()}
meta: map[string]entryMeta{}, trunkBuilt: map[string]delivery.BuildLogEntry{}, placement: delivery.NewPlacementKeys(),
sourceGroups: map[uint64]string{}}
}
// putEntry is put, keeping the value's revision and the time it was written.
@@ -90,6 +75,44 @@ func (w *world) putEntry(bucket, key string, value []byte, revision uint64, at t
if bucket == delivery.PlacementBucket {
_, _ = w.placement.Put(key, value, revision, false) // put took it already; this is its revision
}
if bucket == delivery.SourceBucket && key == delivery.SourceMainKey {
if s, err := delivery.DecodeSource(value); err == nil && s.Group != "" {
w.sourceGroups[revision] = s.Group
}
}
}
// release is the release whose Approve carries module's build fp (ADR 0601 §1), and whether one does: the trunk's value of
// the module names fp, the source revision at which that fingerprint last changed (FPSrc, never the Src of its last merge,
// which a recompute with no merge keeps) names the release that moved it, and the gate's merge of that release, which it
// writes on the operator's Approve, holds the module's own trunk merge among its merged pull requests. Any of them
// unknown: none, and the move is asked of a person.
func (w *world) release(module, fp string) (string, bool) {
t, ok := w.trunk()[module]
if !ok || fp == "" || t.FP2 != fp || t.FPSrc == 0 || t.Commit == "" {
return "", false
}
w.mu.Lock()
group := w.sourceGroups[t.FPSrc]
w.mu.Unlock()
key, err := delivery.MergesKey(group)
if group == "" || err != nil {
return "", false
}
raw := w.value(delivery.MergesBucket, key)
if raw == nil {
return "", false
}
merge, err := delivery.DecodeMerge(raw)
if err != nil {
return "", false
}
for _, pull := range merge.Pulls {
if pull.State == delivery.MergeMerged && pull.Commit == t.Commit && delivery.SameRepository(pull.Repository, t.Repository) {
return group, true
}
}
return "", false
}
// metaOf is the revision and time of the value a bucket holds at key, as the watch said them.
@@ -175,6 +198,19 @@ func kvGetter(kv jetstream.KeyValue) getter {
}
}
// getter reads a key of a bucket the world holds: errNotThere where it holds none.
func (w *world) getter(bucket string) getter {
return func(_ context.Context, key string) ([]byte, error) {
w.mu.Lock()
defer w.mu.Unlock()
v, ok := w.values[bucket][key]
if !ok {
return nil, errNotThere
}
return v, nil
}
}
// manifest reads the manifest of a digest from the indexer's bucket once and keeps it: a digest's manifest never changes.
func (w *world) manifest(ctx context.Context, get getter, digest string) parsed {
w.mu.Lock()
@@ -210,38 +246,15 @@ func (w *world) manifest(ctx context.Context, get getter, digest string) parsed
return p
}
// assigned is each node's modules both the controller's assignments and the seat mesh-placement's placement name, in name order
// (ADR 0549 §5; the assignments alone before any placement was read), with every open assignment by node and module.
// Once the controller's lease is handed away it is the seat's placement alone, with no assignment (ADR 0549 §5, §6).
func (w *world) assigned() (map[string][]string, map[string]assignment) {
// A lease handed away before any placement was read keeps the assignments as they last stood, never an empty mesh:
// the lease write's own check refuses that order (ADR 0547 §11).
placed, seeded := w.placed()
byAssignment, all := w.assignedByController()
if away := w.handedAway(); away && seeded {
both, _ := assign.Placed(byAssignment, placed, seeded, away)
return both, map[string]assignment{}
// assigned is each node's modules the seat mesh-placement's placement names, in name order (ADR 0549 §5, ADR 0567 §8): the
// placement is the one source of where modules run. The controller's assignments are read no more (issue 774).
func (w *world) assigned() map[string][]string {
placed, _ := w.placed()
out := make(map[string][]string, len(placed))
for node, modules := range placed {
out[node] = slices.Sorted(slices.Values(modules))
}
both, _ := assign.Placed(byAssignment, placed, seeded, false)
return both, all
}
// placementDiffers is every pair the controller's assignments and the seat's placement do not both name: none once
// the controller's lease is handed away, since the placement is then the one source.
func (w *world) placementDiffers() []assign.Pair {
placed, seeded := w.placed()
byAssignment, _ := w.assignedByController()
_, differ := assign.Placed(byAssignment, placed, seeded, w.handedAway())
return differ
}
// handedAway says the controller's lease is handed to a holder that is no controller (mesh-sdk buscompose.LeaseHandedAway,
// ADR 0561 §3): from then the seat mesh-placement's placement alone names where modules run (ADR 0549 §5). A lease not read (its
// bucket not granted or not there), a controller's, expired or unreadable, is not handed away.
func (w *world) handedAway() bool {
raw, found := w.read(buscompose.LeaseBucket)[buscompose.ControllerLeaseKey]
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, nil)
return away
return out
}
// placed is each node's modules in the seat's placement (ADR 0567 §1), and whether any placement key was read: a key
@@ -271,24 +284,6 @@ func (w *world) placedNodes() map[string]bool {
return out
}
// assignedByController is each node's modules with an open assignment, read from the controller's assignments, in
// name order.
func (w *world) assignedByController() (map[string][]string, map[string]assignment) {
out, all := map[string][]string{}, map[string]assignment{}
for _, value := range w.read(provider.AssignmentsBucket) {
var a assignment
if json.Unmarshal(value, &a) != nil || a.Node == "" || a.Module == "" || !a.open() {
continue
}
out[a.Node] = append(out[a.Node], a.Module)
all[a.Node+"."+a.Module] = a
}
for _, mods := range out {
sort.Strings(mods)
}
return out, all
}
// trunk is every module the indexer holds at the trunk, with its value.
func (w *world) trunk() map[string]delivery.Module {
out := map[string]delivery.Module{}
@@ -304,63 +299,11 @@ func (w *world) trunk() map[string]delivery.Module {
return out
}
// mesh is the mesh as the trunk holds it, for the composition check.
func (w *world) mesh(ctx context.Context, manifests getter) assign.Mesh {
m := assign.Mesh{Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{}, Digests: map[string]string{}}
m.Assigned, _ = w.assigned()
for name, mod := range w.trunk() {
if mod.Manifest == "" {
continue
}
m.Digests[name] = mod.Manifest
// The indexer's verdict on it, carried in its value (ADR 0578 §3, §5): a refused module is held and named as one
// that cannot be read, and its verdict goes into the composer's input.
if v := mod.VerdictOf(name); v != nil {
if refused, _ := delivery.TakeManifestVerdict(v); refused {
if m.Verdicts == nil {
m.Verdicts = map[string]delivery.ManifestVerdict{}
}
m.Verdicts[name] = *v
m.Unreadable[name] = v.Why()
continue
}
}
p := w.manifest(ctx, manifests, mod.Manifest)
if p.err != "" {
m.Unreadable[name] = p.err
continue
}
m.Manifests[name] = p.manifest
}
// Once the catalogue places the nodes, the check composes over the input they compose from (ADR 0547 §2, ADR 0549
// §3): one composer's input for the check and the nodes. An input that cannot be read is said, and the nodes and their
// assignments alone are composed, as before any placement.
if _, seeded := w.placed(); seeded {
in, err := frombus.Input(ctx, worldReader{w: w, manifests: manifests}, frombus.Mesh{}, frombus.Scope{Every: true})
if err != nil {
m.InputUnread = err.Error()
} else {
m.Input = &in
}
}
return m
}
// worldReader is what the assigner watched as frombus reads the bus: the seat's placement, the indexer's modules, each node's
// record in its own have key, the manifests by digest; the check reads no config value, own secret or credential (the
// composition it asks resolves, and stands in for what it does not carry).
type worldReader struct {
w *world
manifests getter
}
func (r worldReader) Placements(context.Context) (map[string]delivery.Placement, error) {
return r.w.placements(), nil
}
func (r worldReader) Records(context.Context) (map[string]delivery.NodeRecord, error) {
// records is each node's record in its own have key (`<node>.node`): the nodes a placement proposal is answered by,
// with those placement names (ADR 0593 §8).
func (w *world) records() map[string]delivery.NodeRecord {
out := map[string]delivery.NodeRecord{}
for key, value := range r.w.read(delivery.HaveBucket) {
for key, value := range w.read(delivery.HaveBucket) {
node, _, isNode, err := delivery.ParseHaveKey(key)
if err != nil || !isNode {
continue
@@ -369,136 +312,7 @@ func (r worldReader) Records(context.Context) (map[string]delivery.NodeRecord, e
out[node] = *h.Record
}
}
return out, nil
}
func (r worldReader) Modules(context.Context) (map[string]delivery.Module, error) {
return r.w.trunk(), nil
}
func (r worldReader) Manifest(ctx context.Context, digest string) ([]byte, error) {
key, err := delivery.ManifestsKey(digest)
if err != nil {
return nil, err
}
raw, err := r.manifests(ctx, key)
if err != nil {
return nil, err
}
m, err := delivery.DecodeManifest(raw)
if err != nil {
return nil, err
}
return []byte(m.JSON), nil
}
// Values is one module's config values on one node as mesh-config resolved them, each a value its source lays
// (catalogue.LaidValue): the holder composes every node a proposal touches with them (ADR 0567 §9).
func (r worldReader) Values(_ context.Context, node, module string) (map[string]any, error) {
var out map[string]any
prefix := node + "." + module + "."
for key, raw := range r.w.read(buscompose.ResolvedBucket) {
name, ok := strings.CutPrefix(key, prefix)
if !ok || strings.Contains(name, ".") {
continue
}
if v, laid := catalogue.LaidValue(raw); laid {
if out == nil {
out = map[string]any{}
}
out[name] = v
}
}
return out, nil
}
func (worldReader) OwnSecrets(context.Context, string) ([]compose.OwnSecretRow, error) {
return nil, nil
}
func (worldReader) Credentials(context.Context, string, []compose.CredentialKey, []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) {
return nil, nil, nil
}
// Build and Archive are read only for a node composing itself (frombus.Scope without Every): the check of every node
// composes from the manifests the indexer read, and reads no build and no archive.
func (worldReader) Build(context.Context, string, string) (*delivery.BuilderBuild, error) {
return nil, nil
}
func (worldReader) Archive(_ context.Context, url, _ string) ([]byte, error) {
return nil, fmt.Errorf("the composition check reads no archive (%s)", url)
}
func (r worldReader) Served(ctx context.Context, node, module string, keys []string) (map[string]any, error) {
all, err := r.Values(ctx, node, module)
var out map[string]any
for _, k := range keys {
if v, ok := all[k]; ok {
if out == nil {
out = map[string]any{}
}
out[k] = v
}
}
return out, err
}
// change is a candidate's change for the composition check, or why the composition waits for it.
func (w *world) change(ctx context.Context, manifests getter, c delivery.Candidate, trunk map[string]delivery.Module) (assign.Change, string) {
ch := assign.Change{Group: c.Group, Heads: c.HeadsHash, Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{},
Twice: map[string][]string{}, Digests: map[string]string{}, Members: map[string]string{}}
heads := map[string]string{}
for _, p := range c.Pulls {
heads[fmt.Sprintf("%s#%d", p.Repository, p.Number)] = p.Head.Head
}
ch.Held = assign.HeldOf(w.read(keptBucket), heads)
definedIn := map[string]map[string]bool{}
for _, p := range c.Pulls {
if p.Repository == assign.SDKRepository {
ch.SDKHead = p.Head.Head
}
ch.Members[p.Repository] = p.Head.Head
for _, name := range p.Modules {
fp, ok := p.Fps[name]
if !ok || fp.Manifest == "" {
if ok && fp.None2 == noModuleWords {
ch.Removed = append(ch.Removed, name)
continue
}
return ch, fmt.Sprintf("the indexer holds no manifest of %s for %s#%d yet", name, p.Repository, p.Number)
}
if fp.FP2 != "" && fp.FP2 != trunk[name].FP2 {
ch.Rebuilt++
}
got := w.manifest(ctx, manifests, fp.Manifest)
if strings.HasPrefix(got.err, "the indexer holds no manifest") {
return ch, got.err
}
ch.Digests[name] = fp.Manifest
if got.err != "" {
ch.Unreadable[name] = got.err
continue
}
ch.Manifests[got.module] = got.manifest
if definedIn[got.module] == nil {
definedIn[got.module] = map[string]bool{}
}
definedIn[got.module][p.Repository] = true
if t, ok := trunk[got.module]; ok && t.Repository != "" && t.Repository != p.Repository {
definedIn[got.module][t.Repository] = true
}
}
}
for name, repos := range definedIn {
if len(repos) > 1 {
for r := range repos {
ch.Twice[name] = append(ch.Twice[name], r)
}
sort.Strings(ch.Twice[name])
}
}
return ch, ""
return out
}
// online is, by node, whether the watchman holds the node online.
@@ -538,7 +352,8 @@ func (w *world) wantsOf() map[string]map[string]delivery.Want {
// nodes is what the six rules read of the nodes: presence, haves, and every provision's providers with their nodes.
func (w *world) nodes(manifests map[string]catalogue.Manifest, assigned map[string][]string) assign.Nodes {
n := assign.Nodes{Online: map[string]bool{}, Haves: map[string]map[string]delivery.Have{}, Providers: map[string]map[string][]string{}}
n := assign.Nodes{Online: map[string]bool{}, Haves: map[string]map[string]delivery.Have{}, Providers: map[string]map[string][]string{},
Reads: map[string][]string{}}
for key, value := range w.read(delivery.PresenceBucket) {
node, _, _, err := delivery.ParsePresenceKey(key)
if err != nil || node == "" {
@@ -550,7 +365,14 @@ func (w *world) nodes(manifests map[string]catalogue.Manifest, assigned map[stri
}
for key, value := range w.read(delivery.HaveBucket) {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
if err != nil {
continue
}
if isNode {
// What the node's engine reads (ADR 0614 §4); a record that states none is read as the baseline.
if h, err := delivery.DecodeHaveNode(value); err == nil && h.Record != nil {
n.Reads[node] = h.Record.ManifestReads
}
continue
}
if h, err := delivery.DecodeHave(value); err == nil {
@@ -612,49 +434,50 @@ func (w *world) built(module, fp string) bool {
return err == nil && b.State == delivery.BuildDone
}
// composed is the controller's composition of module for node at fp: its digest and the judgement of the move to it
// (mesh-controller_composed `<node>.<module>.<hex>`), and whether it exists and can be read.
//
// From the lease write the controller's copy is no longer read (ADR 0571 §6, a named exception of clause 10 until then): a
// placed node composes its own, and its composition is the digest its own have names; an unplaced node has none, so it gets
// no want (and the pass says so, unplacedFindings).
func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool) {
if w.handedAway() {
if !w.placedNodes()[node] {
return delivery.ComposedDigest{}, false
}
return w.ownComposed(node)
}
key, err := delivery.ComposedKey(node, module, fp)
// buildNeeds are the reference forms the builder's done build of module at fp is written in (the build key's needs, ADR
// 0618 §1): nil when it is not done or names none.
func (w *world) buildNeeds(module, fp string) []string {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return delivery.ComposedDigest{}, false
return nil
}
w.mu.Lock()
value, ok := w.values[delivery.ControllerComposedBucket][key]
w.mu.Unlock()
if !ok {
return delivery.ComposedDigest{}, false
b, err := delivery.DecodeBuilderBuild(w.value(delivery.BuilderBuildsBucket, key))
if err != nil || b.State != delivery.BuildDone {
return nil
}
d, err := delivery.DecodeComposedDigest(value)
return d, err == nil
return b.Needs
}
// nodeComposed is, by node, the digest of the node's own composition (mesh-controller_composed `<node>.node`).
func (w *world) nodeComposed() map[string]delivery.ComposedDigest {
out := map[string]delivery.ComposedDigest{}
if w.handedAway() {
return out // the controller composes no node from the lease write (ADR 0571 §6)
// recordReads is what node's record `<node>.node` states its engine reads (ADR 0614 §4), and whether the assigner holds
// a record of it: the fallback where the engine does not answer the verb manifest-reads (enginereads.FromRecord).
func (w *world) recordReads(node string) ([]string, bool) {
r, ok := w.records()[node]
return r.ManifestReads, ok
}
// knownNodes is every node the assigner hears of: placed, with a record, or with a presence.
func (w *world) knownNodes() []string {
set := map[string]bool{}
for node := range w.placedNodes() {
set[node] = true
}
for key, value := range w.read(delivery.ControllerComposedBucket) {
node, _, _, isNode, err := delivery.ParseComposedKey(key)
if err != nil || !isNode {
continue
}
if d, err := delivery.DecodeComposedDigest(value); err == nil {
out[node] = d
}
for node := range w.records() {
set[node] = true
}
return out
for node := range w.online() {
set[node] = true
}
return slices.Sorted(maps.Keys(set))
}
// composed is the composition of module for node: a placed node composes its own, and its composition is the digest its own
// have names (ADR 0571 §6); an unplaced node has none, so it gets no want (and the pass says so, unplacedFindings). The
// controller's copy (mesh-controller_composed) is read no more (issue 774).
func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool) {
if !w.placedNodes()[node] {
return delivery.ComposedDigest{}, false
}
return w.ownComposed(node)
}
// answer is the teller's answer to the ask at key (mesh-delivery-teller_answers, the same key), and whether there is one.
@@ -684,42 +507,8 @@ func (w *world) changed(bucket, key string) uint64 {
return w.changes[bucket+"|"+key]
}
// logWatch follows the builder's log: handle each entry it keeps, then each new one (process.LogReader.Watch).
type logWatch func(handle func(stdio.LogEntry) error) error
// putBuilt takes one entry of the builder's log in: a done trunk build's end (no ref) naming a full commit, newer than the
// module's last, becomes the module's running commit. It answers whether the commit changed.
func (w *world) putBuilt(raw []byte) bool {
e, err := delivery.DecodeBuildLogEntry(raw)
if err != nil || e.Event != delivery.BuildEndedEvent || e.Outcome != delivery.BuildDone || e.Ref != "" || len(e.Commit) != 40 {
return false
}
w.mu.Lock()
defer w.mu.Unlock()
have, ok := w.trunkBuilt[e.Module]
if ok && !e.At.After(have.At) {
return false
}
w.trunkBuilt[e.Module] = e
return !ok || have.Commit != e.Commit
}
// runningCommits is the commit of module's newest done trunk build in the builder's log (ADR 0571 clause 1): the commit a
// reader of the manifests runs at (novox/hq ADR 0482 clause 5, issue 1623); none when the log holds no such build.
func (w *world) runningCommits(module string) []string {
w.mu.Lock()
defer w.mu.Unlock()
if e, ok := w.trunkBuilt[module]; ok {
return []string{e.Commit}
}
return nil
}
// keptBucket is mesh-config's `kept`: the approved held values of open pull requests (ADR 0565 §5).
var keptBucket = "mesh-config_" + delivery.MeshConfigKeptState
// ownComposed is a placed node's own composition, as the digest its own have names (`<node>.node` of the haves, Composed):
// what rule 7 reads from the lease write. A node whose have names none is not composed yet.
// what rule 7 reads. A node whose have names none is not composed yet.
func (w *world) ownComposed(node string) (delivery.ComposedDigest, bool) {
key, err := delivery.HaveNodeKey(node)
if err != nil {
@@ -737,12 +526,9 @@ func (w *world) ownComposed(node string) (delivery.ComposedDigest, bool) {
Words: "the node composes its own"}, true
}
// unplacedNodes are, from the lease write, the nodes whose engine has a have and that no placement names: they get no
// unplacedNodes are the nodes whose engine has a have and that no placement names: they get no
// want, since nothing composes them.
func (w *world) unplacedNodes() []string {
if !w.handedAway() {
return nil
}
placed := w.placedNodes()
var out []string
for key := range w.read(delivery.HaveBucket) {
@@ -2,16 +2,12 @@ package main
import (
"context"
"os"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign"
)
// fakeManifests is the indexer's manifests bucket in memory.
@@ -46,76 +42,6 @@ func pull(repo string, number int, modules map[string]delivery.CandidateModule)
return p
}
// 1297/B3b: a candidate's change reads each module's manifest by its digest, counts the modules rebuilt, takes a
// module with no manifest in the preview as removed, a mesh-sdk member's head, and a module the trunk holds from
// another repository as defined twice.
func TestTheChangeOfACandidate(t *testing.T) {
theWorldCountsTheValuesAKeyTakes(t)
f := fakeManifests{}
web := f.add(t, "web", `{"module":"web","version":"2"}`)
mail := f.add(t, "mail", `{"module":"mail","version":"1"}`)
trunk := map[string]delivery.Module{"web": {FP2: fp('a'), Repository: "novox/web"}, "mail": {FP2: fp('c'), Repository: "novox/mail"}}
c := delivery.Candidate{Group: "g", HeadsHash: "0123456789ab", Pulls: []delivery.CandidatePull{
pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {FP2: fp('b'), Manifest: web}, "old": {None2: noModuleWords}}),
pull("novox/other", 2, map[string]delivery.CandidateModule{"mail": {FP2: fp('c'), Manifest: mail}}),
pull(assign.SDKRepository, 3, nil),
}}
ch, waits := newWorld().change(context.Background(), f.get, c, trunk)
if waits != "" {
t.Fatalf("the change waits: %s", waits)
}
if ch.Manifests["web"].Version != "2" || ch.Rebuilt != 1 || !slices.Equal(ch.Removed, []string{"old"}) || ch.SDKHead != strings.Repeat("3", 40) {
t.Errorf("change = %+v", ch)
}
if !slices.Equal(ch.Twice["mail"], []string{"novox/mail", "novox/other"}) || len(ch.Twice) != 1 {
t.Errorf("twice = %v, want mail in novox/mail and novox/other", ch.Twice)
}
}
// 1297/B3b: a candidate whose module has no manifest digest yet, or whose manifest the indexer does not hold yet, waits,
// saying for what; a manifest that does not decode is the change's, not a wait (one with a field the assigner does not
// know is decoded with it set aside, novox/hq ADR 0578 §6).
func TestAChangeWaitsForItsManifests(t *testing.T) {
f := fakeManifests{}
bad := f.add(t, "web", `{"module":"web","version":"2","requires":"x"}`)
for _, c := range []delivery.CandidatePull{
pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {None2: "no preview yet"}}),
pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {FP2: fp('b'), Manifest: strings.Repeat("9", 64)}}),
} {
_, waits := newWorld().change(context.Background(), f.get, delivery.Candidate{Group: "g", HeadsHash: "0123456789ab", Pulls: []delivery.CandidatePull{c}}, nil)
if waits == "" {
t.Errorf("a change without its manifest %+v does not wait", c.Fps)
}
}
ch, waits := newWorld().change(context.Background(), f.get, delivery.Candidate{Group: "g", HeadsHash: "0123456789ab",
Pulls: []delivery.CandidatePull{pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {FP2: fp('b'), Manifest: bad}})}}, nil)
if waits != "" || ch.Unreadable["web"] == "" {
t.Errorf("an unreadable manifest: waits %q, change %+v", waits, ch)
}
}
// A manifest of the change whose config value falls back to a predefined key (novox/hq ADR 0348 §7) is read, not
// unreadable for the catalogue's predefined.json the assigner never holds: the merge gate judges the fallback in the
// catalogue, and the indexer reads it the same way (catalogue.ParseManifestForInputs, issue 1751). A group whose
// mesh-catalog member touched pacman and xterm failed its composition on this (group check 18df81380db3). The manifest is
// xterm's module.json from mesh-catalog at 835bf22b.
func TestAManifestFallingBackToAPredefinedKeyIsReadable(t *testing.T) {
raw, err := os.ReadFile("testdata/catalogue-835bf22b-xterm.module.json")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(raw), `"predefined"`) {
t.Fatal("the fixture names no predefined key")
}
f := fakeManifests{}
xterm := f.add(t, "xterm", string(raw))
ch, waits := newWorld().change(context.Background(), f.get, delivery.Candidate{Group: "g", HeadsHash: "0123456789ab",
Pulls: []delivery.CandidatePull{pull("novox/mesh-catalog", 1, map[string]delivery.CandidateModule{"xterm": {FP2: fp('b'), Manifest: xterm}})}}, nil)
if waits != "" || ch.Unreadable["xterm"] != "" || ch.Manifests["xterm"].Module != "xterm" {
t.Errorf("xterm: waits %q, unreadable %q, want read", waits, ch.Unreadable["xterm"])
}
}
// Review round 1 (items 3 and 4): the world counts the values a key takes, so a have written again after an answer is seen.
func theWorldCountsTheValuesAKeyTakes(t *testing.T) {
w := newWorld()
@@ -128,33 +54,22 @@ func theWorldCountsTheValuesAKeyTakes(t *testing.T) {
if got := w.changed(delivery.HaveBucket, "gone"); got != 0 {
t.Errorf("changed = %d for a key never seen", got)
}
// ADR 0571 clause 9: grants are read from the grants part's bucket in every case, a lease held, handed away, absent or
// unreadable alike, and the controller's own grants count for nothing.
// ADR 0571 clause 9: grants are read from the grants part's bucket, the one bucket of grants (ADR 0590).
fp := delivery.SourcePrefix2 + strings.Repeat("d", 64)
gkey, _ := delivery.ControllerGrantKey("web", fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{Accounts: []string{"ace.web"}})
gkey, _ := delivery.PartGrantKey("web", fp)
grant, _ := delivery.EncodeGrant(delivery.Grant{Accounts: []string{"ace.web"}})
for _, row := range []struct {
name, lease string
inPart, inController bool
granted bool
name string
inPart bool
granted bool
}{
{"handed away, the part's grant", `{"handed-to":"the operator's hand-over"}`, true, false, true},
{"handed away, only the controller's grant", `{"handed-to":"x"}`, false, true, false},
{"a controller holds it, the part's grant", `{"instance":"novox/1","epoch":3}`, true, false, true},
{"a controller holds it, only its own grant", `{"instance":"novox/1"}`, false, true, false},
{"no lease read", ``, true, false, true},
{"an unreadable lease", `[1]`, true, false, true},
{"the part's grant", true, true},
{"no grant", false, false},
} {
p := newPasser(buckets{}, time.Now())
if row.lease != "" {
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(row.lease))
}
if row.inPart {
p.w.put(delivery.PartGrantsBucket, gkey, grant)
}
if row.inController {
p.w.put(delivery.ControllerGrantsBucket, gkey, grant)
}
if got := p.w.grantsBucket(); got != delivery.PartGrantsBucket {
t.Errorf("%s: grants are read from %s, want the part's", row.name, got)
}
+17 -5
View File
@@ -3,20 +3,32 @@ module mesh-delivery-assigner
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101025153-c77d17fb42e3
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261104174359-05b3e2e1b94f
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
git.novox.be/novox/mesh-contracts/go v0.0.0-20261104174344-ae5266e77639 // indirect
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/klauspost/compress v1.20.1 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/jwt/v2 v2.8.2 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.org/x/time v0.16.0 // indirect
)
require git.novox.be/novox/mesh-delivery/sdk v0.0.0-00010101000000-000000000000
require (
git.novox.be/novox/mesh-contracts/go/clients/nodeengineclient v0.0.0-20261104174344-ae5266e77639
git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient v0.0.0-20261104174344-ae5266e77639
git.novox.be/novox/mesh-contracts/go/seats/nodeengine v0.0.0-20261104174344-ae5266e77639
git.novox.be/novox/mesh-contracts/go/seats/operatorchannel v0.0.0-20261104174344-ae5266e77639
)
replace git.novox.be/novox/mesh-delivery/sdk => ../../sdk
+20 -28
View File
@@ -1,33 +1,25 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261031084708-acbf80ad910a h1:DcSvfEmdPBKqCg5OmXRNb269EkkPe7BsEp+RykEhkgE=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261031084708-acbf80ad910a/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101005736-7b09de1e5502 h1:XndvkzyAFNde/821lEfYObqMTURzmve/w9Ks+QrBBJ8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101005736-7b09de1e5502/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101012031-7cab079e2ec3 h1:an2f0MLx9OhaDLu5i51OMPBbuGwX+aoTzqOkFLMpVfE=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101012031-7cab079e2ec3/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101013319-25a2cecd989c h1:rlUCZ5ifLojqDUTkcCCcPrugTzV2RT6F6JbIVTJkaC8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101013319-25a2cecd989c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101014012-a20bdf795236 h1:JEkSnrmHs5QIRJGhtrghLC1+C0AxqMpef8a32d/BLPg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101014012-a20bdf795236/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101014310-19c204273bdc h1:kVq7sS4I0EnL8DcAA2a3YstnXS6WmYYzJz7eK7IZ9RY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101014310-19c204273bdc/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101020750-62cddf4efc62 h1:R5C8ICjiaDITHRFR/uIATwARPPXKxssUC70m0svstb0=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101020750-62cddf4efc62/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101022905-ba8b8f28a364 h1:UUDPWfUjyerptar+wPwI891jTUAtbbuXori0j3phTE4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101022905-ba8b8f28a364/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101023317-8311ecb84911 h1:+82ygf2hiY5Mhwfmc1xfCe9e/KY/5ObBuE2LecM37qI=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101023317-8311ecb84911/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101025153-c77d17fb42e3 h1:lb6hz+SBsoCAzsiCCvdtPKyE66Y5Ng/DBAkonAexPzA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101025153-c77d17fb42e3/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
git.novox.be/novox/mesh-contracts/go v0.0.0-20261104174344-ae5266e77639 h1:Zt7CfWt2S2W/fVgWNyW2u2+xwz051LXvGxVTNIyfhdM=
git.novox.be/novox/mesh-contracts/go v0.0.0-20261104174344-ae5266e77639/go.mod h1:0dD1XEJBJ0G5UYDQxSZI84nuujb6m3tOpJl4C8rQVvg=
git.novox.be/novox/mesh-contracts/go/clients/nodeengineclient v0.0.0-20261104174344-ae5266e77639 h1:3FTxvHrv3Hv+ZUbp/ezTrndqdgiS+Z6EUG2XGYJNJr8=
git.novox.be/novox/mesh-contracts/go/clients/nodeengineclient v0.0.0-20261104174344-ae5266e77639/go.mod h1:/V/1wfyQwjE9VbPLsFtkf6uX+SHKUXX34s5JscDuGAs=
git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient v0.0.0-20261104174344-ae5266e77639 h1:19PhtrCOwia711Oqdyafvs5EKTVmmgk5Gdb4leCkq3w=
git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient v0.0.0-20261104174344-ae5266e77639/go.mod h1:Dj6fbL8Evgy72EWO3Gl3L3Ra8N/nJSzgEVwSGmeRam0=
git.novox.be/novox/mesh-contracts/go/seats/nodeengine v0.0.0-20261104174344-ae5266e77639 h1:YQ6eJvc+RYUVIVf+oX+R4TC1dYL8TLLMZ0/JPDU1tp4=
git.novox.be/novox/mesh-contracts/go/seats/nodeengine v0.0.0-20261104174344-ae5266e77639/go.mod h1:TB20TsQuOd5aer+cXL6PbaYyg2RvgL1R4MzD61Oks3A=
git.novox.be/novox/mesh-contracts/go/seats/operatorchannel v0.0.0-20261104174344-ae5266e77639 h1:K/1yIkqAWKGtLEtJ0PibijIdpo5YATL/9IUVXFoEglM=
git.novox.be/novox/mesh-contracts/go/seats/operatorchannel v0.0.0-20261104174344-ae5266e77639/go.mod h1:osh0jQyNqYDM27tbxQo6HeUEXgLq9XUewKzNhUXadog=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261104174359-05b3e2e1b94f h1:jDsfDMd48MXY9Zershc3eRxnhmx2sSpUfQkLq5jAIds=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261104174359-05b3e2e1b94f/go.mod h1:08OomDXdwVN8/87nyDtv5R3rH/6LoP8SFFXz30bTYCM=
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op h1:1BOWQJweNyvZMlpAHXGLiZQn9S+QXGcz3xh94lC0w6E=
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ=
github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/jwt/v2 v2.8.2 h1:XXRgB60MSTnqsRwejQurVDs/hcv2dkt+86GjI+I/bMc=
github.com/nats-io/jwt/v2 v2.8.2/go.mod h1:Ag/56sq9OblL4JgdYufDd16Egb17Kr/8WwwuO/forVc=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
@@ -41,5 +33,5 @@ golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZ
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/time v0.16.0 h1:vMb6ptszcQMkcwiRTAuNNU50gom6++Q/6gY2hDM6VDE=
golang.org/x/time v0.16.0/go.mod h1:rVKOqvZeKvrDKTQiAHJ7wmwP0RzleSphoEA9RcdLA0s=
@@ -0,0 +1,68 @@
package assign
import (
"encoding/json"
"fmt"
"time"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// assignerAsk is the asking-th ask a, as the asks state holds it: the router's ask as JSON (delivery.AssignerAsk.Ask).
func assignerAsk(asking int, a oc.Ask) delivery.AssignerAsk {
raw, err := json.Marshal(a)
if err != nil { // an oc.Ask holds only strings, bools and a slice of them: it cannot fail
panic(fmt.Sprintf("assign: the ask %q does not marshal: %v", a.ID, err))
}
return delivery.AssignerAsk{Asking: asking, Ask: raw}
}
// AskOf is the router's ask an asks-state value carries.
func AskOf(a delivery.AssignerAsk) (oc.Ask, error) {
var ask oc.Ask
if err := json.Unmarshal(a.Ask, &ask); err != nil {
return oc.Ask{}, fmt.Errorf("assign: the value carries no router's ask: %w", err)
}
return ask, nil
}
// EncodeAsk is delivery.EncodeAssignerAsk at key, after the checks every keeper of an ask owes (occ.Carried, ADR 0627 §4):
// the ask's id is delivery.AskingID(key, asking), and it offers at least one option.
func EncodeAsk(key string, a delivery.AssignerAsk) ([]byte, error) {
if _, err := checkedAsk(key, a); err != nil {
return nil, err
}
return delivery.EncodeAssignerAsk(key, a)
}
func checkedAsk(key string, a delivery.AssignerAsk) (oc.Ask, error) {
ask, err := occ.Carried(a.Ask, delivery.AskingID(key, a.Asking))
if err != nil {
return oc.Ask{}, fmt.Errorf("assign: %w", err)
}
return ask, nil
}
// DecodeAsk reads the ask at key (delivery.DecodeAssignerAsk) with the router's ask it carries.
func DecodeAsk(key string, data []byte) (delivery.AssignerAsk, oc.Ask, error) {
a, err := delivery.DecodeAssignerAsk(key, data)
if err != nil {
return delivery.AssignerAsk{}, oc.Ask{}, err
}
ask, err := checkedAsk(key, a)
if err != nil {
return delivery.AssignerAsk{}, oc.Ask{}, err
}
return a, ask, nil
}
// ExpiresAt is when the ask ends unanswered; the zero time for an ask that names none.
func ExpiresAt(a oc.Ask) time.Time {
t, err := occ.Time(a.Expires)
if err != nil {
return time.Time{}
}
return t
}
@@ -6,7 +6,8 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -15,21 +16,21 @@ import (
// record's: the machine whose health check failed and the hold's words (issue 1906). asking numbers the writing: 0 the first, a
// larger number an ask written again after a hold lifted by itself, so the router takes it as new.
func HoldAsk(key, module, fp, node, words string, asking int, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Asking: asking, Ask: asks.Ask{
return assignerAsk(asking, oc.Ask{
ID: delivery.AskingID(key, asking),
Headline: fmt.Sprintf("%s is held at %s", module, short(fp)),
Explanation: fmt.Sprintf("The health check of %s failed on %s at %s (%s), so the assigner holds that version and wants "+
"no node to move to it. It lifts the hold by itself once every node that runs %s runs it healthy; releasing it "+
"lets the wants move to it now.", module, node, short(fp), words, module),
Options: []asks.Option{
{ID: "keep", Label: "Keep it held", Does: "the version stays held", Level: asks.Acknowledge},
{ID: "release", Label: "Release it", Does: "the version is released and the wants move to it by the rules again", Level: asks.Acknowledge},
Options: []oc.Option{
{ID: "keep", Label: "Keep it held", Does: "the version stays held", Level: occ.Acknowledge},
{ID: "release", Label: "Release it", Does: "the version is released and the wants move to it by the rules again", Level: occ.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
Who: occ.Operator,
Expires: occ.FormatTime(now.Add(7 * 24 * time.Hour)),
OnExpiry: "the version stays held",
About: key,
}}
})
}
// The options of the restore ask (ADR 0537 §3, ADR 0539 §19), and what each does on the node.
@@ -42,41 +43,41 @@ const (
// not set back by itself, since a set-back would lose the writes made since. words is the check's words. Restore sets the
// node back to the last healthy fingerprint and loses the newer writes; leaving it stopped keeps the data as it is.
func RestoreAsk(key, node, module, fp, words string, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Ask: asks.Ask{
return assignerAsk(0, oc.Ask{
ID: delivery.AskingID(key, 0),
Headline: fmt.Sprintf("%s failed its health check on %s", module, node),
Explanation: fmt.Sprintf("%s holds data, and its health check failed on %s at %s: %s. Setting it back to the version that ran healthy "+
"would lose the writes made since; leaving it stopped keeps the data as it is.", module, node, short(fp), words),
Options: []asks.Option{
{ID: OptionRestore, Label: "Restore it", Does: "the node is set back to the last healthy version and the newer writes are lost", Level: asks.Acknowledge},
{ID: OptionLeaveStopped, Label: "Leave it stopped", Does: "the module stays as it is on the node and the data is kept", Level: asks.Acknowledge},
Options: []oc.Option{
{ID: OptionRestore, Label: "Restore it", Does: "the node is set back to the last healthy version and the newer writes are lost", Level: occ.Acknowledge},
{ID: OptionLeaveStopped, Label: "Leave it stopped", Does: "the module stays as it is on the node and the data is kept", Level: occ.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
Who: occ.Operator,
Expires: occ.FormatTime(now.Add(7 * 24 * time.Hour)),
OnExpiry: "the module stays as it is on the node",
About: key,
}}
})
}
// HeldCondition is the condition the assigner raises when it holds a fingerprint of a module (ADR 0429 §6, ADR 0539 §19), or
// clears (raised false) when the hold is released. It names the module, the node whose health check failed and the check's
// words, under the assigner's own name (ADR 0437 clause 9), keyed by the held fingerprint. needsYou is true for a module holding
// data, whose restore ask waits for the operator.
func HeldCondition(module, fp, node, words string, raised, needsYou bool) asks.Condition {
func HeldCondition(module, fp, node, words string, raised, needsYou bool) oc.Condition {
key := delivery.AssignerModule + ".held." + module + "." + strings.TrimPrefix(fp, delivery.SourcePrefix2)
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
subject := oc.ConditionSubject{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared}
return oc.Condition{Key: key, Subject: subject, State: occ.StateCleared}
}
c := asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
c := oc.Condition{Key: key, Severity: occ.SeverityWarning, Subject: subject, State: occ.StateRaised,
Summary: fmt.Sprintf("%s is held at %s", module, short(fp)),
Headline: fmt.Sprintf("%s is held at %s", module, short(fp))}
what := fmt.Sprintf("The health check of %s failed on %s at %s: %s. The assigner holds that version and wants no node to move to it.", module, node, short(fp), words)
if needsYou {
c.Needs = "answer whether to restore " + module + " on " + node
c.Explanation = asks.NeedsYou + c.Needs + ". " + what
c.Explanation = occ.NeedsYou + c.Needs + ". " + what
} else {
c.Explanation = asks.NothingToDo + " " + what + " Nodes that wanted it are set back to the version that ran healthy."
c.Explanation = occ.NothingToDo + " " + what + " Nodes that wanted it are set back to the version that ran healthy."
}
return c
}
@@ -85,22 +86,22 @@ func HeldCondition(module, fp, node, words string, raised, needsYou bool) asks.C
// node while other nodes run it healthy (ADR 0557 §1, issue 1496): the failure is the node's, so nothing is held or set back and
// the node keeps its want. It names the node, the module, fp and the have's words, keyed by node and module, and is cleared
// (raised false) once the node no longer fails alone.
func NodeFailedCondition(node, module, fp string, f NodeFailure, raised bool) asks.Condition {
func NodeFailedCondition(node, module, fp string, f NodeFailure, raised bool) oc.Condition {
key := delivery.AssignerModule + ".failed-alone." + node + "." + module
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
subject := oc.ConditionSubject{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
return oc.Condition{Key: key, Subject: subject, State: occ.StateCleared,
Resolved: fmt.Sprintf("%s on %s no longer fails its health check alone.", module, node)}
}
headline := fmt.Sprintf("%s fails on %s alone", module, node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
if n := len([]rune(headline)); n > occ.HeadlineLength {
headline = string([]rune(headline)[:occ.HeadlineLength])
}
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
return oc.Condition{Key: key, Severity: occ.SeverityWarning, Subject: subject, State: occ.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s The health check of %s at %s failed on %s (%s), while it runs healthy on %s. The failure is %s's, "+
"so the assigner holds nothing and sets no node back, and %s keeps its want. This clears once %s runs it healthy or "+
"another version is the newest.", asks.NothingToDo, module, short(fp), node, f.Words, strings.Join(f.Others, ", "), node,
"another version is the newest.", occ.NothingToDo, module, short(fp), node, f.Words, strings.Join(f.Others, ", "), node,
node, node)}
}
@@ -108,36 +109,28 @@ func NodeFailedCondition(node, module, fp string, f NodeFailure, raised bool) as
// gives up every module with a fingerprint of the new form, so nothing applies it there and its files are gone. It is raised (urgent)
// naming the module, the node and the rule holding its first want back with the rule's words, under the assigner's own name keyed by
// node and module, and cleared (raised false) when the node has a want for it again.
func NeitherCondition(node, module string, h Holding, raised bool) asks.Condition {
func NeitherCondition(node, module string, h Holding, raised bool) oc.Condition {
key := delivery.AssignerModule + ".neither." + node + "." + module
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
subject := oc.ConditionSubject{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
return oc.Condition{Key: key, Subject: subject, State: occ.StateCleared,
Resolved: fmt.Sprintf("%s on %s has a want again, so its own composition is applied there.", module, node)}
}
headline := fmt.Sprintf("%s on %s is on no list", module, node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
if n := len([]rune(headline)); n > occ.HeadlineLength {
headline = string([]rune(headline)[:occ.HeadlineLength])
}
if h.Bus {
needs := fmt.Sprintf("put %s on %s with assigner_put", module, node)
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: fmt.Sprintf("%s%s. %s is assigned to %s, but the node holds no want for it and the node's own composition does not "+
"hold it either, so nothing applies it there. The assigner never moves the bus: %s. This clears once the node has a want for it.",
asks.NeedsYou, needs, module, node, h.Words)}
}
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
return oc.Condition{Key: key, Severity: occ.SeverityUrgent, Subject: subject, State: occ.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s %s is assigned to %s, but the node holds no want for it and the node's own composition does not "+
"hold it either, so nothing applies it there. The assigner's rule %d holds its first want back: %s. The assigner writes the want once the "+
"rule holds, and clears this.", asks.NothingToDo, module, node, h.Rule, h.Words)}
"rule holds, and clears this.", occ.NothingToDo, module, node, h.Rule, h.Words)}
}
// NeitherLeftCondition clears the neither condition of module on node when it is no longer on neither list: why is "" when the
// node has a live want for it again, else why it has none (it is held, its move waits for an ask, the node no longer runs it), so
// the clear never says a want the node does not hold (issue 1945: "nats on novox has a want again" while it had none).
func NeitherLeftCondition(node, module, why string) asks.Condition {
func NeitherLeftCondition(node, module, why string) oc.Condition {
c := NeitherCondition(node, module, Holding{}, false)
if why != "" {
c.Resolved = fmt.Sprintf("%s on %s is no longer on neither list and has no want: %s.", module, node, why)
@@ -146,7 +139,7 @@ func NeitherLeftCondition(node, module, why string) asks.Condition {
}
// LiftedCondition clears the condition of a held fingerprint the assigner lifted by itself, saying why (issue 1906).
func LiftedCondition(module, fp string) asks.Condition {
func LiftedCondition(module, fp string) oc.Condition {
c := HeldCondition(module, fp, "", "", false, false)
c.Resolved = LiftedWords(module, fp) + "."
return c
@@ -169,20 +162,20 @@ func RecordAsk(key, node, module, fp string, need RecordNeed, now time.Time) del
delivery.JudgedMovesData: "moves the module's data",
delivery.JudgedCannot: "cannot be judged: " + need.Words,
}[need.Judgement]
return delivery.AssignerAsk{Ask: asks.Ask{
return assignerAsk(0, oc.Ask{
ID: delivery.AskingID(key, 0),
Headline: fmt.Sprintf("%s waits for your yes on %s", module, node),
Explanation: fmt.Sprintf("%s is recorded: a person takes each of its moves. The next one on %s (version %s) %s. "+
"The node is told to apply it only after you say yes.", module, node, short(fp), what),
Options: []asks.Option{
{ID: OptionYes, Label: "Yes, move it", Does: "the node is told to apply the new composition of " + module, Level: asks.Acknowledge},
{ID: OptionNo, Label: "No, not now", Does: "the node keeps what it runs", Level: asks.Acknowledge},
Options: []oc.Option{
{ID: OptionYes, Label: "Yes, move it", Does: "the node is told to apply the new composition of " + module, Level: occ.Acknowledge},
{ID: OptionNo, Label: "No, not now", Does: "the node keeps what it runs", Level: occ.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
Who: occ.Operator,
Expires: occ.FormatTime(now.Add(7 * 24 * time.Hour)),
OnExpiry: "the node keeps what it runs",
About: key,
}}
})
}
// NextBackupAsking is the number of the asking that follows the one numbered after (a negative number: none yet). Numbers 0 and 1
@@ -200,7 +193,11 @@ var backupDigest = regexp.MustCompile(`\(composition (sha256:[0-9a-f]{64})\)`)
// none. An answer is taken for the composition of the ask it answers, and a new revision at the same fingerprint is asked again
// under a new number (ADR 0539 §16).
func BackupAskDigest(a delivery.AssignerAsk) string {
if m := backupDigest.FindStringSubmatch(a.Ask.Explanation); m != nil {
ask, err := AskOf(a)
if err != nil {
return ""
}
if m := backupDigest.FindStringSubmatch(ask.Explanation); m != nil {
return m[1]
}
return ""
@@ -216,33 +213,33 @@ func BackupAnswerWords(option string, asking int) string {
// without a backup. digest is the composition about to be applied and asking the number of the asking, 0 the first; words are the
// backup's own words, from the node's have.
func BackupAsk(key, node, module, fp, digest, words string, asking int, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Asking: asking, Ask: asks.Ask{
return assignerAsk(asking, oc.Ask{
ID: delivery.AskingID(key, asking),
Headline: fmt.Sprintf("The backup of %s failed on %s", module, node),
Explanation: fmt.Sprintf("The backup before applying %s at %s on %s failed: %s. %s keeps running what it has until you answer. (composition %s)",
module, short(fp), node, words, module, digest),
Options: []asks.Option{
{ID: OptionBackupAgain, Label: "Back up again", Does: "the node tries the backup again before it applies", Level: asks.Acknowledge},
{ID: OptionBackupSkip, Label: "Go on without a backup", Does: "the node applies the new version without a backup", Level: asks.Acknowledge},
Options: []oc.Option{
{ID: OptionBackupAgain, Label: "Back up again", Does: "the node tries the backup again before it applies", Level: occ.Acknowledge},
{ID: OptionBackupSkip, Label: "Go on without a backup", Does: "the node applies the new version without a backup", Level: occ.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
Who: occ.Operator,
Expires: occ.FormatTime(now.Add(7 * 24 * time.Hour)),
OnExpiry: "the module keeps running what it has",
About: key,
}}
})
}
// BackupCondition is the condition raised when a backup before an apply fails and the module is held on the node (ADR 0539 §16),
// naming the node, the module and the backup's words; cleared under the same key.
func BackupCondition(node, module, fp, words string, raised bool) asks.Condition {
func BackupCondition(node, module, fp, words string, raised bool) oc.Condition {
key := delivery.AssignerModule + ".backup." + node + "." + module + "." + strings.TrimPrefix(fp, delivery.SourcePrefix2)
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
subject := oc.ConditionSubject{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared}
return oc.Condition{Key: key, Subject: subject, State: occ.StateCleared}
}
headline := fmt.Sprintf("The backup of %s failed on %s", module, node)
needs := "answer whether to back up again or go on without a backup"
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
return oc.Condition{Key: key, Severity: occ.SeverityWarning, Subject: subject, State: occ.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: asks.NeedsYou + needs + ". The backup before applying " + module + " at " + short(fp) + " on " + node + " failed: " + words + "."}
Explanation: occ.NeedsYou + needs + ". The backup before applying " + module + " at " + short(fp) + " on " + node + " failed: " + words + "."}
}
@@ -1,12 +1,14 @@
package assign
import (
"fmt"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
@@ -20,27 +22,42 @@ func TestTheHoldAskIsValid(t *testing.T) {
}
now := time.Now()
a := HoldAsk(key, "mail", fNew, "ace", "health failed, phase running: connection refused", 0, now)
if err := a.Ask.Check(now); err != nil {
if err := occ.Check(mustAsk(t, a), now); err != nil {
t.Errorf("the router refuses the ask: %v", err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
if _, err := EncodeAsk(key, a); err != nil {
t.Errorf("the ask cannot be written: %v", err)
}
// 1906/H3: the ask names the machine whose check failed and the hold's words.
if !strings.Contains(a.Ask.Explanation, "on ace") || !strings.Contains(a.Ask.Explanation, "connection refused") {
t.Errorf("the ask does not name the node and the hold's words: %q", a.Ask.Explanation)
if !strings.Contains(mustAsk(t, a).Explanation, "on ace") || !strings.Contains(mustAsk(t, a).Explanation, "connection refused") {
t.Errorf("the ask does not name the node and the hold's words: %q", mustAsk(t, a).Explanation)
}
// 1906/H2: an ask written again after a lift is another asking, under an id of its own.
again := HoldAsk(key, "mail", fNew, "ace", "x", 7, now)
if again.Asking != 7 || again.Ask.ID != delivery.AskingID(key, 7) || again.Ask.ID == a.Ask.ID {
t.Errorf("the asking after a lift = %d %s", again.Asking, again.Ask.ID)
if again.Asking != 7 || mustAsk(t, again).ID != delivery.AskingID(key, 7) || mustAsk(t, again).ID == mustAsk(t, a).ID {
t.Errorf("the asking after a lift = %d %s", again.Asking, mustAsk(t, again).ID)
}
if _, err := delivery.EncodeAssignerAsk(key, again); err != nil {
if _, err := EncodeAsk(key, again); err != nil {
t.Errorf("the ask asked again cannot be written: %v", err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("the view may not answer it: %v", err)
}
// The join (A1d ruling 1): mesh-sdk no longer refuses another id, no option or no ask in an assigner's ask; EncodeAsk does.
t.Run("EncodeAsk refuses what mesh-sdk no longer does", func(t *testing.T) {
good := HoldAsk(key, "mail", fNew, "ace", "x", 0, time.Unix(100, 0))
other := good
other.Asking = 3 // its id is the asking 0's
empty := good
ask := mustAsk(t, good)
ask.Options = nil
empty.Ask = assignerAsk(0, ask).Ask
for name, a := range map[string]delivery.AssignerAsk{"another id": other, "no option": empty, "no ask": {}} {
if _, err := EncodeAsk(key, a); err == nil {
t.Errorf("%s: written", name)
}
}
})
}
// 1906/D3 (ADR 0539 §19): the ask about a data module held on a node is a valid ask of the router and of mesh-sdk's layout at
@@ -52,24 +69,24 @@ func TestTheRestoreAskIsValid(t *testing.T) {
}
now := time.Now()
a := RestoreAsk(key, "ace", "mail", fNew, "connection refused", now)
if err := a.Ask.Check(now); err != nil {
if err := occ.Check(mustAsk(t, a), now); err != nil {
t.Errorf("the router refuses the ask: %v", err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
if _, err := EncodeAsk(key, a); err != nil {
t.Errorf("the ask cannot be written: %v", err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("the view may not answer it: %v", err)
}
var ids []string
for _, o := range a.Ask.Options {
for _, o := range mustAsk(t, a).Options {
ids = append(ids, o.ID)
}
if len(ids) != 2 || ids[0] != OptionRestore || ids[1] != OptionLeaveStopped {
t.Errorf("options = %v", ids)
}
if !strings.Contains(a.Ask.Explanation, "connection refused") || !strings.Contains(a.Ask.Explanation, "ace") {
t.Errorf("the ask does not name the node and the check's words: %q", a.Ask.Explanation)
if !strings.Contains(mustAsk(t, a).Explanation, "connection refused") || !strings.Contains(mustAsk(t, a).Explanation, "ace") {
t.Errorf("the ask does not name the node and the check's words: %q", mustAsk(t, a).Explanation)
}
}
@@ -78,18 +95,18 @@ func TestTheRestoreAskIsValid(t *testing.T) {
func TestTheAssignersConditions(t *testing.T) {
t.Run("the held condition names module, node and words", heldConditionNamesModuleNodeAndWords)
t.Run("a lifted hold clears its condition saying why", liftedHoldClearsItsConditionSayingWhy)
t.Run("1952/WM6: each said condition is a finding, its clear deletes it, said straight only until the take-over", saidConditionsAreFindings)
t.Run("1952/WM6: each said condition is a finding, its clear deletes it, said straight only with no findings bucket", saidConditionsAreFindings)
}
func heldConditionNamesModuleNodeAndWords(t *testing.T) {
c := HeldCondition("mail", fNew, "ace", "connection refused", true, true)
if _, err := asks.EncodeCondition(c); err != nil {
if _, err := occ.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateRaised || c.Subject.ID != "mail" || c.Subject.Machine != "ace" || !strings.Contains(c.Explanation, "connection refused") || !strings.Contains(c.Key, "mail") {
if c.State != occ.StateRaised || c.Subject.ID != "mail" || c.Subject.Machine != "ace" || !strings.Contains(c.Explanation, "connection refused") || !strings.Contains(c.Key, "mail") {
t.Errorf("condition = %+v", c)
}
if got := HeldCondition("mail", fNew, "", "", false, false); got.State != asks.StateCleared || got.Key != c.Key {
if got := HeldCondition("mail", fNew, "", "", false, false); got.State != occ.StateCleared || got.Key != c.Key {
t.Errorf("cleared = %+v", got)
}
}
@@ -97,52 +114,53 @@ func heldConditionNamesModuleNodeAndWords(t *testing.T) {
// 1906/H2: a hold lifted by itself clears the held condition under its key, saying why.
func liftedHoldClearsItsConditionSayingWhy(t *testing.T) {
c := LiftedCondition("mail", fNew)
if _, err := asks.EncodeCondition(c); err != nil {
if _, err := occ.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateCleared || c.Key != HeldCondition("mail", fNew, "ace", "x", true, false).Key || !strings.Contains(c.Resolved, "lifted its hold by itself") {
if c.State != occ.StateCleared || c.Key != HeldCondition("mail", fNew, "ace", "x", true, false).Key || !strings.Contains(c.Resolved, "lifted its hold by itself") {
t.Errorf("condition = %+v", c)
}
}
// 1952/WM6 (ADR 0564 §2, issue 1952 note 16): every condition the assigner says is a finding in its own bucket, under
// `<scope>.<id>.<kind>`, written at the raise and deleted at the clear; it is said straight to the router while the take-over
// key follows the controller, and only as a finding from the take-over.
// `<scope>.<id>.<kind>`, written at the raise and deleted at the clear, and only as a finding: the keeper is the one
// source since the controller's end (novox/hq ADR 0590 §3, issue 1126 B8). Said straight to the router only where the
// findings bucket cannot be written to at all.
func saidConditionsAreFindings(t *testing.T) {
at := time.Date(2026, 10, 29, 21, 0, 0, 0, time.UTC)
for _, c := range []struct {
raised, cleared asks.Condition
raised, cleared oc.Condition
key string
}{
{HeldCondition("mail", fNew, "ace", "refused", true, true), LiftedCondition("mail", fNew), "module.mail." + strings.TrimPrefix(fNew, delivery.SourcePrefix2) + ".held"},
{NodeFailedCondition("ace", "mail", fNew, NodeFailure{Words: "x", Others: []string{"g14"}}, true), NodeFailedCondition("ace", "mail", fNew, NodeFailure{}, false), "module.ace.mail.failed-alone"},
{NeitherCondition("ace", "mail", Holding{Rule: 3, Words: "w"}, true), NeitherCondition("ace", "mail", Holding{}, false), "module.ace.mail.neither"},
{PlacementCondition(Pair{Node: "ace", Module: "mail", Only: OnlyAssignments}, true), PlacementCondition(Pair{Node: "ace", Module: "mail"}, false), "module.ace.mail.placement"},
{SeatWaitsCondition([]string{"placement"}), SeatWaitsCondition(nil), "module.mesh-delivery-assigner.seat-states"},
{BackupCondition("ace", "mail", fNew, "disk full", true), BackupCondition("ace", "mail", fNew, "", false), "module.ace.mail." + strings.TrimPrefix(fNew, delivery.SourcePrefix2) + ".backup"},
} {
puts, deletes, said := map[string]health.Finding{}, []string{}, 0
keeper := false
f := Findings{Put: func(k string, v []byte) error {
got, err := health.DecodeFinding(v)
puts[k] = got
return err
}, Delete: func(k string) error { deletes = append(deletes, k); return nil }, ToKeeper: func() bool { return keeper }, Now: func() time.Time { return at }}
say := f.Say(func(asks.Condition) error { said++; return nil })
}, Delete: func(k string) error { deletes = append(deletes, k); return nil }, Now: func() time.Time { return at }}
say := f.Say(func(oc.Condition) error { said++; return nil })
if err := say(c.raised); err != nil {
t.Fatalf("%s: %v", c.key, err)
}
got, ok := puts[c.key]
if !ok || got.Severity != health.Severity(c.raised.Severity) || got.Headline != c.raised.Headline || got.Observer != delivery.AssignerModule ||
got.Subject.Machine != c.raised.Subject.Machine || said != 1 {
got.Subject.Machine != c.raised.Subject.Machine || said != 0 {
t.Fatalf("%s: finding %+v (written %v), said %d", c.key, got, puts, said)
}
keeper = true
if err := say(c.cleared); err != nil || !slices.Equal(deletes, []string{c.key}) || said != 1 {
t.Fatalf("%s: deleted %v, said %d after the take-over (%v)", c.key, deletes, said, err)
if err := say(c.cleared); err != nil || !slices.Equal(deletes, []string{c.key}) || said != 0 {
t.Fatalf("%s: deleted %v, said %d (%v)", c.key, deletes, said, err)
}
if err := (Findings{}).Say(func(oc.Condition) error { said++; return nil })(c.raised); err != nil || said != 1 {
t.Fatalf("%s: with no findings bucket said %d (%v), want straight once", c.key, said, err)
}
}
if _, _, _, err := FindingOf(asks.Condition{Key: "another.x", State: asks.StateRaised}, at); err == nil {
if _, _, _, err := FindingOf(oc.Condition{Key: "another.x", State: occ.StateRaised}, at); err == nil {
t.Error("a condition not the assigner's is a finding")
}
}
@@ -163,24 +181,24 @@ func TestTheRecordAndBackupAsksAreValid(t *testing.T) {
dgst := "sha256:" + strings.Repeat("b", 64)
b := BackupAsk(bkey, "ace", "mail", fNew, dgst, "no space left", 0, now)
for key, a := range map[string]delivery.AssignerAsk{rkey: r, bkey: b} {
if err := a.Ask.Check(now); err != nil {
if err := occ.Check(mustAsk(t, a), now); err != nil {
t.Errorf("%s: the router refuses the ask: %v", key, err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
if _, err := EncodeAsk(key, a); err != nil {
t.Errorf("%s: cannot be written: %v", key, err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("%s: the view may not answer it: %v", key, err)
}
}
if !strings.Contains(r.Ask.Explanation, "gone") || r.Ask.Options[0].ID != OptionYes {
t.Errorf("the record ask = %+v", r.Ask)
if !strings.Contains(mustAsk(t, r).Explanation, "gone") || mustAsk(t, r).Options[0].ID != OptionYes {
t.Errorf("the record ask = %+v", mustAsk(t, r))
}
if !strings.Contains(b.Ask.Explanation, "no space left") || b.Ask.Options[0].ID != delivery.BackupAgain || b.Ask.Options[1].ID != delivery.BackupSkip {
t.Errorf("the backup ask = %+v", b.Ask)
if !strings.Contains(mustAsk(t, b).Explanation, "no space left") || mustAsk(t, b).Options[0].ID != delivery.BackupAgain || mustAsk(t, b).Options[1].ID != delivery.BackupSkip {
t.Errorf("the backup ask = %+v", mustAsk(t, b))
}
// the id binds the ask to the composition and to its number (review round 1, items 3 and 4)
if again := BackupAsk(bkey, "ace", "mail", fNew, dgst, "x", NextBackupAsking(0), now); again.Ask.ID == b.Ask.ID {
if again := BackupAsk(bkey, "ace", "mail", fNew, dgst, "x", NextBackupAsking(0), now); mustAsk(t, again).ID == mustAsk(t, b).ID {
t.Errorf("asked again under the same id")
}
if NextBackupAsking(-1) != 0 || NextBackupAsking(0) != 2 || NextBackupAsking(2) != 3 {
@@ -193,7 +211,7 @@ func TestTheRecordAndBackupAsksAreValid(t *testing.T) {
t.Errorf("two answers of the same option are one want value")
}
c := BackupCondition("ace", "mail", fNew, "no space left", true)
if _, err := asks.EncodeCondition(c); err != nil {
if _, err := occ.EncodeCondition(c); err != nil {
t.Errorf("the backup condition is refused: %v", err)
}
// ADR 0546 clauses 7 and 8: a put's ask is valid at the record ask's key; for the bus its yes approves and binds the act,
@@ -205,32 +223,39 @@ func TestTheRecordAndBackupAsksAreValid(t *testing.T) {
if err != nil {
t.Fatalf("bus %v: %v", bus, err)
}
if err := a.Ask.Check(now); err != nil {
if err := occ.Check(mustAsk(t, a), now); err != nil {
t.Errorf("bus %v: the router refuses the put's ask: %v", bus, err)
}
if _, err := delivery.EncodeAssignerAsk(rkey, a); err != nil {
if _, err := EncodeAsk(rkey, a); err != nil {
t.Errorf("bus %v: the put's ask cannot be written: %v", bus, err)
}
viewed := delivery.ViewAnswerFits(delivery.ViewAnswer{Asking: a.Ask.ID, Option: OptionYes}, a)
yes := a.Ask.Options[0]
viewed := delivery.ViewAnswerFits(delivery.ViewAnswer{Asking: mustAsk(t, a).ID, Option: OptionYes}, rkey, a)
yes := mustAsk(t, a).Options[0]
// mesh-sdk gives no refusal by level any more: the teller refuses an option a view cannot answer (its level is not
// acknowledge), as it did through the sdk before.
if viewed == nil {
if o, ok := occ.OptionOf(mustAsk(t, a), OptionYes); ok && o.Level != "" && o.Level != occ.Acknowledge {
viewed = fmt.Errorf("only a channel that proves who answered takes the option %s", o.ID)
}
}
switch {
case !strings.Contains(a.Ask.Explanation, "the bus's user list"):
t.Errorf("bus %v: the put's ask does not say why: %s", bus, a.Ask.Explanation)
case bus && (yes.Level != asks.Approve || yes.Binds == "" || viewed == nil || a.Ask.Expires.After(now.Add(asks.ApproveLasts))):
t.Errorf("bus: the yes is %+v, the view's answer %v, expiring %s", yes, viewed, a.Ask.Expires)
case !bus && (yes.Level != asks.Acknowledge || viewed != nil):
case !strings.Contains(mustAsk(t, a).Explanation, "the bus's user list"):
t.Errorf("bus %v: the put's ask does not say why: %s", bus, mustAsk(t, a).Explanation)
case bus && (yes.Level != occ.Approve || yes.Binds == "" || viewed == nil || expiresAt(t, a).After(now.Add(occ.ApproveLasts))):
t.Errorf("bus: the yes is %+v, the view's answer %v, expiring %s", yes, viewed, expiresAt(t, a))
case !bus && (yes.Level != occ.Acknowledge || viewed != nil):
t.Errorf("a recorded module: the yes is %+v, the view's answer %v", yes, viewed)
}
later := put
later.At = now.Add(time.Minute)
b, _ := PutAsk(rkey, "ace", "mail", later, d, bus, now)
if b.Ask.ID == a.Ask.ID || b.Asking != PutAsking(later) {
t.Errorf("bus %v: a later put's ask is asking %d under id %q, the earlier's %q", bus, b.Asking, b.Ask.ID, a.Ask.ID)
if mustAsk(t, b).ID == mustAsk(t, a).ID || b.Asking != PutAsking(later) {
t.Errorf("bus %v: a later put's ask is asking %d under id %q, the earlier's %q", bus, b.Asking, mustAsk(t, b).ID, mustAsk(t, a).ID)
}
if err := yes.Performs(PutAct("ace", "mail", fNew, dgst)); err != nil {
if err := occ.Performs(yes, PutAct("ace", "mail", fNew, dgst)); err != nil {
t.Errorf("bus %v: the yes does not perform the act it bound: %v", bus, err)
}
if err := yes.Performs(PutAct("ace", "mail", fNew, dig(fOld))); bus && err == nil {
if err := occ.Performs(yes, PutAct("ace", "mail", fNew, dig(fOld))); bus && err == nil {
t.Errorf("bus: the yes performs another composition's act")
}
}
@@ -240,14 +265,29 @@ func TestTheRecordAndBackupAsksAreValid(t *testing.T) {
// under the assigner's own name keyed by node and module, and is cleared under the same key.
func TestTheNeitherConditionNamesModuleNodeAndRule(t *testing.T) {
c := NeitherCondition("g14", "i3status-rust", Holding{Rule: delivery.RuleProviders, Words: "no provider of x11-display runs healthy on g14 or at its canary"}, true)
if _, err := asks.EncodeCondition(c); err != nil {
if _, err := occ.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateRaised || c.Severity != asks.SeverityUrgent || c.Subject.ID != "i3status-rust" || c.Subject.Machine != "g14" ||
if c.State != occ.StateRaised || c.Severity != occ.SeverityUrgent || c.Subject.ID != "i3status-rust" || c.Subject.Machine != "g14" ||
!strings.Contains(c.Explanation, "x11-display") || !strings.Contains(c.Explanation, "rule 5") || c.Key != "mesh-delivery-assigner.neither.g14.i3status-rust" {
t.Errorf("condition = %+v", c)
}
if got := NeitherCondition("g14", "i3status-rust", Holding{}, false); got.State != asks.StateCleared || got.Key != c.Key {
if got := NeitherCondition("g14", "i3status-rust", Holding{}, false); got.State != occ.StateCleared || got.Key != c.Key {
t.Errorf("cleared = %+v", got)
}
}
// mustAsk is the router's ask an asks-state value carries.
func mustAsk(t *testing.T, a delivery.AssignerAsk) oc.Ask {
t.Helper()
ask, err := AskOf(a)
if err != nil {
t.Fatal(err)
}
return ask
}
func expiresAt(t *testing.T, a delivery.AssignerAsk) time.Time {
t.Helper()
return ExpiresAt(mustAsk(t, a))
}
@@ -1,368 +0,0 @@
// Package assign is the assigner's judgement, as functions over what it read from the bus: the composition check of a
// change (novox/hq ADR 0446 clauses 1 to 5, ADR 0528 clause 5, issue 1297), the six rules of what each node should
// run (ADR 0429 §6, ADR 0528 clause 3) and the comparison with what the controller sends (ADR 0528 clause 4). It reads
// and writes nothing itself: the program hands it what it watched and writes what it answers.
package assign
import (
"encoding/json"
"fmt"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// SDKRepository is the repository of mesh-sdk: a group with a member there changes the composition itself.
const SDKRepository = "novox/mesh-sdk"
// NotJudged names, in a composition's warnings, the failures of ADR 0446 clause 1 the assigner cannot judge from the
// bus, because what they need is only in the controller's facts snapshot, which is not on the bus yet (issue 1461;
// the design limit of R4g, issue 1297): said on every verdict, never passed silently.
const NotJudged = "not judged until the facts snapshot is on the bus (issue 1461): a consumer left out of a grant, a " +
"credential left bound elsewhere, a stored setting the changed definition cannot keep, and a new module the " +
"node-engine's validator would refuse"
// Mesh is the mesh as the trunk holds it: every module's manifest by name, the manifests that cannot be read with
// why, and each node's open assignments.
type Mesh struct {
Manifests map[string]catalogue.Manifest
Unreadable map[string]string
Assigned map[string][]string
// Digests is every trunk module's manifest digest in the indexer's manifests bucket, readable or not: what a
// composer at another mesh-sdk reads (issue 1622).
Digests map[string]string
// Input is the rest of the mesh as the nodes compose from it, filled by mesh-sdk's frombus from the catalogue's
// placement and the nodes' records (novox/hq ADR 0547 §2, ADR 0549 §3, issue 1126), so the check and the nodes share
// one composer's input; nil before any placement is read, when the nodes and their assignments alone are composed.
Input *compose.Input
// InputUnread says why that input could not be read from the bus, when it could not: the check then composes the
// nodes and their assignments alone, and says so in its warnings.
InputUnread string
// Verdicts are the indexer's refusals of trunk manifests, by module, as their values carry them (novox/hq ADR 0578
// §3): each such module is in Unreadable too, and the composer's input carries the verdict (§5).
Verdicts map[string]delivery.ManifestVerdict
}
// Change is a group's change as the indexer's candidate holds it: the manifest of every module it moves or adds, those
// it cannot read and why, the modules it removes, every module defined twice with where, the heads of a mesh-sdk
// member (empty when none), and how many modules it rebuilds.
type Change struct {
Group, Heads string
Manifests map[string]catalogue.Manifest
Unreadable map[string]string
Removed []string
Twice map[string][]string
SDKHead string
Rebuilt int
// Digests is the manifest digest of every module the change moves or adds, readable by the assigner or not, and
// Members the head of each member by repository: what a composer at the group's mesh-sdk reads (issues 1622, 1623).
Digests map[string]string
Members map[string]string
// Held are the approved held values of the group's pull requests at the heads judged (mesh-config's `kept`, novox/hq ADR
// 0565 §5, ADR 0502 §4): each laid over its module's values on its node in the change's composition, and named.
Held []delivery.KeptValue
}
// HeldOf are the held values kept for the pull requests the candidate names, each at the head it names: one of another
// pull request, or at another head, is never laid (ADR 0502 §4). kept is mesh-config's `kept` by key.
func HeldOf(kept map[string][]byte, heads map[string]string) []delivery.KeptValue {
var out []delivery.KeptValue
for _, raw := range kept {
var k delivery.KeptValue
if json.Unmarshal(raw, &k) != nil || k.Node == "" || k.Module == "" || k.Key == "" {
continue
}
if head, ok := heads[k.Pull]; ok && head == k.Head {
out = append(out, k)
}
}
sort.Slice(out, func(i, j int) bool {
a, b := out[i], out[j]
return a.Pull+a.Node+a.Module+a.Key < b.Pull+b.Node+b.Module+b.Key
})
return out
}
// layHeld lays each held value over its module's values on its node, as a layer of its own after the rest, and answers
// the words that name each; judged first against the head's declaration of its key (catalogue.ValueProblems), a value the
// declaration refuses, or a key it does not declare, is a failure.
func layHeld(in *compose.Input, held []delivery.KeptValue) (said, fails []string) {
for _, k := range held {
var v any
if err := json.Unmarshal(k.Value, &v); err != nil {
said = append(said, fmt.Sprintf("the held value %s of %s on %s for %s cannot be read, so it is not laid: %v", k.Key, k.Module, k.Node, k.Pull, err))
continue
}
d, declared := in.Catalogue[k.Module].Settings[k.Key]
if !declared {
fails = append(fails, fmt.Sprintf("the held value %s of %s on %s for %s: the head declares no such key", k.Key, k.Module, k.Node, k.Pull))
continue
}
if problems := catalogue.ValueProblems(d, v); len(problems) > 0 {
fails = append(fails, fmt.Sprintf("the held value %s of %s on %s for %s does not hold to the head's declaration: %s", k.Key, k.Module,
k.Node, k.Pull, strings.Join(problems, "; ")))
continue
}
at := slices.IndexFunc(in.Nodes, func(n compose.Node) bool { return n.Name == k.Node })
if at < 0 {
said = append(said, fmt.Sprintf("the held value %s of %s for %s names %s, which runs no module of the mesh: not laid", k.Key, k.Module, k.Pull, k.Node))
continue
}
if in.Nodes[at].Settings == nil {
in.Nodes[at].Settings = map[string][]catalogue.Layer{}
}
in.Nodes[at].Settings[k.Module] = append(in.Nodes[at].Settings[k.Module], catalogue.Layer{From: "held for " + k.Pull, Values: map[string]any{k.Key: v}})
said = append(said, fmt.Sprintf("held value laid: %s of %s on %s, held for %s at %.12s, approved by %s (novox/hq ADR 0502 §4)", k.Key, k.Module, k.Node, k.Pull, k.Head, k.Approved))
}
return said, fails
}
// Touched is the machines a change touches (ADR 0446 clause 3): those running a module the change moves, adds or
// removes, those running a consumer of a provision such a module provides (before or after), and those running a
// provider of a provision such a module requires after the change. A module the change adds that is assigned nowhere
// touches no machine. A change of placement is no change of a repository (ADR 0567 §4): the seat
// mesh-placement judges it.
func Touched(m Mesh, c Change) []string {
moved := map[string]bool{}
for name := range c.Manifests {
moved[name] = true
}
for name := range c.Unreadable {
moved[name] = true
}
for _, name := range c.Removed {
moved[name] = true
}
provided, required := map[string]bool{}, map[string]bool{}
for name := range moved {
for _, man := range []catalogue.Manifest{m.Manifests[name], c.Manifests[name]} {
for _, o := range man.Provides {
provided[o.Name] = true
}
}
for _, r := range c.Manifests[name].Requires {
required[r] = true
}
}
var out []string
for node, modules := range m.Assigned {
for _, mod := range modules {
man := m.Manifests[mod]
if moved[mod] || slices.ContainsFunc(man.Requires, func(r string) bool { return provided[r] }) ||
slices.ContainsFunc(man.Provides, func(o catalogue.Offer) bool { return required[o.Name] }) {
out = append(out, node)
break
}
}
}
sort.Strings(out)
return out
}
// input is the composition's input for the mesh's catalogue and assignments: the nodes by name with what is assigned
// to them, and every seat the assigned modules claim as held there. What the bus does not carry (places on the
// private network, settings, credentials, licences) is left empty, so a trunk composition and a change's are made
// with the same gaps and compared (ADR 0446 clause 1: what was already broken is said and fails nothing).
func input(catalog map[string]catalogue.Manifest, assigned map[string][]string) compose.Input {
in := compose.Input{Catalogue: catalog}
nodes := make([]string, 0, len(assigned))
for n := range assigned {
nodes = append(nodes, n)
}
sort.Strings(nodes)
for _, n := range nodes {
in.Nodes = append(in.Nodes, compose.Node{Name: n, Assigned: slices.Clone(assigned[n])})
for _, mod := range assigned[n] {
for _, cl := range catalog[mod].Claims {
scope := cl.Scope
if seat, ok := catalogue.SeatNamed(cl.Name); ok && scope == "" {
scope = seat.Scope
}
in.Holdings = append(in.Holdings, catalogue.Held{Claim: cl.Name, Scope: scope, Node: n, Module: mod})
}
}
}
return in
}
// Compose is the composition check of change c over the mesh m, composed with the mesh-sdk version sdk: every touched
// machine resolved as the trunk holds it and as the change leaves it. It fails on a manifest of the change that cannot
// be read, a module a machine runs that the change removes, a module defined twice, and a machine that resolves on the
// trunk and not with the change; a machine that does not resolve on either is said and fails nothing. It warns past
// CompositionRebuildWarning modules rebuilt, says what it does not judge (NotJudged), and for a group with a mesh-sdk
// member says it composed with its own mesh-sdk, not the member's head (issue 1621). took is how long it took.
func Compose(m Mesh, c Change, sdk string, took time.Duration) delivery.CompositionVerdict {
v := delivery.CompositionVerdict{Group: c.Group, Heads: c.Heads, SDK: sdk, Took: took.Milliseconds()}
var fails []string
failAt := func(machine, words string) {
if v.Machine == "" && machine != "" {
v.Machine = machine
}
fails = append(fails, words)
}
for _, name := range sortedKeys(c.Unreadable) {
failAt("", fmt.Sprintf("the manifest of %s cannot be read: %s", name, c.Unreadable[name]))
}
for _, name := range sortedKeys(c.Twice) {
failAt("", fmt.Sprintf("%s is defined twice, in %s", name, strings.Join(c.Twice[name], " and ")))
}
machines := Touched(m, c)
v.Machines = machines
for _, node := range machines {
for _, name := range c.Removed {
if slices.Contains(m.Assigned[node], name) {
failAt(node, fmt.Sprintf("%s runs %s, which the change removes from its source", node, name))
}
}
}
trunk := input(m.Manifests, m.Assigned)
changed := map[string]catalogue.Manifest{}
for name, man := range m.Manifests {
changed[name] = man
}
for name, man := range c.Manifests {
changed[name] = man
}
for _, name := range c.Removed {
delete(changed, name)
}
after := input(changed, m.Assigned)
laid, heldFails := layHeld(&after, c.Held)
v.Warnings = append(v.Warnings, laid...)
for _, f := range heldFails {
failAt("", f)
}
for _, node := range machines {
_, before := compose.Resolve(trunk, node)
_, now := compose.Resolve(after, node)
switch {
case now == nil:
case before != nil:
v.Warnings = append(v.Warnings, fmt.Sprintf("%s does not compose on the trunk either, so the change is not blamed: %v", node, before))
default:
failAt(node, fmt.Sprintf("%s composes on the trunk and not with the change: %v", node, now))
}
}
if c.SDKHead != "" {
v.Warnings = append(v.Warnings, fmt.Sprintf("composed with the assigner's own mesh-sdk %s, not the group's mesh-sdk head %s, "+
"and a field the running controller cannot read is not judged (issues 1621, 1622, 1623)", sdk, c.SDKHead))
}
return finish(v, m, c, machines, fails)
}
// finish adds what every composition says beside its own judgement (more modules rebuilt than CompositionRebuildWarning,
// what is not judged, the machines not composed, a change that touches none) and its state from fails.
func finish(v delivery.CompositionVerdict, m Mesh, c Change, machines []string, fails []string) delivery.CompositionVerdict {
if c.Rebuilt > delivery.CompositionRebuildWarning {
v.Warnings = append(v.Warnings, fmt.Sprintf("the merge rebuilds %d modules, more than %d", c.Rebuilt, delivery.CompositionRebuildWarning))
}
v.Warnings = append(v.Warnings, NotJudged)
if could := CouldRun(m, c, machines); len(could) > 0 {
v.Warnings = append(v.Warnings, "not judged until the facts snapshot is on the bus (issue 1461, ADR 0537 clause 1): "+
"the machines that could run a moved or added module and are not assigned it: "+strings.Join(could, ", "))
}
if grants := GrantsAlone(m, c, machines); len(grants) > 0 {
v.Warnings = append(v.Warnings, "not judged until the facts snapshot is on the bus (issue 1461, ADR 0537 clause 1): "+
"the machines whose grants alone the change alters: "+strings.Join(grants, ", "))
}
if len(machines) == 0 {
v.Warnings = append(v.Warnings, "the change touches no machine: no module it moves is assigned anywhere")
}
if len(fails) > 0 {
v.State, v.Words = delivery.CompositionFail, strings.Join(fails, "; ")
return v
}
v.State = delivery.CompositionPass
return v
}
// CouldRun is, by name, every machine of the mesh not composed that is not assigned a module the change moves or adds
// (ADR 0537 clause 1): which of them could run it, the bus does not say (capabilities, settings and places are only in
// the facts snapshot), so each is named as not judged.
func CouldRun(m Mesh, c Change, composed []string) []string {
var out []string
for node, modules := range m.Assigned {
if slices.Contains(composed, node) {
continue
}
for name := range c.Manifests {
if !slices.Contains(modules, name) {
out = append(out, node)
break
}
}
}
sort.Strings(out)
return out
}
// grantsOf is what of a manifest makes grants: its bus declarations and the provisions it requires and offers.
func grantsOf(man catalogue.Manifest) string {
var offers []string
for _, o := range man.Provides {
offers = append(offers, o.Name)
}
return fmt.Sprint(man.Requires, offers, man.Reads, man.State, man.Emits, man.Consumes, man.Invokes, man.Claims, man.Grants)
}
// GrantsAlone is, by name, every machine not composed whose grants alone the change alters (ADR 0537 clause 1): for a
// moved module whose grants differ from the trunk's, the machines assigned the bus's holder (whose user list carries
// every module's bus grants) and those assigned a provider of a provision it requires, before or after.
func GrantsAlone(m Mesh, c Change, composed []string) []string {
provisions := map[string]bool{}
altered := false
for name, man := range c.Manifests {
before, had := m.Manifests[name]
if had && grantsOf(before) == grantsOf(man) {
continue
}
altered = true
for _, r := range append(slices.Clone(man.Requires), before.Requires...) {
provisions[r] = true
}
}
if !altered {
return nil
}
var out []string
for node, modules := range m.Assigned {
if slices.Contains(composed, node) {
continue
}
for _, mod := range modules {
man := m.Manifests[mod]
if man.ClaimsSeat("mesh-broker") || slices.ContainsFunc(man.Provides, func(o catalogue.Offer) bool { return provisions[o.Name] }) {
out = append(out, node)
break
}
}
}
sort.Strings(out)
return out
}
// withOverlay is base with every manifest of over laid over it, a copy.
func withOverlay(base, over map[string]catalogue.Manifest) map[string]catalogue.Manifest {
out := make(map[string]catalogue.Manifest, len(base)+len(over))
for name, man := range base {
out[name] = man
}
for name, man := range over {
out[name] = man
}
return out
}
func sortedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -1,169 +0,0 @@
package assign
import (
"encoding/json"
"slices"
"strconv"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func meshOf() Mesh {
return Mesh{
Manifests: map[string]catalogue.Manifest{
"mail": {Module: "mail", Version: "1", Requires: []string{"database"}},
"store": {Module: "store", Version: "1", Provides: []catalogue.Offer{{Name: "database"}}},
"web": {Module: "web", Version: "1"},
},
Assigned: map[string][]string{"ace": {"mail", "store"}, "g14": {"web"}},
}
}
func hasWarning(v delivery.CompositionVerdict, part string) bool {
return slices.ContainsFunc(v.Warnings, func(w string) bool { return strings.Contains(w, part) })
}
// 1297/AC1 (its unit half): a change touching one module on one machine composes that machine alone, passes, names
// the machine and the mesh-sdk it composed with, and says what it did not judge.
func TestAChangeOfOneModuleComposesItsMachine(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"web": {Module: "web", Version: "2"}}}
v := Compose(meshOf(), c, "v0.1.14-x", 0)
if v.State != delivery.CompositionPass || !slices.Equal(v.Machines, []string{"g14"}) || v.SDK != "v0.1.14-x" {
t.Fatalf("composition = %+v, want a pass on g14 alone", v)
}
if !hasWarning(v, "issue 1461") {
t.Errorf("the verdict does not say what it did not judge: %v", v.Warnings)
}
if _, err := delivery.EncodeCompositionVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err)
}
// 1953/D7 (ADR 0565 §5, ADR 0502 §4): the group's own pull request's held value at its head is laid and named, judged
// against the head's declaration; one of another pull request or another head is not laid.
web, err := catalogue.Decode([]byte(`{"module":"web","version":"2","config":{"port":{"type":"number","default":80,"description":"the port, a number","runtime":true}}}`))
if err != nil {
t.Fatal(err)
}
kept := map[string][]byte{}
for i, k := range []delivery.KeptValue{
{Pull: "novox/x#1", Head: "h1", Node: "g14", Module: "web", Key: "port", Value: json.RawMessage(`81`), Approved: "op"},
{Pull: "novox/x#2", Head: "h2", Node: "g14", Module: "web", Key: "port", Value: json.RawMessage(`82`)},
{Pull: "novox/x#1", Head: "old", Node: "g14", Module: "web", Key: "other", Value: json.RawMessage(`1`)},
} {
kept[strconv.Itoa(i)], _ = json.Marshal(k)
}
held := HeldOf(kept, map[string]string{"novox/x#1": "h1"})
if len(held) != 1 || string(held[0].Value) != "81" {
t.Fatalf("held = %+v, want #1's at its head alone", held)
}
c = Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"web": web}, Held: held}
if v := Compose(meshOf(), c, "x", 0); v.State != delivery.CompositionPass || !hasWarning(v, "held value laid: port of web on g14, held for novox/x#1") {
t.Errorf("composition = %+v", v)
}
c.Held[0].Value = json.RawMessage(`"eighty"`)
if v := Compose(meshOf(), c, "x", 0); v.State != delivery.CompositionFail || !strings.Contains(v.Words, "does not hold to the head's declaration") {
t.Errorf("a held value the declaration refuses: %+v", v)
}
}
// 1297/AC2: a change that removes a module a machine runs fails, naming the machine and the module.
func TestRemovingAModuleAMachineRunsFails(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Removed: []string{"web"}}
v := Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || v.Machine != "g14" || !strings.Contains(v.Words, "g14 runs web") {
t.Fatalf("composition = %+v, want a fail naming g14 and web", v)
}
}
// 1297/B3c: a machine that composes on the trunk and not with the change fails it, naming the machine and why; the
// consumer's machine is touched when its provider changes.
func TestAMachineTheChangeBreaksFails(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"store": {Module: "store", Version: "2"}}}
v := Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || v.Machine != "ace" || !strings.Contains(v.Words, "database") {
t.Fatalf("composition = %+v, want a fail on ace naming the provision nothing provides", v)
}
}
// 1297/B3c: what was already broken on the trunk is said and fails nothing.
func TestWhatWasBrokenBeforeFailsNothing(t *testing.T) {
m := meshOf()
m.Manifests["store"] = catalogue.Manifest{Module: "store", Version: "1"}
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"mail": {Module: "mail", Version: "2", Requires: []string{"database"}}}}
v := Compose(m, c, "sdk", 0)
if v.State != delivery.CompositionPass || !hasWarning(v, "on the trunk either") {
t.Fatalf("composition = %+v, want a pass saying ace was already broken", v)
}
}
// 1297/B3c: an unreadable manifest and a module defined twice fail; more than twelve rebuilt modules warn.
func TestUnreadableTwiceAndWide(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Unreadable: map[string]string{"web": "unknown field \"scope\""}, Rebuilt: 13}
v := Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "web cannot be read") || !hasWarning(v, "rebuilds 13 modules") {
t.Fatalf("composition = %+v", v)
}
c = Change{Group: "g", Heads: "0123456789ab", Twice: map[string][]string{"web": {"novox/a", "novox/b"}}, Rebuilt: 12}
v = Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "web is defined twice") || hasWarning(v, "rebuilds") {
t.Fatalf("composition = %+v", v)
}
}
// 1622's criterion as R4g delivers it: a group with a mesh-sdk member is composed with the assigner's own mesh-sdk, and
// the verdict says so, naming the member's head it did not use; a group without one says nothing of it.
func TestAGroupWithAnSDKMemberSaysWhichSDKComposedIt(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"web": {Module: "web", Version: "2"}}, SDKHead: "f625d98713e4"}
v := Compose(meshOf(), c, "v0.1.14-own", 0)
if !hasWarning(v, "not the group's mesh-sdk head f625d98713e4") || !hasWarning(v, "v0.1.14-own") || !hasWarning(v, "1623") {
t.Errorf("the verdict does not say it composed with its own mesh-sdk: %v", v.Warnings)
}
c.SDKHead = ""
if v := Compose(meshOf(), c, "v0.1.14-own", 0); hasWarning(v, "mesh-sdk head") {
t.Errorf("a group without a mesh-sdk member says it has one: %v", v.Warnings)
}
}
// 1297/B3a: the machines touched are those running a moved module or a consumer of what it provides; an added module
// assigned nowhere touches none, and the verdict says so.
func TestTouched(t *testing.T) {
m := meshOf()
if got := Touched(m, Change{Manifests: map[string]catalogue.Manifest{"store": m.Manifests["store"]}}); !slices.Equal(got, []string{"ace"}) {
t.Errorf("touched by store = %v", got)
}
m.Assigned["g14"] = append(m.Assigned["g14"], "mail")
if got := Touched(m, Change{Manifests: map[string]catalogue.Manifest{"store": m.Manifests["store"]}}); !slices.Equal(got, []string{"ace", "g14"}) {
t.Errorf("touched by store with a consumer on g14 = %v", got)
}
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"new": {Module: "new", Version: "1"}}}
if v := Compose(m, c, "sdk", 0); len(v.Machines) != 0 || !hasWarning(v, "touches no machine") || v.State != delivery.CompositionPass {
t.Errorf("an added module assigned nowhere: %+v", v)
}
}
// ADR 0537 clause 1: every verdict names, by machine, those that could run a moved or added module and are not assigned
// it, and those whose grants alone the change alters (the bus's machine, a provider of what the module requires), as not
// judged; shanks, providing what web now requires, is composed and so named in neither. A change that alters no grant
// names none of the second.
func TestTheMachinesNotJudgedAreNamed(t *testing.T) {
m := meshOf()
m.Manifests["bus"] = catalogue.Manifest{Module: "bus", Version: "1", Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: "mesh"}}}
m.Manifests["db"] = catalogue.Manifest{Module: "db", Version: "1", Provides: []catalogue.Offer{{Name: "cache"}}}
m.Assigned["novox"] = []string{"bus"}
m.Assigned["shanks"] = []string{"db"}
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{
"web": {Module: "web", Version: "2", Reads: []string{"mesh-controller.assignments"}, Requires: []string{"cache"}}}}
v := Compose(m, c, "sdk", 0)
if !hasWarning(v, "could run a moved or added module and are not assigned it: ace, novox") || !slices.Contains(v.Machines, "shanks") {
t.Errorf("the machines that could run web are not named: %v", v.Warnings)
}
if !hasWarning(v, "whose grants alone the change alters: novox") {
t.Errorf("the machines whose grants alone change are not named: %v", v.Warnings)
}
c.Manifests["web"] = catalogue.Manifest{Module: "web", Version: "3"}
if v := Compose(meshOf(), c, "sdk", 0); hasWarning(v, "grants alone") {
t.Errorf("a change altering no grant names machines whose grants change: %v", v.Warnings)
}
}
@@ -1,263 +0,0 @@
package assign
import (
"encoding/json"
"fmt"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The cases of a group's composition at its mesh-sdk head: the trunk as it is, and the trunk with the change.
const (
CaseTrunk = "trunk"
CaseChange = "change"
)
// SDKDir is the directory of mesh-sdk's Go module in its repository, where the composer is built at a head; a reader's
// composer is built at its repository's root, at the mesh-sdk its go.mod pins.
const SDKDir = "go"
// The modules whose mesh-sdk reads the manifests a plan sends (ADR 0482 clause 5): the controller composes and sends
// them, the node-engine applies them. Each is judged at its repository.
var ReaderModules = map[string]string{"mesh-controller": "novox/mesh-controller", "mesh-host": "novox/mesh-host"}
// HeadJob is the compose job of a group with a mesh-sdk member (ADR 0482 clauses 3 and 4, issue 1622): the composer
// built at the member's head composes the machines the change touches, as the trunk holds them and as the change leaves
// them, every manifest named by its digest in the indexer's bucket, the seats held from the claims (the assigner's input).
func HeadJob(m Mesh, c Change) (delivery.ComposeJob, error) {
trunk, change := map[string]composer.Manifest{}, map[string]composer.Manifest{}
for name, digest := range m.Digests {
trunk[name] = composer.Manifest{Digest: digest}
change[name] = composer.Manifest{Digest: digest}
}
for name, digest := range c.Digests {
change[name] = composer.Manifest{Digest: digest}
}
for _, name := range c.Removed {
delete(change, name)
}
in, fromClaims := plainInput(m.Assigned), true
if m.Input != nil {
// The input the nodes compose from, without its catalogue: the cases carry the manifests (issue 1126).
in = *m.Input
in.Catalogue = nil
fromClaims = len(in.Holdings) == 0
}
// Each trunk manifest the indexer refused goes in with its verdict, so the composer holds and names it (ADR 0578 §5);
// a manifest the change brings has no verdict of the indexer yet, and the checker's judge judges it (§2).
changeVerdicts := map[string]delivery.ManifestVerdict{}
for name, v := range m.Verdicts {
if _, changed := c.Digests[name]; !changed {
changeVerdicts[name] = v
}
}
changeCase := composer.Case{Name: CaseChange, Manifests: change, Verdicts: changeVerdicts}
req := composer.Request{V: composer.Version, Input: in, Depth: composer.DepthResolve,
HoldingsFromClaims: fromClaims, Machines: Touched(m, c),
Cases: []composer.Case{{Name: CaseTrunk, Manifests: trunk, Verdicts: m.Verdicts}, changeCase}}
return job(SDKRepository, c.SDKHead, SDKDir, req, "the mesh-sdk head of "+c.Group)
}
// ReaderJob is the compose job that reads the manifests a group changes with the mesh-sdk a reader module pins at
// commit (ADR 0482 clause 5, issue 1623): no machine is composed, every changed manifest is only read.
func ReaderJob(c Change, module, repository, commit string) (delivery.ComposeJob, error) {
changed := map[string]composer.Manifest{}
for name, digest := range c.Digests {
changed[name] = composer.Manifest{Digest: digest}
}
req := composer.Request{V: composer.Version, Cases: []composer.Case{{Name: CaseChange, Manifests: changed}}}
return job(repository, commit, delivery.RootPath, req, module+" at "+commit)
}
func job(repository, commit, dir string, req composer.Request, reader string) (delivery.ComposeJob, error) {
raw, err := json.Marshal(req)
if err != nil {
return delivery.ComposeJob{}, err
}
return delivery.ComposeJob{Fetch: map[string]any{"repository": repository, "commit": commit}, Dir: dir,
Package: composer.Package, Request: raw, Reader: reader}, nil
}
// plainInput is the composition's input without its catalogue: the nodes by name with what is assigned to them (the
// composer derives the seats held from the claims, as input does).
func plainInput(assigned map[string][]string) compose.Input {
var in compose.Input
nodes := make([]string, 0, len(assigned))
for n := range assigned {
nodes = append(nodes, n)
}
sort.Strings(nodes)
for _, n := range nodes {
in.Nodes = append(in.Nodes, compose.Node{Name: n, Assigned: slices.Clone(assigned[n])})
}
return in
}
// Ran is one compose ask's end: its step's state and reason, and the composer's verdict when the step passed.
type Ran struct {
State string
Reason string
Verdict *composer.Verdict
}
// RanOf reads a compose ask's end from its step value; a step that passed with an answer that cannot be read is a
// failure, named.
func RanOf(step delivery.Step) Ran {
r := Ran{State: step.State, Reason: step.Reason}
if step.State != delivery.StepPassed {
return r
}
var a delivery.ComposeAnswer
var v composer.Verdict
if err := json.Unmarshal(step.Result, &a); err != nil {
return Ran{State: delivery.StepFailed, Reason: "the compose step's answer cannot be read: " + err.Error()}
}
if err := json.Unmarshal(a.Verdict, &v); err != nil || v.V != composer.Version {
return Ran{State: delivery.StepFailed, Reason: fmt.Sprintf("the composer's verdict cannot be read (version %d): %v", v.V, err)}
}
r.Verdict = &v
return r
}
func caseOf(v *composer.Verdict, name string) (composer.CaseVerdict, bool) {
if v == nil {
return composer.CaseVerdict{}, false
}
for _, c := range v.Cases {
if c.Name == name {
return c, true
}
}
return composer.CaseVerdict{}, false
}
// Reader is one module whose mesh-sdk reads the group's manifests, at the commit judged: the group's member head when
// the group holds one in its repository, else a build its node runs.
type Reader struct {
Module, Commit string
Member bool
Ran Ran
}
// FromComposer is the composition check of a group with a mesh-sdk member, judged from the composer built at its head
// (ADR 0482 clauses 3 to 5, issues 1622 and 1623): the same failures as Compose, read from the composer's two cases, the
// verdict naming the mesh-sdk the composer was built with; a trunk manifest the head's mesh-sdk cannot decode fails too
// (the group would deliver a reader that cannot read it). A reader that cannot read a changed manifest is a warning,
// never a failure: it decodes leniently and sets a field it does not know aside (novox/hq ADR 0578 §6, which changes
// ADR 0482 §5).
func FromComposer(m Mesh, c Change, head Ran, readers []Reader, took time.Duration) delivery.CompositionVerdict {
v := delivery.CompositionVerdict{Group: c.Group, Heads: c.Heads, SDK: "mesh-sdk " + c.SDKHead, Took: took.Milliseconds()}
machines := Touched(m, c)
v.Machines = machines
var fails []string
if head.State != delivery.StepPassed {
v.Words = fmt.Sprintf("the composer at mesh-sdk %s did not run: %s %s", c.SDKHead, head.State, head.Reason)
v.State = delivery.CompositionFail
return v
}
if head.Verdict.SDK != "" {
v.SDK = head.Verdict.SDK
}
trunk, okT := caseOf(head.Verdict, CaseTrunk)
change, okC := caseOf(head.Verdict, CaseChange)
if !okT || !okC {
v.State, v.Words = delivery.CompositionFail, "the composer's verdict holds no trunk or no change case"
return v
}
failAt := func(machine, words string) {
if v.Machine == "" && machine != "" {
v.Machine = machine
}
fails = append(fails, words)
}
for _, name := range sortedKeys(c.Digests) {
if r := change.Manifests[name]; !r.Accepted {
failAt("", fmt.Sprintf("the manifest of %s cannot be read by mesh-sdk %s: %s", name, c.SDKHead, r.Refused))
}
}
for _, name := range sortedKeys(trunk.Manifests) {
if _, changed := c.Digests[name]; changed || slices.Contains(c.Removed, name) || trunk.Manifests[name].Accepted {
continue
}
if _, before := m.Unreadable[name]; before {
continue // already unreadable on the trunk: not the group's doing
}
failAt("", fmt.Sprintf("mesh-sdk %s refuses the trunk's manifest of %s, which the group does not change: %s", c.SDKHead, name, trunk.Manifests[name].Refused))
}
for _, name := range sortedKeys(c.Twice) {
failAt("", fmt.Sprintf("%s is defined twice, in %s", name, strings.Join(c.Twice[name], " and ")))
}
for _, node := range machines {
for _, name := range c.Removed {
if slices.Contains(m.Assigned[node], name) {
failAt(node, fmt.Sprintf("%s runs %s, which the change removes from its source", node, name))
}
}
before, now := trunk.Machines[node], change.Machines[node]
switch {
case now.Composes:
case !before.Composes:
v.Warnings = append(v.Warnings, fmt.Sprintf("%s does not compose on the trunk either, so the change is not blamed: %s", node, before.Reason))
default:
failAt(node, fmt.Sprintf("%s composes on the trunk and not with the change: %s", node, now.Reason))
}
}
readerFails, readerWarnings := judgeReaders(c, change, readers)
fails = append(fails, readerFails...)
v.Warnings = append(v.Warnings, readerWarnings...)
if m.InputUnread != "" {
v.Warnings = append(v.Warnings, "composed over the nodes and their assignments alone: the input the nodes compose from "+
"cannot be read from the bus ("+m.InputUnread+")")
}
return finish(v, m, c, machines, fails)
}
// NoComposer says whether a compose step failed because the mesh-sdk it was built at has no composer command (a mesh-sdk
// from before issue 1621): Go's words for a package its module does not hold.
func NoComposer(reason string) bool {
for _, words := range []string{"cannot find package", "no required module provides package", "is not in std", "cannot find module providing package", "does not contain package"} {
if strings.Contains(reason, words) {
return true
}
}
return false
}
func judgeReaders(c Change, head composer.CaseVerdict, readers []Reader) (fails, warnings []string) {
for _, r := range readers {
at := r.Module + " at " + r.Commit
if r.Member {
at = r.Module + "'s member head " + r.Commit
}
if r.Ran.State != delivery.StepPassed {
if NoComposer(r.Ran.Reason) {
warnings = append(warnings, fmt.Sprintf("not judged whether %s reads the group's manifests: the mesh-sdk it pins has no "+
"composer command (before issue 1621)", at))
continue
}
fails = append(fails, fmt.Sprintf("the composer of the mesh-sdk %s pins did not run: %s %s", at, r.Ran.State, r.Ran.Reason))
continue
}
read, ok := caseOf(r.Ran.Verdict, CaseChange)
if !ok {
fails = append(fails, fmt.Sprintf("the composer of the mesh-sdk %s pins answered no change case", at))
continue
}
for _, name := range sortedKeys(c.Digests) {
if !head.Manifests[name].Accepted || read.Manifests[name].Accepted {
continue
}
// ADR 0578 §6: no longer a failure; the reader sets what it does not know aside, and the part that needs it takes
// the newer package (ADR 0573 §2).
warnings = append(warnings, fmt.Sprintf("the manifest of %s uses what %s (mesh-sdk %s) cannot read: %s; it reads the rest "+
"once it takes mesh-sdk %s", name, at, r.Ran.Verdict.SDK, read.Manifests[name].Refused, c.SDKHead))
}
}
return fails, warnings
}
@@ -1,225 +0,0 @@
package assign
import (
"encoding/json"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
var testTime = time.Date(2026, 10, 24, 3, 0, 0, 0, time.UTC)
const sdkHead = "1111111111111111111111111111111111111111"
// raws are the manifests of meshOf as their repositories hold them, by digest, as the indexer keeps them; `web2` is web
// with a field `shares` that the assigner's own mesh-sdk does not read (the shape of issue 1593).
var raws = map[string]string{
"d-mail": `{"module": "mail", "version": "1", "requires": ["postgres-database"]}`,
"d-store": `{"module": "store", "version": "1", "provides": [{"name": "postgres-database"}]}`,
"d-web": `{"module": "web", "version": "1"}`,
"d-web2": `{"module": "web", "version": "2", "shares": ["photos"]}`,
}
// asTheHeadReads is how mesh-sdk at the group's head reads each manifest: as the assigner's own reads it, but for the
// field it adds, which it accepts. The head's composer is built and run by the Go build module (ADR 0482 clause 4); here
// its verdict is made by mesh-sdk's composer.Run over the same request, with `web2` read as the head reads it.
var asTheHeadReads = map[string]string{"d-web2": `{"module": "web", "version": "2"}`}
func digestedMesh() Mesh {
m := meshOf()
m.Digests = map[string]string{"mail": "d-mail", "store": "d-store", "web": "d-web"}
return m
}
// webShares is a group of mesh-sdk and mesh-catalog whose catalogue member gives web the field only the SDK head reads:
// the assigner's own reader refuses it (Unreadable), as the indexer's manifest is read in world.change.
func webShares(members map[string]string) Change {
// As an older reader refused it; the field set aside by the lenient decoder is named the same (ADR 0578 §6).
m, _ := catalogue.Decode([]byte(raws["d-web2"]))
return Change{Group: "g", Heads: "0123456789ab", SDKHead: sdkHead, Manifests: map[string]catalogue.Manifest{},
Unreadable: map[string]string{"web": m.SetAside()}, Digests: map[string]string{"web": "d-web2"}, Members: members}
}
// run is the composer's step for a job, as the build module answers it: the request filled from the indexer's digests
// (with read standing in for how that mesh-sdk reads a digest), run, and its verdict put in a passed step.
func run(t *testing.T, job delivery.ComposeJob, read map[string]string) Ran {
t.Helper()
req, err := composer.Decode(job.Request)
if err != nil {
t.Fatal(err)
}
if err := composer.Fill(&req, func(d string) ([]byte, error) {
if r, ok := read[d]; ok {
return []byte(r), nil
}
return []byte(raws[d]), nil
}); err != nil {
t.Fatal(err)
}
v, err := composer.Run(req)
if err != nil {
t.Fatal(err)
}
v.SDK = "(devel) " + job.Fetch["commit"].(string)
vr, _ := json.Marshal(v)
ans, _ := json.Marshal(delivery.ComposeAnswer{Built: true, Key: "composer-0", Verdict: vr})
return RanOf(delivery.Step{State: delivery.StepPassed, Result: ans})
}
// 1622/AC1: a group holding a mesh-sdk pull request is composed by the composer built at the member's head: the job
// fetches mesh-sdk at that head, builds mesh-sdk's composer command in its Go module, and hands it the trunk and the
// change by digest, never the manifests' bytes, composing the machines the change touches.
func TestAGroupWithAnSDKMemberAsksTheComposerAtItsHead(t *testing.T) {
c := webShares(nil)
job, err := HeadJob(digestedMesh(), c)
if err != nil {
t.Fatal(err)
}
if job.Fetch["repository"] != SDKRepository || job.Fetch["commit"] != sdkHead || job.Dir != "go" || job.Package != composer.Package {
t.Fatalf("job = %+v", job)
}
ask := delivery.StepAsk{ID: "mh3k2x9qa1b2c5", Kind: delivery.AskCheck, Language: delivery.AskGo, Asker: "mesh-delivery-assigner",
At: testTime, Compose: &job}
if _, err := delivery.EncodeStepAsk(ask); err != nil {
t.Fatalf("the ask is refused by its layout: %v", err)
}
req, err := composer.Decode(job.Request)
if err != nil {
t.Fatal(err)
}
if len(req.Cases) != 2 || req.Cases[0].Name != CaseTrunk || req.Cases[1].Name != CaseChange {
t.Fatalf("cases = %+v", req.Cases)
}
if req.Cases[0].Manifests["web"].Digest != "d-web" || req.Cases[1].Manifests["web"].Digest != "d-web2" || len(req.Cases[1].Manifests["web"].Raw) != 0 {
t.Errorf("the cases do not name web by its trunk and its change digest: %+v", req.Cases)
}
if !slices.Equal(req.Machines, []string{"g14"}) || req.Depth != composer.DepthResolve || !req.HoldingsFromClaims {
t.Errorf("request = machines %v, depth %q, holdings from claims %v", req.Machines, req.Depth, req.HoldingsFromClaims)
}
// 1126 (ADR 0547 §2, ADR 0549 §3): once the catalogue places the nodes, the check composes over the input the nodes
// compose from (frombus), its catalogue left to the cases, and the seats held as placement says.
m := digestedMesh()
m.Input = &compose.Input{Catalogue: map[string]catalogue.Manifest{"web": {Module: "web"}},
Nodes: []compose.Node{{Name: "g14", Assigned: []string{"web"}, Place: compose.Place{Endpoint: "g14.example:51820"}}},
Holdings: []catalogue.Held{{Claim: "web-seat", Scope: "mesh", Node: "g14", Module: "web"}}}
placedJob, err := HeadJob(m, c)
if err != nil {
t.Fatal(err)
}
placed, err := composer.Decode(placedJob.Request)
if err != nil {
t.Fatal(err)
}
if placed.Input.Catalogue != nil || len(placed.Input.Nodes) != 1 || placed.Input.Nodes[0].Place.Endpoint != "g14.example:51820" ||
len(placed.Input.Holdings) != 1 || placed.HoldingsFromClaims {
t.Errorf("the placed check's request = %+v, holdings from claims %v", placed.Input, placed.HoldingsFromClaims)
}
}
// 1622/AC2, AC3: a catalogue member using a field only the SDK head reads passes with the head's composer, and the
// verdict names the mesh-sdk the composer was built with; with the assigner's own mesh-sdk it fails.
func TestAFieldTheGroupAddsPassesWithTheHeadsComposer(t *testing.T) {
c := webShares(map[string]string{"novox/mesh-controller": "2222222222222222222222222222222222222222"})
job, _ := HeadJob(digestedMesh(), c)
head := run(t, job, asTheHeadReads)
v := FromComposer(digestedMesh(), c, head, nil, 0)
if v.State != delivery.CompositionPass || v.SDK != "(devel) "+sdkHead || !slices.Equal(v.Machines, []string{"g14"}) {
t.Fatalf("with the head's composer: %+v", v)
}
if _, err := delivery.EncodeCompositionVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err)
}
own := Compose(digestedMesh(), c, "v0.1.14-own", 0)
if own.State != delivery.CompositionFail || !strings.Contains(own.Words, "shares") {
t.Fatalf("with the assigner's own mesh-sdk: %+v, want a fail naming the field", own)
}
}
// 1623/AC1, AC2 as ADR 0578 §6 changes them: the group of mesh-sdk and mesh-catalog adding and using a field, without a
// controller member, passes with a warning naming the field and mesh-controller, whose running build pins an older
// mesh-sdk that cannot read it: a reader sets a field it does not know aside, so it fails nothing. With a controller
// member pinned to the SDK head it passes with no such warning.
func TestAFieldTheRunningControllerCannotReadIsAWarning(t *testing.T) {
c := webShares(nil)
job, _ := HeadJob(digestedMesh(), c)
head := run(t, job, asTheHeadReads)
running := "3333333333333333333333333333333333333333"
rjob, err := ReaderJob(c, "mesh-controller", "novox/mesh-controller", running)
if err != nil {
t.Fatal(err)
}
if rjob.Fetch["commit"] != running || rjob.Dir != delivery.RootPath {
t.Fatalf("reader job = %+v", rjob)
}
// The running controller's mesh-sdk, older than the lenient decoder, refuses web2 for its field.
old := Ran{State: delivery.StepPassed, Verdict: &composer.Verdict{V: composer.Version, SDK: "v0.1.13", Cases: []composer.CaseVerdict{
{Name: CaseChange, Manifests: map[string]composer.ManifestRead{"web": {Refused: `json: unknown field "shares"`}}}}}}
v := FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-controller", Commit: running, Ran: old}}, 0)
if v.State != delivery.CompositionPass || !hasWarning(v, "shares") || !hasWarning(v, "mesh-controller at "+running) {
t.Fatalf("without a controller member: %+v, want a pass warning of the field and mesh-controller", v)
}
member := "2222222222222222222222222222222222222222"
pinned := run(t, rjob, asTheHeadReads)
v = FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-controller", Commit: member, Member: true, Ran: pinned}}, 0)
if v.State != delivery.CompositionPass {
t.Fatalf("with a controller member pinned to the head: %+v, want a pass", v)
}
}
// 1623: a reader whose mesh-sdk has no composer command (before issue 1621) is said, not failed; a reader's composer that
// failed for another cause fails the group, loud.
func TestAReaderWithNoComposerIsSaidAndAnotherFailureFails(t *testing.T) {
c := webShares(nil)
job, _ := HeadJob(digestedMesh(), c)
head := run(t, job, asTheHeadReads)
none := Ran{State: delivery.StepFailed, Reason: "go build: no required module provides package git.novox.be/novox/mesh-sdk/go/cmd/mesh-compose"}
v := FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-host", Commit: "abc", Ran: none}}, 0)
if v.State != delivery.CompositionPass || !hasWarning(v, "mesh-host at abc") || !hasWarning(v, "before issue 1621") {
t.Fatalf("a reader with no composer: %+v", v)
}
broke := Ran{State: delivery.StepFailed, Reason: "the fetch could not reach the forge"}
v = FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-host", Commit: "abc", Ran: broke}}, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "could not reach the forge") {
t.Fatalf("a reader whose composer failed: %+v", v)
}
}
// 1622: the head's composer that did not run fails the composition naming why, never passes it; a trunk manifest the
// head's mesh-sdk refuses fails it too, since the group would deliver a reader that refuses it.
func TestTheHeadsComposerFailingOrRefusingTheTrunkFails(t *testing.T) {
c := webShares(nil)
v := FromComposer(digestedMesh(), c, Ran{State: delivery.StepRefused, Reason: "the guard refuses it"}, nil, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "the guard refuses it") || !strings.Contains(v.Words, sdkHead) {
t.Fatalf("a head composer that did not run: %+v", v)
}
job, _ := HeadJob(digestedMesh(), c)
strict := map[string]string{"d-web2": asTheHeadReads["d-web2"], "d-mail": `{"module": "mail", "version": "1", "requires": "x"}`}
v = FromComposer(digestedMesh(), c, run(t, job, strict), nil, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "refuses the trunk's manifest of mail") {
t.Fatalf("a head that refuses a trunk manifest: %+v", v)
}
// ADR 0578 §5: a trunk manifest the indexer refused goes into the composer's input with its verdict, in both cases;
// the composer holds and names it, and the group is not blamed for it.
m := digestedMesh()
m.Verdicts = map[string]delivery.ManifestVerdict{"mail": {Module: "mail", State: delivery.VerdictRefused, Problems: []string{"it is wrong"}}}
m.Unreadable = map[string]string{"mail": m.Verdicts["mail"].Why()}
job, _ = HeadJob(m, c)
var req composer.Request
if err := json.Unmarshal(job.Request, &req); err != nil || !req.Cases[0].Verdicts["mail"].Refused() || !req.Cases[1].Verdicts["mail"].Refused() {
t.Fatalf("the request carries no verdict: %+v %v", req.Cases, err)
}
head := run(t, job, asTheHeadReads)
if trunk, _ := caseOf(head.Verdict, CaseTrunk); !strings.Contains(trunk.Manifests["mail"].Refused, "refused the manifest of mail: it is wrong") {
t.Fatalf("the composer did not name the refusal: %+v", trunk.Manifests["mail"])
}
if v = FromComposer(m, c, head, nil, 0); v.State != delivery.CompositionPass {
t.Fatalf("a refusal on the trunk blamed the group: %+v", v)
}
}
@@ -6,7 +6,8 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
@@ -24,7 +25,7 @@ var FindingsBucket = health.FindingsBucket(delivery.AssignerModule)
// FindingOf is a said condition as a finding: its key `<scope>.<id>.<kind>`, from the condition's own key
// `mesh-delivery-assigner.<kind>.<id…>` (the subject's id when the key names none), its words unchanged. raised is false for
// a clear, whose key is to be deleted; value is nil then.
func FindingOf(c asks.Condition, at time.Time) (key string, value []byte, raised bool, err error) {
func FindingOf(c oc.Condition, at time.Time) (key string, value []byte, raised bool, err error) {
rest, ok := strings.CutPrefix(c.Key, delivery.AssignerModule+".")
if !ok || rest == "" {
return "", nil, false, fmt.Errorf("the condition %q is not the assigner's", c.Key)
@@ -38,10 +39,10 @@ func FindingOf(c asks.Condition, at time.Time) (key string, value []byte, raised
scope = health.ScopeModule
}
key = health.FindingKey(scope, id, kind)
if c.State == asks.StateCleared {
if c.State == occ.StateCleared {
return key, nil, false, nil
}
if c.State != asks.StateRaised {
if c.State != occ.StateRaised {
return "", nil, false, fmt.Errorf("the condition %s is neither raised nor cleared", c.Key)
}
summary := c.Summary
@@ -59,20 +60,19 @@ func FindingOf(c asks.Condition, at time.Time) (key string, value []byte, raised
return key, value, true, err
}
// Findings is where the assigner's findings go: the bucket's put and delete, and whether the keeper's conditions are the
// ones said (the take-over key names the keeper).
// Findings is where the assigner's findings go: the bucket's put and delete.
type Findings struct {
Put func(key string, value []byte) error
Delete func(key string) error
Keys func() ([]string, error)
ToKeeper func() bool
Now func() time.Time
Put func(key string, value []byte) error
Delete func(key string) error
Keys func() ([]string, error)
Now func() time.Time
}
// Say is say with each condition also written as a finding (or its finding deleted), and said straight to the router only
// while the take-over key follows the controller. A finding that cannot be written is the error, beside the say's.
func (f Findings) Say(say func(asks.Condition) error) func(asks.Condition) error {
return func(c asks.Condition) error {
// Say is say with each condition written as a finding (or its finding deleted), which the condition keeper carries, the
// one source since the controller's end (novox/hq ADR 0590 §3); said straight to the router only where there is no
// findings bucket at all. A finding that cannot be written is the error.
func (f Findings) Say(say func(oc.Condition) error) func(oc.Condition) error {
return func(c oc.Condition) error {
var errs []error
if f.Put != nil {
now := time.Now
@@ -93,7 +93,7 @@ func (f Findings) Say(say func(asks.Condition) error) func(asks.Condition) error
}
}
}
if f.ToKeeper == nil || !f.ToKeeper() {
if f.Put == nil {
if say != nil {
if err := say(c); err != nil {
errs = append(errs, err)
@@ -1,132 +1,23 @@
package assign
import (
"fmt"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/asks"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Until ADR 0547's third milestone two sources name which module runs on which node: the controller's assignments and
// the seat mesh-placement's placement (ADR 0549 §5, ADR 0567 §8). The assigner writes wants only for a pair both name:
// for a pair one source names alone it writes no want and touches none it wrote before, so nothing moves and nothing is
// removed on one source's word, and it raises a condition naming the pair until both agree. After an approved proposal
// the assigner keeps the controller in step itself, calling its assign, unassign, pin and unpin; a call refused or not
// yet answered leaves the pair differing, and the condition names the call (Pair.Call). Before the first placement key
// is written the controller's assignments are the one source, as before. From the lease write (the controller's lease
// handed away) the placement is the one source.
// Pair is a module on a node that one source names and the other does not; Only says which: OnlyAssignments or
// OnlyPlacement.
type Pair struct {
Node, Module, Only string
// Call is what became of the assigner's call of the controller's verb for the pair, in words, when it made one.
Call string
}
// The source that names a pair alone.
const (
OnlyAssignments = "the controller's assignments"
OnlyPlacement = "the seat mesh-placement's placement"
)
// Placed is, by node, the modules both sources name, in name order, and every pair one names alone, by node then
// module. With no placement read yet (seeded false) it is the assignments, and no pair differs. Once the controller's
// lease is handed away (handedAway, ADR 0549 §5, §6) it is the placement alone, in name order, and no pair differs: the
// assignments are then a source of nothing.
func Placed(assigned, placed map[string][]string, seeded, handedAway bool) (map[string][]string, []Pair) {
if handedAway && seeded {
out := make(map[string][]string, len(placed))
for node, modules := range placed {
out[node] = slices.Sorted(slices.Values(modules))
}
return out, nil
}
if !seeded {
return assigned, nil
}
set := func(m map[string][]string) map[string]bool {
out := map[string]bool{}
for node, modules := range m {
for _, module := range modules {
out[node+"\x00"+module] = true
}
}
return out
}
inAssigned, inPlaced := set(assigned), set(placed)
both := map[string][]string{}
var differ []Pair
for node, modules := range assigned {
for _, module := range modules {
if inPlaced[node+"\x00"+module] {
both[node] = append(both[node], module)
} else {
differ = append(differ, Pair{Node: node, Module: module, Only: OnlyAssignments})
}
}
}
for node, modules := range placed {
for _, module := range modules {
if !inAssigned[node+"\x00"+module] {
differ = append(differ, Pair{Node: node, Module: module, Only: OnlyPlacement})
}
}
}
for _, modules := range both {
sort.Strings(modules)
}
sort.Slice(differ, func(i, j int) bool {
if differ[i].Node != differ[j].Node {
return differ[i].Node < differ[j].Node
}
return differ[i].Module < differ[j].Module
})
return both, differ
}
// PlacementCondition is the condition of a pair the two sources do not both name: raised while they differ, cleared once
// both name it or neither does.
func PlacementCondition(p Pair, raised bool) asks.Condition {
key := delivery.AssignerModule + ".placement." + p.Node + "." + p.Module
subject := asks.Scope{Scope: "module", ID: p.Module, Machine: p.Node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
Resolved: fmt.Sprintf("the controller's assignments and the seat's placement agree on %s on %s.", p.Module, p.Node)}
}
headline := fmt.Sprintf("%s on %s is placed by one source only", p.Module, p.Node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
var needs string
if p.Only == OnlyAssignments {
needs = fmt.Sprintf("propose placing %s on %s (mesh-placement.propose), or unassign it at the controller", p.Module, p.Node)
} else {
needs = fmt.Sprintf("assign %s to %s at the controller, as the approved placement says, or propose taking it off", p.Module, p.Node)
}
if p.Call != "" {
needs += "; the assigner's call: " + p.Call
}
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: fmt.Sprintf("%s%s. %s names %s on %s and the other source does not. Until ADR 0547's third milestone both must "+
"name a module on a node (ADR 0549 §5): the assigner writes no want for it there and leaves the one it wrote before, so "+
"nothing moves or is removed. This clears once both name it or neither does.", asks.NeedsYou, needs, p.Only, p.Module, p.Node)}
}
// SeatWaitsCondition is the assigner's word on the seat mesh-placement while a state of the seat is not bound (ADR 0567
// §1): raised naming each, cleared when none is left.
func SeatWaitsCondition(unbound []string) asks.Condition {
func SeatWaitsCondition(unbound []string) oc.Condition {
key := delivery.AssignerModule + ".seat-states"
subject := asks.Scope{Scope: "module", ID: delivery.AssignerModule}
subject := oc.ConditionSubject{Scope: "module", ID: delivery.AssignerModule}
if len(unbound) == 0 {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared, Resolved: "the seat mesh-placement holds all its states."}
return oc.Condition{Key: key, Subject: subject, State: occ.StateCleared, Resolved: "the seat mesh-placement holds all its states."}
}
why := "the seat waits for its states: " + strings.Join(unbound, ", ") + " are not on the bus or not granted yet"
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
return oc.Condition{Key: key, Severity: occ.SeverityWarning, Subject: subject, State: occ.StateRaised,
Summary: "the seat waits for its states", Headline: "the seat waits for its states",
Explanation: asks.NothingToDo + " " + why + "; each is bound at the next pass once it is, with no restart."}
Explanation: occ.NothingToDo + " " + why + "; each is bound at the next pass once it is, with no restart."}
}
@@ -7,7 +7,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
@@ -16,9 +16,7 @@ import (
// A person's upgrade policy is the assigner's (novox/hq ADR 0565 §7, §8, issue 1953 D4, D5): kept in its state `policies`, one
// key per module, set and taken back through `mesh-wants.upgrade`. Every reader takes one derivation, mesh-sdk's
// catalogue.Upgrade: a person's choice over the module's word, its irreplaceable data and the default, the bus never rolled.
// Until the lease write the controller still writes a policy beside each composition's digest; where it differs from the
// assigner's, the assigner says so as its finding `policy-split`, naming mesh-wants.upgrade as the remedy.
// catalogue.Upgrade: a person's choice over the module's word, its irreplaceable data and the default, the bus as any module (ADR 0601 §1).
// PoliciesBucket is the assigner's state `policies`.
var PoliciesBucket = delivery.AssignerModule + "_" + delivery.AssignerPoliciesState
@@ -48,26 +46,6 @@ func PolicyOf(m catalogue.Manifest, chosen map[string]catalogue.PersonPolicy) (p
return catalogue.Upgrade(m, c)
}
// ControllerPolicies is, by module, every policy the controller writes beside a composition's digest of it.
func ControllerPolicies(composed map[string][]byte) map[string][]string {
out := map[string][]string{}
for key, raw := range composed {
_, module, _, isNode, err := delivery.ParseComposedKey(key)
if err != nil || isNode {
continue
}
d, err := delivery.DecodeComposedDigest(raw)
if err != nil || d.Policy == "" {
continue
}
if !contains(out[module], d.Policy) {
out[module] = append(out[module], d.Policy)
sort.Strings(out[module])
}
}
return out
}
func contains(list []string, s string) bool {
for _, v := range list {
if v == s {
@@ -77,64 +55,6 @@ func contains(list []string, s string) bool {
return false
}
// Split is a module whose policy at the assigner differs from one the controller writes beside a composition of it.
type Split struct {
Module string `json:"module"`
Policy string `json:"policy"`
From string `json:"from"`
Controller []string `json:"controller"`
}
// Splits is every module of manifests the controller writes a policy for that differs from the assigner's, sorted.
func Splits(manifests map[string]catalogue.Manifest, chosen map[string]catalogue.PersonPolicy, controller map[string][]string) []Split {
var out []Split
for module, theirs := range controller {
m, ok := manifests[module]
if !ok {
continue
}
policy, from, _, _ := PolicyOf(m, chosen)
for _, p := range theirs {
if p != policy {
out = append(out, Split{Module: module, Policy: policy, From: from, Controller: theirs})
break
}
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Module < out[j].Module })
return out
}
// KindPolicySplit is the assigner's finding of a split policy (ADR 0565 §8).
const KindPolicySplit = "policy-split"
// PolicySplitKey is a module's policy-split finding's key.
func PolicySplitKey(module string) string {
return health.FindingKey(health.ScopeModule, module, KindPolicySplit)
}
// PolicySplitFinding is the finding of one split, in mesh-sdk's layout.
func PolicySplitFinding(s Split, at time.Time) ([]byte, error) {
summary := fmt.Sprintf("%s's upgrade policy is %s at the assigner (%s) and %s at the controller", s.Module, s.Policy, s.From,
strings.Join(s.Controller, " and "))
return health.EncodeFinding(health.Finding{Key: PolicySplitKey(s.Module), Kind: KindPolicySplit,
Subject: health.Subject{Scope: health.ScopeModule, ID: s.Module}, Severity: health.Warning, Summary: summary,
Headline: clipHeadline(s.Module + "'s upgrade policy is split"),
Needs: "set it with mesh-wants.upgrade, so the assigner and the controller agree",
Explanation: asks.NeedsYou + "set it with mesh-wants.upgrade, so the assigner and the controller agree." + " The controller still " +
"writes a policy beside each composition until it hands its lease away; a choice made at its upgrade verb after the copy " +
"is not the assigner's.",
Resolved: s.Module + "'s upgrade policy agrees again", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindPolicySplit, Observer: delivery.AssignerModule, At: at.UTC()})
}
func clipHeadline(s string) string {
if len([]rune(s)) <= 60 {
return s
}
return string([]rune(s)[:60])
}
// KindProvisionFailed is the assigner's finding of a provision its module's want waits for that failed (ADR 0570 §4).
const KindProvisionFailed = "provision-failed"
@@ -150,7 +70,7 @@ func ProvisionFailedFinding(key string, p provisions.Provision, at time.Time) ([
return health.EncodeFinding(health.Finding{Key: ProvisionFailedKey(key), Kind: KindProvisionFailed,
Subject: health.Subject{Scope: health.ScopeProvider, ID: key, Machine: p.Consumer.Node, Also: also(p.Provider.Node, p.Consumer.Node)},
Severity: health.Warning, Summary: summary, Headline: clipHeadline(p.Consumer.Module + " waits for a provision that failed"),
Explanation: asks.NothingToDo + " " + summary + ". It clears once the provider provides it.",
Explanation: occ.NothingToDo + " " + summary + ". It clears once the provider provides it.",
Resolved: p.Provision + " for " + p.Consumer.Module + " is provided", Evidence: []health.Evidence{{At: at.UTC(), Said: p.Why}},
Source: "assigner." + KindProvisionFailed, Observer: delivery.AssignerModule, At: at.UTC()})
}
@@ -182,7 +102,7 @@ func ProvisionMissingFinding(key string, bound time.Duration, at time.Time) ([]b
return health.EncodeFinding(health.Finding{Key: ProvisionMissingKey(key), Kind: KindProvisionMissing,
Subject: health.Subject{Scope: health.ScopeProvider, ID: key, Machine: node},
Severity: health.Warning, Summary: summary, Headline: clipHeadline(module + " on " + node + " lacks " + provision),
Explanation: asks.NothingToDo + " " + summary + ". It clears once the provision has its key.",
Explanation: occ.NothingToDo + " " + summary + ". It clears once the provision has its key.",
Resolved: provision + " for " + module + " on " + node + " has its key", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindProvisionMissing, Observer: delivery.AssignerModule, At: at.UTC()})
}
@@ -202,7 +122,14 @@ func UnplacedFinding(node string, at time.Time) ([]byte, error) {
return health.EncodeFinding(health.Finding{Key: UnplacedKey(node), Kind: KindUnplaced,
Subject: health.Subject{Scope: health.ScopeMachine, ID: node, Machine: node}, Severity: health.Warning, Summary: summary,
Headline: clipHeadline(node + " has no placement"), Needs: "place it with mesh-placement.propose.",
Explanation: asks.NeedsYou + "place it with mesh-placement.propose. " + summary + ".",
Explanation: occ.NeedsYou + "place it with mesh-placement.propose. " + summary + ".",
Resolved: node + " is placed", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindUnplaced, Observer: delivery.AssignerModule, At: at.UTC()})
}
func clipHeadline(s string) string {
if len([]rune(s)) <= 60 {
return s
}
return string([]rune(s)[:60])
}
@@ -4,7 +4,8 @@ import (
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -68,7 +69,7 @@ func StepPut(w PutWorld) PutNext {
}
fp := short(p.FP)
if !w.Assigned {
return say(delivery.PutRefused, "%s is not assigned to %s: a put moves a module where it stands assigned (mesh-controller.assign first)", w.Module, w.Node)
return say(delivery.PutRefused, "%s is not assigned to %s: a put moves a module where it stands assigned (propose its placement first)", w.Module, w.Node)
}
switch w.Build {
case delivery.BuildDone:
@@ -210,15 +211,15 @@ func PutAsking(p delivery.Put) int {
}
// PutAct is the act a bus put's yes binds and that is performed when its want is written.
func PutAct(node, module, fp, composed string) asks.Act {
return asks.Act{"put": "want", "node": node, "module": module, "fp": fp, "composed": composed}
func PutAct(node, module, fp, composed string) occ.Act {
return occ.Act{"put": "want", "node": node, "module": module, "fp": fp, "composed": composed}
}
// PutAsk is the ask about a put's move (ADR 0546 clauses 7 and 8), under the record ask's key of the composition so the
// teller relays it and its answer as any record ask's. For the bus its yes is of level approve and binds the act, so only a
// channel that proves who answered takes it (ADR 0259), never the view.
func PutAsk(key, node, module string, p delivery.Put, d delivery.ComposedDigest, bus bool, now time.Time) (delivery.AssignerAsk, error) {
yes := asks.Option{ID: OptionYes, Label: "Yes, put it", Does: "the node is told to apply " + module + " at " + short(p.FP), Level: asks.Acknowledge}
yes := oc.Option{ID: OptionYes, Label: "Yes, put it", Does: "the node is told to apply " + module + " at " + short(p.FP), Level: occ.Acknowledge}
what := map[delivery.Judgement]string{
delivery.JudgedRecreates: "recreates a container",
delivery.JudgedMovesData: "moves the module's data",
@@ -227,26 +228,26 @@ func PutAsk(key, node, module string, p delivery.Put, d delivery.ComposedDigest,
delivery.JudgedUnchanged: "changes nothing the node runs",
}[d.Judgement]
explanation := fmt.Sprintf("Asked to put %s at version %s on %s: %s. The move %s.", module, short(p.FP), node, p.Why, what)
expires := now.Add(7 * 24 * time.Hour)
expires := occ.FormatTime(now.Add(7 * 24 * time.Hour))
if bus {
binds, err := asks.ActDigest(PutAct(node, module, p.FP, d.Digest))
binds, err := occ.ActDigest(PutAct(node, module, p.FP, d.Digest))
if err != nil {
return delivery.AssignerAsk{}, err
}
yes.Level, yes.Binds = asks.Approve, binds
yes.Level, yes.Binds = occ.Approve, binds
yes.Does = "the bus's streams are snapshotted, then " + node + " is told to apply the bus at " + short(p.FP)
explanation += " It is the mesh's bus: every module reconnects while it restarts, and its streams are snapshotted first."
expires = now.Add(asks.ApproveLasts)
expires = occ.FormatTime(now.Add(occ.ApproveLasts))
}
return delivery.AssignerAsk{Ask: asks.Ask{
return assignerAsk(PutAsking(p), oc.Ask{
ID: delivery.AskingID(key, PutAsking(p)),
Headline: fmt.Sprintf("Put %s on %s?", module, node),
Explanation: explanation,
Options: []asks.Option{yes,
{ID: OptionNo, Label: "No", Does: "the node keeps what it runs, and the put is refused", Level: asks.Acknowledge}},
Who: asks.Operator,
Options: []oc.Option{yes,
{ID: OptionNo, Label: "No", Does: "the node keeps what it runs, and the put is refused", Level: occ.Acknowledge}},
Who: occ.Operator,
Expires: expires,
OnExpiry: "the node keeps what it runs",
About: key,
}, Asking: PutAsking(p)}, nil
}), nil
}
@@ -1,13 +1,14 @@
package assign
import (
"encoding/json"
"fmt"
"sort"
"maps"
"slices"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/go/seats/operatorchannel"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -19,30 +20,25 @@ import (
// revision of its key; which modules the indexer holds and which nodes have a record; and, for the seed, the export's
// digest.
type ProposeIn struct {
ID, By, Why string
At time.Time
Changes []delivery.PlacementProposalChange
Current map[string]delivery.Placement
Revisions map[string]uint64
Held func(module string) bool
Recorded func(node string) bool
Seed bool
ExportDigest string
// Leased says the lease write of ADR 0547 has happened (the controller's lease handed away, mesh-sdk
// buscompose.LeaseHandedAway; issue 774), after which the controller's assignments are read no more (§8): until then
// the holder keeps the controller in step and refuses what its verbs cannot take.
Leased bool
ID, By, Why string
At time.Time
Changes []delivery.PlacementProposalChange
Current map[string]delivery.Placement
Revisions map[string]uint64
Held func(module string) bool
Recorded func(node string) bool
// Warrant is the enrolment's warrant an add-node carries (ADR 0587 §7); only that change has one.
Warrant *delivery.AddNodeWarrant
}
// Propose lays a proposal over the current keys (§2): the proposal (state answering, each changed node's base revision),
// the laid placement of every node whose key it changes, and every finding, which refuses it whole: a change the
// controller's verbs cannot take before the lease write (§8), a change that does not fit its node, and what
// the laid placement of every node whose key it changes, and every finding, which refuses it whole: a change that does not fit its node, and what
// CheckPlacements refuses of the result laid over every placement (two modules on one host port, a provider not placed,
// a seat held once held twice, two nodes with one address, two hubs, a module the indexer does not hold, a node with no
// record). A key whose value the proposal leaves as it is changes nothing and is not laid.
func Propose(in ProposeIn) (delivery.PlacementProposal, map[string]*delivery.Placement, []string) {
p := delivery.PlacementProposal{ID: in.ID, By: strings.TrimSpace(in.By), Why: strings.TrimSpace(in.Why), At: in.At.UTC(),
Changes: in.Changes, Seed: in.Seed, ExportDigest: in.ExportDigest, Base: map[string]uint64{}, State: delivery.ProposalAnswering}
Changes: in.Changes, Warrant: in.Warrant, Base: map[string]uint64{}, State: delivery.ProposalAnswering}
var findings []string
if p.By == "" {
findings = append(findings, "the runtime names no caller: a proposal is made by someone")
@@ -54,18 +50,13 @@ func Propose(in ProposeIn) (delivery.PlacementProposal, map[string]*delivery.Pla
findings = append(findings, "a proposal changes something: it names no change")
}
for i, c := range in.Changes {
if (c.Op == delivery.OpSeed) != in.Seed {
findings = append(findings, fmt.Sprintf("change %d: the seed is adopt's alone, and never mixed with other changes", i+1))
continue
}
if !in.Leased {
if why := c.BeforeLease(); why != "" {
findings = append(findings, fmt.Sprintf("change %d: %s", i+1, why))
}
if c.Op == delivery.OpSeed {
findings = append(findings, fmt.Sprintf("change %d: the seed is gone with adopt, the controller's export being gone", i+1))
}
}
laid, lay := delivery.LayChanges(in.Current, in.Changes)
findings = append(findings, lay...)
findings = append(findings, ControllerRefusals(in.Changes, laid)...)
all := map[string]delivery.Placement{}
for node, pl := range in.Current {
all[node] = pl
@@ -80,6 +71,16 @@ func Propose(in ProposeIn) (delivery.PlacementProposal, map[string]*delivery.Pla
out[node] = &pl
p.Base[node] = in.Revisions[node]
}
// A remove-node deletes the key of a joining node that never confirmed (the operator's decision of 2026-11-02 01:05): its
// candidate is no placement, and the checks across nodes read the mesh without it. LayChanges said why one cannot go.
for _, node := range p.RemovedNodes() {
if _, ok := in.Current[node]; !ok {
continue
}
delete(all, node)
out[node] = nil
p.Base[node] = in.Revisions[node]
}
if len(findings) == 0 && len(out) == 0 {
findings = append(findings, "the changes leave every placement as it is")
}
@@ -90,12 +91,55 @@ func Propose(in ProposeIn) (delivery.PlacementProposal, map[string]*delivery.Pla
if recorded == nil {
recorded = func(string) bool { return false }
}
// A joining node writes its record only once it holds its account, which the add-node's write gives it: this one change by
// this one caller is the exception to a node with no record (ADR 0587 §7, changing ADR 0567 §2).
joining := map[string]bool{}
for _, c := range in.Changes {
if c.Op == delivery.OpAddNode {
joining[c.Node] = true
}
}
if len(joining) > 0 {
was := recorded
recorded = func(node string) bool { return joining[node] || was(node) }
}
for _, r := range delivery.CheckPlacements(all, held, recorded) {
findings = append(findings, r)
}
return p, out, findings
}
// ControllerModule is the controller's module, which nothing places again (issue 1126).
const ControllerModule = "mesh-controller"
// ControllerEnded is why a placement of the controller is refused: the parts that acted only once its lease was handed
// (consumers, the user list, enrolment, dead letters) now act without reading the lease, so a controller placed again
// would write beside them, silently.
const ControllerEnded = "the controller ended by its unassignment on 2026-11-02; placing it again would make two writers (issue 1126)"
// ControllerRefusals names every change, and every node as laid, that places the controller: by a place, by a seed's
// whole placement, or in what the changes lay. None when nothing does.
func ControllerRefusals(changes []delivery.PlacementProposalChange, laid map[string]delivery.Placement) []string {
var out []string
nodes := map[string]bool{}
for i, c := range changes {
places := c.Module == ControllerModule && c.Op != delivery.OpTakeOff
if c.Whole != nil && slices.Contains(c.Whole.Assigned(), ControllerModule) {
places = true
}
if places {
out = append(out, fmt.Sprintf("change %d places %s on %s: %s", i+1, ControllerModule, c.Node, ControllerEnded))
nodes[c.Node] = true
}
}
for _, node := range slices.Sorted(maps.Keys(laid)) {
if !nodes[node] && slices.Contains(laid[node].Assigned(), ControllerModule) {
out = append(out, fmt.Sprintf("%s would run %s: %s", node, ControllerModule, ControllerEnded))
}
}
return out
}
// Write is one node a warrant writes (§5): the node, its node key `<node>.node` (the commit of the node's keys,
// mesh-sdk delivery.PlacementKeys.Writes), the revision of the node's placement it is written at (0: it has none, so
// it is created) and its value (nil: it is deleted).
@@ -118,117 +162,24 @@ func Writes(p delivery.PlacementProposal, laid map[string]*delivery.Placement) (
return out, nil
}
// ControllerCall is one call of the controller's seat verbs the holder makes after writing an approved proposal, until
// the lease write (§8): assign, unassign, pin or unpin, with its arguments.
type ControllerCall struct {
Verb string
Args map[string]any
// Node and Module name the pair the call keeps in step (a pin names no module).
Node, Module string
}
// Key names the call, as the condition of its pair says it.
func (c ControllerCall) Key() string {
return fmt.Sprintf("mesh-controller.%s %s %s", c.Verb, c.Node, c.Module)
}
// ControllerCalls is every call that brings the controller's assignments and pins to the placement after: assign for a
// module placed on a node before not, unassign for one taken off, pin for a pin set or changed, unpin for one removed. In
// node order, unassigns before assigns, so a module moved between nodes is first taken off.
//
// Each call carries only the arguments the controller's seat contract declares for its verb (catalogue.ControllerVerbs):
// the controller refuses any other argument and does nothing (issue 774: a "why" added to every call had every approved
// placement refused since 2026-10-28). The proposal and its warrant are in the assigner's own record instead: the
// placement-changed event and the hand-act log.
func ControllerCalls(before map[string]delivery.Placement, after map[string]*delivery.Placement) []ControllerCall {
nodes := make([]string, 0, len(after))
for node := range after {
nodes = append(nodes, node)
}
sort.Strings(nodes)
var off, on, pins []ControllerCall
for _, node := range nodes {
was, now := before[node], delivery.Placement{}
if a := after[node]; a != nil {
now = *a
}
had, has := map[string]bool{}, map[string]bool{}
for _, m := range was.Modules {
had[m.Module] = true
}
for _, m := range now.Modules {
has[m.Module] = true
if !had[m.Module] {
on = append(on, ControllerCall{Verb: "assign", Node: node, Module: m.Module, Args: map[string]any{"node": node, "module": m.Module}})
}
}
for _, m := range was.Modules {
if !has[m.Module] {
off = append(off, ControllerCall{Verb: "unassign", Node: node, Module: m.Module, Args: map[string]any{"node": node, "module": m.Module}})
}
}
for _, provision := range sortedKeys(now.Pins) {
to := now.Pins[provision]
if old, ok := was.Pins[provision]; !ok || old != to {
pins = append(pins, ControllerCall{Verb: "pin", Node: node, Module: provision,
Args: map[string]any{"node": node, "provision": provision, "from": to.Node, "module": to.Module}})
}
}
for _, provision := range sortedKeys(was.Pins) {
if _, ok := now.Pins[provision]; !ok {
pins = append(pins, ControllerCall{Verb: "unpin", Node: node, Module: provision, Args: map[string]any{"node": node, "provision": provision}})
}
}
}
return append(append(off, on...), pins...)
}
// CallKind is the kind of the condition of a refused call: one for a module's place (assign, unassign), one for a pin.
func (c ControllerCall) CallKind() string {
if c.Verb == "pin" || c.Verb == "unpin" {
return KindControllerPin
}
return KindControllerCall
}
// The kinds of a refused controller call's condition (and finding).
// The kinds of the condition (and finding) of a refused controller call, which the assigner raised until it stopped calling
// the controller's verbs (ADR 0567 §8, issue 774).
const (
KindControllerCall = "controller-call"
KindControllerPin = "controller-pin"
)
// CallConditionKey is the key of the condition of a call refused for its pair (node and module, or node and provision).
func CallConditionKey(c ControllerCall) string {
return delivery.AssignerModule + "." + c.CallKind() + "." + c.Node + "." + c.Module
}
// CallRefusedCondition is the condition the assigner raises when the controller refused (or did not answer) a call that
// keeps its assignments in step with an approved placement (ADR 0567 §8, issue 774): the placement is written, the
// controller's assignments are not, so nothing new reaches the node (no bus account, no composition). It names the verb,
// the node, the module (or provision) and the controller's words, keyed by the pair, and is cleared (raised false) when a
// later call for the pair is answered, or both sources agree on it, with resolved saying which.
func CallRefusedCondition(c ControllerCall, refusal string, raised bool, resolved string) asks.Condition {
key := CallConditionKey(c)
subject := asks.Scope{Scope: "module", ID: c.Module, Machine: c.Node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared, Resolved: resolved}
}
headline := fmt.Sprintf("the controller refused %s %s on %s", c.Verb, c.Module, c.Node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
needs := fmt.Sprintf("call mesh-controller.%s for %s on %s by hand once the refusal's cause is fixed", c.Verb, c.Module, c.Node)
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: fmt.Sprintf("%s%s. An approved placement was written, and the assigner's call %s to keep the controller's "+
"assignments in step was refused: %s. Until the controller holds it, the placement and the assignments differ, so "+
"the assigner writes no want for the pair and nothing new reaches %s. This clears when a call for it is answered "+
"or both sources agree.", asks.NeedsYou, needs, c.Key(), refusal, c.Node)}
// CallClearedCondition clears the condition of a refused controller call a previous process raised, for its pair (node and
// module, or node and provision for a pin).
func CallClearedCondition(kind, node, module string) oc.Condition {
return oc.Condition{Key: delivery.AssignerModule + "." + kind + "." + node + "." + module,
Subject: oc.ConditionSubject{Scope: "module", ID: module, Machine: node}, State: occ.StateCleared,
Resolved: "the assigner calls no verb of the controller any more: placement is the one source."}
}
// PlacementAct is the act an approved proposal's yes binds: writing exactly its changes at its base revisions.
func PlacementAct(p delivery.PlacementProposal) asks.Act {
return asks.Act{"placement": "write", "proposal": p.ID, "changes": delivery.ChangesDigest(p)}
func PlacementAct(p delivery.PlacementProposal) occ.Act {
return occ.Act{"placement": "write", "proposal": p.ID, "changes": delivery.ChangesDigest(p)}
}
// PlacementAsking is a proposal's asking number: each judging that raises its ask again (again, a key moved) is
@@ -240,83 +191,29 @@ func PlacementAsking(p delivery.PlacementProposal, at time.Time) int {
// PlacementAsk is the authorising ask about a proposal (§5): an Approve that binds its act, so only a channel that
// proves who answered takes it, with the brief of the verdict by node (judge.Brief) as its explanation.
func PlacementAsk(key string, p delivery.PlacementProposal, brief string, asking int, now time.Time) (delivery.AssignerAsk, error) {
binds, err := asks.ActDigest(PlacementAct(p))
binds, err := occ.ActDigest(PlacementAct(p))
if err != nil {
return delivery.AssignerAsk{}, err
}
nodes := p.Nodes()
headline := fmt.Sprintf("Change placement on %s?", strings.Join(nodes, ", "))
if p.Seed {
headline = "Take placement over from the controller?"
}
if n := len([]rune(headline)); n > asks.HeadlineLength {
if n := len([]rune(headline)); n > occ.HeadlineLength {
headline = fmt.Sprintf("Change placement on %d machines?", len(nodes))
}
return delivery.AssignerAsk{Ask: asks.Ask{
return assignerAsk(asking, oc.Ask{
ID: delivery.AskingID(key, asking),
Headline: headline,
Explanation: brief,
Options: []asks.Option{
Options: []oc.Option{
{ID: OptionYes, Label: "Approve", Does: "the placement of " + strings.Join(nodes, ", ") + " is written as proposed",
Level: asks.Approve, Binds: binds},
{ID: OptionNo, Label: "Decline", Does: "nothing is written, and the proposal ends", Level: asks.Acknowledge},
Level: occ.Approve, Binds: binds},
{ID: OptionNo, Label: "Decline", Does: "nothing is written, and the proposal ends", Level: occ.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(asks.ApproveLasts),
Who: occ.Operator,
Expires: occ.FormatTime(now.Add(occ.ApproveLasts)),
OnExpiry: "nothing is written, and the proposal ends",
About: key,
}, Asking: asking}, nil
}
// ExportOf reads the controller's answer to placement-export (§7): its verb answers what the command printed, the
// JSON of every node's placement, in `output` (the controller's verbAnswer; `answer` when it carries it as data). A
// refused or unreadable answer is an error naming it.
func ExportOf(raw []byte) (map[string]delivery.Placement, error) {
var a struct {
Output string `json:"output"`
OK *bool `json:"ok"`
Answer json.RawMessage `json:"answer"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("the controller's placement-export answered no verb answer: %w", err)
}
if a.OK != nil && !*a.OK {
return nil, fmt.Errorf("the controller's placement-export refused: %s", strings.TrimSpace(a.Output))
}
var byNode map[string]json.RawMessage
if len(a.Answer) > 0 && string(a.Answer) != "null" {
if err := json.Unmarshal(a.Answer, &byNode); err != nil {
return nil, fmt.Errorf("the controller's placement-export is no placement by node: %w", err)
}
} else if err := json.NewDecoder(strings.NewReader(a.Output)).Decode(&byNode); err != nil {
// The command prints its JSON first; what its standard error said after it is left.
return nil, fmt.Errorf("the controller's placement-export printed no placement by node: %w", err)
}
out := map[string]delivery.Placement{}
for node, rawP := range byNode {
p, err := delivery.DecodePlacement(rawP)
if err != nil {
return nil, fmt.Errorf("the controller's placement-export of %s cannot be read: %w", node, err)
}
if p.Node != node {
return nil, fmt.Errorf("the controller's placement-export names %s under %s", p.Node, node)
}
out[node] = p
}
if len(out) == 0 {
return nil, fmt.Errorf("the controller's placement-export names no node")
}
return out, nil
}
// SeedChanges is the seed's changes: every node's placement of the export, whole, in node order.
func SeedChanges(export map[string]delivery.Placement) []delivery.PlacementProposalChange {
var out []delivery.PlacementProposalChange
for _, node := range sortedKeys(export) {
p := export[node]
out = append(out, delivery.PlacementProposalChange{Op: delivery.OpSeed, Node: node, Whole: &p})
}
return out
}), nil
}
// Rest is what a proposal leaves to write once the keys of written are: its changes of the other nodes, to be laid again
@@ -7,8 +7,7 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
occ "git.novox.be/novox/mesh-contracts/go/clients/operatorchannelclient"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -38,39 +37,35 @@ func TestThePlacementSeatDecides(t *testing.T) {
in ProposeIn
want string
}{
"a port before the lease": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "web", Ports: map[string]int{"80": 8080}}), "lease write"},
"a holding": {in(delivery.PlacementProposalChange{Op: delivery.OpHold, Node: "anchor", Module: "store", Claim: "mesh-store", Scope: "mesh"}), "lease write"},
"a binding": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "web", Bindings: map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}}), "lease write"},
"a config place": {in(delivery.PlacementProposalChange{Op: delivery.OpConfig, Node: "anchor", Module: "web", Layer: map[string]any{"places": []any{}}}), "lease write"},
"a config access": {in(delivery.PlacementProposalChange{Op: delivery.OpConfig, Node: "anchor", Module: "web", MeshWide: map[string]any{"accesses": []any{}}}), "lease write"},
"a module not held": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "ghost"}), "the indexer holds no module ghost"},
"a node with no record": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "stranger", Module: "web"}), "stranger is a node with no record"},
"nothing changes": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "anchor", Module: "web"}), "leave every placement as it is"},
"no why": {func() ProposeIn { i := in(move...); i.Why = " "; return i }(), "says why"},
"no caller": {func() ProposeIn { i := in(move...); i.By = ""; return i }(), "names no caller"},
"a seed by hand": {in(delivery.PlacementProposalChange{Op: delivery.OpSeed, Node: "laptop", Whole: &laptop}), "adopt's alone"},
"a pin of nobody placed": {in(delivery.PlacementProposalChange{Op: delivery.OpPin, Node: "laptop", Provision: "db", Provider: &delivery.Provider{Node: "laptop", Module: "store"}}), "not placed there"},
"a module not held": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "ghost"}), "the indexer holds no module ghost"},
"a node with no record": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "stranger", Module: "web"}), "stranger is a node with no record"},
"nothing changes": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "anchor", Module: "web"}), "leave every placement as it is"},
"no why": {func() ProposeIn { i := in(move...); i.Why = " "; return i }(), "says why"},
"no caller": {func() ProposeIn { i := in(move...); i.By = ""; return i }(), "names no caller"},
"a seed by hand": {in(delivery.PlacementProposalChange{Op: delivery.OpSeed, Node: "laptop", Whole: &laptop}), "gone with adopt"},
// 1126: the controller ended by its unassignment; placed again it would be a second writer beside the parts that
// act without its lease.
"the controller placed again": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "mesh-controller"}), "ended by its unassignment"},
"the controller seeded whole": {in(delivery.PlacementProposalChange{Op: delivery.OpSeed, Node: "anchor", Whole: &delivery.Placement{V: delivery.Version, Node: "anchor",
Decided: anchor.Decided, Modules: []delivery.PlacedModule{{Module: "mesh-controller"}}}}), "two writers (issue 1126)"},
"a pin of nobody placed": {in(delivery.PlacementProposalChange{Op: delivery.OpPin, Node: "laptop", Provision: "db", Provider: &delivery.Provider{Node: "laptop", Module: "store"}}), "not placed there"},
} {
if _, _, f := Propose(c.in); !slices.ContainsFunc(f, func(s string) bool { return strings.Contains(s, c.want) }) {
t.Errorf("%s: findings %v, want one naming %q", name, f, c.want)
}
}
// A module's config values may be proposed before the lease write (§13); once the lease is written every change is
// the state's alone, and none waits.
if _, _, f := Propose(in(delivery.PlacementProposalChange{Op: delivery.OpConfig, Node: "anchor", Module: "web", Values: map[string]any{"puid": 1}})); len(f) != 0 {
t.Errorf("config values before the lease write: %v", f)
}
// Every kind of change is taken: the placement is the one source, and the controller's verbs limit nothing (ADR 0567 §8).
for _, c := range []delivery.PlacementProposalChange{
{Op: delivery.OpConfig, Node: "anchor", Module: "web", Values: map[string]any{"puid": 1}},
{Op: delivery.OpPlace, Node: "laptop", Module: "web", Ports: map[string]int{"80": 8080}, Bindings: map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}},
{Op: delivery.OpConfig, Node: "anchor", Module: "web", Layer: map[string]any{"places": []any{}}},
{Op: delivery.OpConfig, Node: "anchor", Module: "web", MeshWide: map[string]any{"accesses": []any{}}},
{Op: delivery.OpDecide, Node: "laptop", Field: "site", Value: json.RawMessage(`"attic"`)},
{Op: delivery.OpHold, Node: "anchor", Module: "store", Claim: "node-shell", Scope: "node"},
} {
leased := in(c)
leased.Leased = true
if _, _, f := Propose(leased); len(f) != 0 {
t.Errorf("after the lease write %s is refused: %v", c.Op, f)
if _, _, f := Propose(in(c)); len(f) != 0 {
t.Errorf("%s is refused: %v", c.Op, f)
}
}
@@ -86,153 +81,23 @@ func TestThePlacementSeatDecides(t *testing.T) {
// The seat says it waits while one of its states is not bound, naming it, and clears once all are.
for _, unbound := range [][]string{{delivery.PlacementPulsesBucket}, nil} {
c := SeatWaitsCondition(unbound)
if _, err := asks.EncodeCondition(c); err != nil || (len(unbound) > 0) != (c.State == asks.StateRaised) ||
if _, err := occ.EncodeCondition(c); err != nil || (len(unbound) > 0) != (c.State == occ.StateRaised) ||
(len(unbound) > 0 && !strings.Contains(c.Explanation, "the seat waits for its states: "+delivery.PlacementPulsesBucket)) {
t.Errorf("waiting for %v: %+v, %v", unbound, c, err)
}
}
// Issue 1126: the seed places the private network, which no repository holds; it is the mesh's own, and passes.
wired := laptop
wired.Modules = append(slices.Clone(laptop.Modules), delivery.PlacedModule{Module: "mesh-wireguard"})
seed := in(SeedChanges(map[string]delivery.Placement{"laptop": wired})...)
seed.Seed, seed.ExportDigest, seed.Current, seed.Revisions = true, "sha256:x", map[string]delivery.Placement{}, map[string]uint64{}
seed.Held = func(m string) bool { return m != "mesh-wireguard" } // the indexer holds every module but it
if _, _, f := Propose(seed); len(f) != 0 {
t.Errorf("a seed placing the private network is refused: %v", f)
}
writes, err := Writes(p, laid)
if err != nil || len(writes) != 2 || writes[0].Key != "anchor.node" || writes[0].Base != 4 || writes[1].Base != 9 {
t.Fatalf("writes = %+v, %v", writes, err)
}
pinned := *laid["laptop"]
pinned.Pins = map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}
calls := ControllerCalls(map[string]delivery.Placement{"anchor": anchor, "laptop": laptop},
map[string]*delivery.Placement{"anchor": laid["anchor"], "laptop": &pinned})
var said []string
for _, c := range calls {
said = append(said, c.Key())
}
if !slices.Equal(said, []string{"mesh-controller.unassign anchor web", "mesh-controller.assign laptop web", "mesh-controller.pin laptop db"}) {
t.Errorf("the controller's calls = %v", said)
}
a, err := PlacementAsk("placement.p-one", p, "move web (proposed by operator@laptop)", 7, time.Unix(100, 0))
if err != nil || a.Ask.Check(time.Unix(100, 0)) != nil || a.Ask.Options[0].Level != asks.Approve || a.Ask.Options[0].Performs(PlacementAct(p)) != nil {
if err != nil || occ.Check(mustAsk(t, a), time.Unix(100, 0)) != nil || mustAsk(t, a).Options[0].Level != occ.Approve || occ.Performs(mustAsk(t, a).Options[0], PlacementAct(p)) != nil {
t.Fatalf("the ask = %+v, %v", a, err)
}
q := p
q.Changes = append(slices.Clone(p.Changes), delivery.PlacementProposalChange{Op: delivery.OpUnpin, Node: "laptop", Provision: "x"})
if a.Ask.Options[0].Performs(PlacementAct(q)) == nil {
if occ.Performs(mustAsk(t, a).Options[0], PlacementAct(q)) == nil {
t.Error("the ask's yes binds changes it was not asked about")
}
export, _ := json.Marshal(map[string]any{"anchor": anchor, "laptop": laptop})
for name, c := range map[string]struct {
raw string
want int
}{
"in output": {`{"ok":true,"output":` + quote(string(export)+"\n") + `}`, 2},
"as answer": {`{"ok":true,"output":"","answer":` + string(export) + `}`, 2},
"refused": {`{"ok":false,"output":"no store"}`, 0},
"no node": {`{"ok":true,"output":"{}"}`, 0},
"no version": {`{"ok":true,"output":"{\"anchor\":{\"node\":\"anchor\",\"modules\":[]}}"}`, 0},
} {
got, err := ExportOf([]byte(c.raw))
if len(got) != c.want || (c.want == 0) != (err != nil) {
t.Errorf("%s: %d nodes, %v", name, len(got), err)
}
}
if seed := SeedChanges(map[string]delivery.Placement{"laptop": laptop, "anchor": anchor}); len(seed) != 2 || seed[0].Node != "anchor" || seed[0].Op != delivery.OpSeed {
t.Errorf("the seed's changes = %+v", seed)
}
}
func quote(s string) string {
raw, _ := json.Marshal(s)
return string(raw)
}
// Issue 774: every call the assigner makes of the controller carries only the arguments the controller's seat contract
// declares for its verb, and every one it requires; the controller refuses any other argument and does nothing. A "why"
// added to every call (2026-10-28) had each approved placement refused.
func TestControllerCallsFollowTheContract(t *testing.T) {
declared := map[string]map[string]any{}
required := map[string][]string{}
for _, v := range catalogue.ControllerVerbs {
props, _ := v.Input["properties"].(map[string]any)
declared[v.Name] = props
switch r := v.Input["required"].(type) {
case []string:
required[v.Name] = r
}
}
db := map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}
moved := map[string]delivery.Provider{"db": {Node: "laptop", Module: "store"}}
placed := func(node string, pins map[string]delivery.Provider, modules ...string) delivery.Placement {
p := delivery.Placement{Node: node, Pins: pins}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
return p
}
ptr := func(p delivery.Placement) *delivery.Placement { return &p }
for name, c := range map[string]struct {
before map[string]delivery.Placement
after map[string]*delivery.Placement
verbs []string
}{
"assign": {map[string]delivery.Placement{}, map[string]*delivery.Placement{"anchor": ptr(placed("anchor", nil, "web"))}, []string{"assign"}},
"unassign": {map[string]delivery.Placement{"anchor": placed("anchor", nil, "web")}, map[string]*delivery.Placement{"anchor": ptr(placed("anchor", nil))}, []string{"unassign"}},
"key deleted": {map[string]delivery.Placement{"anchor": placed("anchor", db, "web")}, map[string]*delivery.Placement{"anchor": nil}, []string{"unassign", "unpin"}},
"pin set": {map[string]delivery.Placement{"anchor": placed("anchor", nil, "web")}, map[string]*delivery.Placement{"anchor": ptr(placed("anchor", db, "web"))}, []string{"pin"}},
"pin changed": {map[string]delivery.Placement{"anchor": placed("anchor", db, "web")}, map[string]*delivery.Placement{"anchor": ptr(placed("anchor", moved, "web"))}, []string{"pin"}},
"moved and pinned": {map[string]delivery.Placement{"anchor": placed("anchor", nil, "web")}, map[string]*delivery.Placement{"anchor": ptr(placed("anchor", nil)), "laptop": ptr(placed("laptop", db, "web"))}, []string{"unassign", "assign", "pin"}},
} {
calls := ControllerCalls(c.before, c.after)
var verbs []string
for _, call := range calls {
verbs = append(verbs, call.Verb)
props, ok := declared[call.Verb]
if !ok {
t.Errorf("%s: the controller's seat has no verb %q", name, call.Verb)
continue
}
for arg := range call.Args {
if _, ok := props[arg]; !ok {
t.Errorf("%s: %s gives %q, which mesh-controller.%s does not take, so the controller refuses it", name, call.Key(), arg, call.Verb)
}
}
for _, arg := range required[call.Verb] {
if _, ok := call.Args[arg]; !ok {
t.Errorf("%s: %s lacks %q, which mesh-controller.%s requires", name, call.Key(), arg, call.Verb)
}
}
}
if !slices.Equal(verbs, c.verbs) {
t.Errorf("%s: the calls = %v, want %v", name, verbs, c.verbs)
}
}
}
// Issue 774: a refused call is a condition keyed by its pair, a module's place apart from a pin, naming the verb, the node, the
// module and the refusal; its clear says why.
func TestCallRefusedCondition(t *testing.T) {
assign := ControllerCall{Verb: "assign", Node: "anchor", Module: "web"}
pin := ControllerCall{Verb: "pin", Node: "anchor", Module: "web"}
c := CallRefusedCondition(assign, `assign takes no argument "why"`, true, "")
if c.State != asks.StateRaised || c.Severity != asks.SeverityUrgent || c.Key != "mesh-delivery-assigner.controller-call.anchor.web" ||
!strings.Contains(c.Headline, "assign web on anchor") || !strings.Contains(c.Explanation, `takes no argument "why"`) {
t.Errorf("the raised condition = %+v", c)
}
if CallConditionKey(pin) == c.Key {
t.Error("a pin's condition shares the key of a module's place")
}
if cl := CallRefusedCondition(assign, "", false, "answered"); cl.State != asks.StateCleared || cl.Key != c.Key || cl.Resolved != "answered" {
t.Errorf("the clear = %+v", cl)
}
if _, _, raised, err := FindingOf(c, time.Unix(0, 0)); err != nil || !raised {
t.Errorf("the condition is no finding: %v", err)
}
}
@@ -0,0 +1,38 @@
package assign
import (
"slices"
"git.novox.be/novox/mesh-delivery/sdk/graph"
)
// verbWaiting says why the module's want on node waits for a verb it invokes (novox/hq ADR 0614 §2): the first seat verb or
// module tool of Invokes (the module graph's verb edges, those it leaves out on a cycle not among them, §3) that no
// running healthy build serves. Nil Served (nothing read of what runs) waits for none, and a verb a running build serves
// already adds no wait. It holds under the provisions' rule (RuleProvisions): no reader knows a rule number of its own
// yet, so the words name the edge (graph.VerbWaitWords, which the delivery view reads).
func (m Module) verbWaiting(node string) (string, bool) {
if m.Served == nil {
return "", false
}
for _, v := range slices.Sorted(slices.Values(m.Invokes)) {
if !m.Served(v) {
return graph.VerbWaitWords(v, m.Name, node), true
}
}
return "", false
}
// busStateWaiting says why the module's want on node waits for bus state (ADR 0614 §5): its manifest asks bus state (a
// bus-state entry, as the module graph reads it) and no build whose manifest offers bus-state runs healthy on a node
// present. The provision bus-state is answered with no credential of a consumer's own, so ADR 0603 §5 counts the
// provider's own have and no provisioner key is waited for, handed or not. Nil Offering (nothing read of what runs) waits
// for none, and neither does a module that offers bus-state itself. It holds under the provisions' rule, its words naming bus state (graph.BusStateWaitWords).
func (m Module) busStateWaiting(node string) (string, bool) {
if len(m.Manifest.BusState) == 0 || m.Offering == nil || m.Offering(graph.BusState) ||
slices.Contains(m.Manifest.Offers(), graph.BusState) { // the bus module offering it waits on nobody for it
return "", false
}
return graph.BusStateWaitWords(m.Name, node), true
}
@@ -11,6 +11,9 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
"git.novox.be/novox/mesh-delivery/sdk/forms"
"git.novox.be/novox/mesh-delivery/sdk/graph"
)
// Module is what the six rules read of one module (ADR 0429 §6, ADR 0528 clause 3): its newest fingerprint F at the
@@ -36,15 +39,34 @@ type Module struct {
// Answering is the manifests of the modules placed somewhere that provide a provision: what catalogue.Awaited
// judges a want's provisions by (issue 774). Nil: none known, and no key is waited for that is not there.
Answering func(provision string) []catalogue.Manifest
// Offered says whether any module offers a provision by name or by its own name (catalogue.Manifest.Offers) among every
// manifest the trunk holds and the modules the mesh provides itself, placed or not. Nil: not known, and no requirement
// is waited on for want of an offer.
Offered func(provision string) bool
// Invokes are the seat verbs and module tools the module graph says the module invokes and keeps as edges to the
// modules serving them, those it leaves out on a cycle not among them (ADR 0614 §2, §3): the want waits until each is
// served (Served). Nil: no graph node is read, and no want waits on a verb.
Invokes []string
// Served says whether a build serving a seat verb or module tool runs healthy, counted from its holder's have (ADR 0614
// §2, as ADR 0603 §5 counts a provider's). Nil: nothing is read of what runs, and no want waits on a verb.
Served func(invoke string) bool
// Offering says whether a build whose manifest offers a provision runs healthy on a node present, counted from the
// provider's own have (ADR 0603 §5): what a want asking bus state waits for (ADR 0614 §5). Nil: nothing is read of what
// runs, and no want waits on bus state.
Offering func(provision string) bool
// Chosen is a person's choice of the module's upgrade policy (the assigner's `policies`, ADR 0565 §7); nil: none.
Chosen *catalogue.PersonPolicy
Nodes []string
// Composed answers the controller's composition of the module for a node at a fingerprint (mesh-controller_composed): its
// digest and the judgement of the move to it, and whether it exists. Nil: none exists (ADR 0539 §1, §6).
// Composed answers the composition of the module for a node at a fingerprint: the digest the node's own have names (a
// placed node composes its own, ADR 0571 §6), and whether it names one. Nil: none exists (ADR 0539 §1, §6).
Composed func(node, fp string) (delivery.ComposedDigest, bool)
// Allowed says whether the operator answered yes to the record ask about the move of this module on node to the composition
// with this digest (ADR 0539 §14). Nil: none did.
Allowed func(node, digest string) bool
// Release names the release whose Approve carries the build fp, as mesh-delivery's gate holds it for the release's heads,
// read by the build's source revision (ADR 0601 §1): that Approve is the yes to a move of the build alone. False when no
// approved release is known to carry it. Nil: none is read.
Release func(fp string) (string, bool)
// Released is a hold on FP the operator released (ADR 0539 §19): FP is not held again by a health check that failed.
Released bool
// Restored says whether the operator answered "restore" to the restore ask of this data module on node: it is then set back
@@ -56,6 +78,13 @@ type Module struct {
Now time.Time
// Puts are the puts of the module, by node (ADR 0546): while one owns a node's want (PutOwns), the rules pass over it.
Puts map[string]delivery.Put
// Needs are the reader capabilities the manifest at FP needs (the indexer's modules value, catalogue.ManifestNeeds, ADR
// 0614 §4): a node's want moves only while that node's engine states each of them. Nil: none.
Needs []string
// FormNeeds are the reference forms the module's build at FP is written in (`form:<name>`, the builder's build key's
// needs, ADR 0618 §1): a node's want is held only while that node's engine reads none of them (ADR 0624 §2). Nil: the
// build names none, and holds nothing.
FormNeeds []string
// Self says whether node composes the module itself: the catalogue's placement names the node (ADR 0547 §2). It is the one
// source of a want's Self: every want the rules write or keep is derived from it (selfDerived), so a want written Self while
// the node was placed does not outlive the placement (issue 1126). Nil: no node does.
@@ -77,6 +106,9 @@ type Nodes struct {
Haves map[string]map[string]delivery.Have
// Providers is, by provision, the modules that provide it, with the nodes each is assigned to.
Providers map[string]map[string][]string
// Reads is, by node, the reader capabilities its node-engine states in its node record (delivery.NodeRecord.ManifestReads,
// ADR 0614 §4). A node with none, or none read, is read as stating the baseline (catalogue.StatedManifestReads).
Reads map[string][]string
}
// healthy says whether a have runs fp healthy.
@@ -130,15 +162,42 @@ func holds(m Module, node string, n Nodes, ordering bool) (bool, int, string) {
return false, delivery.RuleGrant, fmt.Sprintf("the controller has not issued the credential of %s on %s for %s", m.Name, node,
short(m.FP))
}
if words, lacking := m.readerLacking(node, n); lacking {
return false, delivery.RuleComposed, words
}
if _, ok := m.composition(node, m.FP); !ok {
return false, delivery.RuleComposed, fmt.Sprintf("the controller has not composed %s for %s at %s", m.Name, node, short(m.FP))
return false, delivery.RuleComposed, fmt.Sprintf("%s has named no composition of its own for %s at %s yet", node, m.Name, short(m.FP))
}
if words, waiting := m.provisionWaiting(node); waiting {
return false, delivery.RuleProvisions, words
}
if words, waiting := m.verbWaiting(node); waiting {
return false, delivery.RuleProvisions, words
}
if words, waiting := m.busStateWaiting(node); waiting {
return false, delivery.RuleProvisions, words
}
return true, 0, ""
}
// readerLacking says why the module's want on node is held for the node's engine (ADR 0614 §4): a reader capability its
// manifest needs that the engine does not state, read as the baseline when it states none. It is held under rule 7 (the
// composition), since no rule of its own is numbered in this release, with words naming the capability; it moves once the
// engine states it. The node-engine's own new build is held only by a word of its own manifest no engine states.
//
// A build's reference forms hold the want only when the engine reads none of them (ADR 0624 §2): every engine reads a
// first form, so a build written in one is never held by its forms; the words name the forms it reads none of.
func (m Module) readerLacking(node string, n Nodes) (string, bool) {
missing := catalogue.MissingManifestReads(m.Needs, n.Reads[node])
if len(missing) == 0 && forms.Current.Held(m.FormNeeds, n.Reads[node]) {
missing = slices.Clone(m.FormNeeds)
}
if len(missing) == 0 {
return "", false
}
return graph.ReaderWaitWords(node, missing, m.Name, short(m.FP)), true // the words the delivery view names the edge by
}
// provisionWaiting says why the module's want on node waits for a provision (ADR 0570 §4): a provision it requires that has
// no key yet, or the first of its keys that is not provided. Nil Provisions (the seat's state not on the bus) waits for none.
func (m Module) provisionWaiting(node string) (string, bool) {
@@ -149,8 +208,36 @@ func (m Module) provisionWaiting(node string) (string, bool) {
return fmt.Sprintf("provision %s has no key yet for %s on %s", missing[0], m.Name, node), true
}
keyed := m.Provisions(node)
// Only a provision the newest manifest still requires is waited on (ADR 0603 §9): a key of an older revision's edge,
// asked or retiring for the operator's answer (ADR 0570 §5), holds nothing. A requirement catalogue.Awaited leaves out is
// not waited on when something offers it (a module's own name, a module the mesh provides itself, a provision the
// composition does not key: optional, a toolchain on the module's own machine, a login the controller makes). One that
// nothing offers is waited on (ADR 0603 §5: an unknown provider has provided nothing), so a consumer never moves ahead of
// a provider whose module the indexer has not yet taken in; it holds loudly, since the watchman raises the hold past its
// bound (ADR 0603 §8).
needed := m.Manifest.Requires
if m.Answering != nil {
needed = catalogue.Awaited(m.Manifest, m.Answering)
have := map[string]bool{}
for _, p := range keyed {
have[p.Provision] = true
}
for _, r := range slices.Sorted(slices.Values(m.Manifest.Requires)) {
if have[r] && !m.Manifest.OptionalRequires[r] && !slices.Contains(needed, r) {
needed = append(needed, r) // the provisioner keys it from the manifests, whoever is placed yet
continue
}
if m.Offered == nil || m.Manifest.OptionalRequires[r] || len(m.Answering(r)) > 0 || m.Offered(r) || slices.Contains(needed, r) {
continue
}
needed = append(needed, r)
if !have[r] {
return fmt.Sprintf("no module offers %s yet for %s on %s: an unknown provider has provided nothing, so the want waits for one, and the watchman raises it past its bound", r, m.Name, node), true
}
}
}
for _, key := range slices.Sorted(maps.Keys(keyed)) {
if p := keyed[key]; p.State != provisions.Provided {
if p := keyed[key]; p.State != provisions.Provided && slices.Contains(needed, p.Provision) {
words := fmt.Sprintf("the provision %s of %s on %s is %s, not provided", key, m.Name, node, p.State)
if p.Why != "" {
words += ": " + p.Why
@@ -268,29 +355,18 @@ type RecordNeed struct {
Words string
}
// Holding is a rule holding a node back, and why. Bus marks the bar on the bus instead of one of the seven rules (Rule is then
// 0): the bus moves only by the operator's put (ADR 0539 §15, ADR 0546 §8), never by the assigner (issue 1945: nats on novox,
// running an old-form build, got no want and the pass said nothing).
// Holding is a rule holding a node back, and why. The bus module is held back by the rules alone, as every module (ADR 0601 §1,
// §2): no bar of its own.
type Holding struct {
Rule int
Words string
Bus bool
}
// Reason is the holding in words: the rule and its words, or for the bus bar its words alone.
// Reason is the holding in words: the rule and its words.
func (h Holding) Reason() string {
if h.Bus {
return h.Words
}
return fmt.Sprintf("rule %d: %s", h.Rule, h.Words)
}
// BusBarWords say why the assigner writes no want moving the bus on node: it runs prev, the trunk builds fp, and only the
// operator's put moves it (ADR 0539 §15, ADR 0546 §8). The words reach the operator, so they name no record.
func BusBarWords(node, prev, fp string) string {
return fmt.Sprintf("only your put moves the bus: %s runs %s, the trunk builds %s; put it with assigner_put", node, short(prev), short(fp))
}
// Wants is the pass of one module m: for each of its nodes, the want by the six rules, starting from the want written
// before (wants, by node); lastHealthy is, by node, the newest fingerprint other than m.FP the node ran healthy. A failed
// health check of m.FP on a node holds it: every node wanting m.FP is set back to its last healthy fingerprint, but a
@@ -412,7 +488,8 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
next.Newest, next.Rule, next.Words = m.FP, rule, words
at.FP = target
}
if had && !before.Removed && !before.Put && before.FP == target && before.Composed == d.Digest {
// A placed node's want names the build alone: it is the same target while the build is.
if had && !before.Removed && !before.Put && before.FP == target && (before.Composed == d.Digest || (before.Self && m.selfOn(node))) {
next.Backup, next.Words = before.Backup, before.Words // the answer to a failed backup stays while its target stays (ADR 0539 §16)
}
running, ran := n.Haves[node][m.Name]
@@ -424,14 +501,6 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
if need, wait := at.mustAsk(node, asked, hadLive, ran, running.FP, running.Composed == "", d); wait {
if need != nil {
out.Record = append(out.Record, *need)
} else if prev, barred := at.busBar(asked, hadLive, running.FP); barred {
// The bus is barred: no want moves it but the operator's put. Said, never silent (issue 1945): its holding is named,
// and a node that runs it with no live want has it on neither list, so its condition stays raised.
h := Holding{Bus: true, Words: BusBarWords(node, prev, at.FP)}
out.Rules[node] = h
if !hadLive && running.Phase != delivery.PhaseRemoved && !held {
out.Neither[node] = h
}
}
continue
}
@@ -534,7 +603,7 @@ func (m Module) selfDerived(node string, before delivery.Want, had bool, next de
d, there := m.composition(node, next.FP)
if !there || d.Digest == "" {
out.Rules[node] = Holding{Rule: delivery.RuleComposed, Words: fmt.Sprintf("%s on %s was composed by the node while it was placed, "+
"and the controller has not composed it at %s, so its want stays", m.Name, node, short(next.FP))}
"and no composition of its own is named at %s now, so its want stays", m.Name, node, short(next.FP))}
return next, false
}
at := m
@@ -658,10 +727,10 @@ func short(fp string) string {
return part[:12]
}
// mustAsk says whether the move to the composition d on node waits: for the operator's yes (need names the ask) or, for the bus,
// for the operator's hand (ADR 0539 §14, §15). A module the node never ran (it has no want and no have), a move judged neither, a module that is not of
// policy record and a want that stays are written as any want; so is a revision of the bus module's composition at the
// same fingerprint judged neither (a user list only, §7).
// mustAsk says whether the move to the composition d on node waits for the operator's yes (need names the ask, ADR 0539 §14). A
// module the node never ran (it has no want and no have), a move judged neither, a module that is not of policy record and a want
// that stays are written as any want. The bus module is no exception: it moves by the rules (ADR 0601 §1), its node backing up its
// streams before it applies (§2); the assigner asks no snapshot for it.
func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, running string, haveNamesNoComposition bool, d delivery.ComposedDigest) (*RecordNeed, bool) {
if (!had && !ran) || before.Removed || (before.FP == m.FP && before.Composed == d.Digest) {
return nil, false
@@ -673,9 +742,6 @@ func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, runnin
if !had && ran && running == m.FP && haveNamesNoComposition && d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
if _, barred := m.busBar(before, had, running); barred {
return nil, true // the bus is never moved by the assigner; the operator writes its want (§15)
}
if !m.recorded(d) || d.Judgement == delivery.JudgedNeither || d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
@@ -685,17 +751,7 @@ func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, runnin
return &RecordNeed{Node: node, Digest: d.Digest, Judgement: d.Judgement, Words: d.Words}, true
}
// busBar says whether m is the bus and the move would take it off the fingerprint it ran before, prev: its want's, or with no
// readable want the one its have names (ADR 0539 §15 bars a new one).
func (m Module) busBar(before delivery.Want, had bool, running string) (prev string, barred bool) {
prev = before.FP
if !had {
prev = running
}
return prev, catalogue.ProvidesBus(m.Manifest) && prev != m.FP
}
// recorded says whether the module is of policy record: a person's choice at the assigner over everything but the bus (ADR
// recorded says whether the module is of policy record: a person's choice at the assigner over the manifest's word (ADR
// 0565 §7, catalogue.Upgrade); for a module with none, as the controller holds it beside the composition's digest until the
// lease write (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it.
func (m Module) recorded(d delivery.ComposedDigest) bool {
@@ -732,8 +788,8 @@ func (m Module) userListRevision(node string, before delivery.Want, had bool, n
return w, true
}
// GrantNamesNode says whether a grant of module names its accounts on node (mesh-sdk's ControllerGrant.NamesNode, which a
// GrantNamesNode says whether a grant of module names its accounts on node (mesh-sdk's Grant.NamesNode, which a
// put reads by too).
func GrantNamesNode(g delivery.ControllerGrant, node, module string) bool {
func GrantNamesNode(g delivery.Grant, node, module string) bool {
return g.NamesNode(node, module)
}
@@ -9,6 +9,8 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
"git.novox.be/novox/mesh-delivery/sdk/graph"
)
func fpOf(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
@@ -104,6 +106,123 @@ func TestEachRuleHoldsTheWantBack(t *testing.T) {
t.Errorf("provided provisions held g14's want: %+v (%s)", out.Wants["g14"], out.Rules["g14"].Words)
}
}
// ADR 0614 §4: a want is held on a node whose engine does not state a reader capability the manifest needs, under rule 7
// with the capability as its words, and moves once the engine states it; an engine stating nothing reads the baseline,
// so a manifest of the baseline's words, the node-engine's own new build among them, is not held.
hostNeeds, err := catalogue.ManifestNeeds([]byte(`{"module":"mesh-host","version":"1","slug":"host","build":{"artifacts":[` +
`{"name":"host-arch","kind":"bundle","language":"go","system":"arch","from":"cmd/mesh-host"}]},"consumes":["restic.backed-up"],` +
`"state":[{"name":"health"}],"config":{"x":{"type":"number","minimum":1,"default":50,"runtime":true}},` +
`"resources":[{"id":"k","type":"file","path":"/var/lib/k","content":"${config:x}\n"}]}`))
if err != nil {
t.Fatal(err)
}
for _, c := range []struct {
why string
name string
needs []string
reads map[string][]string
held bool
// forms are the build's reference forms (ADR 0624 §2), and names what the hold's words name.
forms []string
names string
}{
{"a word the baseline lacks, on engines stating nothing", "mail", []string{"word:module", "word:tool-runner"}, nil, true, nil, "word:tool-runner"},
{"the same, once both engines state it", "mail", []string{"word:module", "word:tool-runner"},
map[string][]string{"ace": catalogue.ManifestReads(), "g14": catalogue.ManifestReads()}, false, nil, ""},
{"the baseline's words on engines stating nothing", "mail", []string{"word:claims.serves", "seat:mesh-catalog", "form:config"}, nil, false, nil, ""},
{"the node-engine's new build, whose contract has more words than its manifest uses", "mesh-host", hostNeeds, nil, false, nil, ""},
// ADR 0624 §2, with form:new a form no engine reads unless it states it and artifact-store the first form.
{"g14 back stating {old}, a build carrying {old,new}: moves", "mail", nil, map[string][]string{"g14": {"word:module"}}, false,
[]string{"form:artifact-store", "form:new"}, ""},
{"an engine stating nothing, a build carrying the baseline's form: moves", "mail", nil, nil, false, []string{"form:artifact-store"}, ""},
{"a build carrying {new} only: held, naming the form", "mail", nil, map[string][]string{"g14": {"word:module"}}, true,
[]string{"form:new"}, "form:new"},
{"the same, once g14's engine states it", "mail", nil, map[string][]string{"g14": {"form:new"}}, false, []string{"form:new"}, ""},
{"a manifest needing an unstated word, its build in the first form: held, naming the word", "mail", []string{"word:tool-runner"}, nil, true,
[]string{"form:artifact-store"}, "word:tool-runner"},
} {
m, n := ready()
m.Name, m.Needs, n.Reads, m.FormNeeds = c.name, c.needs, c.reads, c.forms
n.Haves["ace"][c.name], n.Haves["g14"][c.name] = n.Haves["ace"]["mail"], n.Haves["g14"]["mail"]
out := Wants(m, n, wantedOld(), nil)
if moved := out.Wants["g14"].FP == fNew; moved == c.held {
t.Errorf("%s: g14's want %+v, held by %+v", c.why, out.Wants["g14"], out.Rules["g14"])
}
if r := out.Rules["g14"]; c.held && (r.Rule != delivery.RuleComposed || !strings.Contains(r.Words, c.names) ||
!strings.Contains(r.Words, "node-engine of g14")) {
t.Errorf("%s: held by %+v, want rule 7 naming the capability", c.why, r)
}
if w := out.Wants["g14"]; c.held && (w.Rule != delivery.RuleComposed || w.Newest != fNew) {
t.Errorf("%s: g14's want %+v, want it kept at what it runs and naming rule 7", c.why, w)
}
}
// ADR 0614 §2, §3: a want waits, under the provisions' rule, until each verb it invokes is served by a running build,
// and moves once one is; a verb a running build serves already adds no wait, and an edge the graph leaves out on a cycle
// is not among Invokes, so it holds nothing; nothing read of what runs waits for none.
for _, c := range []struct {
name string
invokes []string
served []string
nilRead bool
holds string
}{
{"a verb no running build serves", []string{"store.store_put", "seat:mesh-catalog.modules"}, []string{"store.store_put"}, false, "seat:mesh-catalog.modules"},
{"once a build serving it runs", []string{"seat:mesh-catalog.modules"}, []string{"seat:mesh-catalog.modules"}, false, ""},
{"an edge left out on a cycle", nil, nil, false, ""},
{"nothing read of what runs", []string{"seat:mesh-catalog.modules"}, nil, true, ""},
// ADR 0614 §5: a module asking bus state is held while no build offering bus-state runs, and moves once one does.
{"bus state no running build offers", nil, nil, false, graph.BusState},
{"bus state a running build offers", nil, []string{graph.BusState}, false, ""},
{"bus state its own module offers", nil, nil, false, ""},
} {
m, n := ready()
m.Invokes = c.invokes
if !c.nilRead {
m.Served = func(v string) bool { return slices.Contains(c.served, v) }
}
if strings.HasPrefix(c.name, "bus state") {
m.Manifest.BusState = []catalogue.BusStateEntry{{Name: "cache", Storage: "key-value"}}
m.Offering = func(p string) bool { return slices.Contains(c.served, p) }
if c.name == "bus state its own module offers" {
m.Manifest.Provides = []catalogue.Offer{{Name: graph.BusState, Scope: catalogue.ScopeMesh}}
}
}
words := graph.VerbWaitWords(c.holds, "mail", "g14")
if c.holds == graph.BusState {
words = graph.BusStateWaitWords("mail", "g14")
}
out := Wants(m, n, wantedOld(), nil)
if c.holds == "" {
if out.Wants["g14"].FP != fNew {
t.Errorf("%s: held %+v", c.name, out.Rules["g14"])
}
continue
}
if out.Wants["g14"].FP == fNew || out.Rules["g14"].Rule != delivery.RuleProvisions || out.Rules["g14"].Words != words {
t.Errorf("%s: want %+v, held by %+v", c.name, out.Wants["g14"], out.Rules["g14"])
}
}
// ADR 0603 §9: the newest source revision governs. The module now at fNew no longer requires the database an older revision
// asked for and requires oidc instead; the seat's keys follow the newest manifest, so the older edge is no wait: a provided
// oidc moves the want, an asked one holds it and names oidc and never database.
for _, c := range []struct {
name string
state, older string
moves bool
}{{"the replaced edge provided, the older edge retiring", provisions.Provided, provisions.Retiring, true},
{"the replaced edge provided, the older edge asked", provisions.Provided, provisions.Asked, true},
{"the replaced edge asked, the older edge retiring", provisions.Asked, provisions.Retiring, false}} {
m, n := ready()
m.Manifest.Requires = []string{"oidc"}
m.Provisions = func(node string) map[string]provisions.Provision {
return map[string]provisions.Provision{node + ".mail.oidc": {Provision: "oidc", State: c.state},
node + ".mail.database": {Provision: "database", State: c.older, Why: "the operator's answer"}}
}
out := Wants(m, n, wantedOld(), nil)
if got := out.Wants["g14"].FP == fNew; got != c.moves || (!c.moves && (!strings.Contains(out.Rules["g14"].Words, "oidc") || strings.Contains(out.Rules["g14"].Words, "database"))) {
t.Errorf("%s: want %+v, held by %q", c.name, out.Wants["g14"], out.Rules["g14"].Words)
}
}
// 1954/D1 (the lead, 2026-10-29): a required provision with no key yet holds the want, named; and only one the
// composition keys (catalogue.Awaited, issue 774), replaying 2026-10-30: grafana (not handed) waited on ace for
// oidc-client, which the controller still makes, and mesh-build-go (handed) for go-toolchain on its own machine.
@@ -155,6 +274,24 @@ func TestEachRuleHoldsTheWantBack(t *testing.T) {
}
}
}
// ADR 0603 §5: a requirement nobody offers is an unknown provider and waits, loudly; one something offers (a module's own
// name, a module the mesh provides itself) is left to catalogue.Awaited as before and does not.
for _, c := range []struct {
name, requires string
offered bool
}{{"a module's own name (node-tools)", "node-tools", true}, {"a module the mesh provides itself (mesh-addressing)", "mesh-addressing", true},
{"a provision nobody offers", "database", false}} {
m, n := ready()
m.Manifest.Requires = []string{c.requires}
m.Answering = func(string) []catalogue.Manifest { return nil }
m.Offered = func(p string) bool { return c.offered && p == c.requires }
m.Provisions = func(string) map[string]provisions.Provision { return map[string]provisions.Provision{} }
out := Wants(m, n, wantedOld(), nil)
if got := out.Wants["g14"].FP == fNew; got != c.offered || (!c.offered && (out.Rules["g14"].Rule != delivery.RuleProvisions ||
!strings.Contains(out.Rules["g14"].Words, "no module offers database yet") || !strings.Contains(out.Rules["g14"].Words, "past its bound"))) {
t.Errorf("%s: want %+v, held by %+v", c.name, out.Wants["g14"], out.Rules["g14"])
}
}
// 1960: rule 5 holds no want, and no module is left without a first want for a provider down or a node offline; the
// provider down is said (Unreached). novox on 2026-10-29: route-proxy down held de-spiegel and link2pay, minio not
// healthy held photos, and novox offline left a module with no canary.
@@ -293,7 +430,7 @@ func TestEachRuleHoldsTheWantBack(t *testing.T) {
// Rule 6 per node: a grant written for the module's other nodes does not grant a node whose account it does not name
// (issue 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go was wanted with no account on the bus).
m, n := ready()
grant := delivery.ControllerGrant{Accounts: []string{"ace.mail", "ace.mail.send"}}
grant := delivery.Grant{Accounts: []string{"ace.mail", "ace.mail.send"}}
m.GrantedOn = func(node string) bool { return GrantNamesNode(grant, node, "mail") }
out := Wants(m, n, wantedOld(), nil)
if w := out.Wants["ace"]; w.FP != fNew {
@@ -312,7 +449,7 @@ func TestEachRuleHoldsTheWantBack(t *testing.T) {
{[]string{"g14.mailer"}, "g14", "mail", false},
{[]string{"ace.mail"}, "g14", "mail", false},
} {
if got := GrantNamesNode(delivery.ControllerGrant{Accounts: c.accounts}, c.node, c.module); got != c.named {
if got := GrantNamesNode(delivery.Grant{Accounts: c.accounts}, c.node, c.module); got != c.named {
t.Errorf("%v names %s of %s: %v, want %v", c.accounts, c.module, c.node, got, c.named)
}
}
@@ -639,11 +776,13 @@ func TestAMoveJudgedNeitherOrNeverRunOrNotRecordedIsWrittenAtOnce(t *testing.T)
// 1906/D4 (ADR 0539 §7, §15): the bus module's new fingerprint is never written by the assigner, but a revision of its
// composition at the same fingerprint that only changes the user list (judged neither) is written at once.
func TestTheBusIsNotMovedButItsUserListRevisionIsWrittenAtOnce(t *testing.T) {
func TestTheBusMovesByTheRulesAndItsUserListRevisionIsWrittenAtOnce(t *testing.T) {
m, n, _ := movingTo(delivery.JudgedNeither)
m.Manifest.Provides = []catalogue.Offer{{Name: "mesh-bus"}}
if out := Wants(m, n, wantedOld(), nil); out.Wants["ace"].FP == fNew {
t.Errorf("the bus moved to a new fingerprint: %+v", out.Wants["ace"])
// ADR 0601 §1, §2: the bus module's want is written by the rules, as any module's, and no snapshot is asked for it: its node
// backs its streams up before it applies.
if out := Wants(m, n, wantedOld(), nil); out.Wants["ace"].FP != fNew || len(out.Record) != 0 || len(out.Neither) != 0 {
t.Errorf("the bus did not move by the rules: %+v", out)
}
running := map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fOld)}, "g14": {FP: fNew, Composed: dig(fOld)}}
out := Wants(m, n, running, nil)
@@ -696,23 +835,6 @@ func TestTheBusIsNotMovedButItsUserListRevisionIsWrittenAtOnce(t *testing.T) {
t.Errorf("a held bus did not take its user-list revision as a put: %+v", w)
}
m.Puts = nil
// 1945 (the bus bar made loud): a node that runs the bus at an old-form build and holds no want for it gets none, since only
// the operator's put moves the bus, and the pass says so: a holding naming the bar, and the node on neither list so its
// condition stays raised. A node with a live want keeps it and is not on neither list.
m.Composed = composedEverywhere
n.Haves["ace"]["mail"] = delivery.Have{FP: "b-1700000000-oldform", Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
out = Wants(m, n, map[string]delivery.Want{"g14": {FP: fOld, Composed: dig(fOld)}}, nil)
if w, wrote := out.Wants["ace"]; wrote {
t.Errorf("the bus bar wrote ace's want %+v", w)
}
h, on := out.Neither["ace"]
if !on || !h.Bus || out.Rules["ace"] != h || !strings.Contains(h.Words, "only your put moves the bus") || !strings.Contains(h.Words, "ace runs b-1700000000-oldform") ||
!strings.Contains(h.Words, "assigner_put") {
t.Errorf("the barred bus on ace: neither %+v (%v), rules %+v", h, on, out.Rules["ace"])
}
if _, on := out.Neither["g14"]; on || out.Wants["g14"].FP == fNew {
t.Errorf("g14 has a live want: neither %+v, its want %+v", out.Neither, out.Wants["g14"])
}
stepsOfAPut(t)
}
@@ -815,6 +937,13 @@ func TestTheBackupAnswerStaysInTheWantWhileItsTargetStays(t *testing.T) {
if w, ok := Wants(m, n, running, nil).Wants["ace"]; ok {
t.Errorf("the answer was erased: %+v", w)
}
// A placed node composes its own: its want names the build alone, and the answer stays in it all the same (the digest its
// have names is not in the want, so a want kept for its digest would erase the answer at every pass).
m.Self = func(string) bool { return true }
placed := map[string]delivery.Want{"ace": {FP: fNew, Self: true, Backup: delivery.BackupSkip, Words: BackupAnswerWords("skip", 0)}}
if w, ok := Wants(m, n, placed, nil).Wants["ace"]; ok {
t.Errorf("the answer in a placed node's want was erased: %+v", w)
}
}
// 1906/D4 (ADR 0539 §19): a hold released by the operator is not held again by a health check that failed before, and the
@@ -882,8 +1011,8 @@ func TestTheRecordPolicyIsTheOneTheControllerHolds(t *testing.T) {
}
}
// Review round 1, item 2 (ADR 0539 §15): a node that runs the bus with no readable want compares the new fingerprint with the one
// it runs, from its have: the same fingerprint is not a new one (written by the rules), another is barred.
// A node that runs the bus with no readable want gets its want by the rules, at the fingerprint it runs or at a new one alike: the
// bus has no bar of its own (ADR 0601 §1).
func TestTheBusWithNoWantIsComparedWithWhatItRuns(t *testing.T) {
m, n, _ := movingTo(delivery.JudgedNeither)
m.Manifest.Provides = []catalogue.Offer{{Name: "mesh-bus"}}
@@ -892,8 +1021,8 @@ func TestTheBusWithNoWantIsComparedWithWhatItRuns(t *testing.T) {
t.Errorf("the bus runs F with no want and was not given one: %+v %v", w, ok)
}
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
if w, ok := Wants(m, n, nil, nil).Wants["ace"]; ok {
t.Errorf("the bus was moved to a new fingerprint with no want: %+v", w)
if w, ok := Wants(m, n, nil, nil).Wants["ace"]; !ok || w.FP != fNew {
t.Errorf("the bus running an older fingerprint with no want was not given its want by the rules: %+v %v", w, ok)
}
}
@@ -0,0 +1,332 @@
// Package judge is how the seat mesh-placement's holder judges a proposal (novox/hq ADR 0567 §3, §4, as ADR 0593 §8
// changes them): what it changes on each node by its placement (Intended), every node answering it, and the verdict of
// their answers (JudgeAnswers). The assigner's own (ADR 0593 §1), moved back from mesh-sdk's compose/judge: the holder
// composes nothing, since every composition is its node's engine's, and a node answers through the candidates path
// with what it composed. It holds no bus client and no timer: the holder gives it what it read.
package judge
import (
"encoding/json"
"fmt"
"reflect"
"sort"
"strconv"
"strings"
"time"
nec "git.novox.be/novox/mesh-contracts/go/clients/nodeengineclient"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// AnswerBound is how long the holder waits for a node's answer after its candidate was written: the time to live of the
// node's pulse key (delivery.PulseAnswerKey), whose expiry names it as giving no answer, never passed. The same answer
// bound a check gives a node (ADR 0593 §8, §7: 60 seconds, nec.AnswerBound). A variable so a test on the bus
// can shorten it.
var AnswerBound = nec.AnswerBound
// Intended is, by node, the modules whose placed entry the proposal changes there, and the nodes that answer it: every
// node (ADR 0593 §8, changing ADR 0567 §4's list of the nodes that must answer), sorted. current is the placement as it
// stands, laid the proposal's placements by node (nil removes the node's key) and nodes every node with a record. A
// node's set holds the modules the proposal adds, removes or whose placed entry differs on that node; a node with no
// key now gets an empty set (§7: the seed is answered as the same). What a node's compositions change beyond its own
// placement it says itself in its answer, marked in the brief as not computed.
func Intended(current map[string]delivery.Placement, laid map[string]*delivery.Placement, nodes []string) (intended map[string]map[string]bool, must []string) {
intended = map[string]map[string]bool{}
all := map[string]bool{}
for _, n := range nodes {
all[n] = true
}
for n := range laid {
all[n] = true
}
for node := range all {
must = append(must, node)
set := map[string]bool{}
intended[node] = set
before, placed := current[node]
after, changesKey := laid[node]
if !changesKey || !placed {
continue // untouched, or new to placement: every answer must be the same
}
var afterModules []delivery.PlacedModule
if after != nil {
afterModules = after.Modules
}
entries := map[string]delivery.PlacedModule{}
for _, m := range before.Modules {
entries[m.Module] = m
}
seen := map[string]bool{}
for _, m := range afterModules {
seen[m.Module] = true
if e, ok := entries[m.Module]; !ok || !reflect.DeepEqual(e, m) {
set[m.Module] = true
}
}
for name := range entries {
if !seen[name] {
set[name] = true
}
}
}
sort.Strings(must)
return intended, must
}
// JudgeAnswers is the verdict of a proposal's answers (§4). answers is by node its HaveNode.Candidates entry for the
// proposal (nil when it has none) and revisions by node the revision of its candidate key: an answer counts when it
// names the proposal and that revision. It fails on a refusal by a node that must answer; a node of Must with no answer
// that counts is waited for while its pulse key stands, and once it has ended (pulsing false: the bound's expiry, heard
// on the bus) is named as giving no answer and the proposal decided on the answers it has (ADR 0593 §8).
// A difference fails nothing: one Intended computed is listed under Intended, one it did not under Unintended, both
// shown in the ask's brief. The seed alone fails on any difference.
func JudgeAnswers(p delivery.PlacementProposal, intended map[string]map[string]bool, answers map[string]*delivery.CandidateAnswer,
revisions map[string]uint64, online map[string]bool, pulsing map[string]bool, now time.Time) delivery.PlacementVerdict {
v := delivery.PlacementVerdict{At: now.UTC(), Intended: map[string][]string{}, Unintended: map[string][]string{}, Carried: map[string][]string{},
Answers: map[string]delivery.CandidateAnswer{}}
for node, set := range intended {
for m := range set {
v.Intended[node] = append(v.Intended[node], m)
}
sort.Strings(v.Intended[node])
}
must := p.Must
if len(must) == 0 {
must = p.Nodes()
}
for _, node := range must {
a := answers[node]
if a == nil || a.Proposal != p.ID || a.Revision == 0 || a.Revision != revisions[node] {
v.Silent = append(v.Silent, node)
continue
}
v.Answers[node] = *a
if a.Refused != "" {
v.Fails = append(v.Fails, fmt.Sprintf("%s cannot compose the proposal: %s", node, a.Refused))
continue
}
for _, r := range a.Refusals {
v.Fails = append(v.Fails, fmt.Sprintf("%s refuses the proposal for %s: %s", node, r.Module, r.Why))
}
want := intended[node]
differs := func(module, words string) {
if p.Seed {
// The seed carries what its nodes found, each named in the brief for the operator who approves it.
v.Carried[node] = append(v.Carried[node], words)
return
}
if !want[module] {
v.Unintended[node] = append(v.Unintended[node], module)
}
}
for _, d := range a.Differs {
differs(d.Module, differenceWords(d))
}
for _, d := range a.Added {
differs(d.Module, "would run "+d.Module)
}
for _, m := range a.Removed {
differs(m, "would stop running "+m)
}
sort.Strings(v.Unintended[node])
}
if len(v.Intended) == 0 {
v.Intended = nil
}
if len(v.Unintended) == 0 {
v.Unintended = nil
}
if len(v.Carried) == 0 {
v.Carried = nil
}
if len(v.Answers) == 0 {
v.Answers = nil
}
var waiting, gone []string
for _, node := range v.Silent {
named := node
if online != nil && !online[node] {
named += " (offline)"
}
if pulsing[node] {
waiting = append(waiting, named)
} else {
gone = append(gone, named)
}
}
if len(waiting) > 0 {
v.Notes = append(v.Notes, fmt.Sprintf("waits for the answers of %s, each for %s from its candidate's write",
strings.Join(waiting, ", "), AnswerBound))
}
if len(gone) > 0 {
// ADR 0593 §8: decided on the answers it has; that node's own composition judges again when it applies (§9).
v.Notes = append(v.Notes, fmt.Sprintf("no answer from %s within %s of its candidate's write: decided on the answers "+
"it has, and that node's own composition judges again when it applies", strings.Join(gone, ", "), AnswerBound))
}
switch {
case len(v.Fails) > 0:
v.State = delivery.VerdictFail
case len(waiting) > 0:
v.State = delivery.VerdictWaits
default:
v.State = delivery.VerdictPass
}
return v
}
// Brief is the verdict as the ask's brief shows it, by node (§5): what the proposal changes and what each node answered
// beside it, marked, and what stays held as the node holds it today, with the proposal's why.
func Brief(p delivery.PlacementProposal, v delivery.PlacementVerdict) string {
var b strings.Builder
fmt.Fprintf(&b, "%s (proposed by %s)", p.Why, p.By)
nodes := map[string]bool{}
for n := range v.Intended {
nodes[n] = true
}
for n := range v.Unintended {
nodes[n] = true
}
for _, n := range p.Nodes() {
nodes[n] = true
}
for n, a := range v.Answers {
if len(a.Held) > 0 {
nodes[n] = true
}
}
for n := range v.Carried {
nodes[n] = true
}
names := make([]string, 0, len(nodes))
for n := range nodes {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
fmt.Fprintf(&b, "; %s:", n)
if m := v.Intended[n]; len(m) > 0 {
fmt.Fprintf(&b, " changes %s", strings.Join(m, ", "))
} else {
b.WriteString(" composes the same")
}
if m := v.Unintended[n]; len(m) > 0 {
fmt.Fprintf(&b, "; also differs, not computed: %s", strings.Join(m, ", "))
}
if m := v.Carried[n]; len(m) > 0 {
fmt.Fprintf(&b, "; the seed carries: %s", strings.Join(m, "; "))
}
if held := v.Answers[n].Held; len(held) > 0 {
var names []string
for _, h := range held {
today := h.Today
if today == "" {
today = "held"
}
names = append(names, h.Module+" ("+today+")")
}
fmt.Fprintf(&b, "; stays as today: %s", strings.Join(names, ", "))
}
}
return b.String()
}
// differenceWords names one module's difference: the module, the first field that differs in it with its value as it
// runs and as the seed composes it, each cut to about 60 characters ("(value cut)" where a body shown is cut before the
// field), and how many fields more; or the resource and why when no body is shown.
func differenceWords(d delivery.ModuleDiffers) string {
for _, b := range d.Bodies {
if len(b.Paths) == 0 {
continue
}
path := b.Paths[0]
words := fmt.Sprintf("%s: %s %s → %s", d.Module, fieldName(b.Running, b.Composed, path),
valueAt(b.Running, path), valueAt(b.Composed, path))
if n := len(b.Paths) - 1; n > 0 {
words += fmt.Sprintf(" (and %d more fields)", n)
}
return words
}
if d.Resource != "" {
return fmt.Sprintf("%s: %s (%s)", d.Module, d.Resource, d.Why)
}
return fmt.Sprintf("%s: %s", d.Module, d.Why)
}
// shownValue bounds a value named in a brief.
const shownValue = 60
// walk follows a differing path (delivery.DifferingPaths: keys joined by dots, a key may hold dots, and a JSON value
// written in a string is read as JSON) through a body, answering the keys it passed and the value at its end.
func walk(body, path string) (keys []string, value any, ok bool) {
var v any
if json.Unmarshal([]byte(body), &v) != nil {
return nil, nil, false
}
rest := path
for rest != "" {
if s, isString := v.(string); isString {
var inner any
if json.Unmarshal([]byte(s), &inner) != nil {
return nil, nil, false
}
v = inner
}
switch t := v.(type) {
case map[string]any:
best := ""
for k := range t {
if (rest == k || strings.HasPrefix(rest, k+".")) && len(k) > len(best) {
best = k
}
}
if best == "" {
return nil, nil, false
}
keys, v, rest = append(keys, best), t[best], strings.TrimPrefix(strings.TrimPrefix(rest, best), ".")
case []any:
seg, after, _ := strings.Cut(rest, ".")
n, err := strconv.Atoi(seg)
if err != nil || n < 0 || n >= len(t) {
return nil, nil, false
}
keys, v, rest = append(keys, seg), t[n], after
default:
return nil, nil, false
}
}
return keys, v, true
}
// valueAt is the value at path in a shown body, cut to shownValue, or why it is not shown.
func valueAt(body, path string) string {
if body == "" {
return "(none)"
}
_, v, ok := walk(body, path)
if !ok {
return "(value cut)"
}
s, isString := v.(string)
if !isString {
raw, _ := json.Marshal(v)
s = string(raw)
}
if r := []rune(s); len(r) > shownValue {
s = string(r[:shownValue-1]) + "…"
}
return s
}
// fieldName is the differing field as a person reads it: its parent's name and its own, `PARENT[key]`, from whichever
// body reaches it; the path's end when neither does.
func fieldName(running, composed, path string) string {
for _, body := range []string{running, composed} {
if keys, _, ok := walk(body, path); ok && len(keys) > 0 {
if len(keys) == 1 {
return keys[0]
}
return keys[len(keys)-2] + "[" + keys[len(keys)-1] + "]"
}
}
return path
}
@@ -0,0 +1,91 @@
package judge
import (
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ADR 0567 §3, §4 (rows moved from the assigner's check of a pull request's placement, ADR 0559 §5): what a proposal
// changes by node, and the verdict of the answers: differences shown and failing nothing, a refusal and a silent node
// failing, a later answer judging again, the seed failing on any difference.
func TestJudge(t *testing.T) {
current := map[string]delivery.Placement{"anchor": {Node: "anchor", Modules: []delivery.PlacedModule{{Module: "mail"}}}, "server": {Node: "server"}}
laid := map[string]*delivery.Placement{
"anchor": {Node: "anchor", Modules: []delivery.PlacedModule{{Module: "mail", Ports: map[string]int{"25": 2525}}, {Module: "web"}}},
"laptop": {Node: "laptop", Modules: []delivery.PlacedModule{{Module: "web"}}}}
// ADR 0593 §8: every node answers; what the placement changes is read from the placement alone.
got, must := Intended(current, laid, []string{"anchor", "laptop", "server"})
if !got["anchor"]["mail"] || !got["anchor"]["web"] || len(got["laptop"]) != 0 || len(got["server"]) != 0 ||
!slices.Equal(must, []string{"anchor", "laptop", "server"}) {
t.Errorf("intended = %v, must %v: want anchor's mail and web, nothing elsewhere, every node answering", got, must)
}
at := time.Date(2026, 10, 27, 2, 0, 0, 0, time.UTC)
p := delivery.PlacementProposal{ID: "p-one", By: "op", Why: "move web", Written: at, Base: map[string]uint64{"anchor": 3, "laptop": 0},
Must: []string{"anchor", "laptop"}}
revisions := map[string]uint64{"anchor": 7, "laptop": 8}
intended := map[string]map[string]bool{"anchor": {"web": true}}
same := func(rev uint64) *delivery.CandidateAnswer {
return &delivery.CandidateAnswer{Proposal: "p-one", Revision: rev, Verdict: delivery.CandidateSame, Same: []string{"mail"}}
}
differs := func(rev uint64, module string) *delivery.CandidateAnswer {
return &delivery.CandidateAnswer{Proposal: "p-one", Revision: rev, Verdict: delivery.CandidateDiffers,
Differs: []delivery.ModuleDiffers{{Module: module, Why: "the port differs"}}}
}
for _, row := range []struct {
name string
seed bool
answers map[string]*delivery.CandidateAnswer
pulsing map[string]bool
state string
words string
marked string
}{
{"all the same", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": same(8)}, nil, delivery.VerdictPass, "", ""},
{"a module added where intended", false, map[string]*delivery.CandidateAnswer{"anchor": {Proposal: "p-one", Revision: 7, Verdict: delivery.CandidateDiffers,
Added: []delivery.ModuleAdded{{Module: "web", Unissued: []string{"its own secret"}}}}, "laptop": same(8)}, nil, delivery.VerdictPass, "", ""},
{"a difference not computed passes, marked", false, map[string]*delivery.CandidateAnswer{"anchor": differs(7, "mail"), "laptop": same(8)}, nil, delivery.VerdictPass, "", "mail"},
{"the seed carries a difference, named", true, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": {Proposal: "p-one", Revision: 8,
Verdict: delivery.CandidateDiffers, Differs: []delivery.ModuleDiffers{{Module: "node-tools", Resource: "node-tools.runtime", Why: "the resource differs",
Bodies: []delivery.ResourceBodies{{ID: "node-tools.runtime", Paths: []string{"env.MESH_OWNED_PATHS./etc/resolv.conf", "env.MESH_TOOL_MODULES"},
Running: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/hosts\":\"hostname\",\"/etc/resolv.conf\":\"networkmanager (node-uplink)\"}"}}`,
Composed: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/hosts\":\"hostname\",\"/etc/resolv.conf\":\"systemd-resolved (node-resolver)\"}","MESH_TOOL_MODULES":"… (cut at 8192 bytes)`}}}}}}, nil, delivery.VerdictPass, "", ""},
{"a refusal", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": {Proposal: "p-one", Revision: 8, Verdict: delivery.CandidateRefused,
Refusals: []delivery.ModuleRefused{{Module: "web", Why: "no own secret"}}}}, nil, delivery.VerdictFail, "laptop refuses", ""},
{"a node not answering, within its bound", false, map[string]*delivery.CandidateAnswer{"anchor": same(7)}, map[string]bool{"laptop": true}, delivery.VerdictWaits, "laptop (offline)", ""},
{"an answer of an earlier revision, past its bound, is no answer and decided without it", false, map[string]*delivery.CandidateAnswer{"anchor": same(6), "laptop": same(8)}, nil, delivery.VerdictPass, "no answer from anchor", ""},
{"a module held as today passes, shown in the brief", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": {Proposal: "p-one", Revision: 8,
Verdict: delivery.CandidateSame, Held: []delivery.ModuleRefused{{Module: "spiegel", Why: "a required value is unset"}}}}, nil, delivery.VerdictPass, "", ""},
{"its later answer", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": same(8)}, nil, delivery.VerdictPass, "", ""},
} {
q := p
q.Seed = row.seed
v := JudgeAnswers(q, intended, row.answers, revisions, map[string]bool{"anchor": true}, row.pulsing, at)
all := strings.Join(append(append([]string{}, v.Fails...), v.Notes...), "; ")
if v.State != row.state || !strings.Contains(all, row.words) {
t.Errorf("%s: %s, %q", row.name, v.State, all)
}
if row.seed && row.state == delivery.VerdictPass && !strings.Contains(Brief(q, v), "the seed carries: node-tools: MESH_OWNED_PATHS[/etc/resolv.conf] networkmanager (node-uplink) → (value cut) (and 1 more fields)") {
t.Errorf("%s: the brief does not name it: %s", row.name, Brief(q, v))
}
if row.seed && row.state == delivery.VerdictPass {
whole := row.answers["laptop"].Differs[0]
whole.Bodies = []delivery.ResourceBodies{{ID: "r", Paths: []string{"env.MESH_OWNED_PATHS./etc/resolv.conf"},
Running: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/resolv.conf\":\"networkmanager (node-uplink)\"}"}}`,
Composed: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/resolv.conf\":\"systemd-resolved (node-resolver)\"}"}}`}}
if got := differenceWords(whole); got != "node-tools: MESH_OWNED_PATHS[/etc/resolv.conf] networkmanager (node-uplink) → systemd-resolved (node-resolver)" {
t.Errorf("the difference is named %q", got)
}
}
if strings.HasPrefix(row.name, "a module held as today") && !strings.Contains(Brief(q, v), "stays as today: spiegel (held)") {
t.Errorf("%s: the brief does not show it: %s", row.name, Brief(q, v))
}
if row.marked != "" && (!slices.Contains(v.Unintended["anchor"], row.marked) || !strings.Contains(Brief(q, v), "not computed: "+row.marked)) {
t.Errorf("%s: the difference is not marked: %+v, %s", row.name, v.Unintended, Brief(q, v))
}
}
}
+52 -20
View File
@@ -8,6 +8,8 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/views"
)
func load(t *testing.T) catalogue.Manifest {
@@ -33,9 +35,13 @@ func TestManifest(t *testing.T) {
}
reads := slices.Clone(m.Reads)
slices.Sort(reads)
want := []string{"mesh-build-go.steps", "mesh-conditions.takeover", "mesh-config.kept", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.composed", "mesh-controller.lease",
"mesh-delivery-builder.build-log", "mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-grants.grants", "mesh-provisioner.provisions", "node-engine.have"}
// ADR 0593 §8: it composes nothing, so it reads no candidate, held or resolved value, compose step or build log; it reads
// every node's published compositions to compare them, and the gate's source and merges, for the release that carries a build (ADR 0601 §1).
// DLV-F2r (ADR 0615 §1, ADR 0631 §1): the indexer's manifests, modules and module graph (whose verb edges a want waits
// on, ADR 0614 §2, §3) are read through its first views and views family, none of the indexer's states by reads.
want := []string{"mesh-delivery-builder.builds", "mesh-delivery-gate.merges", "mesh-delivery-gate.source",
"mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-grants.grants", "mesh-host.composed",
"mesh-provisioner.provisions", "node-engine.have"}
if !reflect.DeepEqual(reads, want) {
t.Errorf("reads = %v, want exactly %v", reads, want)
}
@@ -43,8 +49,9 @@ func TestManifest(t *testing.T) {
for _, s := range m.State {
states[s.Name] = s.History
}
if want := map[string]int{"asks": 5, "compositions": 5, "differences": 64, "puts": 20, "wants": 20,
"placement": 64, "proposals": 5, "candidates": 5, "pulses": 0, "findings": 5, "policies": 20, "seat-holders": 5}; !reflect.DeepEqual(states, want) {
if want := map[string]int{"asks": 5, "differences": 64, "puts": 20, "wants": 20,
"placement": 64, "proposals": 5, "candidates": 5, "pulses": 0, "findings": 5, "policies": 20, "seat-holders": 5, views.MarksState: 1, views.StatusState: 1,
"engine-reads": 1}; !reflect.DeepEqual(states, want) {
t.Errorf("state = %v, want %v", states, want)
}
if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 {
@@ -53,21 +60,21 @@ func TestManifest(t *testing.T) {
if m.SecretsOwner != "mesh-delivery-assigner" {
t.Errorf("secrets-owner = %q", m.SecretsOwner)
}
// 1906/D3: it says its conditions on the operator channel; 1622: it puts its compose asks on the Go build seat, as the
// checker puts its checks.
if !slices.Equal(m.Uses, []string{"node-build-go", "operator-channel"}) {
t.Errorf("uses = %v, want exactly node-build-go and operator-channel", m.Uses)
// 1906/D3: it says its conditions on the operator channel; it asks the Go build seat nothing since the composition is
// the node-engine's (ADR 0593 §7).
if !slices.Equal(m.Uses, []string{"operator-channel"}) {
t.Errorf("uses = %v, want exactly operator-channel", m.Uses)
}
// ADR 0546: it serves its own tools (put, release, puts) from its bundle through the tool runner (the artifact loads the
// module), says each put's step as the event put, and calls the hand-act seat and the bus node's backup holder.
// ADR 0567 §1, §6-§10: it holds the seat mesh-placement, serving its seven verbs; keeps the seat's events on its stream
// placement for 30 days; calls the controller's export, its four verbs that keep its assignments in step and its
// hand-act log; and reads mesh-config's resolved values to compose every node a proposal touches.
// ADR 0567 §1, §6-§10: it holds the seat mesh-placement, serving its five verbs (adopt and hand-back went with the
// controller, ADR 0590); keeps the seat's events on its stream placement for 30 days; calls the hand-act log; and reads
// mesh-config's resolved values to compose every node a proposal touches.
// ADR 0565 §7, §9 (issue 1953 D4): and the seat mesh-wants, serving upgrade and bus.
if len(m.Claims) != 2 || m.Claims[0].Name != catalogue.PlacementSeat || m.Claims[0].At() != catalogue.ScopeMesh ||
!slices.Equal(m.Claims[0].Serves, []string{"placement", "proposals", "propose", "withdraw", "again", "adopt", "hand-back"}) ||
!slices.Equal(m.Claims[0].Serves, []string{"placement", "proposals", "propose", "withdraw", "again"}) ||
m.Claims[1].Name != catalogue.WantsSeat || m.Claims[1].At() != catalogue.ScopeMesh || !slices.Equal(m.Claims[1].Serves, []string{"upgrade", "bus"}) {
t.Errorf("the assigner claims %+v, want mesh-placement with its seven verbs and mesh-wants with its two", m.Claims)
t.Errorf("the assigner claims %+v, want mesh-placement with its five verbs and mesh-wants with its two", m.Claims)
}
if seat, _ := catalogue.SeatNamed(catalogue.WantsSeat); catalogue.CanHold(m, seat) != nil {
t.Errorf("the assigner cannot hold mesh-wants: %v", catalogue.CanHold(m, seat))
@@ -75,17 +82,42 @@ func TestManifest(t *testing.T) {
if seat, _ := catalogue.SeatNamed(catalogue.PlacementSeat); catalogue.CanHold(m, seat) != nil {
t.Errorf("the assigner cannot hold the seat: %v", catalogue.CanHold(m, seat))
}
if !slices.Equal(m.Tools, []string{"assigner_put", "assigner_release", "assigner_puts"}) {
t.Errorf("the assigner serves tools %v, want its three put tools", m.Tools)
// ADR 0621 §1: the first-view tools placement, seat-holders and wants and design 25 § 12.2's six read tools, of the
// program's bundle, which runs in a runtime of its own on the assigner's account; ADR 0618 §3: and assigner_engine_reads,
// which lists and asks again what each node's engine reads.
if !slices.Equal(m.Tools, []string{"assigner_put", "assigner_release", "assigner_puts", "placement", "seat-holders", "wants",
"assigner_read_placement", "assigner_read_seat_holders", "assigner_read_wants", "assigner_read_findings", "assigner_read_asks", "assigner_read_differences",
"assigner_first_views", "assigner_first_view_again", "assigner_engine_reads"}) {
t.Errorf("the assigner serves tools %v, want its three put tools, three first views, six read tools and assigner_engine_reads", m.Tools)
}
if len(m.Streams) != 1 || m.Streams[0].Name != delivery.PlacementStream || m.Streams[0].MaxAgeDays != delivery.PlacementStreamDays ||
if m.RunsAs != m.Module || m.SecretsOwner != m.RunsAs {
t.Errorf("runs-as = %q: the bundle runs on the assigner's own account (ADR 0621 §1)", m.RunsAs)
}
// ADR 0621 §2: each first-view tool answers exactly its readers, never the assigner.
callers := map[string]string{"placement": "mesh-grants,node-tools,mesh-delivery-indexer", "seat-holders": "node-tools", "wants": "mesh-grants"}
env := m.Build.Artifacts[0].Env
for tool, list := range callers {
if env[views.CallersEnv(tool)] != list || views.ParseCallers(env[views.CallersEnv(tool)]).Allows("n1."+m.Module) {
t.Errorf("%s callers = %q, want %q", tool, env[views.CallersEnv(tool)], list)
}
}
// ADR 0619 §2, ADR 0615 §5: its views family beside placement on its old streams section.
if len(m.Streams) != 2 || m.Streams[1].Name != "assigner-views" || !slices.Equal(m.Streams[1].Events, []string{views.EventPattern}) ||
m.Streams[1].MaxAgeDays != views.StreamDays || m.Streams[1].MaxMiB != views.StreamMiB || m.Streams[1].Retention != "" {
t.Errorf("streams = %+v, want placement and assigner-views", m.Streams)
}
if m.Streams[0].Name != delivery.PlacementStream || m.Streams[0].MaxAgeDays != delivery.PlacementStreamDays ||
!slices.Equal(slices.Sorted(slices.Values(m.Streams[0].Events)), slices.Sorted(slices.Values(delivery.PlacementEvents))) {
t.Errorf("streams = %+v, want placement carrying the seat's four events for %d days", m.Streams, delivery.PlacementStreamDays)
}
if want := []string{"seat:mesh-controller.assign", "seat:mesh-controller.pin",
"seat:mesh-controller.placement-export", "seat:mesh-controller.unassign", "seat:mesh-controller.unpin",
// ADR 0618 §4: and it hears each node's engine say its reads changed.
if !slices.Equal(m.Consumes, []string{"mesh-delivery-indexer.views.graph", "mesh-delivery-indexer.views.manifests", "mesh-delivery-indexer.views.modules", "node-engine.manifest-reads-changed"}) {
t.Errorf("consumes = %v, want the indexer's views of graph, manifests and modules and the engines' reads changed", m.Consumes)
}
if want := []string{"mesh-delivery-indexer.graph", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules",
"seat:mesh-hand-acts.hand-act", "seat:mesh-hand-acts.hand-acts", "seat:mesh-hand-acts.warranted", // ADR 0565 §9, §11: the seat alone
"seat:node-backup.backed-up", "seat:node-backup.now"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) {
"seat:node-backup.backed-up", "seat:node-backup.now",
"seat:node-engine.manifest-reads"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) { // ADR 0618 §3
t.Errorf("invokes = %v, want %v", m.Invokes, want)
}
if !slices.Equal(m.Emits, []string{"put"}) {
+41 -26
View File
@@ -2,6 +2,7 @@
"module": "mesh-delivery-assigner",
"version": "1",
"slug": "assigner",
"runs-as": "mesh-delivery-assigner",
"claims": [
{
"name": "mesh-placement",
@@ -11,9 +12,7 @@
"proposals",
"propose",
"withdraw",
"again",
"adopt",
"hand-back"
"again"
]
},
{
@@ -26,39 +25,35 @@
}
],
"reads": [
"mesh-build-go.steps",
"mesh-conditions.takeover",
"mesh-config.kept",
"mesh-config.resolved",
"mesh-controller.assignments",
"mesh-controller.composed",
"mesh-controller.lease",
"mesh-delivery-builder.build-log",
"mesh-delivery-builder.builds",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
"mesh-delivery-gate.merges",
"mesh-delivery-gate.source",
"mesh-delivery-teller.answers",
"mesh-delivery-watchman.presence",
"mesh-grants.grants",
"mesh-host.composed",
"mesh-provisioner.provisions",
"node-engine.have"
],
"uses": [
"node-build-go",
"operator-channel"
],
"consumes": [
"mesh-delivery-indexer.views.graph",
"mesh-delivery-indexer.views.manifests",
"mesh-delivery-indexer.views.modules",
"node-engine.manifest-reads-changed"
],
"invokes": [
"seat:mesh-controller.assign",
"seat:mesh-controller.pin",
"seat:mesh-controller.placement-export",
"seat:mesh-controller.unassign",
"seat:mesh-controller.unpin",
"mesh-delivery-indexer.graph",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
"seat:mesh-hand-acts.hand-act",
"seat:mesh-hand-acts.hand-acts",
"seat:mesh-hand-acts.warranted",
"seat:node-backup.backed-up",
"seat:node-backup.now"
"seat:node-backup.now",
"seat:node-engine.manifest-reads"
],
"emits": [
"put"
@@ -66,7 +61,19 @@
"tools": [
"assigner_put",
"assigner_release",
"assigner_puts"
"assigner_puts",
"placement",
"seat-holders",
"wants",
"assigner_read_placement",
"assigner_read_seat_holders",
"assigner_read_wants",
"assigner_read_findings",
"assigner_read_asks",
"assigner_read_differences",
"assigner_first_views",
"assigner_first_view_again",
"assigner_engine_reads"
],
"state": [
{ "name": "placement", "history": 64 },
@@ -74,13 +81,15 @@
{ "name": "candidates", "history": 5 },
{ "name": "pulses", "key-ttl": true },
{ "name": "asks", "history": 5 },
{ "name": "compositions", "history": 5 },
{ "name": "differences", "history": 64 },
{ "name": "puts", "history": 20 },
{ "name": "wants", "history": 20 },
{ "name": "findings", "history": 5 },
{ "name": "policies", "history": 20 },
{ "name": "seat-holders", "history": 5 }
{ "name": "seat-holders", "history": 5 },
{ "name": "view-marks", "history": 1 },
{ "name": "first-views", "history": 1 },
{ "name": "engine-reads", "history": 1 }
],
"streams": [
{
@@ -92,7 +101,8 @@
"placement-withdrawn"
],
"max-age-days": 30
}
},
{ "name": "assigner-views", "events": ["views.**"], "max-age-days": 7, "max-mib": 256 }
],
"own-secrets": {
"broker": "${dir:state}/broker"
@@ -142,7 +152,12 @@
"binary": "mesh-delivery-assigner",
"loads": [
"mesh-delivery-assigner"
]
],
"env": {
"MESH_VIEW_CALLERS_PLACEMENT": "mesh-grants,node-tools,mesh-delivery-indexer",
"MESH_VIEW_CALLERS_SEAT_HOLDERS": "node-tools",
"MESH_VIEW_CALLERS_WANTS": "mesh-grants"
}
}
]
}
@@ -0,0 +1,316 @@
package buildrun
import (
"context"
"errors"
"fmt"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **An asker puts an ask on its build seat's stream and waits for its step** (novox/hq ADR 0525 §1, §6; issue 1887 A4,
// A5). It chooses no node: whichever build node pulls the ask writes `<node>.<ask id>` in its build module's steps state,
// and the asker reads the first such key that is not running. The builder and the checker share it.
// AskPublisher puts an encoded ask on subject, de-duplicated by its id.
type AskPublisher interface {
PublishAsk(ctx context.Context, subject, id string, data []byte) error
}
// AskWaiter watches an ask's steps: every value a key `*.<ask id>` of the build module's steps state takes, running or
// ended, on the channel it answers, until ctx ends. A watch, not a poll.
type AskWaiter interface {
WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error)
}
// Asker puts asks and waits for their steps.
type Asker struct {
Publisher AskPublisher
Waiter AskWaiter
// Bound is how long a running step (a build node took the ask, its key written running) may stand still, with no new phase,
// before the asker stops waiting; a step that keeps moving is waited for up to 3 Bounds from its start (issue 1930); zero waits as long as ctx. An ask waiting in its stream is not bounded: the stream's backlog is no fault
// (ADR 0525 §6, §9), and the bound starts again when another node takes the ask.
Bound time.Duration
}
// Put puts a on its stream: the seat's accept subject of its language and kind.
func (k Asker) Put(ctx context.Context, a delivery.StepAsk) error {
raw, err := delivery.EncodeStepAsk(a)
if err != nil {
return err
}
subject, err := delivery.AskSubject(a.Language, a.Kind)
if err != nil {
return err
}
if err := k.Publisher.PublishAsk(ctx, subject, a.ID, raw); err != nil {
return fmt.Errorf("putting the %s ask %s: %w", a.Kind, a.ID, err)
}
return nil
}
// Wait waits for the step of an ask already put (a builder that started again takes over its own asks with it, never
// putting them twice) and answers it when it ended, whatever its state. The bound runs from the step's start on the node
// that took the ask, read from its running key, so an ask that waits in the stream for a free slot waits as long as ctx.
func (k Asker) Wait(ctx context.Context, a delivery.StepAsk) (delivery.Step, error) {
module, err := delivery.BuildModuleOf(a.Language)
if err != nil {
return delivery.Step{}, err
}
watch, cancel := context.WithCancel(ctx)
defer cancel()
steps, err := k.Waiter.WatchAsk(watch, module, a.ID)
if err != nil {
return delivery.Step{}, err
}
var bound, outerCap <-chan time.Time
var timer, capTimer *time.Timer
defer func() {
if timer != nil {
timer.Stop()
}
if capTimer != nil {
capTimer.Stop()
}
}()
running, since, phase, phaseAt := "", time.Time{}, "", time.Time{}
standStill := func() error {
if phase == "" {
return fmt.Errorf("the ask %s's step on %s did not end within %s of its start: it stood still, saying no phase", a.ID, running, k.Bound)
}
return fmt.Errorf("the ask %s's step on %s did not end within %s: it stood still in %q since %s",
a.ID, running, k.Bound, phase, phaseAt.UTC().Format(time.RFC3339))
}
for {
select {
case <-ctx.Done():
return delivery.Step{}, ctx.Err()
case <-bound:
return delivery.Step{}, standStill()
case <-outerCap:
return delivery.Step{}, fmt.Errorf("the ask %s's step on %s did not end within %s of its start: it kept moving (last phase %q at %s) but never ended",
a.ID, running, 3*k.Bound, phase, phaseAt.UTC().Format(time.RFC3339))
case step, ok := <-steps:
if !ok {
return delivery.Step{}, errors.New("the watch of the ask's step ended")
}
if seen := stepSeenOf(ctx); seen != nil {
seen(step)
}
if step.State != delivery.StepRunning {
return step, nil
}
if k.Bound <= 0 {
continue
}
newRun := step.Node != running || !step.Started.Equal(since)
if !newRun && step.Phase == phase && step.PhaseAt.Equal(phaseAt) {
continue
}
// A node took the ask, or took it again after its run stopped (another node, or the same node's build module
// replaced mid-run: a new start), or its step entered a phase. The bound is on standing still: it runs from the
// step's last movement (its phase's start, else its start), so a step that keeps moving is waited for, up to
// three bounds from its start (issue 1930).
running, since, phase, phaseAt = step.Node, step.Started, step.Phase, step.PhaseAt
last := step.Started
if step.PhaseAt.After(last) {
last = step.PhaseAt
}
left := k.Bound
if !last.IsZero() {
left = time.Until(last.Add(k.Bound))
}
if timer != nil {
timer.Stop()
}
timer = time.NewTimer(max(left, 0))
bound = timer.C
if newRun {
if capTimer != nil {
capTimer.Stop()
}
outer := 3 * k.Bound
if !step.Started.IsZero() {
outer = time.Until(step.Started.Add(3 * k.Bound))
}
capTimer = time.NewTimer(max(outer, 0))
outerCap = capTimer.C
}
}
}
}
type stepSeenKey struct{}
// WithStepSeen is ctx carrying seen, which Asker.Wait hands every value the ask's step key takes while it waits: the node
// that took it, each phase it says while running (delivery.Step.Phase), and its end. A try names its asks' progress so
// (issue 1890). seen runs in the waiting goroutine and must not block.
func WithStepSeen(ctx context.Context, seen func(delivery.Step)) context.Context {
return context.WithValue(ctx, stepSeenKey{}, seen)
}
func stepSeenOf(ctx context.Context) func(delivery.Step) {
seen, _ := ctx.Value(stepSeenKey{}).(func(delivery.Step))
return seen
}
// Ask puts a and waits for its step; a step that did not pass is an error naming its node, its state and its reason.
func (k Asker) Ask(ctx context.Context, a delivery.StepAsk) (delivery.Step, error) {
if err := k.Put(ctx, a); err != nil {
return delivery.Step{}, err
}
return k.Passed(k.Wait(ctx, a))
}
// Passed is step when it passed, else an error naming where it ended and why.
func (k Asker) Passed(step delivery.Step, err error) (delivery.Step, error) {
if err != nil {
return step, err
}
if step.State != delivery.StepPassed {
return step, fmt.Errorf("the ask %s on %s %s: %s", step.ID, step.Node, step.State, step.Reason)
}
return step, nil
}
// JetStreamAsks is the bus side of an Asker: a JetStream publish de-duplicated by the ask's id (Nats-Msg-Id), and a watch
// of `*.<ask id>` in the build module's steps bucket.
type JetStreamAsks struct {
JS jetstream.JetStream
}
// PublishAsk is AskPublisher's.
func (s JetStreamAsks) PublishAsk(ctx context.Context, subject, id string, data []byte) error {
msg := &nats.Msg{Subject: subject, Data: data, Header: nats.Header{}}
msg.Header.Set(jetstream.MsgIDHeader, id)
_, err := s.JS.PublishMsg(ctx, msg)
return err
}
// WatchAsk is AskWaiter's: the watch hands over what the keys hold now, so a step that ended before it began is read at
// once, and every change after.
func (s JetStreamAsks) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
pattern, err := delivery.AskStepWatch(askID)
if err != nil {
return nil, err
}
kv, err := s.JS.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
w, err := kv.Watch(ctx, pattern)
if err != nil {
return nil, err
}
out := make(chan delivery.Step)
go func() {
defer close(out)
defer w.Stop()
for {
select {
case <-ctx.Done():
return
case e, ok := <-w.Updates():
if !ok {
return
}
if e == nil || e.Operation() != jetstream.KeyValuePut {
continue
}
step, err := delivery.DecodeStep(e.Value())
if err != nil {
continue // not a step (a node's pause key never matches an ask id; anything else is not this ask's)
}
select {
case out <- step:
case <-ctx.Done():
return
}
}
}
}()
return out, nil
}
// FakeAsks is a test double of the bus for an asker's tests: each ask put is kept and, when Answer is set, its step is
// written at once with what Answer says a build node wrote (running or ended). Write writes a step later. Safe for
// concurrent use.
type FakeAsks struct {
Answer func(a delivery.StepAsk) delivery.Step
mu sync.Mutex
asks []delivery.StepAsk
steps map[string][]delivery.Step
watchers map[string][]chan delivery.Step
}
// PublishAsk is AskPublisher's.
func (f *FakeAsks) PublishAsk(_ context.Context, _, id string, data []byte) error {
a, err := delivery.DecodeStepAsk(data)
if err != nil {
return err
}
f.mu.Lock()
f.asks = append(f.asks, a)
f.mu.Unlock()
if f.Answer != nil {
f.Write(f.Answer(a))
}
return nil
}
// Write writes an ask's step, as a build node does.
func (f *FakeAsks) Write(step delivery.Step) {
f.mu.Lock()
if f.steps == nil {
f.steps, f.watchers = map[string][]delivery.Step{}, map[string][]chan delivery.Step{}
}
f.steps[step.ID] = append(f.steps[step.ID], step)
ws := append([]chan delivery.Step(nil), f.watchers[step.ID]...)
f.mu.Unlock()
for _, w := range ws {
w <- step
}
}
// End is Write: kept for the tests that end a step.
func (f *FakeAsks) End(step delivery.Step) { f.Write(step) }
// WatchAsk is AskWaiter's: what was written so far, then every write.
func (f *FakeAsks) WatchAsk(ctx context.Context, _ string, askID string) (<-chan delivery.Step, error) {
out := make(chan delivery.Step, 64)
f.mu.Lock()
if f.steps == nil {
f.steps, f.watchers = map[string][]delivery.Step{}, map[string][]chan delivery.Step{}
}
for _, s := range f.steps[askID] {
out <- s
}
f.watchers[askID] = append(f.watchers[askID], out)
f.mu.Unlock()
go func() {
<-ctx.Done()
f.mu.Lock()
ws := f.watchers[askID]
for i, w := range ws {
if w == out {
f.watchers[askID] = append(ws[:i], ws[i+1:]...)
break
}
}
f.mu.Unlock()
}()
return out, nil
}
// Asks are the asks put so far.
func (f *FakeAsks) Asks() []delivery.StepAsk {
f.mu.Lock()
defer f.mu.Unlock()
return append([]delivery.StepAsk(nil), f.asks...)
}
@@ -0,0 +1,61 @@
//go:build integration
package buildrun
import (
"context"
"encoding/json"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/testbus"
)
// 1887/A4a, 1887/AC3 on the real bus: an ask put twice under one id is one message, and the asker reads the step a node
// wrote under `<node>.<ask id>`.
func TestAnAskIsPutOnceAndItsStepIsRead(t *testing.T) {
nc, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, _ := jetstream.New(nc)
ctx := context.Background()
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "SEAT_NODE_BUILD_GO", Subjects: []string{"mesh.seat.node-build-go.accept.>"},
Retention: jetstream.WorkQueuePolicy}); err != nil {
t.Fatal(err)
}
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.StepsBucket("mesh-build-go")})
if err != nil {
t.Fatal(err)
}
bus := JetStreamAsks{JS: js}
k := Asker{Publisher: bus, Waiter: bus, Bound: 5 * time.Second}
a := goAsk("mvl0frv18d1960")
for i := 0; i < 2; i++ {
if err := k.Put(ctx, a); err != nil {
t.Fatal(err)
}
}
s, _ := js.Stream(ctx, "SEAT_NODE_BUILD_GO")
if info, _ := s.Info(ctx); info.State.Msgs != 1 {
t.Fatalf("%d messages for one ask put twice", info.State.Msgs)
}
go func() {
time.Sleep(100 * time.Millisecond)
key, _ := delivery.AskStepKey("ace", a.ID)
running, _ := delivery.EncodeStep(delivery.Step{ID: a.ID, Tool: "delivery ask", Node: "ace", State: delivery.StepRunning, Started: time.Now()})
_, _ = kv.Put(ctx, key, running)
passed, _ := delivery.EncodeStep(delivery.Step{ID: a.ID, Tool: "delivery ask", Node: "ace", State: delivery.StepPassed, Started: time.Now(),
Result: json.RawMessage(`{"artifacts":[]}`)})
_, _ = kv.Put(ctx, key, passed)
}()
step, err := k.Passed(k.Wait(ctx, a))
if err != nil || step.Node != "ace" {
t.Fatalf("step %+v, %v", step, err)
}
}
@@ -0,0 +1,234 @@
package buildrun
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
func goAsk(id string) delivery.StepAsk {
return delivery.StepAsk{ID: id, Kind: delivery.AskDelivery, Language: delivery.AskGo, Asker: "mesh-delivery-builder", At: time.Now(),
Build: &delivery.BuildJob{Module: "m", Path: ".", Fetch: map[string]any{"repository": "novox/x", "commit": strings.Repeat("a", 40)},
Artifacts: []delivery.AskArtifact{{Artifact: "program"}}}}
}
// 1887/A4a: an asker puts the ask once and answers the step a build node wrote for it, naming the node; one that did not
// pass is an error naming the node and why; a wait of an ask already put puts nothing.
func TestAnAskerPutsOnceAndReadsTheStep(t *testing.T) {
f := &FakeAsks{Answer: func(a delivery.StepAsk) delivery.Step {
return delivery.Step{ID: a.ID, Node: "ace", State: delivery.StepPassed, Result: []byte(`{"artifacts":[]}`)}
}}
k := Asker{Publisher: f, Waiter: f}
var seen []delivery.Step // issue 1890: a waiter that asks to see each value of the step sees it
step, err := k.Ask(WithStepSeen(context.Background(), func(s delivery.Step) { seen = append(seen, s) }), goAsk("mvl0frv18d1957"))
if err != nil || step.Node != "ace" || len(f.Asks()) != 1 || len(seen) != 1 || seen[0].Node != "ace" {
t.Fatalf("step %+v, %v, asks %d, seen %+v", step, err, len(f.Asks()), seen)
}
f.Answer = func(a delivery.StepAsk) delivery.Step {
return delivery.Step{ID: a.ID, Node: "shanks", State: delivery.StepRefused, Reason: "go 1.27 installed"}
}
if _, err := k.Ask(context.Background(), goAsk("mvl0frv18d1958")); err == nil || !strings.Contains(err.Error(), "shanks refused: go 1.27 installed") {
t.Fatalf("a refused step: %v", err)
}
// review round 2 item 1: an ask that waits in its stream for a free slot is not bounded; once a node writes its step
// running, the step is bounded from its start, and the error names the node.
g := &FakeAsks{}
k = Asker{Publisher: g, Waiter: g, Bound: 20 * time.Millisecond}
waiting, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
if _, err := k.Wait(waiting, goAsk("mvl0frv18d1959")); err != context.DeadlineExceeded {
t.Fatalf("an ask no node took: %v, want it still waiting when ctx ended", err)
}
g.Write(delivery.Step{ID: "mvl0frv18d1961", Node: "ace", State: delivery.StepRunning, Started: time.Now()})
if _, err := k.Wait(context.Background(), goAsk("mvl0frv18d1961")); err == nil || !strings.Contains(err.Error(), "on ace did not end within") {
t.Fatalf("a step that never ends: %v", err)
}
if len(g.Asks()) != 0 {
t.Fatal("a wait put an ask")
}
// Issue 1930: the bound is on a step that stands still, not on its age. A step past one bound from its start that entered a
// new phase inside the bound is waited for until it ends; one that stood a whole bound in a phase fails at once, naming
// the phase; one three bounds from its start ends, said as kept moving. The steps' own times stand for the clock, so the
// bound is an hour no loaded machine reaches and no case waits on a pause (the check of 2026-11-02 22:35 tripped one
// of 60 ms).
k = Asker{Publisher: g, Waiter: g, Bound: time.Hour}
now := time.Now()
g.Write(delivery.Step{ID: "mvl0frv18d1980", Node: "ace", State: delivery.StepRunning, Started: now.Add(-90 * time.Minute), Phase: "pb", PhaseAt: now})
g.Write(delivery.Step{ID: "mvl0frv18d1980", Node: "ace", State: delivery.StepPassed})
if step, err := k.Wait(context.Background(), goAsk("mvl0frv18d1980")); err != nil || step.State != delivery.StepPassed {
t.Fatalf("a step that kept moving past its bound: %+v, %v", step, err)
}
g.Write(delivery.Step{ID: "mvl0frv18d1981", Node: "ace", State: delivery.StepRunning, Started: now.Add(-time.Hour), Phase: "integration: tests", PhaseAt: now.Add(-time.Hour)})
if _, err := k.Wait(context.Background(), goAsk("mvl0frv18d1981")); err == nil || !strings.Contains(err.Error(), `stood still in "integration: tests"`) {
t.Fatalf("a step that stood still in a phase: %v", err)
}
g.Write(delivery.Step{ID: "mvl0frv18d1982", Node: "ace", State: delivery.StepRunning, Started: now.Add(-3 * time.Hour), Phase: "qz", PhaseAt: now})
if _, err := k.Wait(context.Background(), goAsk("mvl0frv18d1982")); err == nil || !strings.Contains(err.Error(), "kept moving") {
t.Fatalf("a step that never stops moving: %v", err)
}
// Issue 734: an ask the build seat gave up on (DEAD_LETTERS keeps its copy with the ask's id) ends its wait as failed
// with the words, though a node had taken it; under NotTakenBound one no node takes ends at the bound; a step written
// first is answered as the node wrote it.
dead := &DeadLetters{}
lw := LostAskWaiter{Inner: g, Dead: dead}
k = Asker{Publisher: g, Waiter: lw, Bound: time.Minute}
g.Write(delivery.Step{ID: "mvl0frv18d1970", Node: "ace", State: delivery.StepRunning, Started: time.Now()})
given, ok := GivenUpOf(map[string][]string{"Mesh-Dead-Msg-Id": {"mvl0frv18d1970"}, "Mesh-Dead-Deliveries": {"5"},
"Mesh-Dead-Stream": {"SEAT_NODE_BUILD_GO"}, "Mesh-Dead-Gave-Up": {"2026-10-26T12:26:40Z"}})
if !ok {
t.Fatal("a dead letter's headers were not read")
}
go dead.Heard(given)
step, err = k.Wait(context.Background(), goAsk("mvl0frv18d1970"))
if err != nil || step.State != delivery.StepFailed || !IsLost(step) ||
!strings.Contains(step.Reason, LostWords+" after 5 deliveries at 2026-10-26T12:26:40Z") {
t.Fatalf("a given-up ask: %+v, %v", step, err)
}
if _, err := k.Passed(step, nil); err == nil || !strings.Contains(err.Error(), "on "+LostNode+" failed: "+LostWords) {
t.Fatalf("a given-up ask's error: %v", err)
}
step, err = k.Wait(NotTakenBound(context.Background(), 20*time.Millisecond), goAsk("mvl0frv18d1971"))
if err != nil || !IsNotTaken(step) || IsLost(step) {
t.Fatalf("an ask no node took: %+v, %v", step, err)
}
g.Write(delivery.Step{ID: "mvl0frv18d1972", Node: "g14", State: delivery.StepPassed})
if step, err = k.Wait(NotTakenBound(context.Background(), time.Minute), goAsk("mvl0frv18d1972")); err != nil || step.Node != "g14" {
t.Fatalf("a step written: %+v, %v", step, err)
}
// Review round 1 of issue 734: a hearing the bus refuses at start is deaf and says so, never holds the start up, and
// tries again on the next ask waited for; heard, it says so and hands the given-up asks on.
fol := &refusingFollower{refuse: 1}
var mu sync.Mutex
var told []Hearing
hearing := NewDeadLetterHearing(context.Background(), fol, []string{delivery.AskGo}, dead, nil,
func(h Hearing) { mu.Lock(); told = append(told, h); mu.Unlock() })
hearing.Wait()
if why := hearing.Deaf(); !strings.Contains(why, "SEAT_NODE_BUILD_GO: permissions violation") {
t.Fatalf("a refused hearing says %q", why)
}
k = Asker{Publisher: g, Waiter: LostAskWaiter{Inner: g, Dead: dead, Hearing: hearing}, Bound: time.Minute}
g.Write(delivery.Step{ID: "mvl0frv18d1973", Node: "g14", State: delivery.StepPassed})
if _, err := k.Wait(context.Background(), goAsk("mvl0frv18d1973")); err != nil {
t.Fatal(err)
}
hearing.Wait()
mu.Lock()
if hearing.Deaf() != "" || len(told) != 2 || told[0].Heard || !told[1].Heard || fol.calls() != 2 {
t.Fatalf("after an ask: deaf %q, told %+v, followed %d times", hearing.Deaf(), told, fol.calls())
}
mu.Unlock()
fol.heard(GivenUp{AskID: "mvl0frv18d1974", Deliveries: 5})
step, err = k.Wait(context.Background(), goAsk("mvl0frv18d1974"))
if err != nil || !IsLost(step) {
t.Fatalf("an ask heard given up after the hearing came back: %+v, %v", step, err)
}
// The lost check of 2026-10-28: a step whose own watch went deaf is read at the next nudge (a step of its build module
// ended) and answered; a nudge of another build module reads nothing; a watch that closes is opened again.
d := &FakeAsks{}
heard := &HeardEnds{Inner: deafWatch{}, Read: d}
k = Asker{Publisher: d, Waiter: heard, Bound: time.Minute}
d.Write(delivery.Step{ID: "mvl0frv18d1980", Node: "shanks", State: delivery.StepPassed})
got := make(chan delivery.Step, 1)
go func() { s, _ := k.Wait(context.Background(), goAsk("mvl0frv18d1980")); got <- s }()
for heard.Waiting() == 0 {
time.Sleep(time.Millisecond)
}
heard.Nudge(NodeBuildModule)
select {
case s := <-got:
t.Fatalf("a nudge of another build module answered %+v", s)
case <-time.After(20 * time.Millisecond):
}
heard.Nudge(GoBuildModule)
if s := <-got; s.State != delivery.StepPassed || s.Node != "shanks" {
t.Fatalf("a step its watch never handed over: %+v", s)
}
closing := &HeardEnds{Inner: &closingWatch{inner: d}, Read: d}
if s, err := (Asker{Publisher: d, Waiter: closing}).Wait(context.Background(), goAsk("mvl0frv18d1980")); err != nil || s.State != delivery.StepPassed {
t.Fatalf("a watch that closed once: %+v, %v", s, err)
}
for _, row := range []struct {
h *Hearing
want string
}{{nil, health.StateUnhealthy}, {&told[0], health.StateUnhealthy}, {&told[1], health.StateHealthy}} {
var raw []byte
if row.h != nil {
raw, _ = json.Marshal(row.h)
}
if a := HearingHealth(raw); a.State != row.want || a.Why == "" {
t.Errorf("health of %+v = %+v, want %s", row.h, a, row.want)
}
}
}
// Review (issue 734 path): an ask taken again by the same node (its build module replaced mid-run) is bounded from the
// new run's start, not the first run's. The steps' own times stand for the clock: the first run started now, the second a
// whole bound ago, so the wait ends at once on the second run's bound where the first run's would wait an hour; no case
// waits on a pause a loaded machine can outlast (the check of 2026-11-02 22:35 tripped one of 60 ms). The context's
// bound is reached only by a wait that kept the first run's start.
func TestARedeliveryToTheSameNodeRestartsTheBound(t *testing.T) {
f := &FakeAsks{}
k := Asker{Publisher: f, Waiter: f, Bound: time.Hour}
const id = "mvl0frv18d1962"
now := time.Now()
f.Write(delivery.Step{ID: id, Node: "ace", State: delivery.StepRunning, Started: now})
f.Write(delivery.Step{ID: id, Node: "ace", State: delivery.StepRunning, Started: now.Add(-time.Hour)})
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if _, err := k.Wait(ctx, goAsk(id)); err == nil || !strings.Contains(err.Error(), "on ace did not end within 1h0m0s of its start") {
t.Fatalf("a redelivered run a bound old: %v, want its own bound to end the wait", err)
}
}
// refusingFollower refuses the first `refuse` follows, as the bus does before the asker's grant is composed.
type refusingFollower struct {
mu sync.Mutex
refuse int
n int
hand func(GivenUp)
}
func (f *refusingFollower) Follow(_ context.Context, _ string, _ time.Time, heard func(GivenUp)) (func(), error) {
f.mu.Lock()
defer f.mu.Unlock()
f.n++
if f.n <= f.refuse {
return nil, errors.New("permissions violation")
}
f.hand = heard
return func() {}, nil
}
func (f *refusingFollower) calls() int { f.mu.Lock(); defer f.mu.Unlock(); return f.n }
func (f *refusingFollower) heard(g GivenUp) { f.mu.Lock(); h := f.hand; f.mu.Unlock(); h(g) }
// deafWatch is a watch of an ask's step that stays open and hears nothing.
type deafWatch struct{}
func (deafWatch) WatchAsk(context.Context, string, string) (<-chan delivery.Step, error) {
return make(chan delivery.Step), nil
}
// closingWatch closes its first watch at once, and hands the later ones to inner.
type closingWatch struct {
inner AskWaiter
closed bool
}
func (c *closingWatch) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
if !c.closed {
c.closed = true
ch := make(chan delivery.Step)
close(ch)
return ch, nil
}
return c.inner.WatchAsk(ctx, module, askID)
}
@@ -0,0 +1,88 @@
package buildrun
import (
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ChooseNodes picks up to want build nodes for a language (ADR 0478 clause 3): the nodes whose toolchains key reports the
// version the nodes agree on, the ones with room under their build capacity first (places), then the fewest lanes the
// part has running or chosen there, a tie broken by the node's name. A place is reserved on each node chosen, in the
// same step as the choice, so builds choosing at once do not choose the same full node while another has room; the
// caller takes the place (Places.Acquire) or gives it up (Places.Cancel). A node at its capacity is still chosen when
// the others are too (fewer than want have room): its lane waits for a place. A node qualifies only when qualifies is true
// of it (online and its build module healthy; nil means every node does). It answers the toolchain all the reporting nodes
// agree on, qualifying or not, as the indexer computes it. When nothing is reported, or the nodes report different
// versions (so the indexer agreed none), it fails naming the language and why; when no reporting node qualifies it fails
// naming the nodes. A node whose build module refuses on its toolchain refuses the step, and the build fails naming it.
func ChooseNodes(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, delivery.Toolchain2, error) {
tc, none := delivery.AgreedToolchain(entries)
if none != "" {
return nil, delivery.Toolchain2{}, fmt.Errorf("no %s build node qualifies: %s", language, none)
}
nodes, err := choose(language, entries, places, qualifies, want)
return nodes, tc, err
}
// ChooseAnyNodes is ChooseNodes for artifacts that need no toolchain, such as an archive (ADR 0478 clause 1): the
// build module's nodes agree on nothing, so any node of the build module that qualifies is chosen, by capacity and load only.
func ChooseAnyNodes(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, error) {
return choose(language, entries, places, qualifies, want)
}
func choose(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, error) {
if places == nil {
places = &Places{}
}
nodes := make([]string, 0, len(entries))
var reporting []string
seen := map[string]bool{}
for _, e := range entries {
if seen[e.Node] {
continue
}
seen[e.Node] = true
reporting = append(reporting, e.Node)
if places.CapacityOf(e.Node) > 0 && (qualifies == nil || qualifies(e.Node)) { // a capacity of 0 keeps the part off a node
nodes = append(nodes, e.Node)
}
}
if len(nodes) == 0 {
sort.Strings(reporting)
return nil, fmt.Errorf("no %s build node qualifies: %s report the toolchain, and none is online with a healthy build module and a build capacity", language, strings.Join(reporting, ", "))
}
return places.choose(nodes, want), nil
}
// DigestsDifferPrefix opens the error of a build whose nodes built an artifact to different digests (ADR 0478 clause 4):
// such a key is not built again as a new revision, since the same source on the same toolchains gave two answers and
// building again settles nothing; it waits for another fingerprint.
const DigestsDifferPrefix = "digests differ:"
// Agree is the one digest every node built an artifact to (ADR 0478 clause 4), or an error naming each node and its
// digest when they differ. One node's digest agrees with itself.
func Agree(artifact string, byNode map[string]string) (string, error) {
nodes := make([]string, 0, len(byNode))
for n := range byNode {
nodes = append(nodes, n)
}
sort.Strings(nodes)
var first string
for i, n := range nodes {
if i == 0 {
first = byNode[n]
continue
}
if byNode[n] != first {
msg := fmt.Sprintf("%s the nodes built %s to different digests:", DigestsDifferPrefix, artifact)
for _, n := range nodes {
msg += fmt.Sprintf(" %s %s;", n, byNode[n])
}
return "", fmt.Errorf("%s", msg[:len(msg)-1])
}
}
return first, nil
}
@@ -0,0 +1,109 @@
package buildrun
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const digest = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
func hex64(c byte) string { return strings.Repeat(string(c), 64) }
func tcs(language, version string, nodes ...string) []delivery.NodeToolchain {
var out []delivery.NodeToolchain
for _, n := range nodes {
out = append(out, delivery.NodeToolchain{Node: n, Toolchain: delivery.Toolchain{Language: language, Version: version, Recipe: "1"}})
}
return out
}
func chooseTakesTheAgreedNodesWithFewestRunningStepsTiesByName(t *testing.T) { // 1607/AC4, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "novox", "ace", "g14", "shanks")
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 3, "ace": 1, "g14": 1, "shanks": 0}), nil, 2)
if err != nil {
t.Fatal(err)
}
if len(nodes) != 2 || nodes[0] != "shanks" || nodes[1] != "ace" {
t.Errorf("nodes = %v, want [shanks ace]: fewest steps first, a tie by name", nodes)
}
if tc.Version != "go1.27.1" || tc.Recipe != "1" || tc.Language != "go" {
t.Errorf("toolchain = %+v", tc)
}
}
func chooseWithOneNodeAnswersOneNode(t *testing.T) { // 1607/AC4
nodes, _, err := ChooseNodes("go", tcs("go", "go1.27.1", "novox"), nil, nil, 2)
if err != nil || len(nodes) != 1 || nodes[0] != "novox" {
t.Errorf("nodes = %v, err %v, want [novox]", nodes, err)
}
}
func chooseFailsNamingTheLanguageAndTheVersionsWhenNoNodeQualifies(t *testing.T) { // 1607/AC4
_, _, err := ChooseNodes("typescript", nil, nil, nil, 2)
if err == nil || !strings.Contains(err.Error(), "typescript") || !strings.Contains(err.Error(), delivery.ToolchainNoReport) {
t.Errorf("no report: error = %v, want it to name typescript and %q", err, delivery.ToolchainNoReport)
}
entries := append(tcs("go", "go1.27.1", "novox"), tcs("go", "go1.26.0", "ace")...)
_, _, err = ChooseNodes("go", entries, nil, nil, 2)
if err == nil || !strings.Contains(err.Error(), "go1.27.1") || !strings.Contains(err.Error(), "go1.26.0") || !strings.Contains(err.Error(), "go") {
t.Errorf("disagreeing nodes: error = %v, want both versions named", err)
}
}
func agreeNeedsEqualDigestsAndNamesBothNodesWhenTheyDiffer(t *testing.T) { // 1607/AC4, ADR 0478 clause 4
d, err := Agree("tool", map[string]string{"ace": digest, "novox": digest})
if err != nil || d != digest {
t.Errorf("Agree = %q, %v, want %q", d, err, digest)
}
_, err = Agree("tool", map[string]string{"ace": digest, "novox": "sha256:" + hex64('9')})
if err == nil || !strings.Contains(err.Error(), "ace") || !strings.Contains(err.Error(), "novox") || !strings.Contains(err.Error(), digest) || !strings.Contains(err.Error(), hex64('9')) || !strings.Contains(err.Error(), "tool") {
t.Errorf("differing digests: error = %v, want the artifact, both nodes and both digests", err)
}
if d, err := Agree("tool", map[string]string{"ace": digest}); err != nil || d != digest {
t.Errorf("one node: %q, %v", d, err)
}
}
func chooseSkipsNodesThatAreNotOnlineAndFailsNamingThemWhenNoneIs(t *testing.T) { // 1607 review 3, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "ace", "g14", "novox")
online := func(n string) bool { return n != "g14" } // a sleeping laptop
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 5}), online, 2)
if err != nil || len(nodes) != 2 || nodes[0] != "ace" || nodes[1] != "novox" || tc.Version != "go1.27.1" {
t.Errorf("nodes = %v, toolchain %+v, err %v, want [ace novox] with g14 left out although it runs nothing", nodes, tc, err)
}
_, tc, err = ChooseNodes("go", entries, nil, func(string) bool { return false }, 2)
if err == nil || !strings.Contains(err.Error(), "online") || !strings.Contains(err.Error(), "g14") || !strings.Contains(err.Error(), "go") {
t.Errorf("none online: error = %v, want it to name the nodes that report and that none is online", err)
}
if tc.Version != "go1.27.1" {
t.Errorf("none online: the agreed toolchain %+v is still answered", tc)
}
}
// placesRunning is Places with these steps counted as running, for a test of the choice.
func placesRunning(running map[string]int) *Places {
return &Places{running: running, changed: make(chan struct{})}
}
func choosePrefersNodesBelowTheirCapacity(t *testing.T) { // round 3
entries := tcs("go", "go1.27.1", "ace", "novox", "shanks")
// shanks has capacity 3 with 2 running: room; ace has the default 1 with 1 running: none; novox has none running.
p := placesRunning(map[string]int{"shanks": 2, "ace": 1})
p.Capacity = map[string]int{"shanks": 3}
nodes, _, err := ChooseNodes("go", entries, p, nil, 2)
if err != nil || len(nodes) != 2 || nodes[0] != "novox" || nodes[1] != "shanks" {
t.Errorf("nodes = %v, err %v, want [novox shanks]: the nodes with room, ace at capacity left out", nodes, err)
}
}
// The choice of the nodes a step runs on and the agreement of their answers, one table of cases (the test budget).
func TestChoosingTheNodesOfAStep(t *testing.T) {
t.Run("choose takes the agreed nodes with fewest running steps ties by name", chooseTakesTheAgreedNodesWithFewestRunningStepsTiesByName)
t.Run("choose with one node answers one node", chooseWithOneNodeAnswersOneNode)
t.Run("choose fails naming the language and the versions when no node qualifies", chooseFailsNamingTheLanguageAndTheVersionsWhenNoNodeQualifies)
t.Run("agree needs equal digests and names both nodes when they differ", agreeNeedsEqualDigestsAndNamesBothNodesWhenTheyDiffer)
t.Run("choose skips nodes that are not online and fails naming them when none is", chooseSkipsNodesThatAreNotOnlineAndFailsNamingThemWhenNoneIs)
t.Run("choose prefers nodes below their capacity", choosePrefersNodesBelowTheirCapacity)
}
@@ -0,0 +1,5 @@
// Package buildrun holds the parts every module that runs steps on the build modules shares: the caller, the steps
// watch and the runner of a long step (ADR 0469), the sets of presence, toolchains and healthy build modules, the
// places a node offers by capacity, and the choice of build nodes (ADR 0478). The builder and the checker of the
// delivery use them; they moved here from the builder's own package so neither keeps a copy (issue 1671, ADR 0484).
package buildrun
@@ -0,0 +1,23 @@
module git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261104174359-05b3e2e1b94f
github.com/nats-io/nats.go v1.54.0
)
require (
git.novox.be/novox/mesh-contracts/go v0.0.0-20261104174344-ae5266e77639 // indirect
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.1 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.2 // indirect
github.com/nats-io/nats-server/v2 v2.11.17 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.16.0 // indirect
)
@@ -0,0 +1,29 @@
git.novox.be/novox/mesh-contracts/go v0.0.0-20261104174344-ae5266e77639 h1:Zt7CfWt2S2W/fVgWNyW2u2+xwz051LXvGxVTNIyfhdM=
git.novox.be/novox/mesh-contracts/go v0.0.0-20261104174344-ae5266e77639/go.mod h1:0dD1XEJBJ0G5UYDQxSZI84nuujb6m3tOpJl4C8rQVvg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261104174359-05b3e2e1b94f h1:jDsfDMd48MXY9Zershc3eRxnhmx2sSpUfQkLq5jAIds=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261104174359-05b3e2e1b94f/go.mod h1:08OomDXdwVN8/87nyDtv5R3rH/6LoP8SFFXz30bTYCM=
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op h1:1BOWQJweNyvZMlpAHXGLiZQn9S+QXGcz3xh94lC0w6E=
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ=
github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.2 h1:XXRgB60MSTnqsRwejQurVDs/hcv2dkt+86GjI+I/bMc=
github.com/nats-io/jwt/v2 v2.8.2/go.mod h1:Ag/56sq9OblL4JgdYufDd16Egb17Kr/8WwwuO/forVc=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.16.0 h1:vMb6ptszcQMkcwiRTAuNNU50gom6++Q/6gY2hDM6VDE=
golang.org/x/time v0.16.0/go.mod h1:rVKOqvZeKvrDKTQiAHJ7wmwP0RzleSphoEA9RcdLA0s=
@@ -0,0 +1,56 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// HaveSet holds what each node's node-engine says of each module on it (node-engine_have, key `<node>.<module>`): the
// builder reads it to know whether a node's build module is running and healthy, since a build module whose installed
// toolchain is not the one its recipe builds with is unhealthy (mesh-build-go build_go_health) and its steps would
// refuse (ADR 0478 clause 3). A module the node-engine has written no key of is not healthy. Safe for use from the watch
// and from a build at once.
type HaveSet struct {
mu sync.Mutex
healthy map[string]bool // by `<node>.<module>`
}
// Put takes a change of the have bucket in and answers whether a module's healthy state changed. A node's own key
// (`<node>.node`) and a value that is not a have are left out.
func (s *HaveSet) Put(key string, value []byte, deleted bool) bool {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
return false
}
ok := false
if !deleted {
h, err := delivery.DecodeHave(value)
if err != nil {
return false
}
ok = h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK
}
k := node + "." + module
s.mu.Lock()
defer s.mu.Unlock()
if s.healthy[k] == ok {
return false
}
if s.healthy == nil {
s.healthy = map[string]bool{}
}
if ok {
s.healthy[k] = true
} else {
delete(s.healthy, k)
}
return true
}
// Healthy is whether the module on the node was last seen running and healthy.
func (s *HaveSet) Healthy(node, module string) bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.healthy[node+"."+module]
}
@@ -0,0 +1,236 @@
package buildrun
import (
"context"
"fmt"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A step's end is never only its own watch's to hear** (the lost check of 2026-10-28: mesh-catalog #553's verdict
// stayed running after its step mvsgthyjed5786 had passed on shanks at 19:09:31Z; the watch of that one key, opened at
// 18:46:05Z, heard neither the step's start nor its end and never closed, while a watch opened at 18:58:48Z heard its own
// step end). HeardEnds stands between the asker (Asker) and the watch of each ask's step key: when it is
// nudged, every ask it waits on reads its step keys now (a read, not the watch) and hands what it reads to its wait, so a
// step that ended is judged even when its own watch went deaf. A watch that closes under a wait is opened again, at most
// reopenTries times. It is nudged on an event (Nudge): any step of the same build module ending (one watch of each steps
// state, HearEnds), a reconnect to the bus, and whatever else the asker hears (the checker: checker_ask_again).
// It holds no timer. Every asker of the build seats waits through it (the checker and the builder, HearEnds).
// StepReader reads, now, every step a build module's steps state holds for an ask: its keys `*.<ask id>`.
type StepReader interface {
AskSteps(ctx context.Context, module, askID string) ([]delivery.Step, error)
}
// reopenTries is how often a wait opens its ask's watch again after it closed, before the wait ends failed.
const reopenTries = 3
// HeardEnds is an AskWaiter over Inner that also reads an ask's steps when nudged. Safe for concurrent use.
type HeardEnds struct {
Inner AskWaiter
Read StepReader
Logf func(format string, args ...any)
mu sync.Mutex
waiting map[string]*waitedAsk // by ask id
}
type waitedAsk struct {
module string
nudge chan struct{}
}
func (h *HeardEnds) logf(format string, args ...any) {
if h.Logf != nil {
h.Logf(format, args...)
}
}
// WatchAsk is AskWaiter's: what Inner's watch hands, and, at each nudge, what the ask's keys hold now.
func (h *HeardEnds) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
in, err := h.Inner.WatchAsk(ctx, module, askID)
if err != nil {
return nil, err
}
w := &waitedAsk{module: module, nudge: make(chan struct{}, 1)}
h.mu.Lock()
if h.waiting == nil {
h.waiting = map[string]*waitedAsk{}
}
h.waiting[askID] = w
h.mu.Unlock()
out := make(chan delivery.Step)
go func() {
defer close(out)
defer func() {
h.mu.Lock()
if h.waiting[askID] == w {
delete(h.waiting, askID)
}
h.mu.Unlock()
}()
send := func(s delivery.Step) bool {
select {
case out <- s:
return true
case <-ctx.Done():
return false
}
}
reopened := 0
for {
select {
case <-ctx.Done():
return
case s, ok := <-in:
if !ok {
if ctx.Err() != nil || reopened >= reopenTries {
h.logf("the watch of the ask %s's step on %s closed %d times; its wait ends", askID, module, reopened+1)
return
}
reopened++
if in, err = h.Inner.WatchAsk(ctx, module, askID); err != nil {
h.logf("the watch of the ask %s's step on %s closed and cannot be opened again, so its wait ends: %v", askID, module, err)
return
}
h.logf("the watch of the ask %s's step on %s closed; opened again (%d of %d)", askID, module, reopened, reopenTries)
continue
}
if !send(s) {
return
}
case <-w.nudge:
if h.Read == nil {
continue
}
steps, err := h.Read.AskSteps(ctx, module, askID)
if err != nil {
h.logf("the steps of the ask %s on %s cannot be read: %v", askID, module, err)
continue
}
for _, s := range steps {
if s.State != delivery.StepRunning {
h.logf("the ask %s's step on %s ended %s, read at a nudge: its watch had not handed it over", askID, s.Node, s.State)
}
if !send(s) {
return
}
}
}
}
}()
return out, nil
}
// Nudge has every ask waited on of module read its steps now; module "" nudges every ask.
func (h *HeardEnds) Nudge(module string) {
h.mu.Lock()
defer h.mu.Unlock()
for _, w := range h.waiting {
if module == "" || w.module == module {
select {
case w.nudge <- struct{}{}:
default: // a nudge is pending already
}
}
}
}
// Waiting is how many asks are waited on now.
func (h *HeardEnds) Waiting() int {
h.mu.Lock()
defer h.mu.Unlock()
return len(h.waiting)
}
// AskSteps is StepReader's: the keys `*.<ask id>` of the build module's steps state, read now.
func (s JetStreamAsks) AskSteps(ctx context.Context, module, askID string) ([]delivery.Step, error) {
pattern, err := delivery.AskStepWatch(askID)
if err != nil {
return nil, err
}
kv, err := s.JS.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
lister, err := kv.ListKeysFiltered(ctx, pattern)
if err != nil {
return nil, err
}
var out []delivery.Step
for key := range lister.Keys() {
e, err := kv.Get(ctx, key)
if err != nil {
continue // deleted between the list and the read
}
if step, err := delivery.DecodeStep(e.Value()); err == nil {
out = append(out, step)
}
}
return out, nil
}
// HearEnds is the HeardEnds every asker of the build seats waits through: over inner (the asks' own watches, or a
// LostAskWaiter's inner), reading with asks, and nudged at each step end of each of modules (one watch of each steps state,
// updates only; one that closes is opened again, and said and left after reopenTries closes in a row that heard nothing). The asker nudges it at a reconnect too (Nudge("")).
func HearEnds(ctx context.Context, asks JetStreamAsks, inner AskWaiter, modules []string, logf func(string, ...any)) *HeardEnds {
if logf == nil {
logf = func(string, ...any) {}
}
h := &HeardEnds{Inner: inner, Read: asks, Logf: logf}
for _, module := range modules {
kv, err := asks.JS.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
logf("the steps of %s cannot be read (%v): an ask whose own watch goes deaf is read only at a reconnect", module, err)
continue
}
go h.follow(ctx, kv, module)
}
return h
}
func (h *HeardEnds) follow(ctx context.Context, kv jetstream.KeyValue, module string) {
silent := 0 // closes in a row having heard nothing
for ctx.Err() == nil {
w, err := kv.WatchAll(ctx, jetstream.UpdatesOnly())
if err != nil {
h.logf("the steps of %s cannot be watched, so an ask whose own watch goes deaf is read only at a reconnect: %v", module, err)
return
}
heard := false
for e := range w.Updates() {
if e == nil || e.Operation() != jetstream.KeyValuePut {
continue
}
heard = true
if step, err := delivery.DecodeStep(e.Value()); err == nil && step.State != delivery.StepRunning {
h.Nudge(module)
}
}
_ = w.Stop()
if ctx.Err() != nil {
return
}
h.Nudge(module) // a step may have ended while the watch was closing
if heard {
silent = 0
} else if silent++; silent >= reopenTries {
h.logf("the watch of the steps of %s closed %d times in a row having heard nothing, so it is left: an ask whose own watch goes deaf is read only at a reconnect", module, silent)
return
}
h.logf("the watch of the steps of %s closed; opened again", module)
}
}
// AskSteps is StepReader's: the last step written for the ask, if any.
func (f *FakeAsks) AskSteps(_ context.Context, _ string, askID string) ([]delivery.Step, error) {
f.mu.Lock()
defer f.mu.Unlock()
if s := f.steps[askID]; len(s) > 0 {
return []delivery.Step{s[len(s)-1]}, nil
}
return nil, nil
}
@@ -0,0 +1,503 @@
package buildrun
import (
"context"
"encoding/json"
"errors"
"fmt"
"strconv"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
)
// **An ask the build seat gave up on ends its step** (novox/hq issue 734). A build seat's worker offers an ask
// delivery.AskMaxDeliver times; one handed out that often without an answer (a bus reload that drops every hand-out, a node
// that dies taking it each time) is given up by the server, and the controller keeps a copy of it in DEAD_LETTERS under
// mesh.events.dead.<seat stream>.<worker>, with the ask's message id (its ask id), how often it was handed out and when it
// was given up. No build node writes a step for it, so an asker that waits for the step waits for ever: a try stays
// running and holds a try place, a check's verdict stays running and its pull request waits.
//
// An asker of the build seats (the checker, the builder) hears those copies (DeadLetterHearing: an ordered consumer on
// DEAD_LETTERS filtered to the build seats' streams, granted to a user of a seat pulled by kind, buscompose
// DeadLetterReads) into DeadLetters, and LostAskWaiter ends the waiting step at once as failed on the node LostNode, its
// reason saying LostWords; the asker decides what follows (the checker ends a try as error and asks a check again, the
// builder asks its part of a build again).
//
// Belt and braces, where the asker says so in the context (NotTakenBound; the checker's tries): an ask no build node took
// within a bound ends the same way, saying NotTakenWords. An ask waits in the stream as long as the backlog is otherwise
// (ADR 0525 §9): a backlog is no fault.
// LostNode is the node a step ended for a lost ask names: no build node wrote it.
const LostNode = "bus"
// The words of a step ended for its ask.
const (
LostWords = "its ask was given up by the build seat"
NotTakenWords = "no build node took its ask"
)
// The headers the controller keeps a given-up message's copy with in DEAD_LETTERS (mesh-controller internal/link
// deadletters.go): the original message id (an ask's is its ask id), how often it was handed out, when it was given up.
const (
DeadHeaderMsgID = "Mesh-Dead-Msg-Id"
DeadHeaderDeliveries = "Mesh-Dead-Deliveries"
DeadHeaderGaveUp = "Mesh-Dead-Gave-Up"
DeadHeaderStream = "Mesh-Dead-Stream"
)
// DeadLettersStream is the stream the controller keeps given-up messages in, and DeadLetterFilter the subjects of one seat
// stream's copies there (buscompose DeadLetterSubject).
const DeadLettersStream = "DEAD_LETTERS"
// DeadLetterFilter is the subjects DEAD_LETTERS keeps a seat stream's given-up messages under.
func DeadLetterFilter(stream string) string { return "mesh.events.dead." + stream + ".>" }
// SeatStream is the stream of a build seat's language (buscompose seatStreamName of node-build-<language>).
func SeatStream(language string) string { return "SEAT_NODE_BUILD_" + strings.ToUpper(language) }
// GivenUp is one ask a build seat gave up on, as DEAD_LETTERS keeps it.
type GivenUp struct {
AskID string
Stream string
Deliveries int
At time.Time
}
// GivenUpOf reads a DEAD_LETTERS copy's headers; false when it names no message id.
func GivenUpOf(header map[string][]string) (GivenUp, bool) {
get := func(k string) string {
if v := header[k]; len(v) > 0 {
return strings.TrimSpace(v[0])
}
return ""
}
g := GivenUp{AskID: get(DeadHeaderMsgID), Stream: get(DeadHeaderStream)}
if g.AskID == "" {
return GivenUp{}, false
}
g.Deliveries, _ = strconv.Atoi(get(DeadHeaderDeliveries))
g.At, _ = time.Parse(time.RFC3339Nano, get(DeadHeaderGaveUp))
return g, true
}
// Words are what a step ended for the given-up ask says.
func (g GivenUp) Words() string {
at := "at an unknown time"
if !g.At.IsZero() {
at = "at " + g.At.UTC().Format(time.RFC3339)
}
return fmt.Sprintf("%s after %d deliveries %s (kept in %s); ask again", LostWords, g.Deliveries, at, DeadLettersStream)
}
// keptGivenUp is how many given-up asks DeadLetters remembers for a waiter that starts after the copy was heard.
const keptGivenUp = 1024
// DeadLetters holds the given-up asks heard, and hands each to the waiter of its ask. Safe for concurrent use.
type DeadLetters struct {
mu sync.Mutex
heard map[string]GivenUp
order []string
watchers map[string][]chan GivenUp
}
// Heard takes one given-up ask in.
func (d *DeadLetters) Heard(g GivenUp) {
d.mu.Lock()
if d.heard == nil {
d.heard, d.watchers = map[string]GivenUp{}, map[string][]chan GivenUp{}
}
if _, ok := d.heard[g.AskID]; !ok {
d.order = append(d.order, g.AskID)
if len(d.order) > keptGivenUp {
delete(d.heard, d.order[0])
d.order = d.order[1:]
}
}
d.heard[g.AskID] = g
ws := d.watchers[g.AskID]
delete(d.watchers, g.AskID)
d.mu.Unlock()
for _, w := range ws {
w <- g // buffered, one each
}
}
// watch answers a channel that gets the ask's given-up copy once (at once when it was heard already), and its cancel.
func (d *DeadLetters) watch(askID string) (<-chan GivenUp, func()) {
ch := make(chan GivenUp, 1)
d.mu.Lock()
defer d.mu.Unlock()
if d.heard == nil {
d.heard, d.watchers = map[string]GivenUp{}, map[string][]chan GivenUp{}
}
if g, ok := d.heard[askID]; ok {
ch <- g
return ch, func() {}
}
d.watchers[askID] = append(d.watchers[askID], ch)
return ch, func() {
d.mu.Lock()
defer d.mu.Unlock()
ws := d.watchers[askID]
for i, w := range ws {
if w == ch {
d.watchers[askID] = append(ws[:i], ws[i+1:]...)
break
}
}
if len(d.watchers[askID]) == 0 {
delete(d.watchers, askID)
}
}
}
type notTakenKey struct{}
// NotTakenBound says, in ctx, that an ask no build node took within bound ends its step (a try's, never a check's).
func NotTakenBound(ctx context.Context, bound time.Duration) context.Context {
return context.WithValue(ctx, notTakenKey{}, bound)
}
func notTakenBoundOf(ctx context.Context) time.Duration {
d, _ := ctx.Value(notTakenKey{}).(time.Duration)
return d
}
// LostAskWaiter is an AskWaiter that ends an ask's step when the build seat gave the ask up (DeadLetters), or, under
// NotTakenBound, when no build node took it within the bound: it hands the asker a failed step whose reason says why, on
// the node "bus". Every step the build module writes is handed on as it is.
type LostAskWaiter struct {
Inner AskWaiter
Dead *DeadLetters
// Hearing, when set, is asked to hear DEAD_LETTERS again each time an ask is waited for while it cannot: an ask put is
// the event that makes hearing matter.
Hearing interface {
Deaf() string
Again()
}
Now func() time.Time
}
func (w LostAskWaiter) now() time.Time {
if w.Now != nil {
return w.Now()
}
return time.Now()
}
// WatchAsk is AskWaiter's.
func (w LostAskWaiter) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
if w.Hearing != nil && w.Hearing.Deaf() != "" {
w.Hearing.Again()
}
inner, err := w.Inner.WatchAsk(ctx, module, askID)
if err != nil {
return nil, err
}
var dead <-chan GivenUp
cancel := func() {}
if w.Dead != nil {
dead, cancel = w.Dead.watch(askID)
}
// The not-taken bound is a context's (ADR 0437 §8: no timer), ended when a node takes the ask.
var notTaken <-chan struct{}
bound := notTakenBoundOf(ctx)
stopBound := func() {}
if bound > 0 {
var nt context.Context
nt, stopBound = context.WithTimeout(context.Background(), bound)
notTaken = nt.Done()
}
out := make(chan delivery.Step)
go func() {
defer close(out)
defer cancel()
defer func() { stopBound() }()
send := func(s delivery.Step) bool {
select {
case out <- s:
return true
case <-ctx.Done():
return false
}
}
ended := func(reason string) delivery.Step {
at := w.now().UTC()
return delivery.Step{ID: askID, Node: LostNode, State: delivery.StepFailed, Reason: reason, Ended: at}
}
for {
select {
case <-ctx.Done():
return
case s, ok := <-inner:
if !ok {
return
}
if notTaken != nil {
stopBound() // a node took it: its step is bounded by the asker from its start
notTaken = nil
}
if !send(s) {
return
}
case g := <-dead:
dead = nil
send(ended(g.Words()))
return
case <-notTaken:
send(ended(fmt.Sprintf("%s within %s of its put; it was lost or waits behind a backlog; ask again", NotTakenWords, bound)))
return
}
}
}()
return out, nil
}
// IsLost says whether a step was ended by LostAskWaiter because the build seat gave its ask up (LostWords).
func IsLost(s delivery.Step) bool { return s.Node == LostNode && strings.Contains(s.Reason, LostWords) }
// IsNotTaken says whether a step was ended by LostAskWaiter because no build node took its ask within the bound.
func IsNotTaken(s delivery.Step) bool {
return s.Node == LostNode && strings.Contains(s.Reason, NotTakenWords)
}
// **The asker hears DEAD_LETTERS, tries again when it cannot, and says whether it hears** (issue 734). The read is
// granted with the asker's seat traffic (buscompose DeadLetterReads); an asker started before its grant was composed is
// refused, and an asker that only tried once would stay deaf until it starts again, silently. DeadLetterHearing tries in
// the background, never holding the start up, and again on an event (Again): the asker's reconnect to the bus, and every
// ask waited for while it is deaf (LostAskWaiter.Hearing); no timer. Each change of whether it hears is said (Said), so
// the module's health tool says it: a deaf asker is not healthy.
// Hearing is what a DeadLetterHearing says: whether it hears every build seat stream it follows, and why not.
type Hearing struct {
Heard bool `json:"heard"`
Streams []string `json:"streams"`
Why string `json:"why,omitempty"`
At time.Time `json:"at"`
}
// HearingKey is the key a module keeps its Hearing under, in its own state (the checker's and the builder's hearing).
const HearingKey = "dead-letters"
// DeadLetterFollower starts following one seat stream's copies in DEAD_LETTERS from since, handing each given-up ask to
// heard; it answers the stop, or why it cannot.
type DeadLetterFollower interface {
Follow(ctx context.Context, stream string, since time.Time, heard func(GivenUp)) (func(), error)
}
// JetStreamDeadLetters is the bus side of a DeadLetterFollower: an ordered consumer on DEAD_LETTERS filtered to the
// stream's copies, which resumes where it was after a reconnect and, reset before it heard any, starts at since again.
type JetStreamDeadLetters struct{ JS jetstream.JetStream }
// Follow is DeadLetterFollower's.
func (j JetStreamDeadLetters) Follow(ctx context.Context, stream string, since time.Time, heard func(GivenUp)) (func(), error) {
cons, err := j.JS.OrderedConsumer(ctx, DeadLettersStream, jetstream.OrderedConsumerConfig{
FilterSubjects: []string{DeadLetterFilter(stream)}, DeliverPolicy: jetstream.DeliverByStartTimePolicy, OptStartTime: &since})
if err != nil {
return nil, err
}
cc, err := cons.Consume(func(m jetstream.Msg) {
if g, ok := GivenUpOf(m.Headers()); ok {
heard(g)
}
})
if err != nil {
return nil, err
}
return cc.Stop, nil
}
// DeadLetterHearing follows the build seats' given-up asks into Dead. Make it with NewDeadLetterHearing.
type DeadLetterHearing struct {
follower DeadLetterFollower
streams []string
dead *DeadLetters
logf func(string, ...any)
said func(Hearing)
ctx context.Context
since time.Time
mu sync.Mutex
stops map[string]func()
why map[string]string
trying bool
again bool
told *Hearing
wg sync.WaitGroup
}
// NewDeadLetterHearing starts hearing the given-up asks of each language's build seat stream, in the background. said
// is told each change of whether it hears (nil tells nothing); ctx ends it.
func NewDeadLetterHearing(ctx context.Context, follower DeadLetterFollower, languages []string, dead *DeadLetters,
logf func(string, ...any), said func(Hearing)) *DeadLetterHearing {
h := &DeadLetterHearing{follower: follower, dead: dead, logf: logf, said: said, ctx: ctx, since: time.Now(),
stops: map[string]func(){}, why: map[string]string{}}
for _, l := range languages {
s := SeatStream(l)
h.streams = append(h.streams, s)
h.why[s] = "not tried yet"
}
if h.logf == nil {
h.logf = func(string, ...any) {}
}
h.Again()
return h
}
// Deaf says why a stream is not heard, or "" when every one is.
func (h *DeadLetterHearing) Deaf() string {
h.mu.Lock()
defer h.mu.Unlock()
return h.deafLocked()
}
func (h *DeadLetterHearing) deafLocked() string {
var out []string
for _, s := range h.streams {
if why, deaf := h.why[s]; deaf {
out = append(out, s+": "+why)
}
}
if len(out) == 0 {
return ""
}
return DeadLettersStream + " is not heard, so an ask a build seat gives up on is waited for until its bound: " +
strings.Join(out, "; ")
}
// Again tries, in the background, every stream not heard yet; a call while it tries makes it try once more after.
func (h *DeadLetterHearing) Again() {
h.mu.Lock()
if h.trying {
h.again = true
h.mu.Unlock()
return
}
h.trying = true
h.mu.Unlock()
h.wg.Add(1)
go func() {
defer h.wg.Done()
for {
h.try()
h.mu.Lock()
if !h.again || h.ctx.Err() != nil {
h.trying = false
h.mu.Unlock()
return
}
h.again = false
h.mu.Unlock()
}
}()
}
// Wait returns when no try runs.
func (h *DeadLetterHearing) Wait() { h.wg.Wait() }
// Stop ends every stream's following.
func (h *DeadLetterHearing) Stop() {
h.mu.Lock()
defer h.mu.Unlock()
for s, stop := range h.stops {
stop()
delete(h.stops, s)
}
}
func (h *DeadLetterHearing) try() {
for _, s := range h.streams {
h.mu.Lock()
_, deaf := h.why[s]
h.mu.Unlock()
if !deaf || h.ctx.Err() != nil {
continue
}
stream := s
stop, err := h.follower.Follow(h.ctx, stream, h.since, func(g GivenUp) {
h.logf("the ask %s on %s was given up after %d deliveries at %s: the wait for its step ends", g.AskID, g.Stream,
g.Deliveries, g.At.Format(time.RFC3339))
h.dead.Heard(g)
})
h.mu.Lock()
if err != nil {
h.why[stream] = err.Error()
h.logf("the asks %s gives up on are not heard (%s of %s); tried again at the next reconnect or ask: %v",
stream, DeadLetterFilter(stream), DeadLettersStream, err)
} else {
delete(h.why, stream)
h.stops[stream] = stop
h.logf("the asks %s gives up on are heard from %s", stream, DeadLettersStream)
}
h.mu.Unlock()
}
h.tell()
}
// tell says the hearing when it changed.
func (h *DeadLetterHearing) tell() {
h.mu.Lock()
why := h.deafLocked()
now := Hearing{Heard: why == "", Streams: append([]string(nil), h.streams...), Why: why, At: time.Now().UTC()}
same := h.told != nil && h.told.Heard == now.Heard && h.told.Why == now.Why
if !same {
h.told = &now
}
h.mu.Unlock()
if !same && h.said != nil {
h.said(now)
}
}
// HearingState is the name of the state an asker keeps its Hearing in (`<module>_hearing`, key HearingKey), which its
// health tool reads.
const HearingState = "hearing"
// HearingKeeper is where a Hearing is kept: a jetstream.KeyValue, or a process.AwaitedBucket for a state that may not be
// on the bus at the part's start (issue 1790), which keeps the latest until it appears and answers process.ErrNotYet.
type HearingKeeper interface {
Put(ctx context.Context, key string, value []byte) (uint64, error)
}
// SayHearing is a DeadLetterHearing's said that keeps each Hearing in kv under HearingKey; nil kv keeps nothing and
// says so once. A write kv keeps for later (process.ErrNotYet) is not an error: the awaited bucket said it waits.
func SayHearing(ctx context.Context, kv HearingKeeper, logf func(string, ...any)) func(Hearing) {
if kv == nil {
if logf != nil {
logf("whether %s is heard is not kept: the state %s is not there, so the health tool cannot say it", DeadLettersStream, HearingState)
}
return nil
}
return func(h Hearing) {
data, err := json.Marshal(h)
if err == nil {
_, err = kv.Put(ctx, HearingKey, data)
}
if err != nil && !errors.Is(err, process.ErrNotYet) && logf != nil {
logf("whether %s is heard could not be kept: %v", DeadLettersStream, err)
}
}
}
// HearingHealth is the health answer of an asker's kept Hearing (raw, empty when none is kept): healthy while it hears
// DEAD_LETTERS, unhealthy saying why while it does not or has not said.
func HearingHealth(raw []byte) health.Answer {
if len(raw) == 0 {
return health.Unhealthy("it has not said whether it hears " + DeadLettersStream + " (issue 734): it is starting, or its state " + HearingState + " is not there")
}
var h Hearing
if err := json.Unmarshal(raw, &h); err != nil {
return health.Unhealthy("what it keeps of hearing " + DeadLettersStream + " is not a hearing: " + err.Error())
}
if !h.Heard {
return health.Unhealthy(h.Why + " (since " + h.At.Format(time.RFC3339) + "; tried again at its next reconnect or ask)")
}
return health.Healthy("it hears " + DeadLettersStream + " for " + strings.Join(h.Streams, ", "))
}
@@ -0,0 +1,7 @@
package buildrun
// The build modules the mesh runs, by the language each builds (ADR 0469).
const (
GoBuildModule = "mesh-build-go"
NodeBuildModule = "mesh-build-node"
)
@@ -0,0 +1,145 @@
package buildrun
import (
"context"
"fmt"
"sort"
"sync"
)
// DefaultCapacity is the build capacity of a node that has no setting: one build step at a time.
const DefaultCapacity = 1
// Places counts the build steps a part runs on each node and holds a lane back while its node is at its
// capacity (the operator's decision of 2026-10-21; ADR 0478 clause 3). A lane is the fetch and the builds of one build
// module on one node, run one step at a time, so a lane holds one place while it runs. A node is chosen for a lane by
// reserving a place on it at the moment of the choice (ChooseNodes), so that builds choosing at once see each other's
// choices and none waits on a full node while another has room. A lane waits for its place by watching the ends of the
// other lanes, not by a timer: ending a lane wakes the waiting ones.
type Places struct {
// Capacity is the number of steps at once by node; a node not in it has DefaultCapacity.
Capacity map[string]int
mu sync.Mutex
running map[string]int // lanes holding a place
reserved map[string]int // lanes chosen for the node that do not hold a place yet
changed chan struct{} // closed, and replaced, each time a lane ends
}
// CapacityOf is how many steps the part runs on the node at once: DefaultCapacity for a node the setting does not
// name, and 0 for one it names so, where the part never builds.
func (p *Places) CapacityOf(node string) int {
if n, ok := p.Capacity[node]; ok && n >= 0 {
return n
}
return DefaultCapacity
}
// Running is the steps in flight now, by node.
func (p *Places) Running() map[string]int {
p.mu.Lock()
defer p.mu.Unlock()
out := make(map[string]int, len(p.running))
for n, c := range p.running {
if c > 0 {
out[n] = c
}
}
return out
}
// Reserved is the lanes chosen for a node that hold no place yet, by node.
func (p *Places) Reserved() map[string]int {
p.mu.Lock()
defer p.mu.Unlock()
out := make(map[string]int, len(p.reserved))
for n, c := range p.reserved {
if c > 0 {
out[n] = c
}
}
return out
}
func (p *Places) init() {
if p.running == nil {
p.running = map[string]int{}
}
if p.reserved == nil {
p.reserved = map[string]int{}
}
if p.changed == nil {
p.changed = make(chan struct{})
}
}
// Acquire takes a place on the node, waiting while the node is at its capacity, and answers the function that gives it
// back. A place reserved for the node by a choice is the one taken. It returns an error when ctx ends first, and then
// gives up the reservation too.
func (p *Places) Acquire(ctx context.Context, node string) (release func(), err error) {
for {
p.mu.Lock()
p.init()
if p.running[node] < p.CapacityOf(node) {
p.running[node]++
if p.reserved[node] > 0 {
p.reserved[node]--
}
p.mu.Unlock()
var once sync.Once
return func() {
once.Do(func() {
p.mu.Lock()
p.running[node]--
close(p.changed)
p.changed = make(chan struct{})
p.mu.Unlock()
})
}, nil
}
wait := p.changed
p.mu.Unlock()
select {
case <-wait:
case <-ctx.Done():
p.Cancel(node)
return nil, fmt.Errorf("waiting for a place on %s: %w", node, ctx.Err())
}
}
}
// Cancel gives up a place reserved for the node that will not be taken.
func (p *Places) Cancel(node string) {
p.mu.Lock()
defer p.mu.Unlock()
p.init()
if p.reserved[node] > 0 {
p.reserved[node]--
}
}
// choose orders the nodes (those with room first, counting the lanes running and the lanes reserved, then the fewest of
// them, then by name), keeps the first want and reserves a place on each, all at once.
func (p *Places) choose(nodes []string, want int) []string {
p.mu.Lock()
defer p.mu.Unlock()
p.init()
load := func(n string) int { return p.running[n] + p.reserved[n] }
sort.SliceStable(nodes, func(i, j int) bool {
ri, rj := load(nodes[i]) < p.CapacityOf(nodes[i]), load(nodes[j]) < p.CapacityOf(nodes[j])
if ri != rj {
return ri
}
if load(nodes[i]) != load(nodes[j]) {
return load(nodes[i]) < load(nodes[j])
}
return nodes[i] < nodes[j]
})
if len(nodes) > want {
nodes = nodes[:want]
}
for _, n := range nodes {
p.reserved[n]++
}
return nodes
}
@@ -0,0 +1,104 @@
package buildrun
import (
"context"
"strings"
"testing"
"time"
)
func acquireAsync(p *Places, node string) chan func() {
got := make(chan func(), 1)
go func() {
release, err := p.Acquire(context.Background(), node)
if err != nil {
close(got)
return
}
got <- release
}()
return got
}
func TestThreeStepsOnANodeOfCapacityTwoRunTwoAtOnceAndTheThirdStartsWhenOneEnds(t *testing.T) { // round 3
p := &Places{Capacity: map[string]int{"ace": 2}}
a, b := acquireAsync(p, "ace"), acquireAsync(p, "ace")
ra, rb := <-a, <-b
third := acquireAsync(p, "ace")
select {
case <-third:
t.Fatal("a third step started on a node of capacity 2 with two running")
case <-time.After(10 * time.Millisecond):
}
if p.Running()["ace"] != 2 {
t.Errorf("running = %v, want 2 on ace", p.Running())
}
ra()
select {
case r := <-third:
r()
case <-time.After(2 * time.Second):
t.Fatal("the third step did not start when one ended")
}
rb()
}
func TestANodeWithNoSettingRunsOneStepAtATime(t *testing.T) { // round 3
p := &Places{Capacity: map[string]int{"shanks": 3}}
first := <-acquireAsync(p, "g14")
second := acquireAsync(p, "g14")
select {
case <-second:
t.Fatal("two steps on a node without a setting")
case <-time.After(10 * time.Millisecond):
}
first()
r := <-second
r()
if p.CapacityOf("g14") != 1 || p.CapacityOf("shanks") != 3 {
t.Errorf("capacities = %d, %d", p.CapacityOf("g14"), p.CapacityOf("shanks"))
}
}
func TestAWaitingStepStopsWhenItsContextEnds(t *testing.T) { // round 3
p := &Places{}
r := <-acquireAsync(p, "ace")
defer r()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { _, err := p.Acquire(ctx, "ace"); done <- err }()
cancel()
select {
case err := <-done:
if err == nil {
t.Error("Acquire returned no error for a cancelled wait")
}
case <-time.After(2 * time.Second):
t.Fatal("a cancelled wait did not return")
}
}
// The operator's decision of 2026-10-21 13:40: the builder builds on shanks and ace only; a capacity of 0 keeps it off a node.
func TestACapacityOfZeroKeepsThePartOffTheNodeAndANodeNotNamedKeepsOne(t *testing.T) { // issue 1661
p := &Places{Capacity: map[string]int{"shanks": 3, "ace": 2, "g14": 0, "novox": 0}}
if p.CapacityOf("g14") != 0 || p.CapacityOf("novox") != 0 || p.CapacityOf("ace") != 2 || p.CapacityOf("other") != DefaultCapacity {
t.Errorf("capacities = %d %d %d %d, want 0 0 2 and the default for a node not named", p.CapacityOf("g14"), p.CapacityOf("novox"), p.CapacityOf("ace"), p.CapacityOf("other"))
}
entries := tcs("go", "go1.27.1", "ace", "g14", "novox", "shanks", "other")
for i := 0; i < 6; i++ {
nodes, _, err := ChooseNodes("go", entries, p, nil, 2)
if err != nil {
t.Fatal(err)
}
for _, n := range nodes {
if n == "g14" || n == "novox" {
t.Fatalf("choice %d: %v includes a node of capacity 0", i, nodes)
}
}
}
// Only nodes of capacity 0 report: nothing qualifies, and the build says why.
_, _, err := ChooseNodes("go", tcs("go", "go1.27.1", "g14", "novox"), p, nil, 2)
if err == nil || !strings.Contains(err.Error(), "capacity") {
t.Errorf("only capacity-0 nodes: error = %v, want it to name the capacity", err)
}
}
@@ -0,0 +1,53 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// PresenceSet holds which nodes are online, as the watchman writes it (mesh-delivery-watchman_presence, key `<node>`, the
// node-engine's connection): a node asleep or off the bus is offline and its build module cannot answer. A node the
// watchman has no key of is not online. Safe for use from the watch and from a build at once.
type PresenceSet struct {
mu sync.Mutex
online map[string]bool
}
// Put takes a change of the presence bucket in and answers whether a node's online state changed. A part's key and a
// value that is not a presence are left out.
func (s *PresenceSet) Put(key string, value []byte, deleted bool) bool {
node, part, _, err := delivery.ParsePresenceKey(key)
if err != nil || part != "" {
return false
}
on := false
if !deleted {
p, err := delivery.DecodePresence(value)
if err != nil {
return false
}
on = p.Online
}
s.mu.Lock()
defer s.mu.Unlock()
if s.online[node] == on {
return false
}
if s.online == nil {
s.online = map[string]bool{}
}
if on {
s.online[node] = true
} else {
delete(s.online, node)
}
return true
}
// Online is whether the watchman last saw the node online.
func (s *PresenceSet) Online(node string) bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.online[node]
}
@@ -0,0 +1,43 @@
package buildrun
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func presenceValue(t *testing.T, online bool) []byte {
t.Helper()
b, err := delivery.EncodePresence(delivery.Presence{Online: online})
if err != nil {
t.Fatal(err)
}
return b
}
func TestPresenceSetKnowsOnlyNodesTheWatchmanSawOnline(t *testing.T) { // 1607 review 3
var s PresenceSet
if s.Online("ace") {
t.Error("a node the watchman never wrote is online")
}
if !s.Put("ace", presenceValue(t, true), false) || !s.Online("ace") {
t.Error("ace online was not taken in")
}
if s.Put("ace", presenceValue(t, true), false) {
t.Error("the same presence again counted as a change")
}
if !s.Put("ace", presenceValue(t, false), false) || s.Online("ace") {
t.Error("ace going offline was not taken in")
}
s.Put("g14", presenceValue(t, true), false)
if !s.Put("g14", nil, true) || s.Online("g14") {
t.Error("a deleted key is still online")
}
// A part's instance (part.<part>.<instance>) is not a node.
if s.Put("part.mesh-delivery-builder.novox", presenceValue(t, true), false) || s.Online("part") {
t.Error("a part's key was taken for a node")
}
if s.Put("novox", []byte("junk"), false) {
t.Error("junk counted")
}
}
@@ -0,0 +1,248 @@
package buildrun
import (
"context"
"errors"
"fmt"
"maps"
"slices"
"sort"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module leaves a node: no part writes another module's state, and the module is gone. A report of a node the
// build module no longer runs on is therefore no builder's (issue 1940: mesh-build-node taken off ace on 2026-10-23 left
// ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// ToolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// ToolchainSet): an entry is passed on only while the placement (the seat mesh-placement's state `placement`, held by
// the assigner, delivery.PlacementBucket, read by delivery.Placed; ADR 0567, ADR 0549 §3) places its build module on
// its node, and only once it was written since that placement; it is passed on as deleted otherwise, said once, and
// passed on when its node writes it again. A report written before the placement is one an earlier run of the module
// left (issue 1940, 2026-10-25 20:56: mesh-build-node put on ace again brought ace.typescript v26.12.0 of 2026-10-23
// back into the agreement before the module ran there). Until the placement is settled, and while no node's placement
// is held (before the assigner's adopt writes the first one, or where the state cannot be read), every entry is passed
// on. Builders that run and report different versions still disagree: that refusal is the guard of novox/hq ADR 0465,
// and it stays.
//
// This replaces the copies of the gate the indexer, the builder and the checker each kept over the controller's
// assignments (issue 1940's subtask, issue 774: those go with the controller's lease, ADR 0549 §6).
type ToolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
placed *delivery.Placed
known bool // the placement is settled
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
// OnPlaced is told, outside the gate's lock, once the placement's history is read and after each revision of it
// FollowPlacement hears from then on (the counted nodes, Nodes, may have changed). Nil tells nobody.
OnPlaced func()
}
// NewToolchainGate is a gate over the toolchains of modules, passing what stands on to forward and saying with logf what
// it leaves out.
func NewToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *ToolchainGate {
g := &ToolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, placed: delivery.NewPlaced(),
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *ToolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Placement takes one revision of the seat mesh-placement's placement in (at is when the bus wrote it) and answers whether what it
// passed on changed. A value that does not read is said, and places nothing on its node.
func (g *ToolchainGate) Placement(key string, value []byte, deleted bool, at time.Time) bool {
g.mu.Lock()
defer g.mu.Unlock()
was := g.placed.Seeded()
node, changed, err := g.placed.Put(key, value, deleted, at)
if err != nil {
g.logf("%s: %v (issue 1940)", delivery.PlacementBucket, err)
}
if !g.known {
return false
}
if was != g.placed.Seeded() {
// The first placement came, or the last went: every report's standing changed.
return g.all()
}
return g.again(node, changed)
}
// Nodes are the nodes the placement places any module on, sorted: the nodes the mesh counts (a node leaves the count
// when the placement no longer names it; ADR 0624 §1). None until the placement is read.
func (g *ToolchainGate) Nodes() []string {
var out []string
for node, modules := range g.placed.ByNode() {
if len(modules) > 0 {
out = append(out, node)
}
}
sort.Strings(out)
return out
}
// Settled says the placement's history is read: from now on only the reports of placed nodes pass, once a node's
// placement is held. It answers whether what it passed on changed.
func (g *ToolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
return g.all()
}
// all passes on every report again. The caller holds mu.
func (g *ToolchainGate) all() bool {
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// again passes on every report of node by a build module among modules. The caller holds mu.
func (g *ToolchainGate) again(node string, modules []string) bool {
changed := false
for _, module := range modules {
if !g.modules[module] {
continue
}
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is placed and wrote it since (or the placement is not
// known), deleted otherwise. The caller holds mu.
func (g *ToolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known && g.placed.Seeded() {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: the catalogue does not place the module on its
// node, or the report was written before it did. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *ToolchainGate) leftOut(module, node string, value []byte) string {
since, placed := g.placed.Since(node, module)
if !placed {
return fmt.Sprintf("the catalogue does not place %s on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s was placed on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// FollowPlacement reads the seat mesh-placement's placement with its history into gate, says it settled, and follows it; changed
// is told whenever what the gate passes on changed. Where the state cannot be read or watched (not there, or this part
// may not read it) it says so and the gate passes every report on: not fatal, since an indexer, builder or checker that
// stops for it stops every delivery (issue 1945). An error of the watch after that ends on the channel.
func FollowPlacement(ctx context.Context, js process.BucketOpener, gate *ToolchainGate, changed func(context.Context), logf func(string, ...any)) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, delivery.PlacementBucket)
if err != nil {
logf("the state %s cannot be read, so every node's toolchain report counts, a build node's the catalogue does not place too (issue 1940): %v", delivery.PlacementBucket, err)
return errs, nil
}
w, err := kv.WatchAll(ctx, jetstream.IncludeHistory())
if err != nil {
logf("the state %s cannot be watched, so every node's toolchain report counts, a build node's the catalogue does not place too (issue 1940): %v", delivery.PlacementBucket, err)
return errs, nil
}
put := func(e jetstream.KeyValueEntry) bool {
return gate.Placement(e.Key(), e.Value(), e.Operation() != jetstream.KeyValuePut, e.Created())
}
for settled := false; !settled; {
select {
case <-ctx.Done():
_ = w.Stop()
return errs, ctx.Err()
case e, ok := <-w.Updates():
if !ok {
return errs, errors.New("the watch of " + delivery.PlacementBucket + " ended before it settled")
}
if e == nil {
settled = true
continue
}
put(e)
}
}
if gate.Settled() {
changed(ctx)
}
if gate.OnPlaced != nil {
gate.OnPlaced()
}
go func() {
defer func() { _ = w.Stop() }()
for {
select {
case <-ctx.Done():
errs <- ctx.Err()
return
case e, ok := <-w.Updates():
if !ok {
errs <- errors.New("the watch of " + delivery.PlacementBucket + " ended")
return
}
if e != nil && put(e) {
changed(ctx)
}
if e != nil && gate.OnPlaced != nil {
gate.OnPlaced()
}
}
}
}()
return errs, nil
}
@@ -0,0 +1,263 @@
package buildrun
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func gateReport(t *testing.T, node, version string, at time.Time) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
// placing is a value of the state `placement` placing modules on node.
func placing(t *testing.T, node string, modules ...string) (string, []byte) {
t.Helper()
p := delivery.Placement{Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
values, err := delivery.PlacementValues(p)
if err != nil {
t.Fatal(err)
}
key, _ := delivery.PlacementNodeKey(node)
return key, values[key]
}
// placingLegacy is placing in the form before, `placement.<node>` (research 072 row 37).
func placingLegacy(t *testing.T, node string, modules ...string) (string, []byte) {
t.Helper()
p := delivery.Placement{Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
key, _ := delivery.LegacyPlacementKey(node)
value, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// Issue 1940, the gate over the catalogue's placement (issue 774, ADR 0549 §3): a build node's report counts only while
// the placement places its build module on its node, and only once written since; before the placement is settled, and
// while no node's placement is held, every report counts. The live cases: mesh-build-node taken off ace left
// ace.typescript at v26.12.0 against v26.12.1 on the three others; put on ace again, its report of 2026-10-23 06:29 came
// back before the module ran there.
func TestToolchainGate(t *testing.T) {
old := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
placed := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node placed on ace again
three := []string{"g14", "novox", "shanks"}
start := func(t *testing.T, seeded bool) (*ToolchainGate, *ToolchainSet, *[]string) {
set := &ToolchainSet{}
said := &[]string{}
gate := NewToolchainGate([]string{GoBuildModule, NodeBuildModule}, set.Put, func(f string, a ...any) { *said = append(*said, fmt.Sprintf(f, a...)) })
for _, node := range three {
if seeded {
key, value := placing(t, node, NodeBuildModule)
gate.Placement(key, value, false, old.Add(-time.Hour))
}
key, value := gateReport(t, node, "v26.12.1", placed.Add(time.Minute))
gate.Toolchain(NodeBuildModule, key, value, false)
}
return gate, set, said
}
t.Run("not placed on ace: its report is no builder's", func(t *testing.T) {
gate, set, said := start(t, true)
key, value := gateReport(t, "ace", "v26.12.0", old)
gate.Toolchain(NodeBuildModule, key, value, false)
if a := delivery.ToolchainAgreement(set.Entries("typescript")); len(a.SittingOut) != 1 || a.SittingOut[0].Node != "ace" { // ADR 0584: the majority is agreed, ace's report counts as the one that sits out
t.Fatalf("before the placement is settled every report counts; got %+v", a)
}
k, v := placing(t, "ace", GoBuildModule) // another build module on ace: no matter
gate.Placement(k, v, false, old)
if !gate.Settled() {
t.Fatal("settling left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the placed nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(*said) != 1 || !strings.Contains((*said)[0], "does not place mesh-build-node on ace") {
t.Fatalf("ace's report is said left out once; said %q", *said)
}
if gate.Toolchain(NodeBuildModule, key, value, false) {
t.Fatal("a report of a node not placed changed what was passed on")
}
// Placed on ace with a report written since: it counts, and the builders really disagree.
k, v = placing(t, "ace", GoBuildModule, NodeBuildModule)
gate.Placement(k, v, false, old.Add(-time.Minute))
if a := delivery.ToolchainAgreement(set.Entries("typescript")); a.Toolchain.Version != "v26.12.1" || len(a.SittingOut) != 1 || a.SittingOut[0].Version != "v26.12.0" {
t.Fatalf("builders that run different versions: the majority is agreed and ace sits out; got %+v", a)
}
// Taken off ace again (its key deleted): its report goes again.
if !gate.Placement(k, nil, true, placed) {
t.Fatal("taking ace's placement away passed nothing on")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
// A report deleted from the bucket goes whether its node is placed or not.
key, _ = gateReport(t, "g14", "v26.12.1", placed)
if !gate.Toolchain(NodeBuildModule, key, nil, true) || len(set.Entries("typescript")) != 2 {
t.Fatalf("deleting g14's report: %d entries left, want novox and shanks", len(set.Entries("typescript")))
}
})
for _, c := range []struct {
name string
at time.Time // when ace wrote its report
counts bool
}{
{"a report left before the placement", old, false},
{"a report written since the placement", placed.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
gate, set, said := start(t, true)
k, v := placing(t, "ace") // ace's key, first heard placing nothing
gate.Placement(k, v, false, old.Add(-time.Hour))
gate.Settled()
k, v = placing(t, "ace", NodeBuildModule)
if gate.Placement(k, v, false, placed) {
t.Fatal("a placement with no report passed something on")
}
key, value := gateReport(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// ace's placement written again with another module beside: its standing does not move, nothing is said again.
k, v = placing(t, "ace", GoBuildModule, NodeBuildModule)
if gate.Placement(k, v, false, placed.Add(time.Minute)) {
t.Fatal("the same placement of mesh-build-node changed what was passed on")
}
if c.counts {
return
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's; got %q, %q", v, none)
}
if len(*said) != 1 || !strings.Contains((*said)[0], "before mesh-build-node was placed on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", *said)
}
key, value = gateReport(t, "ace", "v26.12.1", placed.Add(time.Minute))
if !gate.Toolchain(NodeBuildModule, key, value, false) || len(set.Entries("typescript")) != 4 {
t.Fatal("ace's new report was not passed on")
}
})
}
t.Run("no placement held: every report counts", func(t *testing.T) {
gate, set, _ := start(t, false)
key, value := gateReport(t, "ace", "v26.12.0", old)
gate.Toolchain(NodeBuildModule, key, value, false)
gate.Settled()
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("before the first placement file every report counts; got %d", n)
}
k, v := placing(t, "g14", NodeBuildModule)
if !gate.Placement(k, v, false, old.Add(-time.Hour)) || len(set.Entries("typescript")) != 1 {
t.Fatalf("the first placement leaves every node it does not place out; got %d", len(set.Entries("typescript")))
}
if !gate.Placement(k, nil, true, placed) || len(set.Entries("typescript")) != 4 {
t.Fatal("the last placement gone: every report counts again")
}
})
t.Run("the history: a module in the first revision heard of a key stands since before it", func(t *testing.T) {
p := delivery.NewPlaced()
// ace in the form before, then converted: its module stands since before, the legacy key read no more.
k, v := placingLegacy(t, "ace", NodeBuildModule)
p.Put(k, v, false, old)
k, v = placing(t, "ace", NodeBuildModule)
if _, changed, err := p.Put(k, v, false, placed); len(changed) != 0 || err != nil {
t.Errorf("the conversion of ace's key changed %v, %v", changed, err)
}
if _, changed, _ := p.Put("ace."+NodeBuildModule, []byte("{}"), false, placed); len(changed) != 0 {
t.Errorf("a module key says which modules stand: %v", changed)
}
if k, v := placingLegacy(t, "ace"); true {
if _, changed, _ := p.Put(k, v, false, placed); len(changed) != 0 {
t.Errorf("the legacy key is read after the node key: %v", changed)
}
}
k2, v2 := placing(t, "g14")
p.Put(k2, v2, false, old)
k2, v2 = placing(t, "g14", NodeBuildModule)
if _, changed, _ := p.Put(k2, v2, false, placed); strings.Join(changed, ",") != NodeBuildModule {
t.Errorf("a module placed says changed %v", changed)
}
if _, _, err := p.Put("placement.novox", []byte("not json"), false, old); err == nil {
t.Error("a value that does not read was not said")
}
if s, ok := p.Since("ace", NodeBuildModule); !ok || !s.IsZero() {
t.Errorf("ace's module stands since %v, %v; want the zero time", s, ok)
}
if s, ok := p.Since("g14", NodeBuildModule); !ok || !s.Equal(placed) {
t.Errorf("g14's module stands since %v, %v; want the revision that placed it", s, ok)
}
if got := p.PlacedNodes(NodeBuildModule); strings.Join(got, ",") != "ace,g14" || len(p.ByNode()) != 2 || !p.Seeded() {
t.Errorf("PlacedNodes = %v, ByNode = %v", got, p.ByNode())
}
})
t.Run("a placement the bus refuses is said, and the part goes on", func(t *testing.T) {
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-delivery-assigner_placement")
gate, _, _ := start(t, false)
for _, bus := range []refusingBus{{err: refused}, {refuseOpen: true, err: refused}} {
if _, err := FollowPlacement(context.Background(), bus, gate, func(context.Context) {}, logger.Printf); err != nil {
t.Errorf("a refused placement stopped the start: %v", err)
}
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
})
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
@@ -0,0 +1,67 @@
package buildrun
import (
"context"
"encoding/json"
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Caller calls a tool of a module on one named machine (mesh-sdk process.Conn.CallOn; ADR 0469 clause 3).
type Caller interface {
CallOn(node, key string, body any) (json.RawMessage, error)
}
// Steps waits for the end of a build module's long step: the value of its key `<node>.<id>` in the module's steps state
// (ADR 0469 clause 2) once its state is not running. It is a watch of that key, not a poll; it returns when the step
// ends or ctx does.
type Steps interface {
Wait(ctx context.Context, module, key string) (delivery.Step, error)
}
// Runner runs a build module's long steps: the call answers at once with the step's key, and the result is read from that
// key when the step ends (ADR 0469 clause 2).
type Runner struct {
Caller Caller
Steps Steps
// Bound is how long a step may run before the builder stops waiting for it.
Bound time.Duration
}
// Run calls tool of the build module on node with args, waits for the step's end and answers its result. A step that
// failed or was refused is an error naming the node, the state and the reason.
func (r Runner) Run(ctx context.Context, module, node, tool string, args map[string]any) (json.RawMessage, error) {
raw, err := r.Caller.CallOn(node, module+"."+tool, args)
if err != nil {
return nil, fmt.Errorf("%s on %s: %w", tool, node, err)
}
var answer struct {
Step string `json:"step"`
Key string `json:"key"`
State string `json:"state"`
}
if err := json.Unmarshal(raw, &answer); err != nil {
return nil, fmt.Errorf("%s on %s answered what is not a step: %w", tool, node, err)
}
if answer.Key == "" {
return nil, fmt.Errorf("%s on %s answered no step key: %s", tool, node, raw)
}
if n, _, err := delivery.ParseStepKey(answer.Key); err != nil || n != node {
return nil, fmt.Errorf("%s on %s answered the step key %q, which is not a key of that node", tool, node, answer.Key)
}
wait, cancel := context.WithTimeoutCause(ctx, r.Bound, fmt.Errorf("the step %s did not end within %s", answer.Key, r.Bound))
defer cancel()
step, err := r.Steps.Wait(wait, module, answer.Key)
if err != nil {
if cause := context.Cause(wait); cause != nil && ctx.Err() == nil {
err = cause
}
return nil, fmt.Errorf("%s on %s: %w", tool, node, err)
}
if step.State != delivery.StepPassed {
return nil, fmt.Errorf("%s on %s: the step %s %s: %s", tool, node, answer.Key, step.State, step.Reason)
}
return step.Result, nil
}
@@ -0,0 +1,125 @@
package buildrun
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
type call struct {
node, key string
body any
}
// fakeCaller answers a long tool as the build modules do: at once, with the step and its key.
type fakeCaller struct {
calls []call
answer string
err error
}
func (f *fakeCaller) CallOn(node, key string, body any) (json.RawMessage, error) {
f.calls = append(f.calls, call{node, key, body})
if f.err != nil {
return nil, f.err
}
return json.RawMessage(f.answer), nil
}
// fakeSteps holds the steps values by `<module>/<key>`; Wait on a key it does not hold blocks until the context ends.
type fakeSteps struct {
mu sync.Mutex
steps map[string]delivery.Step
// during, when set, is called as a wait starts and the function it answers as the wait ends.
during func(key string) func()
}
func (f *fakeSteps) Wait(ctx context.Context, module, key string) (delivery.Step, error) {
f.mu.Lock()
s, ok := f.steps[module+"/"+key]
during := f.during
f.mu.Unlock()
if ok {
if during != nil {
defer during(key)()
}
return s, nil
}
<-ctx.Done()
return delivery.Step{}, context.Cause(ctx)
}
func stepAt(node, id, state, reason string, result string) delivery.Step {
s := delivery.Step{ID: id, Tool: "build_go_fetch", Node: node, State: state, Reason: reason}
if result != "" {
s.Result = json.RawMessage(result)
}
return s
}
func runCallsTheToolOnTheNodeAndReadsTheStepsKey(t *testing.T) { // 1607/AC5
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
steps := &fakeSteps{steps: map[string]delivery.Step{"mesh-build-go/ace.abc1": stepAt("ace", "abc1", delivery.StepPassed, "", `{"workspace":"w1","commit":"c"}`)}}
res, err := Runner{Caller: caller, Steps: steps, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", map[string]any{"repository": "novox/r", "commit": "c"})
if err != nil {
t.Fatal(err)
}
if len(caller.calls) != 1 || caller.calls[0].node != "ace" || caller.calls[0].key != "mesh-build-go.build_go_fetch" {
t.Errorf("calls = %+v, want one on ace for mesh-build-go.build_go_fetch", caller.calls)
}
var fr struct{ Workspace string }
if err := json.Unmarshal(res, &fr); err != nil || fr.Workspace != "w1" {
t.Errorf("result = %s (%v), want the step's result", res, err)
}
}
func runFailsOnAFailedOrRefusedStepNamingItsReason(t *testing.T) { // 1607/AC5
for _, state := range []string{delivery.StepFailed, delivery.StepRefused} {
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
steps := &fakeSteps{steps: map[string]delivery.Step{"mesh-build-go/ace.abc1": stepAt("ace", "abc1", state, "go is go1.26.0, the recipe builds with go1.27.1", "")}}
_, err := Runner{Caller: caller, Steps: steps, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_build", nil)
if err == nil || !strings.Contains(err.Error(), "go1.27.1") || !strings.Contains(err.Error(), state) || !strings.Contains(err.Error(), "ace") {
t.Errorf("%s: error = %v, want the node, the state and the reason", state, err)
}
}
}
func runStopsWaitingAtTheBound(t *testing.T) { // 1607/AC5
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
_, err := Runner{Caller: caller, Steps: &fakeSteps{}, Bound: 20 * time.Millisecond}.Run(context.Background(), "mesh-build-go", "ace", "build_go_build", nil)
if err == nil || !strings.Contains(err.Error(), "ace.abc1") || !strings.Contains(err.Error(), "did not end") {
t.Errorf("error = %v, want the step key and that it did not end within its bound", err)
}
}
func runRefusesAnAnswerWithNoKeyAndAKeyOfAnotherNode(t *testing.T) { // 1607/AC5
for _, answer := range []string{`{"state":"running"}`, `{"step":"abc1","key":"novox.abc1","state":"running"}`, `not json`} {
_, err := Runner{Caller: &fakeCaller{answer: answer}, Steps: &fakeSteps{}, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", nil)
if err == nil {
t.Errorf("answer %s: no error", answer)
}
}
}
func runReturnsTheCallsOwnError(t *testing.T) { // 1607/AC5
boom := errors.New("no responders")
_, err := Runner{Caller: &fakeCaller{err: boom}, Steps: &fakeSteps{}, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", nil)
if !errors.Is(err, boom) || !strings.Contains(err.Error(), "ace") {
t.Errorf("error = %v, want the call's error naming the node", err)
}
}
// A step run on its node and read from its steps key, one table of cases (the test budget).
func TestRunningAStepOnANode(t *testing.T) {
t.Run("run calls the tool on the node and reads the steps key", runCallsTheToolOnTheNodeAndReadsTheStepsKey)
t.Run("run fails on a failed or refused step naming its reason", runFailsOnAFailedOrRefusedStepNamingItsReason)
t.Run("run stops waiting at the bound", runStopsWaitingAtTheBound)
t.Run("run refuses an answer with no key and a key of another node", runRefusesAnAnswerWithNoKeyAndAKeyOfAnotherNode)
t.Run("run returns the calls own error", runReturnsTheCallsOwnError)
}
@@ -0,0 +1,64 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// LanguageOf is the language each build module's toolchains state holds (the indexer's ToolchainModules, reversed).
var LanguageOf = map[string]string{GoBuildModule: "go", NodeBuildModule: "typescript"}
// ToolchainSet holds what the build modules' toolchains buckets hold: each node's entry of a language, as the watches
// put them in. Safe for use from the watches and from a build at once.
type ToolchainSet struct {
mu sync.Mutex
by map[string]map[string]delivery.Toolchain // language -> node -> entry
}
// Put takes a change of a build module's toolchains bucket in: a put of a node's entry (key <node>.<language>) or its
// delete. It answers whether the language's entries changed (a node's version or recipe, or a node appearing or going).
// A key or a value that is not an entry of the module's language is left out.
func (s *ToolchainSet) Put(module, key string, value []byte, deleted bool) bool {
language, ok := LanguageOf[module]
if !ok {
return false
}
node, keyLanguage, err := delivery.ParseToolchainKey(key)
if err != nil || keyLanguage != language {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
if deleted {
if _, had := s.by[language][node]; !had {
return false
}
delete(s.by[language], node)
return true
}
tc, err := delivery.DecodeToolchain(value)
if err != nil || tc.Language != language {
return false
}
if s.by == nil {
s.by = map[string]map[string]delivery.Toolchain{}
}
if s.by[language] == nil {
s.by[language] = map[string]delivery.Toolchain{}
}
cur, had := s.by[language][node]
s.by[language][node] = tc
return !had || cur.Version != tc.Version || cur.Recipe != tc.Recipe
}
// Entries are the nodes' entries of a language, in no particular order.
func (s *ToolchainSet) Entries(language string) []delivery.NodeToolchain {
s.mu.Lock()
defer s.mu.Unlock()
var out []delivery.NodeToolchain
for node, tc := range s.by[language] {
out = append(out, delivery.NodeToolchain{Node: node, Toolchain: tc})
}
return out
}
@@ -0,0 +1,49 @@
package buildrun
import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func tcValue(t *testing.T, language, version string) []byte {
t.Helper()
b, err := delivery.EncodeToolchain(delivery.Toolchain{Language: language, Version: version, Recipe: "1", At: time.Now()})
if err != nil {
t.Fatal(err)
}
return b
}
func TestToolchainSetKeepsEachNodesEntryByLanguageAndSaysWhenOneChanged(t *testing.T) { // 1607/AC6
var s ToolchainSet
if !s.Put(GoBuildModule, "novox.go", tcValue(t, "go", "go1.27.1"), false) {
t.Error("a first entry did not count as a change")
}
if s.Put(GoBuildModule, "novox.go", tcValue(t, "go", "go1.27.1"), false) {
t.Error("the same entry again counted as a change")
}
s.Put(GoBuildModule, "ace.go", tcValue(t, "go", "go1.27.1"), false)
s.Put(NodeBuildModule, "ace.typescript", tcValue(t, "typescript", "tsc5"), false)
if got := s.Entries("go"); len(got) != 2 {
t.Errorf("go entries = %+v, want two", got)
}
if got := s.Entries("typescript"); len(got) != 1 || got[0].Node != "ace" || got[0].Version != "tsc5" {
t.Errorf("typescript entries = %+v", got)
}
if !s.Put(GoBuildModule, "ace.go", nil, true) || len(s.Entries("go")) != 1 {
t.Error("a delete did not remove the node's entry")
}
// What is not a toolchain of the module's language is left out.
s = ToolchainSet{}
if s.Put(GoBuildModule, "novox.typescript", tcValue(t, "typescript", "x"), false) {
t.Error("a typescript toolchain in the go build module's bucket counted")
}
if s.Put(GoBuildModule, "novox.go", []byte("not json"), false) || s.Put("other-module", "novox.go", tcValue(t, "go", "x"), false) {
t.Error("garbage or an unknown module counted")
}
if len(s.Entries("go")) != 0 {
t.Error("an entry was kept")
}
}
@@ -14,21 +14,119 @@ import (
"sync"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/views"
"git.novox.be/novox/mesh-delivery/sdk/enginereads"
"mesh-delivery-builder/internal/build"
"mesh-delivery-builder/internal/versions"
)
// EngineReadsTool lists, asks again and drops what each counted node's engine reads, as the builder keeps it.
const EngineReadsTool = "builder_engine_reads"
// queueAsker writes a queues value and waits for the builder's answer to it.
type queueAsker func(key string, value []byte) (json.RawMessage, error)
// Tools are the builder's tools: stuck reads, askAgain writes the reasks value, queue writes a queues value and waits for
// the builder's read, buildAgain writes a reasks value of mode build and waits for the builder's answer.
func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue, buildAgain queueAsker, since build.LogSince, now func() time.Time) []stdio.Tool {
// tags writes a tag-asks value and waits for the builder's answer; ta reads tag-asks.
func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue, buildAgain queueAsker, since build.LogSince, now func() time.Time, vs versions.Reader,
tags queueAsker, ta, fw, of versions.Reader) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) }
// handAct writes a reasks value of a versions hand act and answers the builder's answer, or its refusal as an error.
handAct := func(key string, value []byte, err error) (any, error) {
if err != nil {
return nil, err
}
raw, err := buildAgain(key, value)
if err != nil {
return nil, err
}
a, err := build.DecodeReAsk(raw)
if err != nil {
return nil, err
}
if a.Refused {
return nil, errors.New(a.Answer)
}
return map[string]any{"key": key, "name": a.Name, "do": a.Do, "version": a.Version, "answer": a.Answer}, nil
}
tagAct := func(key string, value []byte, err error) (any, error) {
if err != nil {
return nil, err
}
raw, err := tags(key, value)
if err != nil {
return nil, err
}
return versions.DecodeTagAsk(raw)
}
nameFP := map[string]any{"type": "object", "required": []string{"name", "fp", "why"}, "properties": map[string]any{
"name": map[string]any{"type": "string", "description": "the numbered name, <module>/<artifact>, as builder_versions lists it"},
"fp": map[string]any{"type": "string", "description": "the fingerprint, src2:<hex>"},
"why": map[string]any{"type": "string", "description": "why, in plain words"}}}
branch := func(head bool) map[string]any {
req := []string{"repository", "branch", "why"}
props := map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<name>"},
"branch": map[string]any{"type": "string", "description": "the branch; its cleaned name is the tag"},
"why": map[string]any{"type": "string", "description": "why, in plain words"}}
if head {
req = append(req, "head")
props["head"] = map[string]any{"type": "string", "description": "the commit the tagged version is built from"}
}
return map[string]any{"type": "object", "required": req, "properties": props}
}
return []stdio.Tool{
{Name: versions.PublishAgainTool, Description: versions.PublishAgainDescription, Input: nameFP,
Run: func(args map[string]any) (any, error) {
return handAct(build.NewPublishAgain(str(args, "name"), str(args, "fp"), str(args, "why"), stdio.CallerOf(args), now()))
}},
{Name: versions.FloorAgainTool, Description: versions.FloorAgainDescription,
Input: map[string]any{"type": "object", "required": []string{"name", "why"}, "properties": map[string]any{
"name": map[string]any{"type": "string", "description": "the numbered name, <module>/<artifact>"},
"why": map[string]any{"type": "string", "description": "why, in plain words"}}},
Run: func(args map[string]any) (any, error) {
return handAct(build.NewFloorAgain(str(args, "name"), str(args, "why"), stdio.CallerOf(args), now()))
}},
{Name: versions.TagTool, Description: versions.TagDescription, Input: branch(true),
Run: func(args map[string]any) (any, error) {
return tagAct(versions.NewTagAsk(str(args, "repository"), str(args, "branch"), str(args, "head"), str(args, "why"), stdio.CallerOf(args), now()))
}},
{Name: versions.UntagTool, Description: versions.UntagDescription, Input: branch(false),
Run: func(args map[string]any) (any, error) {
return tagAct(versions.NewUntag(str(args, "repository"), str(args, "branch"), str(args, "why"), stdio.CallerOf(args), now()))
}},
{Name: versions.TagAsksTool, Description: versions.TagAsksDescription,
Input: map[string]any{"type": "object", "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "only this repository's asks, <owner>/<name>"}}},
Run: func(args map[string]any) (any, error) { return versions.AnswerTagAsks(ta, str(args, "repository")) }},
{Name: versions.OldFormTool, Description: versions.OldFormDescription,
Input: map[string]any{"type": "object", "properties": map[string]any{
"drop": map[string]any{"type": "string", "description": "a retired node to drop from the count by hand"},
"why": map[string]any{"type": "string", "description": "with drop: why, in plain words"}}},
Run: func(args map[string]any) (any, error) {
if node := str(args, "drop"); node != "" {
return handAct(build.NewOldFormDrop(node, str(args, "why"), stdio.CallerOf(args), now()))
}
return versions.AnswerOldForm(of)
}},
{Name: versions.FloorWaitsTool, Description: versions.FloorWaitsDescription,
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) { return versions.AnswerFloorWaits(fw) }},
{Name: versions.VersionsTool, Description: versions.VersionsDescription,
Input: map[string]any{"type": "object", "required": []string{"name"}, "properties": map[string]any{
"name": map[string]any{"type": "string", "description": "the package, image, bundle or archive name"},
"version": map[string]any{"type": "string", "description": "one version, to ask whether it is recorded"},
"digest": map[string]any{"type": "string", "description": "with version: the digest the holder received"}}},
Run: func(args map[string]any) (any, error) {
return versions.AnswerVersions(context.Background(), vs, str(args, "name"), str(args, "version"), str(args, "digest"))
}},
{Name: build.StuckTool,
Description: "Every build key that waits (building or queued), oldest first: its module, fingerprint, step words and since when, and each ask it " +
"waits on by language with its id, when it was put, its asking (1 for the first; the builder asks an ask again when it can no longer be " +
@@ -98,7 +196,7 @@ func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, q
Description: "Every build that ended since a time, oldest first, from the builder's log build-log (kept 30 days): per build its module, " +
"fingerprint, source revision, repository, directory, commit and ref, the build node, when it ended, its outcome (done, failed or " +
"queued), the manifest built, the bases it was built against and each artifact made with its digest and reference. more says the " +
"log held more than one answer carries: ask again from the last build's time. What mesh-catalog registers a build from. Read-only.",
"log held more than one answer carries: ask again from the last build's time. What the operator's command nox builds reads. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"since"}, "properties": map[string]any{
"since": map[string]any{"type": "string", "description": "RFC 3339 time; the builds that ended at or after it"}}},
Run: func(args map[string]any) (any, error) {
@@ -264,7 +362,65 @@ func main() {
a, err := build.DecodeReAsk(v)
return err == nil && !a.Answered.IsZero()
})
if err := stdio.Serve("", Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), buildAgain, stdio.Log(delivery.BuilderLog).Since, time.Now)); err != nil {
vs := stdio.State(versions.State)
vr := versions.Reader{Keys: vs.Keys, Get: func(key string) (json.RawMessage, bool, error) {
e, err := vs.Get(key)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}}
ts := stdio.State(versions.TagAsksState)
tags := askKept(ts, versions.TagAsksState, versions.TagAsksBucket, "the ask", 25*time.Second, func(v []byte) bool {
a, err := versions.DecodeTagAsk(v)
return err == nil && !a.Answered.IsZero()
})
tr := versions.Reader{Keys: ts.Keys, Get: func(key string) (json.RawMessage, bool, error) {
e, err := ts.Get(key)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}}
fs := stdio.State(versions.FloorWaitsState)
fr := versions.Reader{Keys: fs.Keys, Get: func(key string) (json.RawMessage, bool, error) {
e, err := fs.Get(key)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}}
ofs := stdio.State(versions.OldFormState)
ofr := versions.Reader{Keys: ofs.Keys, Get: func(key string) (json.RawMessage, bool, error) {
e, err := ofs.Get(key)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}}
bs := stdio.State(versions.BoundsState)
br := versions.Reader{Keys: bs.Keys, Get: func(key string) (json.RawMessage, bool, error) {
e, err := bs.Get(key)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}}
tools := Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), buildAgain, stdio.Log(delivery.BuilderLog).Since, time.Now, vr, tags, tr, fr, ofr)
// The builder's expiring keys (state bounds): what each bounds and when it ends.
tools = append(tools, stdio.Tool{Name: versions.BoundsTool, Description: versions.BoundsDescription,
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) {
rows, err := versions.AnswerBounds(br, time.Now())
return map[string]any{"bounds": rows}, err
}})
// What each counted node's engine reads, as the builder keeps it (ADR 0618 §2, ADR 0624 §1): listed, one node asked
// again, or one node's key dropped.
tools = append(tools, enginereads.Tool(EngineReadsTool, "the builder", enginereads.KeptState(stdio.State("engine-reads")), true, time.Now))
// builder_first_views and builder_first_view_again: where the builder's first views of the indexer's and the assigner's
// states stand, and asking one again by hand (ADR 0615, DLV-M1).
tools = append(tools, views.StatusTools(delivery.BuilderModule, "builder_first_views", "builder_first_view_again", stdio.State(views.StatusState), handacts.Recorder{Ask: stdio.Ask}, time.Now)...)
if err := stdio.Serve("", tools); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] %v\n", err)
os.Exit(1)
}
@@ -0,0 +1,174 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"mesh-delivery-builder/internal/build"
"mesh-delivery-builder/internal/versions"
)
// forgeStream is the forge bridge's stream `forge` as the bus names it, where the builder's durable consumer hears a
// branch's end (design 51 *Tagged versions*).
const forgeStream = "FORGE"
// forgeBranchDeleted is the forge bridge's kind for a branch's end (delivery.ForgeBranchDeleted in the group's mesh-sdk).
const forgeBranchDeleted = "branch-deleted"
// checkerAsksStream is the checker's family `asks` as the bus names it, and tagWantedSubject its event tag-wanted
// (design 51 *Tagged versions*, DLV-C3): on the checker's own stream, so its subject is mesh.stream.<module>.event.*.
const (
checkerAsksStream = "ASKS"
tagWantedSubject = "mesh.stream." + delivery.CheckerModule + ".event.tag-wanted"
)
// forgeEnd is the part of a forge message the builder reads.
type forgeEnd struct {
Kind string `json:"kind"`
Repository string `json:"repository"`
Branch string `json:"branch"`
Action string `json:"action"`
}
// ended says whether the message ends a branch: merged, a pull closed, or branch-deleted.
func (f forgeEnd) ended() bool {
switch f.Kind {
case delivery.ForgeMerged, forgeBranchDeleted:
return f.Branch != ""
case delivery.ForgePull:
return f.Action == delivery.PullClosed && f.Branch != ""
}
return false
}
// branchEnds hears the forge bridge's stream: every message wakes a floor whose bridge_tags was silent, and a branch's
// end removes its tagged versions, acknowledged only once all are removed.
type branchEnds struct {
acts versions.Acts
findings *findings
tagAsks func() jetstream.KeyValue
wake func(ctx context.Context)
logf func(string, ...any)
now func() time.Time
}
func (h *branchEnds) heard(ctx context.Context, m jetstream.Msg) {
if h.wake != nil {
h.wake(ctx)
}
var f forgeEnd
if err := json.Unmarshal(m.Data(), &f); err != nil || !f.ended() {
h.ack(m)
return
}
end, err := h.acts.BranchEnded(ctx, f.Repository, f.Branch)
if err != nil {
h.logf("the end of %s %s: the versions cannot be read, heard again: %v", f.Repository, f.Branch, err)
h.again(m)
return
}
id := f.Repository + "/" + f.Branch
if end.Done {
h.findings.clear(ctx, versions.RemovalNotDone, health.ScopeModule, id)
if kv := h.tagAsks(); kv != nil {
if err := kv.Delete(ctx, versions.TagAskKey(f.Repository, f.Branch)); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
h.logf("the tag ask of %s %s stays in %s: %v", f.Repository, f.Branch, versions.TagAsksState, err)
}
}
h.ack(m)
return
}
since := h.now()
if md, err := m.Metadata(); err == nil {
since = md.Timestamp
}
if h.now().Sub(since) >= versions.RemovalFindingAfter {
h.findings.raise(ctx, finding{kind: versions.RemovalNotDone, scope: health.ScopeModule, id: id,
summary: end.Finding(f.Repository, f.Branch, since),
headline: "A branch's tagged versions are not removed",
explanation: "The branch ended, and the registry seats did not say removed for every version tagged for it within 15 minutes.",
resolved: "Every seat said removed.",
needs: "Look at the seats named: " + strings.Join(end.Seats(), ", ")})
}
h.logf("the end of %s %s: remove asked of %d tagged versions, heard again in a minute", f.Repository, f.Branch, len(end.Left))
h.again(m)
}
func (h *branchEnds) ack(m jetstream.Msg) {
if err := m.Ack(); err != nil {
h.logf("acknowledging %s: %v", m.Subject(), err)
}
}
// again leaves the message for the bus to give again in a minute: the retry is the bus's redelivery, not a timer here.
func (h *branchEnds) again(m jetstream.Msg) {
if err := m.NakWithDelay(versions.RemoveAgainAfter); err != nil {
h.logf("leaving %s for the bus to give again: %v", m.Subject(), err)
}
}
// tagWanted takes the checker's tag-wanted in: written to tag-asks as builder_tag writes it, answered by its watch.
func tagWanted(ctx context.Context, kv jetstream.KeyValue, m jetstream.Msg, by string, now time.Time, logf func(string, ...any)) {
if m.Subject() != tagWantedSubject {
_ = m.Ack() // the family's other events (the verdicts) are not the builder's
return
}
var t versions.TagWanted
err := json.Unmarshal(m.Data(), &t)
var key string
var value []byte
if err == nil {
key, value, err = versions.TagWantedAsk(t, by, now)
}
if err != nil {
logf("%s holds what is no tag-wanted, left: %v", m.Subject(), err)
_ = m.Ack()
return
}
if kv == nil {
logf("tag-wanted of %s %s is heard again: %s is not on the bus yet", t.Repository, t.Branch, versions.TagAsksState)
_ = m.NakWithDelay(versions.RemoveAgainAfter)
return
}
if _, err := kv.Put(ctx, key, value); err != nil {
logf("tag-wanted of %s %s is heard again: %v", t.Repository, t.Branch, err)
_ = m.Nak()
return
}
_ = m.Ack()
}
// tagsVia is build.Builder.Tags over the state tag-asks: the asks of the repository, read when a check's build is
// looked at; none while the state is not bound.
func tagsVia(kv func() jetstream.KeyValue, logf func(string, ...any)) build.TagOf {
return func(ctx context.Context, repository, head string) (string, bool) {
store := kv()
if store == nil {
return "", false
}
lister, err := store.ListKeysFiltered(ctx, versions.EscapeToken(repository)+".>")
if err != nil {
logf("the tag asks of %s cannot be read, so its check builds no tagged version: %v", repository, err)
return "", false
}
var asks []versions.TagAsk
for key := range lister.Keys() {
e, err := store.Get(ctx, key)
if err != nil {
continue
}
if t, err := versions.DecodeTagAsk(e.Value()); err == nil {
asks = append(asks, t)
}
}
return versions.TagFor(asks, repository, head)
}
}
@@ -0,0 +1,174 @@
package main
import (
"context"
"errors"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-delivery/sdk/heard"
"mesh-delivery-builder/internal/versions"
)
// findingsBucket is the builder's findings (novox/hq ADR 0564 §2): what it sees wrong now, written at its raise and
// deleted at its clear, which the condition keeper raises and clears.
var findingsBucket = health.FindingsBucket(delivery.BuilderModule)
// findings writes the builder's findings into its state, once bound; until then each is said in the log only.
type findings struct {
kv *process.AwaitedBucket
logf func(string, ...any)
now func() time.Time
// store stands in for kv in a test.
store findingStore
}
// findingStore is the part of the findings bucket written.
type findingStore interface {
Get(ctx context.Context, key string) (jetstream.KeyValueEntry, error)
Put(ctx context.Context, key string, value []byte) (uint64, error)
Delete(ctx context.Context, key string, opts ...jetstream.KVDeleteOpt) error
}
// finding is one of the builder's findings: its kind, what it is about, and its words.
type finding struct {
kind, scope, id string
summary, headline, explanation, resolved, needs string
severity health.Severity
}
func (f *findings) bucket() findingStore {
switch {
case f == nil:
return nil
case f.store != nil:
return f.store
case f.kv == nil:
return nil
}
if kv := f.kv.KV(); kv != nil {
return kv
}
return nil
}
// BuildFailed is the finding of a failed trunk build (DLV-B7, design 25), about its module.
const BuildFailed = "build-failed"
// buildEnded raises the module's build-failed for a trunk build that failed and clears it at a later done one; a
// candidate's build (its ref named) is the check's to say.
func (f *findings) buildEnded(ctx context.Context, e delivery.BuildLogEntry) {
if e.Event != delivery.BuildEndedEvent || e.Ref != "" {
return
}
switch e.Outcome {
case delivery.BuildFailed:
f.raise(ctx, finding{kind: BuildFailed, scope: health.ScopeModule, id: e.Module,
summary: "the trunk build of " + e.Module + " at " + e.FP + " (commit " + e.Commit + ") failed",
headline: "A module's trunk build failed",
explanation: "The builder built " + e.Module + " from the trunk and the build failed, so its new version is not delivered.",
resolved: "A later trunk build of " + e.Module + " is done.",
needs: "builder_builds_since shows the build; builder_build_again builds it again."})
case delivery.BuildDone:
f.clear(ctx, BuildFailed, health.ScopeModule, e.Module)
}
}
// raise writes fd; it is written again only when its words changed, so the keeper's condition moves only then.
func (f *findings) raise(ctx context.Context, fd finding) {
key := health.FindingKey(fd.scope, fd.id, fd.kind)
kv := f.bucket()
if kv == nil {
f.logf("finding %s (not written, %s is not on the bus yet): %s", key, findingsBucket, fd.summary)
return
}
if e, err := kv.Get(ctx, key); err == nil {
if old, err := health.DecodeFinding(e.Value()); err == nil && old.Summary == fd.summary {
return
}
}
at := f.now().UTC()
sev := fd.severity
if sev == "" {
sev = health.Warning
}
value, err := health.EncodeFinding(health.Finding{Key: key, Kind: fd.kind, Subject: health.Subject{Scope: fd.scope, ID: fd.id},
Severity: sev, Summary: fd.summary, Headline: fd.headline, Explanation: fd.explanation, Resolved: fd.resolved, Needs: fd.needs,
Evidence: []health.Evidence{{At: at, Said: fd.summary}}, Source: "builder." + fd.kind, Observer: delivery.BuilderModule, At: at})
if err == nil {
_, err = kv.Put(ctx, key, value)
}
if err != nil {
f.logf("the finding %s cannot be written: %v", key, err)
}
}
// clear deletes the finding of kind about scope and id; one not there is nothing to clear.
func (f *findings) clear(ctx context.Context, kind, scope, id string) {
kv := f.bucket()
if kv == nil {
return
}
key := health.FindingKey(scope, id, kind)
if _, err := kv.Get(ctx, key); errors.Is(err, jetstream.ErrKeyNotFound) {
return
}
if err := kv.Delete(ctx, key); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
f.logf("the finding %s cannot be cleared: %v", key, err)
}
}
// notHeard is the writer of the builder's finding of a family it does not hear (heard.Writer, ADR 0615 §5): written into
// its findings once bound, said in the log until then.
func (f *findings) notHeard(ctx context.Context) heard.Writer {
on := func(do func(findingStore)) {
switch {
case f.store != nil:
do(f.store)
case f.kv != nil:
f.kv.Ready(func(kv jetstream.KeyValue) { do(kv) })
}
}
return heard.Writer{Observer: delivery.BuilderModule, Logf: f.logf, Now: f.now,
Put: func(key string, value []byte) error {
on(func(kv findingStore) {
if _, err := kv.Put(ctx, key, value); err != nil {
f.logf("the finding %s cannot be written: %v", key, err)
}
})
return nil
},
Delete: func(key string) error {
on(func(kv findingStore) {
if _, err := kv.Get(ctx, key); err != nil {
return // none stands
}
if err := kv.Delete(ctx, key); err != nil {
f.logf("the finding %s cannot be cleared: %v", key, err)
}
})
return nil
}}
}
// floorSaid is the floor finding (design 51 *Health and conditions*: a floor source silent): raised for a name waiting on
// its floor, about that name, cleared once all three sources answered.
func (f *findings) floorSaid(ctx context.Context) func(versions.FloorWait, bool) {
return func(w versions.FloorWait, raised bool) {
if !raised {
f.clear(ctx, versions.FloorSilent, health.ScopeModule, w.Name)
return
}
f.raise(ctx, finding{kind: versions.FloorSilent, scope: health.ScopeModule, id: w.Name, summary: w.Finding(),
headline: "A version's floor is not known",
explanation: "The builder numbers a name only once its registry seat, the forge bridge's tags and the indexer's requirements all answered; one of them did not.",
resolved: "All three answered.",
needs: "builder_floor_waits names the silent source; builder_floor_again asks again by hand."})
}
}
@@ -12,6 +12,7 @@ import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -19,7 +20,7 @@ import (
// (novox/hq issue 1937): on 2026-10-24 the store's 03:30 collection deleted the bare blobs of three done keys the
// controller had never taken in, and the builder kept them done.
// heldVerb answers the controller seat's verb held on the bus, as the serving controller does: {result: {output, ok,
// heldVerb answers the seat mesh-artifact-sweep's verb held on the bus, as its holder does: {result: {output, ok,
// answer}}; a reference is gone while gone says so.
type heldVerb struct {
mu sync.Mutex
@@ -29,7 +30,7 @@ type heldVerb struct {
func (h *heldVerb) serve(t *testing.T, b *bed) {
t.Helper()
sub, err := b.js.Conn().Subscribe("mesh.seat.mesh-controller.call.held.>", func(m *nats.Msg) {
sub, err := b.js.Conn().Subscribe("mesh.seat."+catalogue.ArtifactSweepSeat+".call.held.>", func(m *nats.Msg) {
var args struct {
References []string `json:"references"`
}
@@ -9,9 +9,10 @@ import (
"encoding/json"
"errors"
"fmt"
"io"
"maps"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"sync/atomic"
@@ -22,12 +23,20 @@ import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
// The build runner the build modules pull and run asks with: mesh-build-go's own package (novox/hq ADR 0593 §1),
// another repository's, by its published version, so the builder is tested against the real pulling code.
runner "git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep"
"git.novox.be/novox/mesh-contracts/go/seats/gopackageregistry"
"git.novox.be/novox/mesh-contracts/go/seats/ociartifactregistry"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/views/viewstest"
"mesh-delivery-builder/internal/build"
"mesh-delivery-builder/internal/versions"
)
// busConf is the shape of the composed accounts file (mesh-controller internal/broker): one account with JetStream, the
@@ -53,7 +62,7 @@ const (
)
// fakeBuildModules are the build modules on the nodes as the real ones run since R4g (ADR 0525; issue 1887 A4): each pulls
// its build seat's asks on its nodes (buildstep.Puller over the seat's workers) and runs an ask's job with step bodies that
// its build seat's asks on its nodes (runner.Puller over the seat's workers) and runs an ask's job with step bodies that
// answer as the real tools do, ending the step on <node>.<ask id> of its steps bucket.
type fakeBuildModules struct {
js jetstream.JetStream
@@ -68,14 +77,17 @@ type fakeBuildModules struct {
// closed, while set, keeps every node's capacity at 0: no ask is pulled until open is called (a test of the asks
// waiting in their streams).
closed atomic.Bool
pullers []*buildstep.Puller
pullers []*runner.Puller
// hang, while it holds a channel, keeps every build step running until the channel is closed: a step that never ends,
// so the builder's step bound runs out (design 51's round-2 case).
hang atomic.Pointer[chan struct{}]
}
// open lets the nodes pull again and wakes their pullers.
func (f *fakeBuildModules) open() {
f.closed.Store(false)
f.mu.Lock()
ps := append([]*buildstep.Puller(nil), f.pullers...)
ps := append([]*runner.Puller(nil), f.pullers...)
f.mu.Unlock()
for _, p := range ps {
p.Wake()
@@ -107,6 +119,12 @@ func (f *fakeBuildModules) step(module, name string, args map[string]any) (any,
f.builds = append(f.builds, args)
}
f.mu.Unlock()
if h := f.hang.Load(); h != nil && name == "build" {
select {
case <-*h:
case <-f.ctx.Done():
}
}
switch {
case name == "fetch":
merge := ""
@@ -139,26 +157,26 @@ func (f *fakeBuildModules) serve(t *testing.T, module string, nodes ...string) {
t.Fatal(err)
}
language := strings.TrimPrefix(module, "mesh-build-")
if err := buildstep.DeclareAskWorkers(f.ctx, f.js, language, 30*time.Second); err != nil {
if err := runner.DeclareAskWorkers(f.ctx, f.js, language, 30*time.Second); err != nil {
t.Fatal(err)
}
put := func(key string, value any) (uint64, error) {
raw, _ := value.(json.RawMessage)
return kv.Put(f.ctx, key, raw)
}
call := func(name string) buildstep.StepFunc {
call := func(name string) runner.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) { return f.step(module, name, args) }
}
steps := buildstep.AskSteps{Fetch: call("fetch"), Build: call("build"), Copy: call("copy")}
steps := runner.AskSteps{Fetch: call("fetch"), Build: call("build"), Copy: call("copy")}
for _, node := range nodes {
p := &buildstep.Puller{Source: &buildstep.JetStreamAsks{JS: f.js, Language: language}, Language: language, Node: node, Put: put,
p := &runner.Puller{Source: &runner.JetStreamAsks{JS: f.js, Language: language}, Language: language, Node: node, Put: put,
Capacity: func() int {
if f.closed.Load() {
return 0
}
return 2
}, PullWait: 200 * time.Millisecond,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return buildstep.RunAsk(ctx, steps, a) }}
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return runner.RunAsk(ctx, steps, a) }}
f.mu.Lock()
f.pullers = append(f.pullers, p)
f.mu.Unlock()
@@ -192,7 +210,11 @@ type bed struct {
builds jetstream.KeyValue
tools map[string]jetstream.KeyValue
instances jetstream.KeyValue
stop func() // stops the builder started last
tagAsks jetstream.KeyValue
tagBounds jetstream.KeyValue
packages jetstream.KeyValue // the indexer's packages, served by its read tool and views family
out *saidLog // the builder's process log, every instance's
stop func() // stops the builder started last
fake *fakeBuildModules
srvURL string
dir string
@@ -267,8 +289,23 @@ func startBedWith(t *testing.T, closed bool) *bed {
b.candidates = mk(delivery.CandidatesBucket, delivery.CandidatesHistory)
b.source = mk(delivery.SourceBucket, delivery.SourceHistory)
mk(delivery.ManifestsBucket, delivery.ManifestsHistory)
// The indexer and the assigner as the owners of the states the builder follows: their read tools and views families
// (ADR 0615, DLV-M1); n1 and n2 each have their durable consumer.
readers := []string{"n1_" + delivery.BuilderModule, "n2_" + delivery.BuilderModule}
for _, st := range []string{modulesState, manifestsState, candidatesState} {
viewstest.Serve(t, ctx, js, ctl, delivery.IndexerModule, st, delivery.IndexerModule+".indexer_read_"+st, indexerViews, readers...)
}
b.packages = viewstest.Serve(t, ctx, js, ctl, delivery.IndexerModule, delivery.PackagesState, delivery.IndexerModule+".indexer_read_packages", indexerViews, readers...)
viewstest.Serve(t, ctx, js, ctl, delivery.AssignerModule, delivery.PlacementState, delivery.AssignerModule+".assigner_read_placement", assignerViews, readers...)
b.builds = mk(delivery.BuilderBuildsBucket, delivery.BuilderBuildsHistory)
b.instances = mk(instancesBucket, 1)
// The tag asks and their bounds (design 51 *Tagged versions*): the bounds declared key-ttl, so an ended bound leaves the
// bus's marker a watch hears.
b.tagAsks = mk(versions.TagAsksBucket, 1)
if b.tagBounds, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: versions.TagAskBoundsBucket, History: 1, LimitMarkerTTL: time.Minute}); err != nil {
t.Fatal(err)
}
b.out = &saidLog{}
// The builder's module log, as the runtime raises it (ADR 0297): its stream on the log's subjects.
logStream := buscompose.LogStreamName(delivery.BuilderModule, delivery.BuilderLog)
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: logStream, Subjects: []string{buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog) + ".>"},
@@ -328,15 +365,24 @@ func (b *bed) report(t *testing.T, module, node, language, version, recipe strin
// module writes modules.<name> as the indexer does for a module with a new-form fingerprint, and its manifest by digest.
func (b *bed) module(t *testing.T, name, fixture, path string) string {
return b.moduleWith(t, name, fixture, path, map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}})
return b.moduleWith(t, name, fixture, path, bothToolchains)
}
// bothToolchains are the toolchains the bed's nodes report, as a module's fingerprint names them.
var bothToolchains = map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}
// moduleWith is module with the toolchains its fingerprint was computed with.
func (b *bed) moduleWith(t *testing.T, name, fixture, path string, toolchains map[string]delivery.ToolchainUsed) string {
return b.moduleCommit(t, name, fixture, path, toolchains, commit)
}
func (b *bed) moduleCommit(t *testing.T, name, fixture, path string, toolchains map[string]delivery.ToolchainUsed, at string) string {
t.Helper()
return b.moduleFP(t, name, fixture, path, toolchains, at, fp, 0)
}
// moduleFP is moduleCommit with the new-form fingerprint fp2 and the source revision src of its merge (0: none).
func (b *bed) moduleFP(t *testing.T, name, fixture, path string, toolchains map[string]delivery.ToolchainUsed, at, fp2 string, src uint64) string {
t.Helper()
raw, err := os.ReadFile(filepath.Join("testdata", fixture))
if err != nil {
@@ -354,8 +400,8 @@ func (b *bed) moduleCommit(t *testing.T, name, fixture, path string, toolchains
if _, err := manifests.Put(b.ctx, digest, mv); err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: at, FP: "src1:" + strings.Repeat("f", 64), FP2: fp,
Manifest: digest, Path: path, Toolchains: toolchains})
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: at, FP: "src1:" + strings.Repeat("f", 64), FP2: fp2,
Manifest: digest, Path: path, Toolchains: toolchains, Src: src})
if err != nil {
t.Fatal(err)
}
@@ -386,16 +432,58 @@ func (b *bed) start(t *testing.T, node string, env map[string]string) {
}
runCtx, stop := context.WithCancel(b.ctx)
done := make(chan error, 1)
go func() { done <- run(runCtx, func(k string) string { return e[k] }, io.Discard) }()
go func() { done <- run(runCtx, func(k string) string { return e[k] }, b.out) }()
var once sync.Once
b.stop = func() { once.Do(func() { stop(); <-done }) }
t.Cleanup(b.stop)
}
// saidLog is the builder's process log as its instances write it.
type saidLog struct {
mu sync.Mutex
buf strings.Builder
}
func (l *saidLog) Write(p []byte) (int, error) {
l.mu.Lock()
defer l.mu.Unlock()
return l.buf.Write(p)
}
// awaitSaid reads the process log until it holds want, within 20 s, and answers the line.
func (b *bed) awaitSaid(t *testing.T, want string) string {
t.Helper()
deadline := time.Now().Add(20 * time.Second)
for {
b.out.mu.Lock()
all := b.out.buf.String()
b.out.mu.Unlock()
for _, line := range strings.Split(all, "\n") {
if strings.Contains(line, want) {
return line
}
}
if time.Now().After(deadline) {
t.Fatalf("the builder never said %q; it said:\n%s", want, all)
}
select {
case <-b.ctx.Done():
t.Fatal(b.ctx.Err())
case <-time.After(50 * time.Millisecond):
}
}
}
// await reads the build key until pred holds of it, within 20 s.
func (b *bed) await(t *testing.T, module string, pred func(delivery.BuilderBuild) bool) delivery.BuilderBuild {
t.Helper()
key, err := delivery.BuilderBuildKey(module, fp)
return b.awaitFP(t, module, fp, pred)
}
// awaitFP is await of the build key of fingerprint at.
func (b *bed) awaitFP(t *testing.T, module, at string, pred func(delivery.BuilderBuild) bool) delivery.BuilderBuild {
t.Helper()
key, err := delivery.BuilderBuildKey(module, at)
if err != nil {
t.Fatal(err)
}
@@ -422,9 +510,31 @@ func isState(state string) func(delivery.BuilderBuild) bool {
// the digests of both bundles, both build modules' toolchains, after a fetch and a build on one node (ADR 0478 clause 4).
func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
b := startBed(t)
digest := b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
b.start(t, "n1", nil)
// DLV-B10 (design 51, round 2): the builder's versions, findings and the floor's three sources are on the bus; the seat
// answers 0.1.4, so netcheck's bundles are numbered from 0.1.5. Two merges close together, the first replaced before
// its claim: the second is numbered at once, and the first never.
reg := b.registry(t, "n1", "0.1.4")
const netcheck = "catalogue-a67f0196-netcheck.module.json"
first := "src2:" + strings.Repeat("1", 64)
b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, commit, first, 1)
digest := b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, commit, fp, 2)
// DLV-B6: a tag ask kept and never numbered (no check builds its head) is narrated once its bound ends on the bus, with
// what it waits on, and its bound is written again; no timer runs in the builder.
key, ask, err := versions.NewTagAsk("novox/mesh-sdk", "feat/x", "abc", "a group member's check", "me@shanks", time.Now())
if err != nil {
t.Fatal(err)
}
if _, err := b.tagAsks.Put(b.ctx, key, ask); err != nil {
t.Fatal(err)
}
b.start(t, "n1", map[string]string{"MESH_BUILDER_TAG_ASK_BOUND": "1s", "MESH_BUILDER_STEP_BOUND": "1s"})
v := b.await(t, "netcheck", isState(delivery.BuildDone))
for _, a := range []string{"tools-go", "tools-typescript"} {
es := reg.of(t, "netcheck/"+a)
if len(es) != 1 || es[0].Version != "0.1.5" || es[0].Fingerprint != fp || es[0].State != versions.Numbered || es[0].Digest != digestOf(a) {
t.Errorf("versions of netcheck/%s = %+v, want the second merge numbered 0.1.5 at once with its digest, the first never", a, es)
}
}
if v.By != "n1.mesh-delivery-builder" || v.Manifest != digest || len(v.Artifacts) != 2 {
t.Fatalf("build = %+v", v)
}
@@ -436,9 +546,21 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
if v.Toolchains["go"].Version != "go1.27.1" || v.Toolchains["typescript"].Version != "tsc5" {
t.Errorf("toolchains = %+v", v.Toolchains)
}
// ADR 0618 §1: the done key names the reference forms it is written in.
if len(v.Needs) != 1 || v.Needs[0] != "form:artifact-store" {
t.Errorf("needs = %v, want the first form", v.Needs)
}
// 1955/L7 (ADR 0571 §1): the build's start and end are in the module log build-log under the module's key, read as a
// reader with the log's read reads them, the end naming the manifest and what was made.
ended := b.logged(t, "netcheck")
// DLV-B10: between them the build log holds the build's narrations (build.said), each version numbered, and why.
var ended []delivery.BuildLogEntry
for _, e := range b.logged(t, "netcheck") {
if e.Event != delivery.BuildSaidEvent {
ended = append(ended, e)
} else if e.Why != versions.SaidNumbered || !strings.Contains(e.Line, "numbered 0.1.5") || e.FP != fp {
t.Errorf("said = %+v, want netcheck's versions numbered", e)
}
}
if len(ended) != 2 || ended[0].Event != delivery.BuildStartedEvent || ended[1].Event != delivery.BuildEndedEvent ||
ended[1].Manifest != digest || len(ended[1].Made) != 2 || ended[1].Node != "n1" {
t.Errorf("logged = %+v, want the start and the end of netcheck's build", ended)
@@ -448,34 +570,169 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
t.Errorf("%s called %d times, want %d (on one node, ADR 0478 clause 4)", tool, got, want)
}
}
said := b.awaitSaid(t, "the tag ask of novox/mesh-sdk feat/x at abc (feat-x) is not answered after")
if !strings.Contains(said, "the check's build of abc is not claimed yet") {
t.Errorf("the narration = %q, want what the ask waits on", said)
}
// And in the build log, under the builder's own key (the ruling of 13:17: build.said).
if !slices.ContainsFunc(b.logged(t, delivery.BuilderModule), func(e delivery.BuildLogEntry) bool {
return e.Event == delivery.BuildSaidEvent && e.Why == versions.SaidUnanswered && e.FP == key && e.Commit == "abc" &&
strings.Contains(e.Line, "the check's build of abc is not claimed yet")
}) {
t.Errorf("the build log of %s = %+v, want the unanswered tag ask said", delivery.BuilderModule, b.logged(t, delivery.BuilderModule))
}
if e, err := b.tagAsks.Get(b.ctx, key); err != nil {
t.Fatal(err)
} else if a, _ := versions.DecodeTagAsk(e.Value()); a.Answered.IsZero() || !a.Unanswered() {
t.Errorf("the tag ask = %+v, want kept and answered, no version yet", a)
}
for deadline := time.Now().Add(10 * time.Second); ; time.Sleep(50 * time.Millisecond) {
if _, err := b.tagBounds.Get(b.ctx, key); err == nil {
break
} else if time.Now().After(deadline) {
t.Fatalf("the bound is not written again after its narration: %v", err)
}
}
// Design 32, ADR 0615 §1: a package's name the indexer writes later reaches the builder by its packages' views family
// (never their bucket), and looks at the module again; the done bundles are kept, not numbered twice.
if _, err := b.packages.Put(b.ctx, "netcheck.0.1", []byte(`{"v":1,"name":"netcheck","package":"netcheck/lib","series":"0.1"}`)); err != nil {
t.Fatal(err)
}
b.awaitSaid(t, "names a package of netcheck")
if got := reg.numberedOf("netcheck/tools-go"); len(got) != 1 || got[0] != "0.1.5" {
t.Errorf("version.numbered of netcheck/tools-go = %v, want 0.1.5 once", got)
}
// DLV-B10: builder_publish_again after a published takes a new number, and says it numbered.
for _, a := range []string{"tools-go", "tools-typescript"} {
reg.published(t, "netcheck/"+a, "0.1.5", digestOf(a))
}
reg.awaitOf(t, "netcheck/tools-go", func(es []versions.Entry) bool { return len(es) == 1 && es[0].State == versions.Published })
key, again, err := build.NewPublishAgain("netcheck/tools-go", fp, "the holder lost it", "me@shanks", time.Now())
if err != nil {
t.Fatal(err)
}
if a := reg.answered(t, key, again); a.Refused || a.Do != versions.AgainRenumber || a.Version != "0.1.6" {
t.Errorf("builder_publish_again = %+v, want a new number, 0.1.6", a)
}
// Running-testing note (review round 1): two claims never get one number on the bus's own compare-and-set, as
// the builder's state versions keeps them: eight claims of one name at once, eight versions.
store := versions.Versions{Store: awaitedVersions{process.AwaitBucket(b.ctx, b.js, nil, versions.Bucket, "", t.Logf)}}
numbers := make(chan string, 8)
var claims sync.WaitGroup
for i := range 8 {
claims.Go(func() {
d, _, err := store.Number(b.ctx, versions.Claim{Name: "netcheck/race", Kind: versions.KindBundle, Fingerprint: fmt.Sprintf("f%d", i)})
if err != nil {
t.Error(err)
}
numbers <- d.Version
})
}
claims.Wait()
close(numbers)
got := map[string]bool{}
for v := range numbers {
got[v] = true
}
if len(got) != 8 {
t.Errorf("eight claims of one name at once were numbered %v, want eight versions", slices.Sorted(maps.Keys(got)))
}
if got := reg.numberedOf("netcheck/tools-go"); len(got) != 2 || got[1] != "0.1.6" {
t.Errorf("version.numbered of netcheck/tools-go = %v, want 0.1.6 said after 0.1.5", got)
}
// DLV-B10: a build module's bound running out fails the trunk build, which is the builder's finding build-failed; the
// fingerprint built again keeps its number, said with why, and its done build clears the finding.
release := make(chan struct{})
b.fake.hang.Store(&release)
third := "src2:" + strings.Repeat("4", 64)
b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, commit, third, 3)
if f := b.awaitFP(t, "netcheck", third, isState(delivery.BuildFailed)); !strings.Contains(f.Error, "did not end within") {
t.Errorf("the failed build = %+v, want its bound run out", f)
}
failed := health.FindingKey(health.ScopeModule, "netcheck", BuildFailed)
reg.awaitFinding(t, failed, true)
b.fake.hang.Store(nil)
close(release)
b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, commit, third, 3) // the indexer writes it again
b.awaitFP(t, "netcheck", third, isState(delivery.BuildDone))
b.awaitSaid(t, "kept: fingerprint "+third+" is numbered under it")
if !slices.ContainsFunc(b.logged(t, "netcheck"), func(e delivery.BuildLogEntry) bool {
return e.Event == delivery.BuildSaidEvent && e.Why == versions.SaidKept && e.FP == third && strings.Contains(e.Line, "kept: fingerprint "+third)
}) {
t.Errorf("the build log of netcheck says no version kept at %s", third)
}
reg.awaitFinding(t, failed, false)
// DLV-B4: the seats' refused and removed, heard on the builder's durable consumer, move a numbered version: a retention
// removal turns the entry removed with its reason, a refusal refused with the holder's reason and the bytes it refused.
reg.outcome(t, ociartifactregistry.SubjectRemoved, versions.Outcome{Name: "netcheck/tools-typescript", Version: "0.1.5", Kind: "bundle", Holder: "distribution", Reason: "retention"})
reg.outcome(t, ociartifactregistry.SubjectRefused, versions.Outcome{Name: "netcheck/tools-go", Version: "0.1.6", Digest: digestOf("other"), Kind: "bundle", Holder: "distribution", Reason: "the cap"})
reg.awaitOf(t, "netcheck/tools-typescript", func(es []versions.Entry) bool {
return slices.ContainsFunc(es, func(e versions.Entry) bool {
return e.Version == "0.1.5" && e.State == versions.Removed && e.Reason == "retention"
})
})
reg.awaitOf(t, "netcheck/tools-go", func(es []versions.Entry) bool {
return slices.ContainsFunc(es, func(e versions.Entry) bool {
return e.Version == "0.1.6" && e.State == versions.Refused && e.Reason == "the cap" && e.RefusedDigest == digestOf("other")
})
})
// DLV-B6: a branch's end (the forge bridge's merged) asks the seat to remove each version tagged for it, the reason
// tag-gone, and the seat's removed turns it removed.
tagged := versions.Entry{Name: "mesh-sdk/go", Kind: versions.KindPackage, Series: "0.1", Version: "0.1.15-feat-x.1", Tag: "feat-x", Module: "mesh-sdk",
Fingerprint: "src2:" + strings.Repeat("5", 64), Repository: "novox/mesh-sdk", Seat: gopackageregistry.Seat, State: versions.Published, At: time.Now().UTC()}
raw, _ := json.Marshal(tagged)
if _, err := reg.versions.Put(b.ctx, versions.Key(tagged.Name, tagged.Version), raw); err != nil {
t.Fatal(err)
}
if _, err := b.js.Publish(b.ctx, process.ModuleStreamSubject(delivery.ForgeBridgeModule, "forge"),
[]byte(`{"kind":"`+delivery.ForgeMerged+`","repository":"novox/mesh-sdk","branch":"feat/x"}`)); err != nil {
t.Fatal(err)
}
for deadline := time.Now().Add(20 * time.Second); ; time.Sleep(50 * time.Millisecond) {
reg.mu.Lock()
asked := slices.ContainsFunc(reg.removes, func(a map[string]any) bool {
return a["name"] == tagged.Name && a["version"] == tagged.Version && a["reason"] == "tag-gone"
})
reg.mu.Unlock()
if asked {
break
}
if time.Now().After(deadline) {
t.Fatalf("the seat's remove of %s@%s was not asked at the branch's end: %v", tagged.Name, tagged.Version, reg.removes)
}
}
reg.outcome(t, gopackageregistry.SubjectRemoved, versions.Outcome{Name: tagged.Name, Version: tagged.Version, Kind: versions.KindPackage, Holder: "gitea", Reason: "tag-gone"})
reg.awaitOf(t, tagged.Name, func(es []versions.Entry) bool { return len(es) == 1 && es[0].State == versions.Removed })
}
// TestTwoBuilderInstancesStartingTogetherEndWithOneDoneKey: two builders on two nodes start together on one module key of an
// archive only (no toolchain named, as the indexer writes it); the second takes over the key the first holds building
// (ADR 0478 clause 6), and the key ends done, written once.
func TestTwoBuilderInstancesStartingTogetherEndWithOneDoneKey(t *testing.T) { // 1607/AC7
b := startBed(t)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
b.start(t, "n2", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildDone))
if len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "archive" || v.Artifacts[0].Digest != digestOf("files") {
t.Fatalf("build = %+v, want the archive's digest", v)
}
if n := b.fake.count("mesh-build-go.build_go_build"); n < 1 { // each instance that built, on one node
t.Errorf("build_go_build was called %d times, want one for each instance that built", n)
}
}
// TestAnArchiveOnlyModuleNamingNoToolchainIsBuilt: the indexer gives an archive no language, so modules.ca-trust has no
// toolchains; the builder builds it on the build module's nodes all the same (review round 4).
func TestAnArchiveOnlyModuleNamingNoToolchainIsBuilt(t *testing.T) {
b := startBed(t)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildDone))
if len(v.Toolchains) != 0 || len(v.Artifacts) != 1 || strings.HasPrefix(v.Step, "one node (") {
t.Errorf("build = %+v, want done naming no toolchain", v)
// TestAnArchiveOnlyModuleAndARootModuleAreBuilt: the indexer gives an archive no language, so modules.ca-trust has no
// toolchains; the builder builds it on the build module's nodes all the same (review round 4). A module whose path is "."
// is given to the build tools as it is (1615/AC4). Two builders on two nodes start together on the archive's key: the
// second takes over the key the first holds building (ADR 0478 clause 6), and the key ends done, written once (1607/AC7).
// One table: three tests were merged, to pay for the enrolment's and to bring mesh-delivery's count down (DLV-B8).
func TestAnArchiveOnlyModuleAndARootModuleAreBuilt(t *testing.T) {
for _, c := range []struct {
name, path string
two bool
}{{"no toolchain", "modules/ca-trust", false}, {"root", delivery.RootPath, false}, {"two instances together", "modules/ca-trust", true}} {
t.Run(c.name, func(t *testing.T) {
b := startBed(t)
b.archive(t, "ca-trust", c.path, commit)
b.start(t, "n1", nil)
if c.two {
b.start(t, "n2", nil)
}
v := b.await(t, "ca-trust", isState(delivery.BuildDone))
if c.name == "no toolchain" && (len(v.Toolchains) != 0 || len(v.Artifacts) != 1 || strings.HasPrefix(v.Step, "one node (")) {
t.Errorf("build = %+v, want done naming no toolchain", v)
}
if c.two && (len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "archive" || v.Artifacts[0].Digest != digestOf("files")) {
t.Errorf("build = %+v, want the archive's digest", v)
}
if n := b.fake.count("mesh-build-go.build_go_build"); c.two && n < 1 { // each instance that built, on one node
t.Errorf("build_go_build was called %d times, want one for each instance that built", n)
}
})
}
}
@@ -530,14 +787,6 @@ func TestAToolchainChangeDoesNotBuildTheOldFingerprint(t *testing.T) { // 1615/A
}
}
// TestARootModuleIsBuilt: a module whose path is "." is given to the build tools as it is.
func TestARootModuleIsBuilt(t *testing.T) { // 1615/AC4
b := startBed(t)
b.archive(t, "ca-trust", delivery.RootPath, commit)
b.start(t, "n1", nil)
b.await(t, "ca-trust", isState(delivery.BuildDone))
}
func mustKey(t *testing.T, module string) string {
k, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
@@ -791,7 +1040,12 @@ func TestTheBuilderAsksTheBuildSeatsAndCallsNoNode(t *testing.T) {
}
defer sub.Close()
var calls atomic.Int64
s, err := sub.Subscribe("mesh.mod.*.call.>", func(*nats.Msg) { calls.Add(1) })
// A build module's tool, never the owners' read tools the builder's first views ask (DLV-M1).
s, err := sub.Subscribe("mesh.mod.*.call.>", func(m *nats.Msg) {
if strings.HasPrefix(m.Subject, "mesh.mod.mesh-build-") {
calls.Add(1)
}
})
if err != nil {
t.Fatal(err)
}
@@ -910,3 +1164,184 @@ func TestARestartedBuilderPutsNoSecondAsk(t *testing.T) {
t.Errorf("done = %+v", done)
}
}
// registry is the part of the bed design 51's builder cases need: the builder's states versions, findings, floor-waits and
// reasks, EVENTS with the seat oci-artifact-registry's outcomes and the builder's durable consumer of it, the builder's
// own events kept, and the floor's three sources served on the bus as their holders answer through the tool runner
// ({result}): the seat's versions, the forge bridge's bridge_tags and the indexer's indexer_requirements.
type registry struct {
b *bed
versions, findings, reasks jetstream.KeyValue
mu sync.Mutex
numbered []delivery.VersionNumbered
removes []map[string]any
}
func (b *bed) registry(t *testing.T, node string, seatVersions ...string) *registry {
t.Helper()
r := &registry{b: b}
mk := func(name string) jetstream.KeyValue {
kv, err := b.js.CreateKeyValue(b.ctx, jetstream.KeyValueConfig{Bucket: name, History: 1})
if err != nil {
t.Fatal(err)
}
return kv
}
r.versions, r.findings, r.reasks = mk(versions.Bucket), mk(findingsBucket), mk(build.ReAsksBucket)
mk(versions.FloorWaitsBucket)
// EVENTS with two seats' outcomes, and the forge bridge's stream (a branch's end): each with the builder's durable consumer.
for stream, subjects := range map[string][]string{
eventsStream: {delivery.ArtifactsGoneSubject, ociartifactregistry.SubjectPublished, ociartifactregistry.SubjectRefused, ociartifactregistry.SubjectRemoved,
gopackageregistry.SubjectPublished, gopackageregistry.SubjectRefused, gopackageregistry.SubjectRemoved},
forgeStream: {process.ModuleStreamSubject(delivery.ForgeBridgeModule, ">")},
} {
if _, err := b.js.CreateStream(b.ctx, jetstream.StreamConfig{Name: stream, Subjects: subjects}); err != nil {
t.Fatal(err)
}
if _, err := b.js.CreateOrUpdateConsumer(b.ctx, stream, jetstream.ConsumerConfig{Durable: node + "_" + delivery.BuilderModule,
AckPolicy: jetstream.AckExplicitPolicy}); err != nil {
t.Fatal(err)
}
}
// The Go seat's remove, as its holder answers it; each call is kept.
rm, err := b.js.Conn().Subscribe("mesh.seat."+gopackageregistry.Seat+".call.remove.>", func(m *nats.Msg) {
var args map[string]any
_ = json.Unmarshal(m.Data, &args)
r.mu.Lock()
r.removes = append(r.removes, args)
r.mu.Unlock()
_ = m.Respond([]byte(`{"result":{},"node":"ctl"}`))
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = rm.Unsubscribe() })
own := "mesh.mod." + delivery.BuilderModule + ".event.>"
if _, err := b.js.CreateStream(b.ctx, jetstream.StreamConfig{Name: "BUILDER_EVENTS", Subjects: []string{own}}); err != nil {
t.Fatal(err)
}
nc := b.js.Conn()
// As a build module hears it: on the contract's subject, read by the contract's decoder (BG2's late publish).
sub, err := nc.Subscribe(delivery.VersionNumberedSubject, func(m *nats.Msg) {
if nb, err := delivery.DecodeVersionNumbered(m.Data); err == nil {
r.mu.Lock()
r.numbered = append(r.numbered, nb)
r.mu.Unlock()
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
var listed []map[string]string
for _, v := range seatVersions {
listed = append(listed, map[string]string{"version": v})
}
for subject, answer := range map[string]any{
"mesh.seat." + ociartifactregistry.Seat + ".call.versions.>": map[string]any{"versions": listed},
"mesh.mod.mesh-delivery-forge-bridge.call.bridge_tags.>": map[string]any{"tags": []string{}},
"mesh.mod." + delivery.IndexerModule + ".call.indexer_requirements.>": map[string]any{"requirements": []string{}},
} {
reply, _ := json.Marshal(map[string]any{"result": answer, "node": "ctl"})
s, err := nc.Subscribe(subject, func(m *nats.Msg) { _ = m.Respond(reply) })
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = s.Unsubscribe() })
}
return r
}
// of is every entry versions holds of name.
func (r *registry) of(t *testing.T, name string) []versions.Entry {
t.Helper()
l, err := r.versions.ListKeysFiltered(r.b.ctx, versions.NameFilter(name))
if err != nil {
t.Fatal(err)
}
defer func() { _ = l.Stop() }()
var out []versions.Entry
for k := range l.Keys() {
e, err := r.versions.Get(r.b.ctx, k)
if err != nil {
t.Fatal(err)
}
var v versions.Entry
if err := json.Unmarshal(e.Value(), &v); err != nil {
t.Fatal(err)
}
out = append(out, v)
}
return out
}
// awaitOf reads name's entries until pred holds of them, within 20 s.
func (r *registry) awaitOf(t *testing.T, name string, pred func([]versions.Entry) bool) []versions.Entry {
t.Helper()
var es []versions.Entry
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if es = r.of(t, name); pred(es) {
return es
}
}
t.Fatalf("versions of %s = %+v within 20 s", name, es)
return es
}
// published says the artifact seat's published for name@version, as its holder does.
func (r *registry) published(t *testing.T, name, version, digest string) {
t.Helper()
r.outcome(t, ociartifactregistry.SubjectPublished, versions.Outcome{Name: name, Version: version, Digest: digest, Kind: "bundle", Holder: "distribution"})
}
// outcome says a seat's outcome on subject, as its holder does.
func (r *registry) outcome(t *testing.T, subject string, o versions.Outcome) {
t.Helper()
data, _ := json.Marshal(o)
msg := nats.NewMsg(subject)
msg.Header.Set("x-event-id", subject+"-"+o.Name+"-"+o.Version)
msg.Data = data
if _, err := r.b.js.PublishMsg(r.b.ctx, msg); err != nil {
t.Fatal(err)
}
}
// answered writes a reasks value and reads it until the builder answered it, within 20 s.
func (r *registry) answered(t *testing.T, key string, value []byte) build.ReAsk {
t.Helper()
if _, err := r.reasks.Put(r.b.ctx, key, value); err != nil {
t.Fatal(err)
}
var a build.ReAsk
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := r.reasks.Get(r.b.ctx, key); err == nil && json.Unmarshal(e.Value(), &a) == nil && !a.Answered.IsZero() {
return a
}
}
t.Fatalf("reasks.%s = %+v, not answered within 20 s", key, a)
return a
}
// awaitFinding reads the builder's finding key until it is there (raised) or gone, within 20 s.
func (r *registry) awaitFinding(t *testing.T, key string, raised bool) {
t.Helper()
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if _, err := r.findings.Get(r.b.ctx, key); (err == nil) == raised {
return
}
}
t.Fatalf("the finding %s raised = %v within 20 s, want %v", key, !raised, raised)
}
// numberedOf are the version.numbered the builder said for name, in order.
func (r *registry) numberedOf(name string) []string {
r.mu.Lock()
defer r.mu.Unlock()
var out []string
for _, nb := range r.numbered {
if nb.Name == name {
out = append(out, nb.Version)
}
}
return out
}
@@ -5,6 +5,7 @@
package main
import (
"cmp"
"context"
"fmt"
"io"
@@ -18,15 +19,18 @@ import (
"syscall"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-delivery/sdk/forms"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/views"
"mesh-delivery-builder/internal/build"
"mesh-delivery-builder/internal/versions"
)
const (
@@ -61,22 +65,53 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err != nil {
return err
}
modules, err := process.EssentialBucket(ctx, js, "builder", delivery.ModulesBucket, "know any fingerprint to build")
// The indexer's modules, manifests, candidates and packages and the assigner's placement are their owners' states, read by the
// first view of their read tools and then their views families, never their buckets (ADR 0615 §1, DLV-M1). An owner that
// is down never holds the start: each is read on a goroutine once its first view is answered, and until then the
// unanswered first view is the builder's finding.
calls, err := process.Connect(cred, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
manifests, err := process.EssentialBucket(ctx, js, "builder", delivery.ManifestsBucket, "read what a module builds")
defer calls.Close()
reader := &views.ProcessReader{Module: delivery.BuilderModule, Call: calls.Call, Logf: logger.Printf}
findingsKV := process.AwaitBucket(ctx, js, conn, findingsBucket, "the builder's findings, a first view unanswered past its bound among them, are said in its log only", logger.Printf)
reader.Findings = findingsKV.KV
modules, err := reader.Follow(delivery.IndexerModule, modulesState, delivery.IndexerModule+".indexer_read_modules", indexerViews)
if err != nil {
return err
}
manifests, err := reader.Follow(delivery.IndexerModule, manifestsState, delivery.IndexerModule+".indexer_read_manifests", indexerViews)
if err != nil {
return err
}
// The candidates' fingerprints are built too (issue 1684).
candidates, err := reader.Follow(delivery.IndexerModule, candidatesState, delivery.IndexerModule+".indexer_read_candidates", indexerViews)
if err != nil {
return err
}
// The indexer's packages, whose names a package is numbered under (design 32).
packages, err := reader.Follow(delivery.IndexerModule, delivery.PackagesState, delivery.IndexerModule+".indexer_read_packages", indexerViews)
if err != nil {
return err
}
placement, err := reader.Follow(delivery.AssignerModule, delivery.PlacementState, delivery.AssignerModule+".assigner_read_placement", assignerViews)
if err != nil {
return err
}
// The candidates' fingerprints are built too (issue 1684): a builder that cannot read them builds the trunk's.
candidates := process.OptionalBucket(ctx, js, delivery.CandidatesBucket, "no candidate's fingerprint is built until the builder starts again; the trunk's are", logger.Printf)
bound := defaultStepBound
if v := strings.TrimSpace(getenv("MESH_BUILDER_STEP_BOUND")); v != "" {
if bound, err = time.ParseDuration(v); err != nil {
return fmt.Errorf("MESH_BUILDER_STEP_BOUND: %w", err)
}
}
// An unanswered tag ask is narrated every TagAskBound (design 51: five minutes); a test gives a shorter one.
tagAskBound := versions.TagAskBound
if v := strings.TrimSpace(getenv("MESH_BUILDER_TAG_ASK_BOUND")); v != "" {
if tagAskBound, err = time.ParseDuration(v); err != nil {
return fmt.Errorf("MESH_BUILDER_TAG_ASK_BOUND: %w", err)
}
}
tools := &buildrun.ToolchainSet{}
// Each build module's part of a build is an ask on its build seat's stream, pulled by a build node with a free slot
// (novox/hq ADR 0525): the builder chooses no node, so it reads neither presence nor have and holds no capacity.
@@ -103,12 +138,31 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
heard := buildrun.HearEnds(ctx, asks, asks, buildModules, logger.Printf)
conn.SetReconnectHandler(func(*nats.Conn) { hearing.Again(); heard.Nudge("") })
// The builder asks the controller, as its own account, whether the artifact store still holds a done key's blobs
// (the seat verb held, novox/hq issue 1937) before it leaves the key done.
calls, err := process.Connect(cred, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer calls.Close()
// (the seat verb held, novox/hq issue 1937) before it leaves the key done, on its calls connection.
// The builder numbers each trunk build at its claim (novox/hq design 51 *The versions*): its state versions, bound once
// the bus holds it; the floor's three sources asked as its own account; each new number said as version.numbered.
versionsKV := process.AwaitBucket(ctx, js, conn, versions.Bucket, "no trunk build is numbered: each is built unnumbered and published by its digest", logger.Printf)
found := &findings{kv: findingsKV, logf: logger.Printf, now: time.Now}
floorWaits := &versions.FloorWaits{Logf: logger.Printf, Said: found.floorSaid(context.WithoutCancel(ctx))}
// The names waiting on their floor are kept in the state floor-waits (DLV-B4): a restarted builder takes them in, so
// their silent sources' events still wake them; until the state is bound they are kept here only.
process.AwaitBucket(ctx, js, conn, versions.FloorWaitsBucket, "the names waiting on their floor are kept in memory only, and builder_floor_waits shows none", logger.Printf).
Ready(func(kv jetstream.KeyValue) {
floorWaits.Keep(kvKeeper{ctx: ctx, kv: kv}, heldFloorWaits(ctx, kv, logger.Printf))
})
// A package is numbered under its name in its ecosystem, read from the indexer's packages state (design 32).
numberer := &versions.Numberer{Versions: versions.Versions{Store: awaitedVersions{versionsKV}}, Floor: floorVia(calls.Call), Waits: floorWaits,
Names: versions.NamesIn(indexerPackages{packages}),
Said: func(nb delivery.VersionNumbered) {
if err := emitNumbered(context.WithoutCancel(ctx), js, cred, nb); err != nil {
logger.Printf("%v", err)
}
}}
// A publish that ends badly is a finding (design 51 *Health and conditions*): a holder's refused, a version no holder
// answered within its bound (a key of the state bounds with its own time to live), a seat's outcome that does not read.
pubBoundsKV := process.AwaitBucket(ctx, js, conn, versions.BoundsBucket, "a publish no holder answers, and a node holding the old form seven days, are not raised", logger.Printf)
pubs := &publishes{bounds: pubBoundsKV.KV, versions: numberer.Versions, findings: found, bound: versions.PublishBound, logf: logger.Printf}
numberer.Publishing = pubs.arm
b := &build.Builder{
Claimer: build.Claimer{Store: kvStore{builds}, By: cred.Node + "." + cred.Module},
Asks: buildrun.Asker{Publisher: asks, Waiter: buildrun.LostAskWaiter{Inner: heard, Dead: dead, Hearing: hearing}, Bound: bound},
@@ -122,20 +176,52 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
Stored: storedVia(calls.Call),
Node: cred.Node,
}
tagAsksKV := process.AwaitBucket(ctx, js, conn, versions.TagAsksBucket, "builder_tag and builder_untag are not answered", logger.Printf)
// A check's build of a head a kept tag ask names numbers its packages tagged (design 51 *Tagged versions*), and the ask
// is marked numbered with its versions.
b.Tags = tagsVia(tagAsksKV.KV, logger.Printf)
b.Numbers = build.VersionNumbers{N: numberer, TagNumbered: tagNumbered(tagAsksKV.KV, logger.Printf, time.Now)}
// An ask not answered within five minutes is narrated with what it waits on: its bound's end on the bus says when.
boundsKV := process.AwaitBucket(ctx, js, conn, versions.TagAskBoundsBucket, "an unanswered tag ask is not narrated", logger.Printf)
bounds := &tagBounds{asks: tagAsksKV.KV, bounds: boundsKV.KV, bound: tagAskBound, waits: floorWaits.Waiting, logf: logger.Printf, now: time.Now,
log: func(e delivery.BuildLogEntry) { b.Log(e) }}
// A state bound later numbers the builds looked at without it.
versionsKV.Ready(func(jetstream.KeyValue) { b.Reconsider(ctx) })
// The versions' hand acts (builder_publish_again, builder_floor_again) come through reasks, as builder_build_again.
acts := versions.Acts{Numberer: numberer, Done: doneVia(b), Reconsider: b.Reconsider, Remove: removeVia(calls.Call),
BuildAgain: func(ctx context.Context, module, fp, why, by string) (string, bool) {
r := b.BuildAgain(ctx, module, fp, why, by, buildAgainBound)
if r.Refused != "" {
return r.Refused, true
}
if r.Failed != "" {
return "built again, and it failed before its asks were put: " + r.Failed, false
}
return fmt.Sprintf("built again as revision %d", r.Revision), false
}}
// A candidate's preview is merged onto the trunk head the indexer previewed it onto: source.main at the candidate's base
// revision (the gate's source state). The source's revisions are watched with their history (a direct get of one revision
// is not what a read grants): a revision heard after the candidate that names it builds that candidate then. Where it
// cannot be read the trunk's keys are built and the candidates wait (issue 1945).
errs := make(chan error, 6)
trunks := &trunkSource{ctx: ctx, js: js, heads: &sourceHeads{}, moved: b.TrunkMoved, errs: errs, logf: logger.Printf}
errs := make(chan error, 10)
trunks := &trunkSource{ctx: ctx, js: js, heads: &sourceHeads{}, moved: func(ctx context.Context) {
// A source revision heard is the forge's merge: it wakes a floor whose bridge_tags was silent (design 51 *The floor*).
(&floorWake{b: b, waits: floorWaits, logf: logger.Printf}).wake(ctx, versions.WakeForge, "", "")
b.TrunkMoved(ctx)
}, errs: errs, logf: logger.Printf}
trunks.start()
b.Trunk = trunks.head
// The builder's log (ADR 0571 §1): each build's start and end in its module log `build-log`, read by the view, the
// indexer and the assigner through their reads, and by mesh-catalog through the tool builder_builds_since.
// indexer and the assigner through their reads, and by the operator's command nox builds through the tool builder_builds_since.
b.Log = func(e delivery.BuildLogEntry) {
if e.Node == "" { // a narration of no one build (a tag ask, the old form's window) is this instance's
e.Node = cred.Node
}
if err := appendLog(context.WithoutCancel(ctx), js, cred, e); err != nil {
logger.Printf("%v", err)
}
// A failed trunk build is the builder's finding build-failed, cleared by a later done one (DLV-B7).
found.buildEnded(context.WithoutCancel(ctx), e)
}
b.Said = func(e build.AskedAgain) {
if err := emitAskedAgain(context.WithoutCancel(ctx), js, cred, e); err != nil {
@@ -165,42 +251,219 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// placement is settled first, so the toolchains settle through the gate already knowing it.
gate := buildrun.NewToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, b.NodesChanged, logger.Printf)
if err != nil {
return err
// What each counted node's engine reads (ADR 0618 §2, ADR 0624 §1): asked through the node-engine seat's verb as the
// builder's own account, for every node the placement names, and kept in the state engine-reads once it is bound.
reads := newEngineReads(calls.CallOn, gate, logger.Printf)
// The old reference form's window over them (DLV-B9), kept in the state old-form once it is bound; builder_old_form
// reads it and drops a retired node by hand.
reads.ctx = context.WithoutCancel(ctx)
reads.window = &oldForm{forms: forms.Current, findings: found, logf: logger.Printf, now: time.Now, log: func(e delivery.BuildLogEntry) { b.Log(e) },
bounds: pubBoundsKV.KV,
said: func(body versions.OldFormClosedBody) {
if err := emitEvent(context.WithoutCancel(ctx), js, cred, versions.OldFormClosedEvent, "old-form-closed-"+body.At.Format(time.RFC3339Nano), body); err != nil {
logger.Printf("saying %s: %v", versions.OldFormClosedEvent, err)
}
}}
acts.DropOldForm = func(ctx context.Context, node, by, why string, at time.Time) (string, error) {
answer, err := reads.window.drop(ctx, node, by, why, at)
if err == nil {
reads.stepWindow()
}
return answer, err
}
process.AwaitBucket(ctx, js, conn, versions.OldFormBucket, "the old form's window is kept in memory only, and builder_old_form shows none", logger.Printf).
Ready(func(kv jetstream.KeyValue) {
reads.window.bind(context.WithoutCancel(ctx), kv)
reads.stepWindow()
})
gate.OnPlaced = reads.placed
b.Forms = reads.built
// Until the placement's first view is answered the gate passes every report on, as before the placement is settled.
placementErrs := make(chan error, 1)
go func() {
select {
case <-ctx.Done():
return
case <-placement.Follower().Ready():
}
pe, err := buildrun.FollowPlacement(ctx, reader, gate, b.NodesChanged, logger.Printf)
if err != nil {
placementErrs <- err
return
}
placementErrs <- <-pe
}()
if err := watch.start(); err != nil {
return err
}
// A build module says each time it wrote its toolchains key: its stream, and the builder's durable consumer on it, are
// made by the controller. Where the stream is not there yet the builder says so and is started again with the build
// module.
for _, module := range []string{buildrun.GoBuildModule, buildrun.NodeBuildModule} {
stream := toolchainStreams[module]
cons, err := cred.EventConsumer(ctx, js, stream)
if err != nil {
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s's reports are not heard until the builder starts again: %v", cred.Node, cred.Module, stream, module, err)
continue
}
cc, err := process.HearEvents(cons, func(m jetstream.Msg) { watch.Reported(m) }, logger.Printf)
if err != nil {
return err
}
defer cc.Stop()
// Each consumer that is not there yet is the finding family-not-heard naming its stream until it binds at a later
// start (heard.Writer; ADR 0615 §5); one there that cannot be heard stops the builder, as before.
notHeard := found.notHeard(ctx)
var stopped error
hear := func(stream, consequence string, each func(jetstream.Msg)) {
_ = notHeard.Bind(stream, consequence, func() error {
cons, err := cred.EventConsumer(ctx, js, stream)
if err != nil {
return fmt.Errorf("the consumer %s_%s on %s does not exist yet: %w", cred.Node, cred.Module, stream, err)
}
cc, err := process.HearEvents(cons, each, logger.Printf)
if err != nil {
stopped = cmp.Or(stopped, err)
return err
}
go func() { <-ctx.Done(); cc.Stop() }()
return nil
})
}
// The controller says artifacts-gone when its take-in found a key's blob gone (issue 1937): the builder's durable
// consumer on EVENTS, made by the controller from the manifest's consumes. Where it is not there yet the builder says
for _, module := range []string{buildrun.GoBuildModule, buildrun.NodeBuildModule} {
hear(toolchainStreams[module], module+"'s toolchain reports are not heard until the builder starts again", func(m jetstream.Msg) { watch.Reported(m) })
}
// The seat mesh-artifact-sweep says artifacts-gone when it found a key's blob gone (issue 1937, ADR 0571 §2): the
// builder's durable consumer on EVENTS, made by the grants part from the manifest's consumes. Where it is not there yet the builder says
// so; it still asks held at its start and at every module value.
switch cons, err := cred.EventConsumer(ctx, js, eventsStream); {
case err != nil:
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s is not heard until the builder starts again: %v",
cred.Node, cred.Module, eventsStream, delivery.ArtifactsGoneEvent, err)
default:
cc, err := process.HearEvents(cons, func(m jetstream.Msg) { heardGone(ctx, b, m, logger.Printf) }, logger.Printf)
if err != nil {
return err
// The same consumer carries what wakes a floor (design 51 *The floor*): the indexer's manifest-accepted and the
// condition keeper's condition-cleared.
floor := floorWake{b: b, waits: floorWaits, logf: logger.Printf}
// And the four registry seats' published, refused and removed (DLV-B4), which move a numbered version.
seats := seatHearer{n: numberer, b: b, logf: logger.Printf, pub: pubs}
hear(eventsStream, delivery.ArtifactsGoneEvent+", what wakes a floor, the registry seats' words and manifest-reads-changed are not heard until the builder starts again", func(m jetstream.Msg) {
if reads.heard(m, logger.Printf) || floor.heard(ctx, m) || seats.heard(ctx, m) { // reads: the node-engine seat's manifest-reads-changed (ADR 0618 §4)
return
}
heardGone(ctx, b, m, logger.Printf)
})
if stopped != nil {
return stopped
}
reader.Start(ctx, js, cred, process.AwaitBucket(ctx, js, conn, firstViewsBucket, "builder_first_views reads nothing and builder_first_view_again is not heard", logger.Printf))
// The modules, once their first views and the manifests' are answered: what a module builds is read from the manifests.
go func() { errs <- buildFromModules(ctx, b, modules, manifests, candidates) }()
// A package's name heard in the indexer's packages looks at its module again, so a done build is numbered then.
go func() { errs <- followPackages(ctx, packages, b, logger.Printf) }()
// A person asks a key's build again with builder_ask_again, which writes the reasks state (issue 1945): watched, as
// every other input. The builder asks again by itself without it.
// A state not on the bus yet at this start is watched once it appears (issue 1790).
process.AwaitBucket(ctx, js, conn, build.ReAsksBucket, "builder_ask_again is not heard; an ask that can no longer be answered is still asked again by itself", logger.Printf).Ready(func(kv jetstream.KeyValue) {
eachReAsk := func(e jetstream.KeyValueEntry) { reAsked(ctx, b, acts, kv, e, logger.Printf) }
if rw, err := settle(ctx, kv, eachReAsk); err != nil {
process.NotWatched(logger.Printf, build.ReAsksBucket, err, "builder_ask_again is not heard until the builder starts again")
} else {
go func() { errs <- follow(ctx, rw, eachReAsk) }()
}
})
// A tag ask or a branch's removal by hand (builder_tag, builder_untag) is written to tag-asks and answered there
// (design 51 *Tagged versions*); the checker's tag-wanted and the branch's end join it on their consumers (DLV-B6).
tagAsksKV.Ready(func(kv jetstream.KeyValue) {
each := func(e jetstream.KeyValueEntry) {
tagAsked(ctx, acts, kv, e, logger.Printf)
if !opOf(e) {
bounds.disarm(ctx, e.Key())
} else if t, err := versions.DecodeTagAsk(e.Value()); err == nil {
bounds.asked(ctx, e.Key(), t)
}
}
if tw, err := settle(ctx, kv, each); err != nil {
process.NotWatched(logger.Printf, versions.TagAsksBucket, err, "builder_tag and builder_untag are not answered until the builder starts again")
} else {
go func() { errs <- follow(ctx, tw, each) }()
}
})
pubBoundsKV.Ready(func(kv jetstream.KeyValue) {
each := func(e jetstream.KeyValueEntry) {
if versions.IsOldFormBound(e.Key()) {
if e.Operation() == jetstream.KeyValuePurge {
reads.stepWindow() // a node's seven days ended: the step raises old-form-held now
}
return
}
pubs.ended(ctx, e)
}
watchBounds(ctx, kv, each, found.notHeard(ctx), errs, logger.Printf)
})
boundsKV.Ready(func(kv jetstream.KeyValue) {
each := func(e jetstream.KeyValueEntry) { bounds.ended(ctx, e) }
if bw, err := settle(ctx, kv, each); err != nil {
process.NotWatched(logger.Printf, versions.TagAskBoundsBucket, err, "an unanswered tag ask is not narrated until the builder starts again")
} else {
go func() { errs <- follow(ctx, bw, each) }()
}
})
// A person reads a build seat's stream with builder_seat_queue, which writes the queues state (issue 1945 note 62): the
// builder alone may read the stream, so it answers there.
process.AwaitBucket(ctx, js, conn, build.QueuesBucket, "builder_seat_queue is not answered", logger.Printf).Ready(func(kv jetstream.KeyValue) {
eachRead := func(e jetstream.KeyValueEntry) { queueRead(ctx, heldOnTheBus{js}, kv, e, logger.Printf) }
if qw, err := settle(ctx, kv, eachRead); err != nil {
process.NotWatched(logger.Printf, build.QueuesBucket, err, "builder_seat_queue is not answered until the builder starts again")
} else {
go func() { errs <- follow(ctx, qw, eachRead) }()
}
})
// A branch's end removes its tagged versions (design 51 *Tagged versions*, DLV-B6): the forge bridge's merged, a pull's
// closed or branch-deleted on the builder's durable consumer of its stream; each forge message also wakes a floor whose
// bridge_tags was silent. The checker's tag-wanted is kept in tag-asks. A consumer not there yet is said.
ends := &branchEnds{acts: acts, findings: found, tagAsks: tagAsksKV.KV, logf: logger.Printf, now: time.Now,
wake: func(ctx context.Context) {
(&floorWake{b: b, waits: floorWaits, logf: logger.Printf}).wake(ctx, versions.WakeForge, "", "")
}}
for _, c := range []struct {
stream, what string
hear func(jetstream.Msg)
}{
{forgeStream, "a branch's end", func(m jetstream.Msg) { ends.heard(ctx, m) }},
{checkerAsksStream, "the checker's tag-wanted", func(m jetstream.Msg) {
tagWanted(ctx, tagAsksKV.KV(), m, delivery.CheckerModule, time.Now(), logger.Printf)
}},
} {
hear(c.stream, c.what+" is not heard until the builder starts again", c.hear)
}
if stopped != nil {
return stopped
}
// The state engine-reads, new with ADR 0618 (issue 1790's kind): until it is on the bus each statement is kept in memory
// only, and builder_engine_reads answers nothing.
process.AwaitBucket(ctx, js, conn, engineReadsBucket, "what each node's engine reads is kept in memory only", logger.Printf).Ready(func(kv jetstream.KeyValue) {
reads.bind(ctx, kv, errs, logger.Printf)
})
logger.Printf("started on %s", cred.Node)
select {
case <-ctx.Done():
b.Wait()
return nil
case err := <-errs:
return err
case err := <-placementErrs:
return err
case err := <-watch.errs:
return err
}
}
// The builder's own state of its first views (views.StatusState); its findings (findingsBucket) are where the condition
// keeper raises a first view unanswered past its bound (ADR 0615 §6).
var firstViewsBucket = delivery.BuilderModule + "_" + views.StatusState
// The indexer's states the builder follows (delivery.ModulesBucket, ManifestsBucket, CandidatesBucket), and the owners'
// views families as their manifests' `streams` name them.
const (
indexerViews = "indexer-views"
assignerViews = "assigner-views"
modulesState = "modules"
manifestsState = "manifests"
candidatesState = "candidates"
)
// buildFromModules builds from the indexer's modules and candidates once the first views of the modules, the manifests and
// the candidates are answered (never at the start: an indexer that is down holds no other part of it), and follows them.
func buildFromModules(ctx context.Context, b *build.Builder, modules, manifests, candidates *views.Picture) error {
for _, p := range []*views.Picture{modules, manifests, candidates} {
select {
case <-ctx.Done():
return nil
case <-p.Follower().Ready():
}
defer cc.Stop()
}
each := func(e jetstream.KeyValueEntry) {
if opOf(e) {
@@ -225,48 +488,12 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
b.ForgetCandidate(e.Key())
}
}
if candidates != nil {
cw, err := settle(ctx, candidates, eachCandidate)
if err != nil {
process.NotWatched(logger.Printf, delivery.CandidatesBucket, err, "no candidate's fingerprint is built until the builder starts again")
} else {
go func() { errs <- follow(ctx, cw, eachCandidate) }()
}
}
// A person asks a key's build again with builder_ask_again, which writes the reasks state (issue 1945): watched, as
// every other input. The builder asks again by itself without it.
// A state not on the bus yet at this start is watched once it appears (issue 1790).
process.AwaitBucket(ctx, js, conn, build.ReAsksBucket, "builder_ask_again is not heard; an ask that can no longer be answered is still asked again by itself", logger.Printf).Ready(func(kv jetstream.KeyValue) {
eachReAsk := func(e jetstream.KeyValueEntry) { reAsked(ctx, b, kv, e, logger.Printf) }
if rw, err := settle(ctx, kv, eachReAsk); err != nil {
process.NotWatched(logger.Printf, build.ReAsksBucket, err, "builder_ask_again is not heard until the builder starts again")
} else {
go func() { errs <- follow(ctx, rw, eachReAsk) }()
}
})
// A person reads a build seat's stream with builder_seat_queue, which writes the queues state (issue 1945 note 62): the
// builder alone may read the stream, so it answers there.
process.AwaitBucket(ctx, js, conn, build.QueuesBucket, "builder_seat_queue is not answered", logger.Printf).Ready(func(kv jetstream.KeyValue) {
eachRead := func(e jetstream.KeyValueEntry) { queueRead(ctx, heldOnTheBus{js}, kv, e, logger.Printf) }
if qw, err := settle(ctx, kv, eachRead); err != nil {
process.NotWatched(logger.Printf, build.QueuesBucket, err, "builder_seat_queue is not answered until the builder starts again")
} else {
go func() { errs <- follow(ctx, qw, eachRead) }()
}
})
logger.Printf("started on %s", cred.Node)
go func() { errs <- follow(ctx, w, each) }()
select {
case <-ctx.Done():
b.Wait()
return nil
case err := <-errs:
return err
case err := <-placementErrs:
return err
case err := <-watch.errs:
cw, err := settle(ctx, candidates, eachCandidate)
if err != nil {
return err
}
go func() { _ = follow(ctx, cw, eachCandidate) }()
return follow(ctx, w, each)
}
// sourceHeads are the trunk heads of each revision of the gate's source.main the watch delivered, the newest
@@ -0,0 +1,187 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-delivery/sdk/forms"
"mesh-delivery-builder/internal/versions"
)
// oldForm keeps the old reference form's window (design 51 *Every build keeps the old reference form*, DLV-B9) in the
// state old-form: stepped on every change of what the counted nodes state, of the placement, on a hand drop and at each
// build key written, never on a timer. A node holding it seven days is the finding old-form-held; its close is said once
// as version.old-form-closed and in the builder's log.
type oldForm struct {
forms forms.ArtifactForms
findings *findings
// said emits version.old-form-closed.
said func(versions.OldFormClosedBody)
// log writes the close into the builder's build log (build.said); nil writes none.
log func(delivery.BuildLogEntry)
logf func(string, ...any)
now func() time.Time
// bounds is the builder's state bounds: each node holding the window has a key there whose time to live ends at its
// seven days, so old-form-held is raised then, not at the next change (review round 1); nil keeps none.
bounds func() jetstream.KeyValue
mu sync.Mutex
w versions.OldFormWindow
kv jetstream.KeyValue
// overdue are the nodes the finding is raised for now.
overdue []string
}
// bind takes the kept window in, so a restarted builder keeps each node's since and the nodes dropped by hand, and keeps
// every later step in kv.
func (o *oldForm) bind(ctx context.Context, kv jetstream.KeyValue) {
e, err := kv.Get(ctx, versions.OldFormKey)
o.mu.Lock()
defer o.mu.Unlock()
switch {
case errors.Is(err, jetstream.ErrKeyNotFound):
case err != nil:
o.logf("%s.%s cannot be read, so the window starts again: %v", versions.OldFormState, versions.OldFormKey, err)
default:
w, err := versions.DecodeOldFormWindow(e.Value())
if err != nil {
o.logf("%v; the window starts again", err)
break
}
// Taken in under what was stepped before the state was bound: the kept since and drops are the older truth.
for n, s := range o.w.Since {
if _, ok := w.Since[n]; !ok && w.Open && slices.Contains(w.Holding, n) {
w.Since[n] = s
}
}
for _, d := range o.w.Dropped {
w, _ = w.Drop(d.Node, d.By, d.Why, d.At)
}
o.w = w
}
o.kv = kv
o.keep(ctx)
}
// dropped is the nodes dropped by hand.
func (o *oldForm) dropped() map[string]bool {
o.mu.Lock()
defer o.mu.Unlock()
return o.w.DroppedSet()
}
// step moves the window to what counted states.
// named is every node the read named (the placement's nodes before any is dropped), counted the statements of those
// still counted: a read naming no node judges nothing (delivery.StepWindow), so it never closes the window.
func (o *oldForm) step(ctx context.Context, named []string, counted map[string][]string) {
o.mu.Lock()
defer o.mu.Unlock()
before := o.w
next, closed, overdue := o.w.Step(named, versions.Holding(o.forms, counted, o.w.DroppedSet()), o.now())
o.w = next
if !slices.Equal(before.Holding, next.Holding) || before.Open != next.Open || closed {
o.keep(ctx)
}
for _, n := range o.overdue {
if !slices.Contains(overdue, n) {
o.findings.clear(ctx, versions.OldFormHeld, health.ScopeMachine, n)
}
}
for _, n := range overdue {
o.findings.raise(ctx, finding{kind: versions.OldFormHeld, scope: health.ScopeMachine, id: n,
summary: fmt.Sprintf("the node-engine of %s has read only an older reference form since %s, so every build key still carries it",
n, next.Since[n].Format(time.RFC3339)),
headline: "A node holds the old reference form",
explanation: "The builder writes an older reference form into every build key while a counted node's engine reads none of the newest (ADR 0624 §1); this node has held it seven days.",
resolved: "The node's engine states the newest form, the node leaves the placement, or it is dropped as retired.",
needs: "builder_old_form shows the window; a retired node is dropped there by hand; builder_engine_reads asks the node again."})
}
o.overdue = overdue
o.keepBounds(ctx, before, next, overdue)
if closed {
body := next.Closing()
line := fmt.Sprintf("the old reference form's window closed at %s: no counted node-engine reads only an older form (last held by %s; dropped by hand: %d); new build keys leave it out",
body.At.Format(time.RFC3339), strings.Join(body.LastHolding, ", "), len(body.Dropped))
o.logf("%s", line)
// In the build log too (design 51 *The node-engine's own build*), under the builder's own key, the window's key
// standing for the fingerprint: the close is no one build's.
if o.log != nil {
o.log(delivery.BuildLogEntry{Event: delivery.BuildSaidEvent, Module: delivery.BuilderModule, FP: versions.OldFormKey,
At: body.At.UTC(), Line: line, Why: versions.SaidOldFormClosed})
}
if o.said != nil {
o.said(body)
}
}
}
// keepBounds writes a bound for each node holding the window and not yet overdue, ending at its seven days, and deletes
// the bound of each node that no longer holds it. A standing bound keeps its time (a Create). Called with mu held.
func (o *oldForm) keepBounds(ctx context.Context, before, next versions.OldFormWindow, overdue []string) {
if o.bounds == nil {
return
}
kv := o.bounds()
if kv == nil {
return
}
for _, n := range before.Holding {
if !slices.Contains(next.Holding, n) {
if err := kv.Delete(ctx, versions.OldFormBoundKey(n)); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
o.logf("the old form's bound of %s is not deleted: %v", n, err)
}
}
}
for _, n := range next.Holding {
left := next.Since[n].Add(versions.FindingAfter).Sub(o.now())
if slices.Contains(overdue, n) || left <= 0 {
continue
}
if left < time.Second {
left = time.Second // the bus's least time to live
}
if _, err := kv.Create(ctx, versions.OldFormBoundKey(n), versions.EncodeBound("the seven days "+n+" holds the old form's window", o.now().Add(left)), jetstream.KeyTTL(left)); err != nil && !errors.Is(err, jetstream.ErrKeyExists) {
o.logf("the old form's bound of %s is not written, so its seven days are seen at the next change only: %v", n, err)
}
}
}
// drop is builder_old_form's drop: the node leaves the count, the window is stepped by the caller.
func (o *oldForm) drop(ctx context.Context, node, by, why string, at time.Time) (string, error) {
o.mu.Lock()
defer o.mu.Unlock()
w, err := o.w.Drop(node, by, why, at)
if err != nil {
return "", err
}
o.w = w
o.keep(ctx)
return node + " dropped from the old form's count by hand; the window is stepped now", nil
}
// keep writes the window, once the state is bound; until then it is kept here only. Called with mu held.
func (o *oldForm) keep(ctx context.Context) {
if o.kv == nil {
return
}
raw, err := json.Marshal(o.w)
if err == nil {
_, err = o.kv.Put(ctx, versions.OldFormKey, raw)
}
if err != nil {
o.logf("%s.%s cannot be written: %v", versions.OldFormState, versions.OldFormKey, err)
}
}
@@ -0,0 +1,129 @@
package main
import (
"context"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-delivery/sdk/heard"
"mesh-delivery-builder/internal/versions"
)
// publishes raises and clears the builder's findings of a publish that ended badly (versions/publish.go): a holder's
// refused, a version unanswered past its bound, kept as a key of the state bounds with its own time to live, and a seat's
// outcome that does not read.
type publishes struct {
bounds func() jetstream.KeyValue
versions versions.Versions
findings *findings
bound time.Duration
logf func(string, ...any)
}
// arm writes the version's bound when none stands, so a standing bound keeps its time. A bound that cannot be written
// would never end, so the version is publish-unanswered at once (review round 2): the finding says why.
func (p *publishes) arm(ctx context.Context, e versions.Entry) {
kv := p.bounds()
var err error
if kv == nil {
err = fmt.Errorf("%s is not on the bus yet", versions.BoundsBucket)
} else if _, err = kv.Create(ctx, versions.PublishBoundKey(e.Name, e.Version), versions.EncodeBound("the publish of "+e.Name+"@"+e.Version+" to "+e.Seat, time.Now().Add(p.bound)), jetstream.KeyTTL(p.bound)); errors.Is(err, jetstream.ErrKeyExists) {
err = nil
}
if err == nil {
return
}
p.logf("the publish bound of %s@%s is not written, so it is raised unanswered now: %v", e.Name, e.Version, err)
p.unanswered(ctx, e, fmt.Sprintf("its bound could not be written (%v), so the builder cannot wait on the holder's word", err))
}
// unanswered raises publish-unanswered about e's name, saying why.
func (p *publishes) unanswered(ctx context.Context, v versions.Entry, why string) {
p.findings.raise(ctx, finding{kind: versions.PublishUnanswered, scope: health.ScopeModule, id: v.Name,
summary: fmt.Sprintf("%s said neither published nor refused of %s@%s: %s", v.Seat, v.Name, v.Version, why),
headline: "A registry seat did not answer a publish",
explanation: "The builder built and numbered " + v.Name + " " + v.Version + " and asked " + v.Seat + " to publish it; no word of its holder is known, so the build says done while the version may not be in the registry.",
resolved: "That version is published, or a later version of " + v.Name + " is.",
needs: "a look at the holder of " + v.Seat + " (its health and log); builder_publish_again asks the publish again."})
}
// heard is told each holder's word the numberer took in: published clears the name's findings, refused raises
// publish-refused with the holder's reason, and either ends the version's bound.
func (p *publishes) heard(ctx context.Context, seat, event string, o versions.Outcome) {
if o.Version != "" {
if kv := p.bounds(); kv != nil {
if err := kv.Delete(ctx, versions.PublishBoundKey(o.Name, o.Version)); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
p.logf("the publish bound of %s@%s is not ended: %v", o.Name, o.Version, err)
}
}
}
switch event {
case versions.HeardPublished:
p.findings.clear(ctx, versions.PublishRefused, health.ScopeModule, o.Name)
p.findings.clear(ctx, versions.PublishUnanswered, health.ScopeModule, o.Name)
case versions.HeardRefused:
reason := o.Reason
if reason == "" {
reason = "it gave no reason"
}
p.findings.raise(ctx, finding{kind: versions.PublishRefused, scope: health.ScopeModule, id: o.Name,
summary: fmt.Sprintf("%s refused to publish %s@%s (digest %s): %s", seat, o.Name, o.Version, o.Digest, reason),
headline: "A registry seat refused a publish",
explanation: "The builder built and numbered " + o.Name + " " + o.Version + ", and the holder of " + seat + " refused to store it, so that version is not in the registry.",
resolved: "That version is published, or a later version of " + o.Name + " is.",
needs: "builder_versions shows the version and why; builder_publish_again publishes it again once the cause is mended."})
}
}
// unread raises seat-outcome-unread about seat: what it said on subject is no outcome. read clears it.
func (p *publishes) unread(ctx context.Context, seat, subject string, why error) {
p.findings.raise(ctx, finding{kind: versions.SeatOutcomeUnread, scope: health.ScopeModule, id: seat,
summary: fmt.Sprintf("%s said on %s what is no outcome of a publish: %v", seat, subject, why),
headline: "A registry seat said what the builder cannot read",
explanation: "The builder hears each registry seat's published, refused and removed to move a numbered version; this one does not read, so the version it was about does not move.",
resolved: "The next outcome of " + seat + " reads.",
needs: "a look at the seat's holder: the outcome's body is its contract's (mesh-contracts)."})
}
func (p *publishes) read(ctx context.Context, seat string) {
p.findings.clear(ctx, versions.SeatOutcomeUnread, health.ScopeModule, seat)
}
// ended is told each change of the state bounds: a bound whose time ended (the bus's marker) on a version still numbered
// raises publish-unanswered.
func (p *publishes) ended(ctx context.Context, e jetstream.KeyValueEntry) {
key, ok := versions.PublishBoundOf(e.Key())
if e.Operation() != jetstream.KeyValuePurge || !ok {
return
}
raw, _, found, err := p.versions.Store.Get(ctx, key)
if err != nil || !found {
return
}
v, err := versions.Decode(raw)
if err != nil || v.State != versions.Numbered {
return
}
p.unanswered(ctx, v, fmt.Sprintf("nothing within %s of its build", p.bound))
}
// watchBounds watches the state bounds, handing each change to each: a watch that cannot start loses every bound's end,
// so it is the finding family-not-heard naming the state until a later start watches it (review round 2).
func watchBounds(ctx context.Context, kv jetstream.KeyValue, each func(jetstream.KeyValueEntry), w heard.Writer, errs chan<- error, logf func(string, ...any)) {
const consequence = "a publish no holder answers, and a node holding the old form seven days, are not raised until the builder starts again"
pw, err := settle(ctx, kv, each)
if err != nil {
process.NotWatched(logf, versions.BoundsBucket, err, consequence)
w.NotHeard(versions.BoundsState, consequence, err)
return
}
w.Heard(versions.BoundsState)
go func() { errs <- follow(ctx, pw, each) }()
}
@@ -0,0 +1,151 @@
package main
import (
"context"
"slices"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-delivery/sdk/enginereads"
)
// EngineReadsState is the builder's state of what each counted node's engine states it reads (novox/hq ADR 0618 §2, ADR
// 0624 §1): one key `<node>`, written from the node-engine seat's verb manifest-reads at the builder's start, on the
// seat's event manifest-reads-changed for that node, and when a person asks again (builder_engine_reads); a node that
// is away keeps its last statement, and its key is removed when the placement no longer names the node. A counted node
// with no key reads as stating the baseline: its first forms. No node's record is read in its place.
const EngineReadsState = "engine-reads"
// engineReadsBucket is that state's bucket.
const engineReadsBucket = delivery.BuilderModule + "_" + EngineReadsState
// engineReads keeps the statements and the counted nodes for the builder's forms (build.Builder.Forms).
type engineReads struct {
keeper *enginereads.Keeper
gate *buildrun.ToolchainGate
mu sync.Mutex
counted []string
// window is the old reference form's window over these statements (DLV-B9); nil keeps none.
window *oldForm
ctx context.Context
}
// newEngineReads asks through call (process.Conn.CallOn) and keeps each answer in the state once it is bound.
func newEngineReads(call enginereads.Caller, gate *buildrun.ToolchainGate, logf func(string, ...any)) *engineReads {
r := &engineReads{gate: gate, keeper: &enginereads.Keeper{Call: call, Logf: logf}}
r.keeper.Changed = r.stepWindow
return r
}
// stepWindow moves the old form's window to what the counted nodes state now (design 51 *Every build keeps the old
// reference form*): on a statement's change, the placement's and a hand drop, and at each build key written.
func (r *engineReads) stepWindow() {
if r.window == nil {
return
}
ctx := r.ctx
if ctx == nil {
ctx = context.Background()
}
_, counted := r.forms()
r.window.step(ctx, r.named(), counted)
}
// named are the nodes the placement names, none dropped: what the window's step judges a read by.
func (r *engineReads) named() []string {
r.mu.Lock()
defer r.mu.Unlock()
return slices.Clone(r.counted)
}
// bind keeps the statements in kv and hears it: every node's last statement at the start, a person's ask again and drop
// later. Bound once the state is on the bus (process.AwaitBucket).
func (r *engineReads) bind(ctx context.Context, kv jetstream.KeyValue, errs chan<- error, logf func(string, ...any)) {
r.keeper.Put = func(node string, v []byte) error { _, err := kv.Put(ctx, node, v); return err }
r.keeper.Delete = func(node string) error { return kv.Delete(ctx, node) }
each := func(e jetstream.KeyValueEntry) {
r.keeper.Heard(e.Key(), e.Value(), e.Operation() != jetstream.KeyValuePut)
}
w, err := settle(ctx, kv, each)
if err != nil {
process.NotWatched(logf, engineReadsBucket, err, "builder_engine_reads is not answered until the builder starts again")
return
}
go func() { errs <- follow(ctx, w, each) }()
// Every counted node asked again, so what was asked before the state was bound is kept in it too.
r.mu.Lock()
nodes := slices.Clone(r.counted)
r.mu.Unlock()
for _, node := range nodes {
r.keeper.AskNode(node, "the state engine-reads is bound")
}
}
// placed takes the placement in: a node it names newly is asked, one it no longer names is forgotten, key and all. The
// first call asks every counted node (the builder's start).
func (r *engineReads) placed() {
nodes := r.gate.Nodes()
r.mu.Lock()
before, first := r.counted, r.counted == nil
r.counted = nodes
r.mu.Unlock()
for _, node := range nodes {
if first || !slices.Contains(before, node) {
r.keeper.AskNode(node, "counted: the placement names it")
}
}
for _, node := range before {
if !slices.Contains(nodes, node) {
r.keeper.Forget(node)
}
}
r.stepWindow()
}
// heard takes an event of the builder's consumer on EVENTS: manifest-reads-changed asks its node again, and answers true.
func (r *engineReads) heard(m jetstream.Msg, logf func(string, ...any)) bool {
node, ours, err := enginereads.NodeOf(m.Subject(), m.Data())
if !ours {
return false
}
if ackErr := m.Ack(); ackErr != nil {
logf("acknowledging %s: %v", m.Subject(), ackErr)
}
if err != nil {
logf("%s holds what is not %s, and %s is asked again all the same: %v", m.Subject(), enginereads.Event, node, err)
}
r.keeper.AskNode(node, enginereads.Event)
return true
}
// forms is the counted nodes' statements, and those of the nodes whose engine answered; a retired node dropped by hand
// (builder_old_form) counts no more.
func (r *engineReads) forms() (present, counted map[string][]string) {
r.mu.Lock()
nodes := slices.Clone(r.counted)
r.mu.Unlock()
if r.window != nil {
dropped := r.window.dropped()
nodes = slices.DeleteFunc(nodes, func(n string) bool { return dropped[n] })
}
return r.keeper.Present(nodes), r.keeper.Reads(nodes)
}
// built is build.Builder.Forms: forms, and the window stepped with them, so a node's seven days are seen at the next
// build key at the latest.
func (r *engineReads) built() (present, counted map[string][]string) {
present, counted = r.forms()
if r.window != nil {
ctx := r.ctx
if ctx == nil {
ctx = context.Background()
}
r.window.step(ctx, r.named(), counted)
}
return present, counted
}
@@ -15,6 +15,7 @@ import (
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-builder/internal/build"
"mesh-delivery-builder/internal/versions"
)
// askAgainer is the part of the builder a reasks value asks through.
@@ -31,7 +32,7 @@ const buildAgainBound = 15 * time.Second
// reAsked answers one value of the reasks state (the tool builder_ask_again, issue 1945): a value not yet answered asks the
// key's waiting lanes again and is written back, by compare-and-set at its revision, with what the builder did. A value
// answered already (this builder's own write) or deleted is left.
func reAsked(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstream.KeyValueEntry, logf func(string, ...any)) {
func reAsked(ctx context.Context, b askAgainer, acts versions.Acts, kv jetstream.KeyValue, e jetstream.KeyValueEntry, logf func(string, ...any)) {
if e.Operation() != jetstream.KeyValuePut {
return
}
@@ -52,6 +53,11 @@ func reAsked(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstre
announcedAgain(ctx, b, kv, e, r, logf)
return
}
if r.Mode == build.ModePublishAgain || r.Mode == build.ModeFloorAgain || r.Mode == build.ModeOldFormDrop {
// A version's hand act (design 51 *Tools of the builder*) may build again, so it is answered beside the watch.
go versionActed(ctx, acts, kv, e, r, logf)
return
}
asked := b.AskAgain(e.Key(), r.Why)
if len(asked) == 0 {
r.Answer = "no ask of this key waits in this builder now (it is not building, or its build has not reached its ask yet): nothing was asked; builder_stuck shows where it stands"
@@ -72,21 +78,7 @@ func reAsked(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstre
// emitAskedAgain says the builder's event build.asked-again (issue 1945): mesh.mod.<module>.event.build.asked-again,
// with the SDK's envelope headers and the new ask's id as the bus's dedup id.
func emitAskedAgain(ctx context.Context, js jetstream.JetStream, cred process.Account, e build.AskedAgain) error {
data, err := json.Marshal(e)
if err != nil {
return err
}
msg := nats.NewMsg(cred.EventSubject(build.AskedAgainEvent))
id := "asked-again-" + e.New
msg.Header.Set("x-event-id", id)
msg.Header.Set("x-source", cred.Module)
msg.Header.Set("x-node", cred.Node)
msg.Header.Set("x-time", time.Now().UTC().Format(time.RFC3339Nano))
msg.Header.Set("content-type", "application/json")
msg.Data = data
pub, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if _, err := js.PublishMsg(pub, msg, jetstream.WithMsgID(id)); err != nil {
if err := emitEvent(ctx, js, cred, build.AskedAgainEvent, "asked-again-"+e.New, e); err != nil {
return fmt.Errorf("saying %s for %s: %w", build.AskedAgainEvent, e.Key, err)
}
return nil
@@ -163,3 +155,33 @@ func appendLog(ctx context.Context, js jetstream.JetStream, cred process.Account
}
return nil
}
// versionActed answers a reasks value of builder_publish_again, builder_floor_again or builder_old_form's drop, a hand act said in the log.
func versionActed(ctx context.Context, acts versions.Acts, kv jetstream.KeyValue, e jetstream.KeyValueEntry, r build.ReAsk, logf func(string, ...any)) {
var err error
if r.Mode == build.ModePublishAgain {
var ag versions.Again
ag, r.Answer, err = acts.PublishAgain(ctx, r.Name, r.FP, r.Why, r.By, r.At)
r.Do, r.Version = ag.Do, ag.Entry.Version
} else if r.Mode == build.ModeOldFormDrop {
if acts.DropOldForm == nil {
err = errors.New("this builder keeps no old-form window")
} else {
r.Answer, err = acts.DropOldForm(ctx, r.Name, r.By, r.Why, r.At)
}
} else {
r.Answer, err = acts.FloorAgain(ctx, r.Name)
}
if err != nil {
r.Answer, r.Refused = err.Error(), true
}
r.Answered = time.Now().UTC()
logf("%s: %s of %s by %s (%s): %s", e.Key(), r.Mode, r.Name, r.By, r.Why, r.Answer)
data, err := json.Marshal(r)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the answer could not be written: %v", build.ReAsksState, e.Key(), err)
}
}
@@ -3,17 +3,31 @@ package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"log"
"maps"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-delivery/sdk/heard"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-delivery/sdk/enginereads"
"git.novox.be/novox/mesh-delivery/sdk/forms"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-builder/internal/build"
"mesh-delivery-builder/internal/versions"
)
// 1945 (a part exits at start for a bucket it works without): a bus that refuses every toolchains bucket leaves the
@@ -21,6 +35,9 @@ import (
// its grant. The gate over the seat's placement, and its refusal, are mesh-sdk's (buildrun.ToolchainGate,
// issue 774).
func TestARefusedBucketIsSaidAndThePartGoesOn(t *testing.T) {
t.Run("a floor's wake is heard on the events consumer (design 51 The floor)", aFloorsWakeIsHeard)
t.Run("a failed trunk build is a finding until a later done one (DLV-B7)", aFailedTrunkBuildIsAFinding)
t.Run("a branch's end is acknowledged once its tagged versions are removed (design 51 Tagged versions)", aBranchsEndIsHeard)
set := &buildrun.ToolchainSet{}
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
@@ -45,6 +62,100 @@ func TestARefusedBucketIsSaidAndThePartGoesOn(t *testing.T) {
if src.opened {
t.Error("a refused source counts as opened, so it is never opened again")
}
t.Run("ADR 0624: the counted nodes' engine reads", theCountedNodesEngineReads)
}
// theCountedNodesEngineReads: the builder asks the node-engine seat's verb of every node the placement names; a node away
// and never asked has no statement and reads as the baseline's first forms, with no node record read in its place; an
// away node keeps its last statement; a node the placement no longer names leaves the count, its key removed; and
// builder_engine_reads drops one node's key.
func theCountedNodesEngineReads(t *testing.T) {
gate := buildrun.NewToolchainGate(nil, func(string, string, []byte, bool) bool { return false }, t.Logf)
place := func(nodes ...string) {
for _, node := range []string{"n1", "n2"} {
pl := delivery.Placement{Node: node}
if slices.Contains(nodes, node) {
pl.Modules = []delivery.PlacedModule{{Module: "web"}}
}
values, err := delivery.PlacementValues(pl)
if err != nil {
t.Fatal(err)
}
for key, raw := range values {
gate.Placement(key, raw, false, time.Now())
}
if !slices.Contains(nodes, node) {
key, _ := delivery.PlacementModuleKey(node, "web")
gate.Placement(key, nil, true, time.Now())
}
}
}
away := map[string]bool{"n2": true}
asked := map[string]int{}
r := newEngineReads(func(node, key string, _ any) (json.RawMessage, error) {
asked[node]++
if away[node] {
return nil, nats.ErrNoResponders
}
return json.Marshal(nodeengine.ManifestReadsAnswer{V: 1, Node: node, ManifestReads: []string{"form:new"}, StatedAt: time.Now().Format(time.RFC3339Nano)})
}, gate, t.Logf)
r.keeper.Sync = true
kept := map[string][]byte{}
r.keeper.Put = func(node string, v []byte) error { kept[node] = v; return nil }
r.keeper.Delete = func(node string) error { delete(kept, node); return nil }
place("n1", "n2")
r.placed()
present, counted := r.forms()
if !slices.Equal(present["n1"], []string{"form:new"}) || len(present) != 1 || len(counted) != 2 || counted["n2"] != nil {
t.Errorf("away and never asked: present %v, counted %v", present, counted)
}
if s := r.keeper.Statements()["n2"]; s.From != enginereads.FromNone || s.Answered || asked["n2"] != 1 {
t.Errorf("n2, away and never asked, is %+v (asked %d)", s, asked["n2"])
}
// n2 back, stating {word:module}, then away again: it keeps that statement.
away["n2"] = false
r.keeper.Heard("n2", mustStatement(t, enginereads.Statement{Node: "n2", From: enginereads.FromVerb, Answered: true,
AskAgain: &enginereads.Again{By: "n1.ada", Why: "back"}}), false)
away["n2"] = true
r.keeper.AskNode("n2", "its event")
if _, counted = r.forms(); !slices.Equal(counted["n2"], []string{"form:new"}) || r.keeper.Statements()["n2"].Answered {
t.Errorf("away again, n2 reads %v (%+v)", counted["n2"], r.keeper.Statements()["n2"])
}
// n2 retired: it leaves the placement, and its key goes.
place("n1")
r.placed()
if _, counted = r.forms(); len(counted) != 1 || kept["n2"] != nil {
t.Errorf("retired n2: counted %v, its key %s", counted, kept["n2"])
}
// builder_engine_reads drops one key.
st := &memReads{v: map[string][]byte{"n1": kept["n1"]}}
tool := enginereads.Tool("builder_engine_reads", "the builder", st, true, time.Now)
if _, err := tool.Run(map[string]any{"node": "n1", "drop": "true", "why": "a test"}); err != nil || st.v["n1"] != nil {
t.Errorf("drop: %v, the key %s", err, st.v["n1"])
}
}
func mustStatement(t *testing.T, s enginereads.Statement) []byte {
t.Helper()
raw, err := enginereads.Encode(s)
if err != nil {
t.Fatal(err)
}
return raw
}
// memReads is the builder's state of engine reads in memory, as the tool reads it.
type memReads struct{ v map[string][]byte }
func (m *memReads) Keys() ([]string, error) { return slices.Collect(maps.Keys(m.v)), nil }
func (m *memReads) Get(key string) ([]byte, uint64, bool, error) {
v, ok := m.v[key]
return v, 1, ok, nil
}
func (m *memReads) Put(key string, value []byte) (uint64, error) { m.v[key] = value; return 2, nil }
func (m *memReads) Delete(key string) error { delete(m.v, key); return nil }
func (m *memReads) Await(string, uint64, func([]byte) bool, time.Duration) ([]byte, error) {
return nil, errors.New("no part answers here")
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
@@ -71,3 +182,347 @@ func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstrea
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
// wakeMsg is a message of the builder's events consumer: its subject and body, acknowledged.
type wakeMsg struct {
jetstream.Msg
subject string
data []byte
acked bool
naked bool
delayed time.Duration
}
func (m *wakeMsg) Subject() string { return m.subject }
func (m *wakeMsg) Data() []byte { return m.data }
func (m *wakeMsg) Ack() error { m.acked = true; return nil }
func (m *wakeMsg) Nak() error { m.naked = true; return nil }
func (m *wakeMsg) NakWithDelay(d time.Duration) error {
m.delayed = d
return nil
}
func (m *wakeMsg) Metadata() (*jetstream.MsgMetadata, error) {
return nil, errors.New("no metadata")
}
func (m *wakeMsg) Headers() nats.Header {
return nil
}
func aFloorsWakeIsHeard(t *testing.T) {
var logs bytes.Buffer
waits := &versions.FloorWaits{}
waits.Wait("m/lib", versions.SeatGoPackages, "", versions.Floor{Silent: []string{versions.SourceRequirements}}, nil, time.Unix(0, 0))
waits.Wait("m/img", versions.SeatImages, "", versions.Floor{Silent: []string{versions.SourceTags}}, nil, time.Unix(0, 0))
f := &floorWake{b: &build.Builder{}, waits: waits, logf: log.New(&logs, "", 0).Printf}
accepted := &wakeMsg{subject: manifestAcceptedSubject}
if !f.heard(context.Background(), accepted) || !accepted.acked || !strings.Contains(logs.String(), "the floor of m/lib is asked again: manifest-accepted") {
t.Fatalf("manifest-accepted: acked %v, log %q", accepted.acked, logs.String())
}
logs.Reset()
cleared := &wakeMsg{subject: conditionClearedSubject, data: []byte(`{"subject":{"scope":"probe","id":"ace/mesh-delivery-forge-bridge"}}`)}
if !f.heard(context.Background(), cleared) || !strings.Contains(logs.String(), "the floor of m/img is asked again: condition-cleared") {
t.Fatalf("condition-cleared: log %q", logs.String())
}
if f.heard(context.Background(), &wakeMsg{subject: delivery.ArtifactsGoneSubject}) {
t.Fatal("artifacts-gone was taken as a floor's wake")
}
// DLV-B4: a registry seat's outcome on the same consumer moves a version; a published wakes a name whose seat was
// silent; one the state cannot take is left for the bus to give again.
logs.Reset()
waits.Wait("m/pkg", versions.SeatGoPackages, "", versions.Floor{Silent: []string{versions.SourceSeat}}, nil, time.Unix(0, 0))
h := &seatHearer{n: &versions.Numberer{Versions: versions.Versions{Store: noVersions{}}, Waits: waits}, b: &build.Builder{}, logf: log.New(&logs, "", 0).Printf}
byDigest := &wakeMsg{subject: "mesh.seat.go-package-registry.event.published", data: []byte(`{"name":"m/other","digest":"sha256:aa","kind":"package","holder":"gitea"}`)}
if !h.heard(context.Background(), byDigest) || !byDigest.acked || !strings.Contains(logs.String(), "the floor of m/pkg is asked again: go-package-registry said published") {
t.Fatalf("published: acked %v, log %q", byDigest.acked, logs.String())
}
refused := &wakeMsg{subject: "mesh.seat.oci-artifact-registry.event.refused", data: []byte(`{"name":"m/program","version":"0.1.3","digest":"sha256:bb","reason":"bytes differ"}`)}
if !h.heard(context.Background(), refused) || refused.acked || !refused.naked {
t.Fatalf("a refusal the state cannot take: acked %v, naked %v, log %q", refused.acked, refused.naked, logs.String())
}
if h.heard(context.Background(), &wakeMsg{subject: manifestAcceptedSubject}) {
t.Fatal("manifest-accepted was taken as a seat's outcome")
}
}
// noVersions is a state versions that holds nothing.
type noVersions struct{}
func (noVersions) Get(context.Context, string) ([]byte, uint64, bool, error) {
return nil, 0, false, nil
}
func (noVersions) Create(context.Context, string, []byte) (uint64, error) {
return 0, errors.New("read only")
}
func (noVersions) Update(context.Context, string, []byte, uint64) (uint64, error) {
return 0, errors.New("read only")
}
func (noVersions) KeysOf(context.Context, string) ([]string, error) { return nil, nil }
func aBranchsEndIsHeard(t *testing.T) {
for _, c := range []struct {
f forgeEnd
want bool
}{
{forgeEnd{Kind: delivery.ForgeMerged, Repository: "o/r", Branch: "feat/x"}, true},
{forgeEnd{Kind: forgeBranchDeleted, Repository: "o/r", Branch: "feat/x"}, true},
{forgeEnd{Kind: delivery.ForgePull, Repository: "o/r", Branch: "feat/x", Action: delivery.PullClosed}, true},
{forgeEnd{Kind: delivery.ForgePull, Repository: "o/r", Branch: "feat/x", Action: delivery.PullOpened}, false},
{forgeEnd{Kind: delivery.ForgeMerged, Repository: "o/r"}, false},
} {
if got := c.f.ended(); got != c.want {
t.Errorf("%+v ended = %v, want %v", c.f, got, c.want)
}
}
var logs bytes.Buffer
woken := 0
h := &branchEnds{acts: versions.Acts{Numberer: &versions.Numberer{Versions: versions.Versions{Store: noVersions{}}}}, findings: &findings{logf: t.Logf},
tagAsks: func() jetstream.KeyValue { return nil }, wake: func(context.Context) { woken++ }, logf: log.New(&logs, "", 0).Printf, now: time.Now}
open := &wakeMsg{subject: "mesh.mod.mesh-delivery-forge-bridge.event.forge.o.r.pull", data: []byte(`{"kind":"pull","repository":"o/r","branch":"feat/x","action":"opened"}`)}
h.heard(context.Background(), open)
merged := &wakeMsg{subject: "mesh.mod.mesh-delivery-forge-bridge.event.forge.o.r.merged", data: []byte(`{"kind":"merged","repository":"o/r","branch":"feat/x"}`)}
h.heard(context.Background(), merged)
if !open.acked || !merged.acked || woken != 2 {
t.Errorf("an open pull acked %v, a merge with nothing tagged acked %v, floors woken %d times", open.acked, merged.acked, woken)
}
// The checker's family holds its verdicts beside tag-wanted: a verdict is acknowledged and not kept; a tag-wanted with
// tag-asks not bound is heard again.
verdict := &wakeMsg{subject: "mesh.stream.mesh-delivery-checker.event.verdict", data: []byte(`{"repository":"o/r"}`)}
wanted := &wakeMsg{subject: tagWantedSubject, data: []byte(`{"repository":"o/r","branch":"feat/x","head":"abc","tag":"feat-x"}`)}
for _, m := range []*wakeMsg{verdict, wanted} {
tagWanted(context.Background(), nil, m, delivery.CheckerModule, time.Now(), t.Logf)
}
if !verdict.acked || verdict.delayed != 0 || wanted.acked || wanted.delayed != versions.RemoveAgainAfter || !strings.HasPrefix(tagWantedSubject, "mesh.stream.") {
t.Errorf("a verdict acked %v (delayed %v), a tag-wanted acked %v (delayed %v)", verdict.acked, verdict.delayed, wanted.acked, wanted.delayed)
}
unreadable := &wakeMsg{subject: merged.subject, data: merged.data}
h.acts.Numberer.Versions.Store = failingVersions{}
h.heard(context.Background(), unreadable)
if unreadable.acked || unreadable.delayed != versions.RemoveAgainAfter {
t.Errorf("a branch's end the versions cannot answer: acked %v, given again after %v", unreadable.acked, unreadable.delayed)
}
}
// failingVersions is a state versions that cannot be read.
type failingVersions struct{ noVersions }
func (failingVersions) KeysOf(context.Context, string) ([]string, error) {
return nil, errors.New("down")
}
func aFailedTrunkBuildIsAFinding(t *testing.T) {
st := &memFindings{vals: map[string][]byte{}}
f := &findings{store: st, logf: t.Logf, now: func() time.Time { return time.Unix(0, 0) }}
ended := func(outcome, ref string) delivery.BuildLogEntry {
return delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: "m", FP: "f1", Commit: "c", Ref: ref, Outcome: outcome}
}
key := health.FindingKey(health.ScopeModule, "m", BuildFailed)
f.buildEnded(context.Background(), ended(delivery.BuildFailed, "refs/pull/3/head"))
if len(st.vals) != 0 {
t.Fatalf("a candidate's failed build is a finding: %v", st.vals)
}
f.buildEnded(context.Background(), ended(delivery.BuildFailed, ""))
got, err := health.DecodeFinding(st.vals[key])
if err != nil || got.Observer != delivery.BuilderModule || !strings.Contains(got.Summary, "trunk build of m at f1") {
t.Fatalf("the finding = %+v, %v", got, err)
}
f.buildEnded(context.Background(), ended(delivery.BuildDone, ""))
if _, ok := st.vals[key]; ok {
t.Error("a later done trunk build leaves build-failed")
}
// The floor finding: raised while a name waits on its floor, cleared once it is known.
fw := &versions.FloorWaits{Said: f.floorSaid(context.Background())}
fw.Wait("m/lib", versions.SeatGoPackages, "", versions.Floor{Silent: []string{versions.SourceTags}}, nil, time.Unix(0, 0))
floorKey := health.FindingKey(health.ScopeModule, "m/lib", versions.FloorSilent)
if got, err := health.DecodeFinding(st.vals[floorKey]); err != nil || !strings.Contains(got.Summary, "bridge_tags did not answer") {
t.Fatalf("the floor finding = %+v, %v", got, err)
}
fw.Answered("m/lib")
if _, ok := st.vals[floorKey]; ok {
t.Error("a known floor leaves floor-silent")
}
// Design 51 *Health and conditions* (review round 1): a holder's refused is publish-refused with its reason, a version
// no holder answered within its bound publish-unanswered, a seat's outcome that does not read seat-outcome-unread;
// a later published of the name clears the first two, the seat's next outcome that reads the third.
vs := versions.Versions{Store: memVersions{}}
for _, v := range []string{"0.1.3", "0.1.4"} {
raw, err := versions.Encode(versions.Entry{Name: "m/program", Kind: versions.KindBundle, Series: "0.1", Version: v, Seat: "oci-artifact-registry", State: versions.Numbered})
if err != nil {
t.Fatal(err)
}
vs.Store.(memVersions)[versions.Key("m/program", v)] = raw
}
pubs := &publishes{bounds: func() jetstream.KeyValue { return nil }, versions: vs, findings: f, bound: versions.PublishBound, logf: t.Logf}
h := &seatHearer{n: &versions.Numberer{Versions: vs}, b: &build.Builder{}, logf: t.Logf, pub: pubs}
refusedKey := health.FindingKey(health.ScopeModule, "m/program", versions.PublishRefused)
refused := &wakeMsg{subject: "mesh.seat.oci-artifact-registry.event.refused", data: []byte(`{"name":"m/program","version":"0.1.3","digest":"sha256:bb","reason":"bytes differ"}`)}
if !h.heard(context.Background(), refused) || !refused.acked {
t.Fatalf("a refusal: acked %v", refused.acked)
}
if got, err := health.DecodeFinding(st.vals[refusedKey]); err != nil || !strings.Contains(got.Summary, "refused to publish m/program@0.1.3") || !strings.Contains(got.Summary, "bytes differ") {
t.Fatalf("a holder's refused: the finding = %+v, %v", got, err)
}
pubs.ended(context.Background(), purged{key: versions.PublishBoundKey("m/program", "0.1.4")})
unansweredKey := health.FindingKey(health.ScopeModule, "m/program", versions.PublishUnanswered)
if got, err := health.DecodeFinding(st.vals[unansweredKey]); err != nil || !strings.Contains(got.Summary, "neither published nor refused of m/program@0.1.4") {
t.Fatalf("a bound ended unanswered: the finding = %+v, %v", got, err)
}
// Review round 2: a bound that cannot be written is publish-unanswered at once; a watch of bounds that cannot start
// is family-not-heard naming the state.
delete(st.vals, unansweredKey)
pubs.arm(context.Background(), versions.Entry{Name: "m/program", Version: "0.1.4", Seat: "oci-artifact-registry"})
if got, err := health.DecodeFinding(st.vals[unansweredKey]); err != nil || !strings.Contains(got.Summary, "its bound could not be written") {
t.Fatalf("a bound not written: the finding = %+v, %v", got, err)
}
watchBounds(context.Background(), refusedWatch{err: errors.New("permissions violation")}, func(jetstream.KeyValueEntry) {}, f.notHeard(context.Background()), nil, t.Logf)
if got, err := health.DecodeFinding(st.vals[heard.Key(delivery.BuilderModule, versions.BoundsState)]); err != nil || !strings.Contains(got.Summary, "permissions violation") {
t.Fatalf("bounds not watched: the finding = %+v, %v", got, err)
}
delete(st.vals, heard.Key(delivery.BuilderModule, versions.BoundsState))
malformed := &wakeMsg{subject: "mesh.seat.oci-artifact-registry.event.published", data: []byte(`{"version":"0.1.4"}`)}
unreadKey := health.FindingKey(health.ScopeModule, "oci-artifact-registry", versions.SeatOutcomeUnread)
if !h.heard(context.Background(), malformed) || !malformed.acked || st.vals[unreadKey] == nil {
t.Fatalf("an outcome that does not read: acked %v, findings %v", malformed.acked, st.vals)
}
published := &wakeMsg{subject: "mesh.seat.oci-artifact-registry.event.published", data: []byte(`{"name":"m/program","version":"0.1.4","digest":"sha256:cc"}`)}
if !h.heard(context.Background(), published) || st.vals[refusedKey] != nil || st.vals[unansweredKey] != nil || st.vals[unreadKey] != nil {
t.Fatalf("a later published leaves the findings: %v", st.vals)
}
// ADR 0615 §5 (review round 1): a consumer not there yet is family-not-heard naming its stream, cleared when it binds.
notHeard := f.notHeard(context.Background())
unheardKey := heard.Key(delivery.BuilderModule, "FORGE")
_ = notHeard.Bind("FORGE", "a branch's end is not heard", func() error { return errors.New("consumer not found") })
if got, err := health.DecodeFinding(st.vals[unheardKey]); err != nil || got.Kind != heard.Kind || !strings.Contains(got.Summary, "does not hear FORGE") {
t.Fatalf("a consumer not there: the finding = %+v, %v", got, err)
}
_ = notHeard.Bind("FORGE", "a branch's end is not heard", func() error { return nil })
if _, ok := st.vals[unheardKey]; ok {
t.Error("a consumer bound leaves family-not-heard")
}
// DLV-B9, the old form's window: one counted node reading only the first form holds it; seven days on it is
// old-form-held about that node; the node stating the newest form closes it, said once, the finding cleared; a
// retired node dropped by hand leaves the count.
now := time.Unix(0, 0)
fs := forms.ArtifactForms{{Name: "registry", First: true, Kinds: []string{"image"}}, {Name: "oci", Kinds: []string{"image"}}}
var closings []versions.OldFormClosedBody
var logged []delivery.BuildLogEntry
marks := &memBounds{keys: map[string]bool{}}
window := &oldForm{forms: fs, findings: f, logf: t.Logf, now: func() time.Time { return now }, bounds: func() jetstream.KeyValue { return marks },
said: func(b versions.OldFormClosedBody) { closings = append(closings, b) },
log: func(e delivery.BuildLogEntry) { logged = append(logged, e) }}
reads := &engineReads{keeper: &enginereads.Keeper{}, window: window, counted: []string{"a", "b"}}
newest := []string{forms.FormNeed("oci")}
counted := map[string][]string{"a": newest, "b": nil}
window.step(context.Background(), []string{"a", "b"}, counted)
heldKey := health.FindingKey(health.ScopeMachine, "b", versions.OldFormHeld)
if !window.w.Open || !slices.Equal(window.w.Holding, []string{"b"}) || st.vals[heldKey] != nil {
t.Fatalf("one node lacking the newest form: %+v", window.w)
}
// Review round 1: the node holding it has a bound in the state bounds ending at its seven days, so old-form-held is
// raised then, not at the next change.
if !marks.keys[versions.OldFormBoundKey("b")] || marks.keys[versions.OldFormBoundKey("a")] {
t.Fatalf("the old form's bounds = %v, want b's alone", marks.keys)
}
// Security-upgrade F3 (review round 1): an empty node_haves read, naming no node, leaves the window open.
window.step(context.Background(), nil, map[string][]string{})
if !window.w.Open || !slices.Equal(window.w.Holding, []string{"b"}) || len(closings) != 0 {
t.Fatalf("an empty read closed the window: %+v, closings %+v", window.w, closings)
}
now = now.Add(versions.FindingAfter)
window.step(context.Background(), []string{"a", "b"}, counted)
if got, err := health.DecodeFinding(st.vals[heldKey]); err != nil || !strings.Contains(got.Summary, "the node-engine of b") {
t.Fatalf("seven days held: the finding = %+v, %v", got, err)
}
counted["b"] = newest
window.step(context.Background(), []string{"a", "b"}, counted)
window.step(context.Background(), []string{"a", "b"}, counted)
if marks.keys[versions.OldFormBoundKey("b")] {
t.Error("a node no longer holding the window keeps its bound")
}
if window.w.Open || len(closings) != 1 || !slices.Equal(closings[0].LastHolding, []string{"b"}) || st.vals[heldKey] != nil {
t.Errorf("the node back with the newest form: %+v, closings %+v, finding left %v", window.w, closings, st.vals[heldKey] != nil)
}
if len(logged) != 1 || logged[0].Event != delivery.BuildSaidEvent || logged[0].Why != versions.SaidOldFormClosed ||
!strings.Contains(logged[0].Line, "last held by b") {
t.Errorf("the build log = %+v, want the close said once (build.said)", logged)
} else if _, err := delivery.EncodeBuildLogEntry(logged[0]); err != nil {
t.Errorf("the close is no entry the build log takes: %v", err)
}
if _, err := window.drop(context.Background(), "b", "op@shanks", "retired", now); err != nil {
t.Fatal(err)
}
if _, c := reads.forms(); len(c) != 1 || c["b"] != nil {
t.Errorf("a node dropped by hand is still counted: %v", c)
}
}
// memVersions is a state versions in memory.
type memVersions map[string][]byte
func (m memVersions) Get(_ context.Context, key string) ([]byte, uint64, bool, error) {
v, ok := m[key]
return v, 1, ok, nil
}
func (m memVersions) Create(_ context.Context, key string, v []byte) (uint64, error) {
m[key] = v
return 1, nil
}
func (m memVersions) Update(_ context.Context, key string, v []byte, _ uint64) (uint64, error) {
m[key] = v
return 2, nil
}
func (m memVersions) KeysOf(context.Context, string) ([]string, error) {
return slices.Collect(maps.Keys(m)), nil
}
// memBounds is the state bounds in memory: which keys stand.
type memBounds struct {
jetstream.KeyValue
keys map[string]bool
}
func (m *memBounds) Create(_ context.Context, key string, _ []byte, _ ...jetstream.KVCreateOpt) (uint64, error) {
if m.keys[key] {
return 0, jetstream.ErrKeyExists
}
m.keys[key] = true
return 1, nil
}
func (m *memBounds) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
delete(m.keys, key)
return nil
}
// purged is the bus's marker of a key whose time to live ended.
type purged struct {
jetstream.KeyValueEntry
key string
}
func (p purged) Key() string { return p.key }
func (p purged) Operation() jetstream.KeyValueOp { return jetstream.KeyValuePurge }
// memFindings is a findings bucket in memory.
type memFindings struct{ vals map[string][]byte }
type memEntry struct {
jetstream.KeyValueEntry
v []byte
}
func (e memEntry) Value() []byte { return e.v }
func (m *memFindings) Get(_ context.Context, key string) (jetstream.KeyValueEntry, error) {
v, ok := m.vals[key]
if !ok {
return nil, jetstream.ErrKeyNotFound
}
return memEntry{v: v}, nil
}
func (m *memFindings) Put(_ context.Context, key string, v []byte) (uint64, error) {
m.vals[key] = v
return 1, nil
}
func (m *memFindings) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
delete(m.vals, key)
return nil
}
@@ -0,0 +1,148 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-contracts/go/seats/containerimageregistry"
"git.novox.be/novox/mesh-contracts/go/seats/gopackageregistry"
"git.novox.be/novox/mesh-contracts/go/seats/npmpackageregistry"
"git.novox.be/novox/mesh-contracts/go/seats/ociartifactregistry"
"mesh-delivery-builder/internal/build"
"mesh-delivery-builder/internal/versions"
)
// seatEvent is one registry seat's event the builder hears: the seat and the event's name.
type seatEvent struct{ seat, event string }
// seatEvents are the four registry seats' published, refused, removed and kept, by subject (design 51 *How a publish ends*,
// DLV-B4): each a seat event `<seat>.<event>` on the shared events stream, heard on the builder's durable consumer of
// it through the manifest's consumes. The subjects are the seats' contracts' (mesh-contracts).
var seatEvents = map[string]seatEvent{
containerimageregistry.SubjectPublished: {containerimageregistry.Seat, containerimageregistry.EventPublished},
containerimageregistry.SubjectRefused: {containerimageregistry.Seat, containerimageregistry.EventRefused},
containerimageregistry.SubjectRemoved: {containerimageregistry.Seat, containerimageregistry.EventRemoved},
containerimageregistry.SubjectKept: {containerimageregistry.Seat, containerimageregistry.EventKept},
ociartifactregistry.SubjectPublished: {ociartifactregistry.Seat, ociartifactregistry.EventPublished},
ociartifactregistry.SubjectRefused: {ociartifactregistry.Seat, ociartifactregistry.EventRefused},
ociartifactregistry.SubjectRemoved: {ociartifactregistry.Seat, ociartifactregistry.EventRemoved},
ociartifactregistry.SubjectKept: {ociartifactregistry.Seat, ociartifactregistry.EventKept},
gopackageregistry.SubjectPublished: {gopackageregistry.Seat, gopackageregistry.EventPublished},
gopackageregistry.SubjectRefused: {gopackageregistry.Seat, gopackageregistry.EventRefused},
gopackageregistry.SubjectRemoved: {gopackageregistry.Seat, gopackageregistry.EventRemoved},
gopackageregistry.SubjectKept: {gopackageregistry.Seat, gopackageregistry.EventKept},
npmpackageregistry.SubjectPublished: {npmpackageregistry.Seat, npmpackageregistry.EventPublished},
npmpackageregistry.SubjectRefused: {npmpackageregistry.Seat, npmpackageregistry.EventRefused},
npmpackageregistry.SubjectRemoved: {npmpackageregistry.Seat, npmpackageregistry.EventRemoved},
npmpackageregistry.SubjectKept: {npmpackageregistry.Seat, npmpackageregistry.EventKept},
}
// seatHearer takes the registry seats' outcomes in: each moves a numbered version (versions.Numberer.Heard), and a
// `published` that wakes a name waiting on its floor looks at every module again.
type seatHearer struct {
n *versions.Numberer
b *build.Builder
logf func(string, ...any)
// pub raises and clears the findings of a publish that ended badly; nil raises none.
pub *publishes
}
// heard takes a message of the events consumer in when it is a registry seat's outcome, and answers whether it was.
// One the state could not take is left unacknowledged (Nak), so the bus gives it again; one that is no outcome is said
// and dropped.
func (h *seatHearer) heard(ctx context.Context, m jetstream.Msg) bool {
se, ok := seatEvents[m.Subject()]
if !ok {
return false
}
var o versions.Outcome
if err := json.Unmarshal(m.Data(), &o); err != nil || o.Name == "" {
if err == nil {
err = errors.New("it names no name")
}
h.logf("%s holds what is no outcome of %s, left: %v", m.Subject(), se.seat, err)
if h.pub != nil {
h.pub.unread(ctx, se.seat, m.Subject(), err)
}
h.ack(m)
return true
}
if h.pub != nil {
h.pub.read(ctx, se.seat)
}
id := ""
if m.Headers() != nil {
id = m.Headers().Get("x-event-id")
}
wake, err := h.n.Heard(ctx, se.seat, se.event, id, o)
if err != nil {
h.logf("%v; heard again", err)
if nakErr := m.Nak(); nakErr != nil {
h.logf("leaving %s for the bus to give again: %v", m.Subject(), nakErr)
}
return true
}
if h.pub != nil {
h.pub.heard(ctx, se.seat, se.event, o)
}
h.ack(m)
if len(wake) > 0 {
h.logf("the floor of %s is asked again: %s said %s", strings.Join(wake, ", "), se.seat, se.event)
h.b.Reconsider(ctx)
}
return true
}
func (h *seatHearer) ack(m jetstream.Msg) {
if err := m.Ack(); err != nil {
h.logf("acknowledging %s: %v", m.Subject(), err)
}
}
// kvKeeper is the state floor-waits as versions.FloorKeeper.
type kvKeeper struct {
ctx context.Context
kv jetstream.KeyValue
}
func (k kvKeeper) Put(key string, value []byte) error {
_, err := k.kv.Put(k.ctx, key, value)
return err
}
func (k kvKeeper) Delete(key string) error {
err := k.kv.Delete(k.ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil
}
return err
}
// heldFloorWaits are the waits floor-waits holds at the builder's start; a value that is no wait is said and left.
func heldFloorWaits(ctx context.Context, kv jetstream.KeyValue, logf func(string, ...any)) []versions.FloorWait {
l, err := kv.ListKeys(ctx)
if err != nil {
logf("%s cannot be listed, so a restart's waits are asked at their next claim: %v", versions.FloorWaitsState, err)
return nil
}
defer func() { _ = l.Stop() }()
var out []versions.FloorWait
for k := range l.Keys() {
e, err := kv.Get(ctx, k)
if err != nil {
continue
}
w, err := versions.DecodeFloorWait(e.Value())
if err != nil {
logf("%s.%s holds what is no floor wait, left: %v", versions.FloorWaitsState, k, err)
continue
}
out = append(out, w)
}
return out
}
@@ -15,8 +15,8 @@ import (
"mesh-delivery-builder/internal/build"
)
// eventsStream is the stream every module's and role's events are on, where the controller makes the builder's durable
// consumer of artifacts-gone (mesh-controller internal/broker EventsStream).
// eventsStream is the stream every module's and role's events are on, where the grants part makes the builder's durable
// consumer of artifacts-gone (mesh-sdk buscompose EventsStream).
const eventsStream = "EVENTS"
// SeatHeld is the seat mesh-artifact-sweep's verb held (ADR 0571 §2).
@@ -27,8 +27,7 @@ type toolCaller func(key string, body any) (json.RawMessage, error)
// storedVia is build.Builder's Stored: it asks the seat mesh-artifact-sweep's `held` (novox/hq ADR 0571 §2, issue 1937)
// and answers the references the store no longer holds. One it could not ask about, with none gone, is an error: the key
// is then left done, and the controller's take-in still refuses a gone blob. The controller's `held` is not asked: it
// is no longer a fallback (issue 1955).
// is then left done, and asked again at the next module value or artifacts-gone.
func storedVia(call toolCaller) func(ctx context.Context, references []string) ([]string, error) {
return func(_ context.Context, references []string) ([]string, error) {
body := map[string]any{"references": references}
@@ -0,0 +1,132 @@
package main
import (
"context"
"encoding/json"
"errors"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/versions"
)
// tagBounds narrates a tag ask the builder has not answered within five minutes, with what it waits on (design 51
// *Tagged versions*): each unanswered ask has a key in the state tag-ask-bounds written with its own time to live, and
// the bus's marker of its end, heard on the watch, is what narrates it and writes the bound again. No timer.
type tagBounds struct {
asks func() jetstream.KeyValue
bounds func() jetstream.KeyValue
waits func() []versions.FloorWait
logf func(string, ...any)
// log writes a narration into the builder's build log (build.said); nil writes none.
log func(delivery.BuildLogEntry)
now func() time.Time
bound time.Duration
}
// arm writes key's bound when none stands: a Create, so a standing bound keeps its time and a restart arms each
// unanswered ask whose bound ended while the builder was away.
func (tb *tagBounds) arm(ctx context.Context, key string) {
kv := tb.bounds()
if kv == nil {
return
}
if _, err := kv.Create(ctx, key, []byte(`{}`), jetstream.KeyTTL(tb.bound)); err != nil && !errors.Is(err, jetstream.ErrKeyExists) {
tb.logf("the bound of the tag ask %s is not written, so it is not narrated past five minutes: %v", key, err)
}
}
// disarm removes key's bound: the ask was answered or ended.
func (tb *tagBounds) disarm(ctx context.Context, key string) {
if kv := tb.bounds(); kv != nil {
_ = kv.Delete(ctx, key)
}
}
// asked is told each value of tag-asks: an unanswered ask is armed, any other disarmed.
func (tb *tagBounds) asked(ctx context.Context, key string, t versions.TagAsk) {
if t.Unanswered() {
tb.arm(ctx, key)
return
}
tb.disarm(ctx, key)
}
// ended is told each change of tag-ask-bounds: a bound whose time ended (the bus's marker, a purge on the watch) on an
// ask still unanswered narrates it and arms it again.
func (tb *tagBounds) ended(ctx context.Context, e jetstream.KeyValueEntry) {
if e.Operation() != jetstream.KeyValuePurge {
return
}
kv := tb.asks()
if kv == nil {
return
}
got, err := kv.Get(ctx, e.Key())
if err != nil {
return // the ask ended with its branch
}
t, err := versions.DecodeTagAsk(got.Value())
if err != nil || !t.Unanswered() {
return
}
var waits []versions.FloorWait
if tb.waits != nil {
waits = tb.waits()
}
line := versions.NarrateUnanswered(t, waits, tb.now())
tb.logf("%s", line)
// In the build log too (design 51 *Tagged versions*): under the builder's own key, no build being claimed yet, the
// ask's key standing for the fingerprint.
if tb.log != nil {
tb.log(delivery.BuildLogEntry{Event: delivery.BuildSaidEvent, Module: delivery.BuilderModule, FP: e.Key(), Commit: t.Head,
At: tb.now().UTC(), Line: line, Why: versions.SaidUnanswered})
}
tb.arm(ctx, e.Key())
}
// tagNumbered marks the tag ask of a check's build's head numbered with its versions in tag-asks, by compare-and-set.
func tagNumbered(kv func() jetstream.KeyValue, logf func(string, ...any), now func() time.Time) func(context.Context, versions.BuildClaim, versions.Numbering) {
return func(ctx context.Context, c versions.BuildClaim, got versions.Numbering) {
store := kv()
if store == nil {
return
}
lister, err := store.ListKeysFiltered(ctx, versions.EscapeToken(c.Repository)+".>")
if err != nil {
logf("the tag ask of %s at %s is not marked numbered: %v", c.Repository, c.Head, err)
return
}
for key := range lister.Keys() {
for range 3 {
e, err := store.Get(ctx, key)
if err != nil {
break
}
t, err := versions.DecodeTagAsk(e.Value())
if err != nil {
break
}
t, changed := t.WithNumbered(c.Module, c.Head, got.Versions, got.Names, now())
if !changed {
break
}
data, err := json.Marshal(t)
if err != nil {
break
}
if _, err = store.Update(ctx, key, data, e.Revision()); err == nil {
logf("%s: tag ask of %s %s: %s", key, t.Repository, t.Branch, t.Answer)
break
}
if !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("the tag ask %s is not marked numbered: %v", key, err)
break
}
}
}
}
}
@@ -8,7 +8,7 @@ import (
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
@@ -0,0 +1,352 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-contracts/go/seats/ociartifactregistry"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-builder/internal/build"
"mesh-delivery-builder/internal/versions"
)
// The floor's two module tools (novox/hq design 51 *The floor*); the seat's is `seat:<registry>.versions`.
const (
bridgeTagsTool = "mesh-delivery-forge-bridge.bridge_tags"
requirementsTool = "mesh-delivery-indexer.indexer_requirements"
)
// awaitedVersions is the state versions as versions.Store: bound once the bus holds it (the grants part makes it, issue
// 1790); until then every use is an error, so a trunk build is built unnumbered and said (build.Builder.number).
type awaitedVersions struct{ b *process.AwaitedBucket }
var errVersionsMissing = errors.New("the state " + versions.Bucket + " is not on the bus yet")
func (s awaitedVersions) kv() (jetstream.KeyValue, error) {
if kv := s.b.KV(); kv != nil {
return kv, nil
}
return nil, errVersionsMissing
}
func (s awaitedVersions) Get(ctx context.Context, key string) ([]byte, uint64, bool, error) {
kv, err := s.kv()
if err != nil {
return nil, 0, false, err
}
e, err := kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, 0, false, nil
}
if err != nil {
return nil, 0, false, err
}
return e.Value(), e.Revision(), true, nil
}
// indexerPackages is the indexer's packages state as the builder reads a package's name in its ecosystem from it (design
// 32): the picture of its read tool's first view and its views family (ADR 0615 §1), never its bucket. Until the first
// view is answered a package is built unnumbered and said (versions.Numberer.Names), and numbered once its name is heard
// (followPackages).
type indexerPackages struct{ kv jetstream.KeyValue }
func (s indexerPackages) Keys(ctx context.Context) ([]string, error) {
keys, err := s.kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
func (s indexerPackages) Get(ctx context.Context, key string) ([]byte, bool, error) {
e, err := s.kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) || errors.Is(err, jetstream.ErrKeyDeleted) {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
return e.Value(), true, nil
}
// followPackages watches the picture of the indexer's packages state: each series key heard that names a package's name
// (versions.NamedModule), its first view's included, looks at its module again (build.Builder.PackageNamed), so a done
// build that left the package unnumbered is numbered then with its digests. Woken by the views family, never a timer; a
// key heard again naming the name and module it named (the indexer writes a series at each publish) looks at nothing.
func followPackages(ctx context.Context, kv jetstream.KeyValue, b *build.Builder, logf func(string, ...any)) error {
w, err := kv.WatchAll(ctx)
if err != nil {
return err
}
heard := map[string]string{}
return follow(ctx, w, func(e jetstream.KeyValueEntry) {
if e.Operation() != jetstream.KeyValuePut {
delete(heard, e.Key())
return
}
module, name, named := versions.NamedModule(e.Key(), e.Value())
if named && heard[e.Key()] != module+" "+name {
heard[e.Key()] = module + " " + name
logf("%s.%s names a package of %s: its build is looked at again", delivery.PackagesBucket, e.Key(), module)
b.PackageNamed(ctx, module)
}
})
}
func versionsConflict(err error) error {
if errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
return versions.ErrConflict
}
return err
}
func (s awaitedVersions) Create(ctx context.Context, key string, value []byte) (uint64, error) {
kv, err := s.kv()
if err != nil {
return 0, err
}
rev, err := kv.Create(ctx, key, value)
return rev, versionsConflict(err)
}
func (s awaitedVersions) Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error) {
kv, err := s.kv()
if err != nil {
return 0, err
}
rev, err := kv.Update(ctx, key, value, revision)
return rev, versionsConflict(err)
}
func (s awaitedVersions) KeysOf(ctx context.Context, filter string) ([]string, error) {
kv, err := s.kv()
if err != nil {
return nil, err
}
l, err := kv.ListKeysFiltered(ctx, filter)
if err != nil {
return nil, err
}
defer func() { _ = l.Stop() }()
var out []string
for k := range l.Keys() {
out = append(out, k)
}
return out, nil
}
// floorVia is the floor's three sources as calls of the builder's own account (design 51 *The floor*): the seat's verb
// versions, the forge bridge's bridge_tags and the indexer's indexer_requirements. A call that fails is that source
// silent.
func floorVia(call toolCaller) versions.FloorSources {
return versions.FloorSources{
Seat: func(_ context.Context, seat, name string) ([]string, error) {
raw, err := call("seat:"+seat+".versions", map[string]any{"name": name})
if err != nil {
return nil, err
}
var a struct {
Versions []struct {
Version string `json:"version"`
} `json:"versions"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("%s's versions answered what is no list of versions: %w", seat, err)
}
var out []string
for _, v := range a.Versions {
if v.Version != "" {
out = append(out, v.Version)
}
}
return out, nil
},
Tags: func(_ context.Context, repository, prefix string) ([]string, error) {
raw, err := call(bridgeTagsTool, map[string]any{"repository": repository, "prefix": prefix})
if err != nil {
return nil, err
}
var a struct {
Tags []string `json:"tags"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("bridge_tags answered what is no list of tags: %w", err)
}
return a.Tags, nil
},
Requirements: func(_ context.Context, name string) ([]string, error) {
raw, err := call(requirementsTool, map[string]any{"package": name})
if err != nil {
return nil, err
}
var a struct {
Requirements []string `json:"requirements"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("indexer_requirements answered what is no list of versions: %w", err)
}
return a.Requirements, nil
},
}
}
// emitNumbered says the builder's event version.numbered (design 51 *Events of the builder*), on its own family apart
// from its builds' ends; the name and version are the bus's dedup id.
func emitNumbered(ctx context.Context, js jetstream.JetStream, cred process.Account, b delivery.VersionNumbered) error {
id := "numbered-" + versions.Key(b.Name, b.Version)
if b.Again != "" {
id += "-again-" + b.Again
}
if err := emitEvent(ctx, js, cred, delivery.VersionNumberedEvent, id, b); err != nil {
return fmt.Errorf("saying %s for %s@%s: %w", delivery.VersionNumberedEvent, b.Name, b.Version, err)
}
return nil
}
// emitEvent says one of the builder's events, mesh.mod.<module>.event.<event>, with the SDK's envelope headers and id as
// the bus's dedup id.
func emitEvent(ctx context.Context, js jetstream.JetStream, cred process.Account, event, id string, body any) error {
data, err := json.Marshal(body)
if err != nil {
return err
}
msg := nats.NewMsg(cred.EventSubject(event))
msg.Header.Set("x-event-id", id)
msg.Header.Set("x-source", cred.Module)
msg.Header.Set("x-node", cred.Node)
msg.Header.Set("x-time", time.Now().UTC().Format(time.RFC3339Nano))
msg.Header.Set("content-type", "application/json")
msg.Data = data
pub, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
_, err = js.PublishMsg(pub, msg, jetstream.WithMsgID(id))
return err
}
// The events on the builder's consumer of EVENTS that wake a floor (design 51 *The floor*, DLV-B3b).
var (
manifestAcceptedSubject = "mesh.mod." + delivery.IndexerModule + ".event." + delivery.ManifestAcceptedEvent
conditionClearedSubject = "mesh.mod." + health.KeeperModule + ".event." + health.EventCleared
)
// floorWake asks the floor of the names an event wakes again, by looking at every module again: a trunk build is
// numbered before its claim (build.Builder.number).
type floorWake struct {
b *build.Builder
waits *versions.FloorWaits
logf func(string, ...any)
}
// heard takes a message of the consumer in when it is one that wakes a floor, acknowledged, and answers whether it was.
func (f *floorWake) heard(ctx context.Context, m jetstream.Msg) bool {
switch m.Subject() {
case manifestAcceptedSubject:
f.ack(m)
f.wake(ctx, versions.WakeAccepted, "", "")
case conditionClearedSubject:
f.ack(m)
var e health.Event
if err := json.Unmarshal(m.Data(), &e); err != nil {
f.logf("%s holds what is no condition, left: %v", m.Subject(), err)
return true
}
for _, module := range conditionModules(e.Subject.ID) {
f.wake(ctx, versions.WakeCleared, "", module)
}
default:
return false
}
return true
}
func (f *floorWake) ack(m jetstream.Msg) {
if err := m.Ack(); err != nil {
f.logf("acknowledging %s: %v", m.Subject(), err)
}
}
// wake looks at every module again when the event wakes a name, and says which.
func (f *floorWake) wake(ctx context.Context, kind, seat, module string) {
names := f.waits.Wake(kind, seat, module, "")
if len(names) == 0 {
return
}
f.logf("the floor of %s is asked again: %s", strings.Join(names, ", "), kind)
f.b.Reconsider(ctx)
}
// conditionModules are the tokens of a condition's subject id (`<node>/<module>`, `<module>.<check>`): a module named
// among them is the one whose condition cleared.
func conditionModules(id string) []string {
return strings.FieldsFunc(id, func(r rune) bool { return r == '/' || r == '.' || r == ':' })
}
// doneVia says whether a build key is done, read from the builder's own builds state: the build's file is held.
func doneVia(b *build.Builder) func(ctx context.Context, module, fp string) (bool, error) {
return func(ctx context.Context, module, fp string) (bool, error) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return false, err
}
raw, _, found, err := b.Claimer.Store.Get(ctx, key)
if err != nil || !found {
return false, err
}
cur, err := delivery.DecodeBuilderBuild(raw)
if err != nil {
return false, err
}
return cur.State == delivery.BuildDone, nil
}
}
// removeVia calls a registry seat's remove for one tagged version, as the builder's own account (design 51 *Tagged
// versions*): the reason tag-gone; the seat says removed on its events, or answers kept when another tag names the
// digest (the registry part's RemoveAnswer: kept, reason, tags), which ends the remove.
func removeVia(call toolCaller) func(ctx context.Context, seat string, e versions.Entry) (versions.RemoveOutcome, error) {
return func(_ context.Context, seat string, e versions.Entry) (versions.RemoveOutcome, error) {
raw, err := call("seat:"+seat+".remove", map[string]any{"name": e.Name, "version": e.Version, "tag": e.Tag, "reason": "tag-gone"})
if err != nil {
return versions.RemoveOutcome{}, err
}
var a ociartifactregistry.RemoveAnswer // the four registry seats share the part's answer
if len(raw) > 0 && json.Unmarshal(raw, &a) != nil {
return versions.RemoveOutcome{}, fmt.Errorf("%s's remove answered what is no answer of it: %s", seat, raw)
}
return versions.RemoveOutcome{Kept: a.Kept, Reason: a.Reason, Tags: a.Tags}, nil
}
}
// tagAsked answers an unanswered value of tag-asks by compare-and-set at its revision, said in the log.
func tagAsked(ctx context.Context, acts versions.Acts, kv jetstream.KeyValue, e jetstream.KeyValueEntry, logf func(string, ...any)) {
if e.Operation() != jetstream.KeyValuePut {
return
}
t, err := versions.DecodeTagAsk(e.Value())
if err != nil {
logf("%s.%s holds what is not a tag ask, left: %v", versions.TagAsksState, e.Key(), err)
return
}
if !t.Answered.IsZero() {
return
}
t = acts.TagAsked(ctx, t, time.Now())
logf("%s: %s of %s %s by %s (%s): %s", e.Key(), map[bool]string{true: "untag", false: "tag ask"}[t.Untag], t.Repository, t.Branch, t.By, t.Why, t.Answer)
data, err := json.Marshal(t)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the answer could not be written: %v", versions.TagAsksState, e.Key(), err)
}
}
+29 -5
View File
@@ -3,20 +3,44 @@ module mesh-delivery-builder
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101023317-8311ecb84911
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261104174359-05b3e2e1b94f
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
require git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep v0.0.0-20261104174638-125929dbc459
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
git.novox.be/novox/mesh-contracts/go v0.0.0-20261104174344-ae5266e77639 // indirect
git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun v0.0.0-00010101000000-000000000000
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/klauspost/compress v1.20.1 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/jwt/v2 v2.8.2 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.org/x/time v0.16.0 // indirect
)
replace git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun => ./buildrun
require (
git.novox.be/novox/mesh-contracts/go/seats/containerimageregistry v0.0.0-20261104174344-ae5266e77639
git.novox.be/novox/mesh-contracts/go/seats/gopackageregistry v0.0.0-20261104174344-ae5266e77639
git.novox.be/novox/mesh-contracts/go/seats/nodeengine v0.0.0-20261104174344-ae5266e77639
git.novox.be/novox/mesh-contracts/go/seats/npmpackageregistry v0.0.0-20261104174344-ae5266e77639
git.novox.be/novox/mesh-contracts/go/seats/ociartifactregistry v0.0.0-20261104174344-ae5266e77639
git.novox.be/novox/mesh-delivery/sdk v0.0.0-00010101000000-000000000000
)
require (
git.novox.be/novox/mesh-contracts/go/clients/containerimageregistryclient v0.0.0-20261104174344-ae5266e77639 // indirect
git.novox.be/novox/mesh-contracts/go/clients/gopackageregistryclient v0.0.0-20261104174344-ae5266e77639 // indirect
git.novox.be/novox/mesh-contracts/go/clients/npmpackageregistryclient v0.0.0-20261104174344-ae5266e77639 // indirect
git.novox.be/novox/mesh-contracts/go/clients/ociartifactregistryclient v0.0.0-20261104174344-ae5266e77639 // indirect
)
replace git.novox.be/novox/mesh-delivery/sdk => ../../sdk
+32 -10
View File
@@ -1,15 +1,37 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101023317-8311ecb84911 h1:+82ygf2hiY5Mhwfmc1xfCe9e/KY/5ObBuE2LecM37qI=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261101023317-8311ecb84911/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep v0.0.0-20261104174638-125929dbc459 h1:DA4SFG/gKyG+aPJbfUpAUad2VeITIiKjJ6qOPmDhSA8=
git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep v0.0.0-20261104174638-125929dbc459/go.mod h1:zR1HsAKZVbQsBIY5wPyrqI2WHInK268tITzGl2oqHro=
git.novox.be/novox/mesh-contracts/go v0.0.0-20261104174344-ae5266e77639 h1:Zt7CfWt2S2W/fVgWNyW2u2+xwz051LXvGxVTNIyfhdM=
git.novox.be/novox/mesh-contracts/go v0.0.0-20261104174344-ae5266e77639/go.mod h1:0dD1XEJBJ0G5UYDQxSZI84nuujb6m3tOpJl4C8rQVvg=
git.novox.be/novox/mesh-contracts/go/clients/containerimageregistryclient v0.0.0-20261104174344-ae5266e77639 h1:I6uVDO9E+i3D9/n6KQ2RWYRs3kXYCqueiKXynifQo8w=
git.novox.be/novox/mesh-contracts/go/clients/containerimageregistryclient v0.0.0-20261104174344-ae5266e77639/go.mod h1:ER2NVrd81uJo1lN0YCGA7OBxL910IHRgWYtxpsnHGF0=
git.novox.be/novox/mesh-contracts/go/clients/gopackageregistryclient v0.0.0-20261104174344-ae5266e77639 h1:Rs5jOa7EWlpKuaH6+ExyGLN9k9UUi8N7upP9Ac3k+GY=
git.novox.be/novox/mesh-contracts/go/clients/gopackageregistryclient v0.0.0-20261104174344-ae5266e77639/go.mod h1:fNfjFkP1+suYb12HJkapQe5W3cF98JsTcLBI/syoLHc=
git.novox.be/novox/mesh-contracts/go/clients/npmpackageregistryclient v0.0.0-20261104174344-ae5266e77639 h1:VG+sZltyBIKrd9cwFJyKkInz04SxWBxUlJWfAuo6774=
git.novox.be/novox/mesh-contracts/go/clients/npmpackageregistryclient v0.0.0-20261104174344-ae5266e77639/go.mod h1:Nq7glLZSZ8zOLTh1pXDnKZi+MxsMRyHw/bZODC3mvrY=
git.novox.be/novox/mesh-contracts/go/clients/ociartifactregistryclient v0.0.0-20261104174344-ae5266e77639 h1:ROS9UG/Dg0NpsaDuEmDFV9CsIC1R92sOm3QauF7IvL0=
git.novox.be/novox/mesh-contracts/go/clients/ociartifactregistryclient v0.0.0-20261104174344-ae5266e77639/go.mod h1:/Pke1v2VnSFcR8ic+fo/kNCg3JFYK6ILTal72Ca9520=
git.novox.be/novox/mesh-contracts/go/seats/containerimageregistry v0.0.0-20261104174344-ae5266e77639 h1:5HeZzk6tKzcSwVivD3IcIvQ/2WIvysZvNvNKyq/oIBA=
git.novox.be/novox/mesh-contracts/go/seats/containerimageregistry v0.0.0-20261104174344-ae5266e77639/go.mod h1:pPxwhTMb3hIk7UUrBicJnGQCqfTEoBGJvNFspEK2U18=
git.novox.be/novox/mesh-contracts/go/seats/gopackageregistry v0.0.0-20261104174344-ae5266e77639 h1:oAXU1t8A1rGIr82JPBaNYSGLicXqhWglKaEAtPmrD0c=
git.novox.be/novox/mesh-contracts/go/seats/gopackageregistry v0.0.0-20261104174344-ae5266e77639/go.mod h1:Y5kgBh0hBbL/hgc+abCF9Vbhov+vCsB0ckpRzUw1D+0=
git.novox.be/novox/mesh-contracts/go/seats/nodeengine v0.0.0-20261104174344-ae5266e77639 h1:YQ6eJvc+RYUVIVf+oX+R4TC1dYL8TLLMZ0/JPDU1tp4=
git.novox.be/novox/mesh-contracts/go/seats/nodeengine v0.0.0-20261104174344-ae5266e77639/go.mod h1:TB20TsQuOd5aer+cXL6PbaYyg2RvgL1R4MzD61Oks3A=
git.novox.be/novox/mesh-contracts/go/seats/npmpackageregistry v0.0.0-20261104174344-ae5266e77639 h1:ILh7uctNwgx/89qHoR/H2kz5TpWeKEeIH0O2pDPE7Iw=
git.novox.be/novox/mesh-contracts/go/seats/npmpackageregistry v0.0.0-20261104174344-ae5266e77639/go.mod h1:JRtTjIjMTkzn59IN4vCYei9ZwJGFnN1Puwame4V/sD0=
git.novox.be/novox/mesh-contracts/go/seats/ociartifactregistry v0.0.0-20261104174344-ae5266e77639 h1:+bkN8op2yrAlBYrOB+hYnIoIZkbjXsNmc/4//uMnL4E=
git.novox.be/novox/mesh-contracts/go/seats/ociartifactregistry v0.0.0-20261104174344-ae5266e77639/go.mod h1:ooQa2CiP8JOvhP0bdQvwurV5wskRE7Cm68ecURBbCwU=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261104174359-05b3e2e1b94f h1:jDsfDMd48MXY9Zershc3eRxnhmx2sSpUfQkLq5jAIds=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261104174359-05b3e2e1b94f/go.mod h1:08OomDXdwVN8/87nyDtv5R3rH/6LoP8SFFXz30bTYCM=
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op h1:1BOWQJweNyvZMlpAHXGLiZQn9S+QXGcz3xh94lC0w6E=
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ=
github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/jwt/v2 v2.8.2 h1:XXRgB60MSTnqsRwejQurVDs/hcv2dkt+86GjI+I/bMc=
github.com/nats-io/jwt/v2 v2.8.2/go.mod h1:Ag/56sq9OblL4JgdYufDd16Egb17Kr/8WwwuO/forVc=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
@@ -23,5 +45,5 @@ golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZ
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/time v0.16.0 h1:vMb6ptszcQMkcwiRTAuNNU50gom6++Q/6gY2hDM6VDE=
golang.org/x/time v0.16.0/go.mod h1:rVKOqvZeKvrDKTQiAHJ7wmwP0RzleSphoEA9RcdLA0s=
@@ -6,13 +6,14 @@ import (
"fmt"
"regexp"
"slices"
"sort"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// **An ask that can no longer be answered is asked again** (novox/hq issue 1945). A build seat's stream gives an ask up
@@ -177,6 +178,42 @@ func (b *Builder) drop(ctx context.Context, ask delivery.StepAsk, old string) {
}
}
// DefaultDeclineBound is how long one ask may go on being declined, from its first decline, before its key fails (the
// checker's DefaultDeclineBound, issue 890); DefaultDeclinePause is how long the builder waits before it puts a declined
// ask again, so a lone node off the agreed toolchain cannot spin on it.
const (
DefaultDeclineBound = 30 * time.Minute
DefaultDeclinePause = 5 * time.Second
)
func (b *Builder) declineBound() time.Duration {
if b.DeclineBound > 0 {
return b.DeclineBound
}
return DefaultDeclineBound
}
func (b *Builder) declinePause() time.Duration {
if b.DeclinePause != 0 {
return max(b.DeclinePause, 0)
}
return DefaultDeclinePause
}
// declinedWords names the nodes that declined an ask and why, in the order of their names.
func declinedWords(by map[string]string) string {
nodes := make([]string, 0, len(by))
for n := range by {
nodes = append(nodes, n)
}
sort.Strings(nodes)
var parts []string
for _, n := range nodes {
parts = append(parts, n+": "+by[n])
}
return strings.Join(parts, "; ")
}
func (b *Builder) askings() int {
if b.Askings > 0 {
return b.Askings
@@ -255,6 +292,8 @@ func (b *Builder) unlisten(key, language string, c chan string) {
var (
errCheck = errors.New("the check bound passed")
errAgain = errors.New("asked again")
// errDeclined: a build node on another toolchain than the agreed one sat out of the ask (ADR 0584 clause 2).
errDeclined = errors.New("declined")
)
type byHand struct{ why string }
@@ -268,11 +307,42 @@ func (b *Builder) answer(ctx context.Context, l *lane, ask delivery.StepAsk, ask
again := b.listen(key, ask.Language)
defer b.unlisten(key, ask.Language, again)
asking := askingOf(asked.step, l.module)
var firstDecline time.Time
declines := map[string]string{} // node -> why it declined, the last
for {
step, why, err := b.waitOnce(ctx, ask, again)
switch {
case errors.Is(err, errCheck):
continue
case errors.Is(err, errDeclined):
// Not an asking: a node that sits out would use them all while it idles and pulls the ask again at once.
// Bounded by time from the first decline, then failed loudly naming the nodes and their reasons (as the
// checker's decline loop is, issue 890).
now := b.now()
if firstDecline.IsZero() {
firstDecline = now
}
declines[step.Node] = why
if bound := b.declineBound(); now.Sub(firstDecline) > bound {
return step, ask, fmt.Errorf("the ask %s of %s was declined for %s and no build node took it: %s", ask.ID, l.module, bound.Round(time.Second), declinedWords(declines))
}
// The pause is a context's bound, not a timer (ADR 0437 §8).
pause, stop := context.WithTimeout(ctx, b.declinePause())
<-pause.Done()
stop()
if ctx.Err() != nil {
return step, ask, ctx.Err()
}
old := ask.ID
ask.ID, ask.At = buildstep.NewID(), time.Now()
if err := asked.note(ctx, ask.Language, ask.ID, fmt.Sprintf("put %s again after %s declined it", l.module, step.Node)); err != nil {
return step, ask, fmt.Errorf("the ask of %s put again after a decline could not be written to the key: %w", l.module, err)
}
if err := b.Asks.Put(ctx, ask); err != nil {
return step, ask, err
}
b.drop(ctx, ask, old)
b.logf("%s: the ask %s of %s was declined on %s and is put again as %s: %s", key, old, l.module, step.Node, ask.ID, why)
case errors.Is(err, errAgain):
if asking >= b.askings() {
return step, ask, fmt.Errorf("the ask %s could not be answered and was asked %d times, the most it is: %s", ask.ID, asking, why)
@@ -317,6 +387,11 @@ func (b *Builder) waitOnce(ctx context.Context, ask delivery.StepAsk, again <-ch
defer stop()
}
step, err := b.Asks.Wait(wait, ask)
if _, declined := delivery.DeclinedOf(step); err == nil && declined {
// A build node off the agreed toolchain sat out (ADR 0584 clause 2). It is no asking: answer() puts the ask again
// after a pause, within DeclineBound.
return step, step.Reason, errDeclined
}
if err == nil && buildrun.IsLost(step) {
// The build seat gave the ask up (issue 734): it can no longer be answered, so it is asked again now, with the
// words of why, not at the check bound.
@@ -0,0 +1,94 @@
package build
import (
"context"
"encoding/json"
"fmt"
"maps"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The build runner a build node runs an ask with is mesh-build-go's (novox/hq ADR 0593 §1), another repository's,
// which the builder does not import. These tests run an ask as the builder needs it run: fetch once at the job's
// fetch arguments with its package versions, build or copy each artifact, answer each one's digest and reference. They
// test the builder, not the runner.
// stepFunc is one step of a build module, as its tool answers it.
type stepFunc func(ctx context.Context, args map[string]any) (any, error)
// askSteps are a build module's steps an ask is run with.
type askSteps struct{ Fetch, Build, Copy stepFunc }
// runAsk runs a build ask with steps and answers its delivery.BuildAnswer, or the error that fails the step.
func runAsk(ctx context.Context, steps askSteps, ask delivery.StepAsk) (any, error) {
if ask.Build == nil {
return nil, fmt.Errorf("the ask %s holds no build job", ask.ID)
}
job := *ask.Build
fetch := job.Fetch
if len(ask.Packages) > 0 && fetch != nil {
fetch = maps.Clone(fetch)
fetch["packages"] = maps.Clone(ask.Packages)
}
answer := delivery.BuildAnswer{}
workspace := ""
for _, a := range job.Artifacts {
var out any
var err error
if a.From != "" {
if steps.Copy == nil {
return nil, fmt.Errorf("this build module copies no artifact (%s)", a.Artifact)
}
out, err = steps.Copy(ctx, map[string]any{"module": job.Module, "artifact": a.Artifact, "from": a.From})
} else {
if workspace == "" {
raw, err := asRaw(steps.Fetch(ctx, fetch))
if err != nil {
return nil, err
}
var f struct {
Workspace string `json:"workspace"`
Merge string `json:"merge"`
Conflicts []string `json:"conflicts"`
}
if err := json.Unmarshal(raw, &f); err != nil || f.Workspace == "" {
return nil, fmt.Errorf("the fetch answered no workspace: %s", raw)
}
if _, candidate := fetch["base"]; candidate && (len(f.Conflicts) > 0 || f.Merge == "") {
return nil, fmt.Errorf("the head does not merge into the trunk: %s", strings.Join(f.Conflicts, ", "))
}
workspace = f.Workspace
}
args := map[string]any{"workspace": workspace, "module": job.Path, "artifact": a.Artifact}
if job.Publish {
args["publish"] = true
}
out, err = steps.Build(ctx, args)
}
raw, err := asRaw(out, err)
if err != nil {
return nil, fmt.Errorf("%s: %w", a.Artifact, err)
}
var built struct {
Digest string `json:"digest"`
Reference string `json:"reference"`
}
if err := json.Unmarshal(raw, &built); err != nil || built.Digest == "" {
return nil, fmt.Errorf("the build of %s answered no digest: %s", a.Artifact, raw)
}
answer.Artifacts = append(answer.Artifacts, delivery.AskBuilt{Artifact: a.Artifact, Digest: built.Digest, Reference: built.Reference})
}
return answer, nil
}
func asRaw(v any, err error) (json.RawMessage, error) {
if err != nil {
return nil, err
}
if r, ok := v.(json.RawMessage); ok {
return r, nil
}
return json.Marshal(v)
}
@@ -7,8 +7,8 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// **A failed key is built again by hand** (novox/hq issue 1126 note 72; ADR 0429 clause 18: "a fingerprint is built again
@@ -5,8 +5,8 @@ import (
"fmt"
"sort"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// How a candidate's fingerprint is built (novox/hq ADR 0484 clause 3, issue 1684): the indexer writes, for each module a
@@ -8,9 +8,9 @@ import (
"sort"
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// ErrConflict is what a Store answers when a Create finds the key or an Update finds another revision than the one given.
@@ -114,12 +114,11 @@ func TestClaimCreatesTheKeyBuildingAndASecondBuilderGetsNothing(t *testing.T) {
if second, err := b.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{}); err != nil || second != nil {
t.Errorf("second builder's claim = %v, %v, want none: the same source is never built twice", second, err)
}
}
func TestClaimLosingTheCreateRaceGetsNothing(t *testing.T) { // 1607/AC3
st := newMem()
// A claim losing the create race gets nothing (merged into this test for the test budget, design 51).
st = newMem()
st.before = func() { seed(t, st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.b"}) }
got, err := Claimer{Store: st, By: "novox.a"}.Claim(context.Background(), "m", fp1, "", ClaimOptions{})
got, err = Claimer{Store: st, By: "novox.a"}.Claim(context.Background(), "m", fp1, "", ClaimOptions{})
if err != nil || got != nil {
t.Errorf("claim = %v, %v, want none: the create failed because the key exists", got, err)
}
@@ -197,12 +196,11 @@ func TestFinishMovesTheKeyByCompareAndSetAndWritesOnlyADifferentValue(t *testing
if st.writes != writes {
t.Errorf("the same value was written again (%d writes, was %d)", st.writes, writes)
}
}
func TestFinishAfterTheKeyChangedIsAConflictNotAnOverwrite(t *testing.T) { // 1607/AC3
st := newMem()
c := Claimer{Store: st, By: "novox.a"}
got, _ := c.Claim(context.Background(), "m", fp1, "", ClaimOptions{})
// Finish after the key changed is a conflict, not an overwrite (merged into this test for the test budget).
st = newMem()
c = Claimer{Store: st, By: "novox.a"}
got, _ = c.Claim(context.Background(), "m", fp1, "", ClaimOptions{})
st.put(got.Key, []byte(`{"v":1,"state":"failed","error":"someone else"}`))
err := c.Finish(context.Background(), got, delivery.BuilderBuild{State: delivery.BuildFailed, Error: "mine"})
if !errors.Is(err, ErrConflict) {
@@ -29,7 +29,7 @@ func captured(t *testing.T, name string) json.RawMessage {
return step.Result
}
// The captured image steps, run as a pulled ask runs them (buildstep.RunAsk), answer what the builder reads: the trunk's
// The captured image steps, run as a pulled ask runs them (runAsk), answer what the builder reads: the trunk's
// build its digest and reference, a candidate's its digest and no reference (issue 1887 A4a).
func TestCapturedImageStepsDecodeWithTheBuildersOwnTypes(t *testing.T) { // review r2, mesh-catalog 2; ADR 0487 clause 1
for _, tc := range []struct {
@@ -37,7 +37,7 @@ func TestCapturedImageStepsDecodeWithTheBuildersOwnTypes(t *testing.T) { // revi
publish bool
ref bool
}{{"build-trunk.json", true, true}, {"build-candidate.json", false, false}} {
steps := buildstep.AskSteps{
steps := askSteps{
Fetch: func(context.Context, map[string]any) (any, error) {
return json.RawMessage(captured(t, "fetch.json")), nil
},
@@ -47,7 +47,7 @@ func TestCapturedImageStepsDecodeWithTheBuildersOwnTypes(t *testing.T) { // revi
At: time.Now(), Build: &delivery.BuildJob{Module: "m", Path: "modules/m", Publish: tc.publish,
Fetch: map[string]any{"repository": "novox/mesh-catalog", "commit": strings.Repeat("a", 40)},
Artifacts: []delivery.AskArtifact{{Artifact: "server"}}}}
out, err := buildstep.RunAsk(context.Background(), steps, ask)
out, err := runAsk(context.Background(), steps, ask)
if err != nil {
t.Fatalf("%s: %v", tc.build, err)
}
@@ -0,0 +1,78 @@
package build
import (
"encoding/json"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ADR 0593 §2 (F3): the imports a Go build module answers at a build's end reach the done key and the log's end entry,
// where the indexer keeps them on the module.
func importsABuildAnswersReachTheDoneKeyAndTheLog(t *testing.T) {
r := newRig(t, goBundle)
imports := []string{"git.novox.be/novox/mesh-sdk/go", "git.novox.be/novox/mesh-sdk/go/announce"}
r.asks.Answer = func(a delivery.StepAsk) delivery.Step {
step := r.answer(a)
var answer map[string]any
if err := json.Unmarshal(step.Result, &answer); err == nil && a.Language == "go" {
answer["imports"] = imports
step.Result, _ = json.Marshal(answer)
}
return step
}
var logged []delivery.BuildLogEntry
var mu sync.Mutex
r.b.Node, r.b.Log = "novox", func(e delivery.BuildLogEntry) { mu.Lock(); logged = append(logged, e); mu.Unlock() }
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || strings.Join(v.Imports, ",") != strings.Join(imports, ",") {
t.Fatalf("done key %+v, want the imports", v)
}
if len(logged) != 2 || strings.Join(logged[1].Imports, ",") != strings.Join(imports, ",") || len(logged[0].Imports) != 0 {
t.Fatalf("log %+v, want the imports on the end alone", logged)
}
}
// Design 26 (the lead's ruling on A2i's finding 2): the out-of-step files a contracts module's build module answers, in a
// failed step's result too, reach the log's end entry, sorted and each once, where the indexer raises its finding; a
// build that answers none names none.
func outOfStepFilesABuildAnswersReachTheLog(t *testing.T) {
for _, row := range []struct {
name string
fail bool
files []string
want string
}{
{"a passed build naming files", false, []string{"go/b.go", "go/a.go", "go/b.go"}, "go/a.go,go/b.go"},
{"a failed build naming files", true, []string{"ts/x.ts"}, "ts/x.ts"},
{"a build naming none", false, nil, ""},
} {
t.Run(row.name, func(t *testing.T) {
r := newRig(t, goBundle)
r.asks.Answer = func(a delivery.StepAsk) delivery.Step {
step := r.answer(a)
answer := map[string]any{}
_ = json.Unmarshal(step.Result, &answer)
if row.files != nil {
answer["out-of-step"] = row.files
}
step.Result, _ = json.Marshal(answer)
if row.fail {
step.State, step.Reason = delivery.StepFailed, "the generated files are out of step"
}
return step
}
var logged []delivery.BuildLogEntry
var mu sync.Mutex
r.b.Node, r.b.Log = "novox", func(e delivery.BuildLogEntry) { mu.Lock(); logged = append(logged, e); mu.Unlock() }
r.consider(t, "modules/m")
mu.Lock()
defer mu.Unlock()
if len(logged) != 2 || strings.Join(logged[1].OutOfStep, ",") != row.want || len(logged[0].OutOfStep) != 0 {
t.Fatalf("log %+v, want the end naming %q", logged, row.want)
}
})
}
}
@@ -0,0 +1,140 @@
package build
import (
"context"
"fmt"
"strconv"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/versions"
)
// Numbers numbers a trunk build at its claim (novox/hq design 51 *The versions*, ADR 0606, ADR 0617 §1) and records the
// digests it was built with. Nil numbers nothing.
type Numbers interface {
// tag is a check's build's tag (TagOf), empty for a trunk build; done the digests of its key when it is done already
// (versions.BuildClaim.Done), nil otherwise.
Number(ctx context.Context, module string, m delivery.Module, plan Plan, tag string, done map[string]string) (versions.Numbering, error)
Built(ctx context.Context, module string, m delivery.Module, plan Plan, got versions.Numbering, made []delivery.BuiltArtifact) error
}
// VersionNumbers is Numbers over the builder's state versions.
type VersionNumbers struct {
N *versions.Numberer
// TagNumbered is told the versions a check's build of a tag ask's head was numbered (the ask is then marked numbered
// in tag-asks, design 51 *Tagged versions*); nil tells nobody.
TagNumbered func(ctx context.Context, c versions.BuildClaim, got versions.Numbering)
}
// ClaimOf is the claim a module value's trunk build is numbered by: a package per source revision (Src), the rest per
// fingerprint, in merge order (Src).
func ClaimOf(module string, m delivery.Module, plan Plan) versions.BuildClaim {
c := versions.BuildClaim{Module: module, Path: m.Path, Repository: delivery.RepositoryName(m.Repository), Fingerprint: m.FP2,
Commit: m.Commit, Merge: m.Src}
if m.Src != 0 {
c.Revision = strconv.FormatUint(m.Src, 10)
}
for _, t := range plan.Tasks {
c.Artifacts = append(c.Artifacts, versions.ClaimArtifact{Artifact: t.Artifact, Kind: t.Kind, Language: t.Language})
}
return c
}
func (v VersionNumbers) Number(ctx context.Context, module string, m delivery.Module, plan Plan, tag string, done map[string]string) (versions.Numbering, error) {
c := ClaimOf(module, m, plan)
c.Done = done
if tag != "" {
// A check's build of a head a tag ask names: its packages tagged, below the next version (design 51 *Tagged versions*).
c.Tag, c.Head, c.Revision, c.Merge = tag, m.Commit, "", 0
}
got, err := v.N.Number(ctx, c)
if err == nil && tag != "" && len(got.Versions) > 0 && v.TagNumbered != nil {
v.TagNumbered(ctx, c, got)
}
return got, err
}
func (v VersionNumbers) Built(ctx context.Context, module string, m delivery.Module, plan Plan, got versions.Numbering, made []delivery.BuiltArtifact) error {
digests := map[string]string{}
for _, a := range made {
digests[a.Name] = a.Digest
}
return v.N.Built(ctx, ClaimOf(module, m, plan), got, digests)
}
// TagOf answers the cleaned tag a kept tag ask names for a repository's head (the builder's state tag-asks, design 51
// *Tagged versions*): a check's build of that head numbers its packages tagged at its claim.
type TagOf func(ctx context.Context, repository, head string) (tag string, ok bool)
// number numbers a build before its claim, a trunk build's artifacts or a check's build's packages under its tag: false
// when it waits for its floor (a package, design 51 *The floor*), and then nothing is claimed; the floor's wake looks at
// the module again.
func (b *Builder) number(ctx context.Context, module string, m delivery.Module, plan Plan, tag string) (versions.Numbering, bool) {
if b.Numbers == nil {
return versions.Numbering{}, true
}
var done map[string]string
if tag == "" {
done = b.doneDigests(ctx, module, m.FP2)
}
got, err := b.Numbers.Number(ctx, module, m, plan, tag, done)
for _, s := range got.Said {
b.logf("%s at %s: %s", module, m.FP2, s.Line)
// And in the build log (design 51 *The versions*: it says a version was kept, and why), between the build's start and end.
// A look at a key already done builds nothing and has no start or end: a version kept there says nothing new, and is
// not logged (each later look, a first view answered or a name heard, would add one).
if b.Log != nil && !(done != nil && s.Why == versions.SaidKept) {
b.Log(delivery.BuildLogEntry{Event: delivery.BuildSaidEvent, Module: module, FP: m.FP2, Commit: m.Commit, Node: b.Node,
At: time.Now().UTC(), Line: s.Line, Why: s.Why})
}
}
if err != nil {
// Said, and built as before: an unnumbered build is published by its digest (ADR 0617 §1), numbered at its next look.
b.logf("%s at %s: %v", module, m.FP2, fmt.Errorf("not numbered, the versions cannot be used: %w", err))
return versions.Numbering{}, true
}
if got.Wait != "" {
b.logf("%s at %s is not built yet: %s", module, m.FP2, got.Wait)
return got, false
}
return got, true
}
// doneDigests are the digests of a trunk build's key by artifact when the key is done, nil otherwise or when it cannot be
// read: a look at a done build numbers what it left unnumbered with them (versions.BuildClaim.Done).
func (b *Builder) doneDigests(ctx context.Context, module, fp string) map[string]string {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil || b.Claimer.Store == nil {
return nil
}
raw, _, found, err := b.Claimer.Store.Get(ctx, key)
if err != nil || !found {
return nil
}
cur, err := delivery.DecodeBuilderBuild(raw)
if err != nil || cur.State != delivery.BuildDone {
return nil
}
out := map[string]string{}
for _, a := range cur.Artifacts {
if a.Digest != "" {
out[a.Name] = a.Digest
}
}
return out
}
// PackageNamed looks at a module again when the indexer's packages state names a name for one of its packages
// (versions.NamedModule): a done trunk build that left the package unnumbered, its name not known at its claim, is
// numbered then with the done build's digests, and version.numbered says them for its build module's late publish.
func (b *Builder) PackageNamed(ctx context.Context, module string) {
b.init()
b.mu.Lock()
m, ok := b.known[module]
b.mu.Unlock()
if ok {
b.consider(ctx, module, m)
}
}
@@ -8,9 +8,9 @@ import (
"sort"
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// ImageBuildModule is the build module that builds an image artifact (novox/hq ADR 0487): buildah, rootless, on the nodes that
@@ -3,11 +3,12 @@ package build
import (
"errors"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"strings"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
@@ -305,10 +305,9 @@ func TestTheFetchOfAModuleNamingItsPathsCarriesItsDirectoryAndThosePaths(t *test
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
paths := map[string]string{"src": hex40('1'), "package.json": hex40('2')}
tree, err := delivery.NamedTree(manifestD, paths)
if err != nil {
t.Fatal(err)
}
// The named tree the indexer computes from those paths (its own, ADR 0593 §2), given as data: the builder reads it,
// never computes it.
const tree = delivery.NamedTreePrefix + "5b5e0f3b6f3c1f2a9d8e7c6b5a4938271605f4e3d2c1b0a9f8e7d6c5b4a39281"
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-sdk", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP2: fp1, Manifest: manifestD, Path: ".",
Toolchains: agreedToolchains, Inputs2: &delivery.KeptInputs2{Tree: tree, Paths: paths}})
if err != nil {
@@ -2,9 +2,10 @@ package build
import (
"context"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -6,16 +6,20 @@ import (
"errors"
"fmt"
"maps"
"slices"
"sort"
"strconv"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-delivery/sdk/forms"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-builder/internal/versions"
)
// ToolchainEntries are the nodes' toolchain entries of a language (buildrun.ToolchainSet).
@@ -67,13 +71,27 @@ type Builder struct {
// Askings is how many times one build module's part of a build is asked, the first included, before its key is failed
// with why; DefaultAskings when 0.
Askings int
// DeclineBound and DeclinePause bound and pace the asks build nodes off the agreed toolchain declined (ADR 0584 clause 2):
// they are no askings. DefaultDeclineBound and DefaultDeclinePause when 0; a negative DeclinePause puts it again at once.
DeclineBound, DeclinePause time.Duration
// Said is told each ask asked again (the builder's event build.asked-again). Nil tells nobody.
Said func(AskedAgain)
// Log is told each build's start and end, the entries of the builder's log (novox/hq ADR 0571 §1). Nil tells nobody.
Log func(delivery.BuildLogEntry)
// Now is the clock of the log; nil is time.Now. Node is the node the builder runs on, which asked the build.
// Now is the builder's clock: of the log, and of the decline bound (DeclineBound runs from the first decline); nil is
// time.Now. Node is the node the builder runs on, which asked the build.
Now func() time.Time
Node string
// Numbers numbers each trunk build at its claim and records what it was built with (design 51 *The versions*); nil
// numbers nothing.
Numbers Numbers
// Tags answers a check's build's tag from the kept tag asks; nil numbers no check's build.
Tags TagOf
// Forms answers what each node's engine states it reads (ADR 0618 §2, ADR 0624 §1): present, the nodes whose engine
// answered when last asked, and counted, every node the placement names, present or away, a node with no statement
// stating nothing (the baseline's first forms). A done key names each artifact in the forms these keep, the oldest in
// its single reference, and its needs. Nil: no node states anything, and the first forms are kept.
Forms func() (present, counted map[string][]string)
once sync.Once
sem chan struct{}
@@ -85,8 +103,10 @@ type Builder struct {
plans map[string]planned // by manifest digest
cands map[string]delivery.Candidate // by candidates key (candidates.go)
via map[string]via // by build key: a candidate's fetch (candidates.go)
trunkFP map[string]bool // by build key: a trunk module value carries this fingerprint
tried map[string]bool // by build key: looked at since this process started
// outOfStep are, by build key, the out-of-step files a build's lanes answered, until its end entry takes them.
outOfStep map[string][]string
trunkFP map[string]bool // by build key: a trunk module value carries this fingerprint
tried map[string]bool // by build key: looked at since this process started
// published are the package versions `<name>@<version>` a done key of this bucket names as a package's reference: a key
// waiting for one is built once it is here (ADR 0500 clause 6).
published map[string]bool
@@ -257,8 +277,8 @@ func (b *Builder) storedOnce(key string) func(ctx context.Context, references []
}
}
// ArtifactsGone takes the controller's event artifacts-gone in (issue 1937): a take-in found that the store does not
// hold what the module's key names. The module is looked at again, which asks the store (ClaimOptions.Stored) and
// ArtifactsGone takes the seat mesh-artifact-sweep's event artifacts-gone in (issue 1937, ADR 0571 §2): the sweep found
// that the store does not hold what the module's key names. The module is looked at again, which asks the store (ClaimOptions.Stored) and
// builds the key again; an event for a fingerprint the module no longer has is said and left. The event only wakes the
// builder: what is built rests on the store's answer, never on the event alone.
func (b *Builder) ArtifactsGone(ctx context.Context, e delivery.ArtifactsGone) {
@@ -415,7 +435,7 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
if planErr == nil && !nothing {
if why := b.stale(m, plan); why != "" {
// The fingerprint names a toolchain the build nodes no longer agree on: a new fingerprint is coming, and a build
// now would be of this source with another toolchain (ADR 0477 clause 2). Nothing is written.
// now would be of this source with another toolchain (ADR 0584 clause 1, which changes ADR 0477 clause 2). Nothing is written.
b.logf("%s at %s is not built: %s", module, m.FP2, why)
return
}
@@ -425,6 +445,17 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
first := !b.tried[key]
b.tried[key] = true
b.mu.Unlock()
var numbering versions.Numbering
tag := ""
if !trunk && b.Tags != nil && b.refOf(module, m.FP2) != "" {
tag, _ = b.Tags(ctx, delivery.RepositoryName(m.Repository), m.Commit)
}
if (trunk || tag != "") && planErr == nil && !nothing {
var goOn bool
if numbering, goOn = b.number(ctx, module, m, plan, tag); !goOn {
return
}
}
cl, err := b.Claimer.Claim(ctx, module, m.FP2, m.Manifest, ClaimOptions{TakeOver: first, ImageBuilder: imageSeat, Trunk: trunk, Published: b.isPublished,
Stored: b.storedOnce(key)})
if err != nil {
@@ -485,7 +516,14 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
b.logf("%s: the build could not be written: %v", cl.Key, err)
return
}
b.logBuild(LogEnded(module, m, b.refOf(module, m.FP2), b.Node, b.now(), result))
ended := LogEnded(module, m, b.refOf(module, m.FP2), b.Node, b.now(), result)
ended.OutOfStep = b.takeOutOfStep(key) // the generated files the build module's compare found out of step (design 26)
b.logBuild(ended)
if result.State == delivery.BuildDone && b.Numbers != nil && len(numbering.Versions) > 0 {
if err := b.Numbers.Built(ctx, module, m, plan, numbering, result.Artifacts); err != nil {
b.logf("%s: %v", cl.Key, err)
}
}
if result.State == delivery.BuildFailed {
b.logf("%s: failed: %s", cl.Key, result.Error)
}
@@ -506,10 +544,11 @@ func (b *Builder) stale(m delivery.Module, plan Plan) string {
continue // an archive is packed with no toolchain
}
language := buildrun.LanguageOf[bm]
tc, none := delivery.AgreedToolchain(b.Toolchains.Entries(language))
if none != "" {
return fmt.Sprintf("the %s toolchain is not agreed: %s", language, none)
ag := delivery.ToolchainAgreement(b.Toolchains.Entries(language))
if ag.Wait != "" {
return fmt.Sprintf("the %s toolchain is not agreed: %s", language, ag.Wait)
}
tc := ag.Toolchain
want, ok := m.Toolchains[language]
if !ok || want.Version != tc.Version || want.Recipe != tc.Recipe {
return fmt.Sprintf("the fingerprint names %s %s (recipe %s) and the nodes agree on %s (recipe %s)", language, want.Version, want.Recipe, tc.Version, tc.Recipe)
@@ -529,6 +568,8 @@ type lane struct {
toolchain *delivery.ToolchainUsed
digests map[string]string // artifact -> digest
references map[string]string // artifact -> the reference an image was pushed to (a trunk build only)
imports []string // the mesh packages the lane's programs import, as its build module answered them
outOfStep []string // the generated files its build module's compare found out of step, passed or failed
err error
}
@@ -543,7 +584,8 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
repository := delivery.RepositoryName(m.Repository)
var lanes []*lane
used := map[string]delivery.ToolchainUsed{}
var kept []string // build modules whose artifacts stay as the prior done key has them
var sitting []string // the nodes that sit out of a language's build, said in the key's step (ADR 0584 clause 2)
var kept []string // build modules whose artifacts stay as the prior done key has them
for _, bm := range plan.Modules() {
if prior != nil && len(only) > 0 && !only[bm] {
kept = append(kept, bm)
@@ -553,10 +595,12 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
if bm != ImageBuildModule && plan.NeedsToolchain(bm) {
// The ask names the toolchain the fingerprint was computed with; a node building with another refuses it.
language := buildrun.LanguageOf[bm]
tc, none := delivery.AgreedToolchain(b.Toolchains.Entries(language))
if none != "" {
return failed("the %s toolchain is not agreed: %s", language, none)
ag := delivery.ToolchainAgreement(b.Toolchains.Entries(language))
if ag.Wait != "" {
return failed("the %s toolchain is not agreed: %s", language, ag.Wait)
}
tc := ag.Toolchain
sitting = append(sitting, SittingOutWords(language, ag)...)
if want := m.Toolchains[language]; want.Version != tc.Version || want.Recipe != tc.Recipe {
return delivery.BuilderBuild{} // the agreed toolchain moved since try looked: nothing is written
}
@@ -581,6 +625,9 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
go func() { defer wg.Done(); b.runLane(ctx, l, repository, m, module, asked) }()
}
wg.Wait()
if key, err := delivery.BuilderBuildKey(module, m.FP2); err == nil {
b.keepOutOfStep(key, lanes)
}
var errs []string
for _, l := range lanes {
if l.err != nil {
@@ -596,6 +643,10 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
return failed("%s", keyError(errs))
}
var artifacts []delivery.BuiltArtifact
var present, counted map[string][]string
if b.Forms != nil {
present, counted = b.Forms()
}
keptModule := map[string]bool{}
for _, bm := range kept {
keptModule[bm] = true
@@ -624,6 +675,7 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
return failed("%s", err)
}
built := delivery.BuiltArtifact{Name: t.Artifact, Kind: t.Kind, Digest: d}
var answered []string
for _, l := range lanes {
if l.module != t.Module {
continue
@@ -635,10 +687,11 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
return failed("%s%s on %s: the build answered no reference, so its push to the artifact store was not answered", delivery.NotStored, t.Artifact, l.node)
}
if r != "" {
built.Reference = r
answered = append(answered, r)
}
}
artifacts = append(artifacts, built)
// The forms a counted node's engine reads are kept, the oldest in the single reference (ADR 0624 §1, §3).
artifacts = append(artifacts, forms.Current.Choose(built, answered, present, counted))
}
var steps []string
for _, l := range lanes {
@@ -653,10 +706,14 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
}
}
step := "built " + strings.Join(steps, "; ")
if len(sitting) > 0 {
step += " (" + strings.Join(sitting, "; ") + ")"
}
if len(used) == 0 {
used = nil
}
done := delivery.BuilderBuild{State: delivery.BuildDone, By: b.Claimer.By, Manifest: m.Manifest, Step: step, Artifacts: artifacts, Toolchains: used}
done := delivery.BuilderBuild{State: delivery.BuildDone, By: b.Claimer.By, Manifest: m.Manifest, Step: step, Artifacts: artifacts, Toolchains: used,
Imports: importsOf(lanes), Needs: forms.Current.Needs(artifacts)} // the forms it is written in (ADR 0618 §1)
if _, err := delivery.EncodeBuilderBuild(done); err != nil {
// A reference the build module answered that does not name the artifact and its digest is no reference.
return failed("%s%v", delivery.NotStored, err)
@@ -751,7 +808,7 @@ func (n *askNotes) note(ctx context.Context, language, id, step string) error {
n.mu.Lock()
defer n.mu.Unlock()
v := n.cl.Build
v.State, v.By, v.Step, v.Error, v.Artifacts = delivery.BuildBuilding, n.b.Claimer.By, step, "", nil
v.State, v.By, v.Step, v.Error, v.Artifacts, v.Needs = delivery.BuildBuilding, n.b.Claimer.By, step, "", nil, nil
v.Asks = maps.Clone(v.Asks)
if v.Asks == nil {
v.Asks = map[string]string{}
@@ -847,6 +904,12 @@ func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m del
return
}
l.node = step.Node
// The build module's compare of a contracts module's generated files (design 26, CAT-BG5/BN2) answers the files out
// of step as delivery.BuildAnswer.OutOfStep, in a failed step's result too: read before the step's state is.
var compared delivery.BuildAnswer
if len(step.Result) > 0 && json.Unmarshal(step.Result, &compared) == nil {
l.outOfStep = compared.OutOfStep
}
if step.State != delivery.StepPassed {
l.err = fmt.Errorf("%s on %s: the ask %s %s: %s", l.module, step.Node, ask.ID, step.State, step.Reason)
return
@@ -856,6 +919,7 @@ func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m del
l.err = fmt.Errorf("the ask of %s on %s answered what is not a build: %s", l.module, step.Node, step.Result)
return
}
l.imports = answer.Imports
built := map[string]delivery.AskBuilt{}
for _, a := range answer.Artifacts {
built[a.Artifact] = a
@@ -929,7 +993,7 @@ func LogStarted(module string, m delivery.Module, ref, by string, at time.Time)
}
// LogEnded is the log's entry of that build ending: the outcome (done, failed or queued), the manifest built, the bases it
// was built against and each artifact made, as the done key has them: what mesh-catalog registers a build from.
// was built against and each artifact made, as the done key has them: what the operator's command nox builds shows of a build.
func LogEnded(module string, m delivery.Module, ref, by string, at time.Time, result delivery.BuilderBuild) delivery.BuildLogEntry {
e := LogStarted(module, m, ref, by, at)
e.Event, e.Outcome, e.Manifest = delivery.BuildEndedEvent, result.State, m.Manifest
@@ -940,5 +1004,62 @@ func LogEnded(module string, m delivery.Module, ref, by string, at time.Time, re
e.Against = append([]string(nil), m.Inputs2.Bases...)
}
e.Made = append([]delivery.BuiltArtifact(nil), result.Artifacts...)
e.Imports = append([]string(nil), result.Imports...) // the indexer keeps them on the module (novox/hq ADR 0593 §2)
return e
}
// keepOutOfStep holds, by build key, the out-of-step files the lanes' build modules answered, sorted and each once, until
// the build's end entry takes them (takeOutOfStep); a build with none holds nothing.
func (b *Builder) keepOutOfStep(key string, lanes []*lane) {
seen := map[string]bool{}
for _, l := range lanes {
for _, f := range l.outOfStep {
seen[f] = true
}
}
b.mu.Lock()
defer b.mu.Unlock()
if len(seen) == 0 {
delete(b.outOfStep, key)
return
}
if b.outOfStep == nil {
b.outOfStep = map[string][]string{}
}
b.outOfStep[key] = slices.Sorted(maps.Keys(seen))
}
// takeOutOfStep answers the out-of-step files held for a build key and forgets them.
func (b *Builder) takeOutOfStep(key string) []string {
b.mu.Lock()
defer b.mu.Unlock()
files := b.outOfStep[key]
delete(b.outOfStep, key)
return files
}
// importsOf are the mesh packages the lanes' programs import, as their build modules answered them (novox/hq ADR 0593
// §2): sorted and each once; nil when none answered any. A lane kept from a prior build answered none, so a build that
// kept one reports only what it built.
func importsOf(lanes []*lane) []string {
seen := map[string]bool{}
for _, l := range lanes {
for _, p := range l.imports {
seen[p] = true
}
}
if len(seen) == 0 {
return nil
}
return slices.Sorted(maps.Keys(seen))
}
// SittingOutWords says, per node that is not on the agreed toolchain of a language, that it sits out and which version
// it has against the agreed one: "ace sits out: typescript 26.12.1, agreed 26.13.0" (novox/hq ADR 0584 clause 2).
func SittingOutWords(language string, a delivery.Agreement) []string {
var out []string
for _, s := range a.SittingOut {
out = append(out, fmt.Sprintf("%s sits out: %s %s, agreed %s", s.Node, language, s.Version, a.Toolchain.Version))
}
return out
}
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"reflect"
"slices"
"strconv"
"strings"
"sync"
@@ -13,11 +14,13 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-delivery/sdk/forms"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-builder/internal/versions"
)
// buildModules simulates the two build modules on the nodes: a long tool answers a step and key at once and writes the
@@ -140,7 +143,7 @@ type rig struct {
// toolPrefix is the start of each build module's tool names, as the fake build modules answer them.
var toolPrefix = map[string]string{buildrun.GoBuildModule: "build_go_", buildrun.NodeBuildModule: "build_node_", ImageBuildModule: "build_image_"}
// answer runs an ask as the build module that pulls it does (buildstep.RunAsk), with the fake build modules' tools on the
// answer runs an ask as the build module that pulls it does (runAsk), with the fake build modules' tools on the
// rig's pulling node, and answers the step that node writes.
func (r *rig) answer(a delivery.StepAsk) delivery.Step {
node := r.puller
@@ -149,7 +152,7 @@ func (r *rig) answer(a delivery.StepAsk) delivery.Step {
}
module, _ := delivery.BuildModuleOf(a.Language)
runner := buildrun.Runner{Caller: r.bm, Steps: r.bm.steps, Bound: time.Second}
tool := func(name string) buildstep.StepFunc {
tool := func(name string) stepFunc {
return func(ctx context.Context, args map[string]any) (any, error) {
raw, err := runner.Run(ctx, module, node, toolPrefix[module]+name, args)
if err != nil {
@@ -158,8 +161,8 @@ func (r *rig) answer(a delivery.StepAsk) delivery.Step {
return raw, nil
}
}
steps := buildstep.AskSteps{Fetch: tool("fetch"), Build: tool("build"), Copy: tool("copy")}
out, err := buildstep.RunAsk(context.Background(), steps, a)
steps := askSteps{Fetch: tool("fetch"), Build: tool("build"), Copy: tool("copy")}
out, err := runAsk(context.Background(), steps, a)
step := delivery.Step{ID: a.ID, Tool: a.Kind + " ask", Node: node, State: delivery.StepPassed}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
@@ -222,7 +225,18 @@ var (
)
func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7, ADR 0478 clauses 1, 4
t.Run("the imports a build answers reach the done key and the log", importsABuildAnswersReachTheDoneKeyAndTheLog)
t.Run("the out-of-step files a build answers reach the log", outOfStepFilesABuildAnswersReachTheLog)
t.Run("ADR 0624: a done key keeps each form a counted node reads", aDoneKeyKeepsEachFormACountedNodeReads)
r := newRig(t, goBundle, archiveA, tsBundle)
// ADR 0584: a third node agrees with novox and ace's typescript is the odd one; the build is done and its step says ace sits out.
r.tools.Put(buildrun.NodeBuildModule, "g14.typescript", tcValue(t, "typescript", "tsc5"), false)
r.tools.Put(buildrun.NodeBuildModule, "ace.typescript", tcValue(t, "typescript", "tsc4"), false)
defer func() {
if st := r.st.vals[r.key]; !strings.Contains(string(st), "ace sits out: typescript tsc4, agreed tsc5") {
t.Errorf("the key does not say ace sits out: %s", st)
}
}()
var logged []delivery.BuildLogEntry
var lmu sync.Mutex
r.b.Node, r.b.Log = "novox", func(e delivery.BuildLogEntry) { lmu.Lock(); logged = append(logged, e); lmu.Unlock() }
@@ -270,6 +284,11 @@ func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7
if v.Artifacts[0].Kind != "archive" || v.Artifacts[1].Kind != "bundle" {
t.Errorf("kinds = %+v", v.Artifacts)
}
// ADR 0618 §1: the done key names the forms its references are written in, here the first form alone, and keeps no list
// beside the single reference while no counted node needs another form.
if !slices.Equal(v.Needs, []string{"form:artifact-store"}) || v.Artifacts[0].References != nil {
t.Errorf("needs = %v, references = %v", v.Needs, v.Artifacts[0].References)
}
if got := v.Toolchains["go"]; got.Version != "go1.27.1" || got.Recipe != "1" {
t.Errorf("toolchains = %+v, want go and typescript named", v.Toolchains)
}
@@ -310,7 +329,7 @@ func TestATrunkFingerprintIsOneAskOnTheDeliveryStream(t *testing.T) {
}
r.b.Module(context.Background(), "m", data)
var asks []delivery.StepAsk
for deadline := time.Now().Add(2 * time.Second); len(asks) == 0; time.Sleep(time.Millisecond) {
for deadline := time.Now().Add(10 * time.Second); len(asks) == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no ask was put")
}
@@ -410,8 +429,19 @@ func TestACandidatesImageCallCarriesNoPublishAndItsKeyNoReference(t *testing.T)
r := newRig(t, imageArt)
r.puller = "ace"
trunkModule(t, r)
// DLV-B6: a check's build of a head a kept tag ask names is numbered under that tag at its claim.
nums := &fakeNumbers{}
var tagAsked []string
r.b.Numbers = nums
r.b.Tags = func(_ context.Context, repository, head string) (string, bool) {
tagAsked = append(tagAsked, repository+"@"+head)
return "feat-x", head == headX
}
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
if !slices.Equal(nums.tags, []string{"feat-x"}) || len(tagAsked) != 1 || !strings.HasSuffix(tagAsked[0], "@"+headX) {
t.Errorf("the check's build numbered with tags %v (asked %v), want feat-x for its head", nums.tags, tagAsked)
}
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 || v.Artifacts[0].Reference != "" {
t.Fatalf("build = %+v, want done with no reference", v)
@@ -425,14 +455,69 @@ func TestACandidatesImageCallCarriesNoPublishAndItsKeyNoReference(t *testing.T)
}
}
func TestATrunkImageCallCarriesPublish(t *testing.T) { // 1705b
func TestATrunkImageCallCarriesPublish(t *testing.T) { // 1705b; design 51 *The versions*: numbered at the claim
r := newRig(t, imageArt)
r.puller = "ace"
nums := &fakeNumbers{wait: "m/img: the package waits for its floor"}
r.b.Numbers = nums
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
if _, ok := r.st.vals[r.key]; ok || nums.numbered != 1 {
t.Fatalf("a build waiting for its floor was claimed (numbered %d times)", nums.numbered)
}
nums.wait, nums.versions = "", map[string]string{"img": "0.1.4"}
r.consider(t, "modules/m")
calls := r.bm.callsOf("build_image_build")
if len(calls) != 1 || calls[0].body.(map[string]any)["publish"] != true {
t.Errorf("calls = %+v, want publish true", calls)
}
if nums.built["img"] != digest {
t.Errorf("the built digest recorded = %v, want %s for img", nums.built, digest)
}
if nums.done != nil {
t.Errorf("a build still to make was numbered with done digests %v", nums.done)
}
// The indexer's packages state names a package of the module (PackageNamed): the done build is looked at again and
// numbered with its key's digests, so version.numbered carries them for the late publish; nothing is built again.
looks := nums.numbered
r.b.PackageNamed(context.Background(), "m")
r.b.Wait()
if nums.numbered != looks+1 || nums.done["img"] != digest || len(r.bm.callsOf("build_image_build")) != 1 {
t.Errorf("a named package's done build: numbered %d more times with %v, %d image builds; want once with img %s and no build",
nums.numbered-looks, nums.done, len(r.bm.callsOf("build_image_build")), digest)
}
r.b.PackageNamed(context.Background(), "other")
r.b.Wait()
if nums.numbered != looks+1 {
t.Errorf("a module the builder does not know was looked at")
}
}
// fakeNumbers is Numbers that answers what it is set to and keeps the digests a done build recorded.
type fakeNumbers struct {
wait string
versions map[string]string
numbered int
built map[string]string
tags []string
done map[string]string // the done key's digests the last look numbered with
}
func (f *fakeNumbers) Number(_ context.Context, _ string, _ delivery.Module, _ Plan, tag string, done map[string]string) (versions.Numbering, error) {
f.tags = append(f.tags, tag)
f.done = done
f.numbered++
return versions.Numbering{Wait: f.wait, Versions: f.versions}, nil
}
func (f *fakeNumbers) Built(_ context.Context, _ string, _ delivery.Module, _ Plan, got versions.Numbering, made []delivery.BuiltArtifact) error {
f.built = map[string]string{}
for _, a := range made {
if got.Versions[a.Name] != "" {
f.built[a.Name] = a.Digest
}
}
return nil
}
func TestAnUnpinnedUpstreamFailsNotPinnedAndIsNeverBuiltAgain(t *testing.T) { // B3, ADR 0498 clause 4
@@ -557,6 +642,7 @@ func TestAKeyIsBuiltOnlyWithTheToolchainItsFingerprintNamesAndNothingIsWrittenOt
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", old))
r.b.Wait()
if len(r.st.vals) != 0 || len(r.bm.calls) != 0 {
t.Run("a recipe that moves builds the module again under its new fingerprint", aRecipeThatMovesBuildsTheModuleAgainUnderItsNewFingerprint)
t.Errorf("keys %v and %d calls for a fingerprint of go1.26.0 while the nodes agree on go1.27.1, want neither", r.st.vals, len(r.bm.calls))
}
// A recipe that differs is a different toolchain too, and a value that names none (written before) is not built.
@@ -581,6 +667,45 @@ func TestAKeyIsBuiltOnlyWithTheToolchainItsFingerprintNamesAndNothingIsWrittenOt
}
}
// ADR 0603 §3 (D9): a source revision changing the Go build module's recipe and a Go module builds the module at once with
// the recipe the nodes run and again, a second key under the new fingerprint, once the agreement moves to the new recipe;
// the first key is neither rebuilt nor waited on.
func aRecipeThatMovesBuildsTheModuleAgainUnderItsNewFingerprint(t *testing.T) {
r := newRig(t, goBundle)
r.consider(t, "modules/m")
calls := len(r.bm.calls)
fp2 := "src2:" + hex64('2')
for _, n := range []string{"ace", "novox"} {
b, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "go", Version: "go1.27.1", Recipe: "2", At: time.Now()})
if err != nil {
t.Fatal(err)
}
r.tools.Put(buildrun.GoBuildModule, n+".go", b, false)
}
r.b.Reconsider(context.Background())
r.b.Wait()
if len(r.bm.calls) != calls {
t.Errorf("%d calls, was %d: the first fingerprint was built again for the new recipe", len(r.bm.calls), calls)
}
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP: "src1:" + hex64('f'), FP2: fp2,
Manifest: mdigest('a'), Path: "modules/m", Toolchains: map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "2"}}})
if err != nil {
t.Fatal(err)
}
r.b.Module(context.Background(), "m", data)
r.b.Wait()
second, err := delivery.BuilderBuildKey("m", fp2)
if err != nil {
t.Fatal(err)
}
if v := valueOf(t, r.st, second); v.State != delivery.BuildDone || v.Toolchains["go"].Recipe != "2" {
t.Errorf("second build = %+v, want done with recipe 2", v)
}
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone {
t.Errorf("first build = %+v, want it left done", v)
}
}
func TestNodesTheyDisagreeOnLeaveTheKeyUnwritten(t *testing.T) { // 1615/AC4
r := newRig(t, goBundle)
r.tools.Put(buildrun.GoBuildModule, "ace.go", tcValue(t, "go", "go1.26.0"), false)
@@ -697,7 +822,7 @@ func (g *goneAsks) asked() int {
// waitForAsks waits until n asks were put.
func waitForAsks(t *testing.T, r *rig, n int) []delivery.StepAsk {
t.Helper()
for deadline := time.Now().Add(2 * time.Second); len(r.asks.Asks()) < n; time.Sleep(time.Millisecond) {
for deadline := time.Now().Add(10 * time.Second); len(r.asks.Asks()) < n; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("%d asks put, want %d", len(r.asks.Asks()), n)
}
@@ -717,7 +842,13 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
r.b.Gone, r.b.CheckAfter = gone, 5*time.Millisecond
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
waitForAsks(t, r, 1)
time.Sleep(100 * time.Millisecond)
// Five check bounds passed (five reads of whether it can be answered), so the ask waited past the step bound: an event
// to wait on, where a pause of 100 ms said nothing on a loaded machine.
for deadline := time.Now().Add(10 * time.Second); gone.asked() < 5; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("whether the ask can still be answered was read %d times within 10 s, want 5", gone.asked())
}
}
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildBuilding {
t.Fatalf("the key = %+v while its ask waits, want building", v)
}
@@ -730,6 +861,30 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
t.Fatalf("the key = %+v after %d asks, want done from the one ask", v, len(r.asks.Asks()))
}
})
t.Run("declined by a node off the agreed toolchain: put again, no asking used, the bound ends it loudly", func(t *testing.T) { // ADR 0584 clause 2
r := newRig(t, goBundle)
r.asks.Answer = nil
// The builder's clock steps a second at each read, so the bound of a minute ends the declines after many asks however
// slow the machine (a bound of 150 ms of wall clock let a loaded one put fewer than DefaultAskings).
var clock atomic.Int64
start := time.Now()
r.b.Now = func() time.Time { return start.Add(time.Duration(clock.Add(1)) * time.Second) }
r.b.DeclinePause, r.b.DeclineBound = -1, time.Minute
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
declined, _ := json.Marshal(delivery.StepDeclined{Declined: true, Lacks: []string{"the agreed toolchain go1.27.1 (recipe 1)"}})
waitForAsks(t, r, 1)
for ended, deadline := 0, time.Now().Add(20*time.Second); valueOf(t, r.st, r.key).State == delivery.BuildBuilding && time.Now().Before(deadline); time.Sleep(5 * time.Millisecond) {
if asks := r.asks.Asks(); len(asks) > ended { // each ask put again is declined in its turn
r.asks.End(delivery.Step{ID: asks[ended].ID, Node: "ace", State: delivery.StepRefused, Reason: "this node builds with go1.26.0", Result: declined})
ended++
}
}
r.b.Wait()
v := valueOf(t, r.st, r.key)
if n := len(r.asks.Asks()); v.State != delivery.BuildFailed || n <= DefaultAskings || !strings.Contains(v.Error, "declined for") || !strings.Contains(v.Error, "ace: this node builds with go1.26.0") {
t.Fatalf("the key = %+v after %d asks, want it failed after more than %d asks, naming ace and its reason", v, n, DefaultAskings)
}
})
for _, c := range []struct {
name string
askings int
@@ -899,7 +1054,7 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
}
if c.byHand {
waitForAsks(t, r, 1)
for deadline := time.Now().Add(2 * time.Second); len(r.b.AskAgain(r.key, "stuck since 19:06")) == 0; time.Sleep(time.Millisecond) {
for deadline := time.Now().Add(10 * time.Second); len(r.b.AskAgain(r.key, "stuck since 19:06")) == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no lane of the key waits")
}
@@ -907,7 +1062,7 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
}
if c.deaf {
r.asks.End(r.answer(waitForAsks(t, r, 1)[0]))
for deadline := time.Now().Add(2 * time.Second); heard.Waiting() == 0; time.Sleep(time.Millisecond) {
for deadline := time.Now().Add(10 * time.Second); heard.Waiting() == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no ask is waited on")
}
@@ -1156,3 +1311,117 @@ type silentWatch struct{}
func (silentWatch) WatchAsk(context.Context, string, string) (<-chan delivery.Step, error) {
return make(chan delivery.Step), nil
}
// aDoneKeyKeepsEachFormACountedNodeReads holds the builder's rows of ADR 0624's check: every build keeps each form a
// counted node's engine reads, present or away, a node stating nothing reading the baseline's first form; the oldest kept
// form is the single reference; a retired node leaves the count. old is the first form, new a fixture second form read
// only by an engine stating form:new.
func aDoneKeyKeepsEachFormACountedNodeReads(t *testing.T) {
fs := forms.ArtifactForms{
{Name: "old", First: true, Kinds: []string{"bundle"}, Is: func(r string) bool { return strings.HasPrefix(r, "artifact-store://") }},
{Name: "new", Kinds: []string{"bundle"}, Is: func(r string) bool { return strings.HasPrefix(r, "10.0.0.1:5000/") }},
}
both, nothing, onlyNew := []string{"form:new", "word:module"}, []string(nil), []string{"form:new"}
oldRef, newRef := "artifact-store://m/code/blobs/"+digest, "10.0.0.1:5000/m/code@"+digest
for _, row := range []struct {
name string
present, counted map[string][]string
kept []string
reference string
references []string
}{
{"A present {old,new}, B away {old}: both forms, the old in the single reference",
map[string][]string{"a": both}, map[string][]string{"a": both, "b": {"word:module"}}, []string{"old", "new"}, oldRef, []string{oldRef, newRef}},
{"A present {new}, B away stating nothing: the baseline's form stays",
map[string][]string{"a": onlyNew}, map[string][]string{"a": onlyNew, "b": nothing}, []string{"old", "new"}, oldRef, []string{oldRef, newRef}},
{"A present {old,new}, B away and never asked (no key): the baseline's form stays",
map[string][]string{"a": both}, map[string][]string{"a": both, "b": nil}, []string{"old", "new"}, oldRef, []string{oldRef, newRef}},
{"B retired: the new form only", map[string][]string{"a": both}, map[string][]string{"a": both}, []string{"new"}, newRef, nil},
{"B present stating nothing: the new form is not written",
map[string][]string{"a": both, "b": nothing}, map[string][]string{"a": both, "b": nothing}, []string{"old"}, oldRef, nil},
{"no node present: the first form", nil, map[string][]string{"a": both}, []string{"old"}, oldRef, nil},
} {
if got := fs.Kept("bundle", row.present, row.counted); !slices.Equal(got, row.kept) {
t.Errorf("%s: kept %v, want %v", row.name, got, row.kept)
}
a := fs.Choose(delivery.BuiltArtifact{Name: "code", Kind: "bundle", Digest: digest}, []string{newRef, oldRef}, row.present, row.counted)
if a.Reference != row.reference || !slices.Equal(a.References, row.references) {
t.Errorf("%s: %q %v, want %q %v", row.name, a.Reference, a.References, row.reference, row.references)
}
// An engine that predates the list reads the single reference, in a form every counted node reads (ADR 0624 §3).
if form, _ := fs.FormOf("bundle", a.Reference); form != row.kept[0] {
t.Errorf("%s: the single reference is in %q, want the oldest kept %q", row.name, form, row.kept[0])
}
}
// An answered reference no kept form names still names where the artifact is (ADR 0498 clause 1).
if a := fs.Choose(delivery.BuiltArtifact{Name: "code", Kind: "bundle", Digest: digest}, []string{oldRef}, map[string][]string{"a": both},
map[string][]string{"a": both}); a.Reference != oldRef {
t.Errorf("a lone answered reference was dropped: %+v", a)
}
withBoth := []delivery.BuiltArtifact{{Name: "code", Kind: "bundle", Digest: digest, Reference: oldRef, References: []string{oldRef, newRef}}}
if got := fs.Needs(withBoth); !slices.Equal(got, []string{"form:new", "form:old"}) {
t.Errorf("needs %v", got)
}
// The builder's own forms (forms.Current): a bundle a build step answers in the registry seat's form (mesh-build-go's
// oci-artifact-registry/<module>/<artifact>@<digest>) keeps the artifact-store form it derives for each counted node
// whose engine does not state form:seat-reference, that one in the single reference (ADR 0624 §1, §3).
seat := []string{forms.FormNeed(forms.FormSeatReference), "word:module"}
// Each kind the seat form names (a bundle through oci-artifact-registry, an image and an upstream copy through
// container-image-registry, the copy under the holder's own upstream/<host>/<path>), with the artifact-store form it derives.
for _, k := range []struct{ kind, seatRef, storeRef string }{
{"bundle", "oci-artifact-registry/m/code@" + digest, "artifact-store://m/code/blobs/" + digest},
{"image", "container-image-registry/m/code@" + digest, "artifact-store://m/code@" + digest},
{"upstream", "container-image-registry/upstream/quay.io:443/org/app@" + digest, "artifact-store://upstream/quay.io:443/org/app@" + digest},
} {
seatRef, storeRef := k.seatRef, k.storeRef
for _, row := range []struct {
name string
present, counted map[string][]string
reference string
references []string
needs []string
}{
{"seat form answered, B away stating nothing: the derived store form stays, in the single reference",
map[string][]string{"a": seat}, map[string][]string{"a": seat, "b": nil}, storeRef, []string{storeRef, seatRef},
[]string{"form:artifact-store", "form:seat-reference"}},
{"seat form answered, every counted engine reads it: the seat form only",
map[string][]string{"a": seat}, map[string][]string{"a": seat, "b": seat}, seatRef, nil, []string{"form:seat-reference"}},
{"seat form answered, a present engine stating nothing: the store form only",
map[string][]string{"a": seat, "b": nil}, map[string][]string{"a": seat, "b": nil}, storeRef, nil, []string{"form:artifact-store"}},
{"seat form answered, no node present: the store form", nil, map[string][]string{"a": seat}, storeRef, nil, []string{"form:artifact-store"}},
} {
a := forms.Current.Choose(delivery.BuiltArtifact{Name: "code", Kind: k.kind, Digest: digest}, []string{seatRef}, row.present, row.counted)
if a.Reference != row.reference || !slices.Equal(a.References, row.references) {
t.Errorf("%s %s: %q %v, want %q %v", k.kind, row.name, a.Reference, a.References, row.reference, row.references)
}
needs := forms.Current.Needs([]delivery.BuiltArtifact{a})
if !slices.Equal(needs, row.needs) {
t.Errorf("%s %s: needs %v, want %v", k.kind, row.name, needs, row.needs)
}
for node, stated := range row.counted {
if forms.Current.Held(needs, stated) {
t.Errorf("%s %s: the want of %s is held", k.kind, row.name, node)
}
}
if _, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, Manifest: strings.TrimPrefix(digest, "sha256:"), Artifacts: []delivery.BuiltArtifact{a}, Needs: needs}); err != nil {
t.Errorf("%s %s: %v", k.kind, row.name, err)
}
}
}
// What the assigner holds a want by (ADR 0624 §2), from the same forms.
for _, row := range []struct {
name string
needs []string
stated []string
held bool
}{
{"a node stating {old}, a build carrying {old,new}: moves", []string{"form:new", "form:old"}, []string{"word:module"}, false},
{"a node stating nothing, a build carrying the baseline's form: moves", []string{"form:old"}, nil, false},
{"a build carrying {new} only, the node stating {old}: held", []string{"form:new"}, []string{"word:module"}, true},
{"a build naming no form: moves", nil, nil, false},
} {
if got := fs.Held(row.needs, row.stated); got != row.held {
t.Errorf("%s: held %v", row.name, got)
}
}
}
@@ -1,6 +1,8 @@
package build
import (
"mesh-delivery-builder/internal/versions"
"encoding/json"
"errors"
"fmt"
@@ -11,8 +13,8 @@ import (
"sync/atomic"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// The builder's tools (novox/hq issue 1945; the instruction "tools are how a module is spoken to"): builder_stuck reads
@@ -54,6 +56,67 @@ type ReAsk struct {
Failed string `json:"failed,omitempty"`
Revision uint64 `json:"revision,omitempty"`
Asks map[string]string `json:"asks,omitempty"`
// Name is the numbered name (`<module>/<artifact>`) of a ModePublishAgain or ModeFloorAgain value; Do and Version the
// builder's answer: what it did (versions.Again*) and the version published.
Name string `json:"name,omitempty"`
Do string `json:"do,omitempty"`
Version string `json:"version,omitempty"`
}
// ModePublishAgain is the Mode of a reasks value of builder_publish_again (design 51 *Tools of the builder*).
const ModePublishAgain = "publish-again"
// ModeFloorAgain is the Mode of a reasks value of builder_floor_again (design 51 *The floor*).
const ModeFloorAgain = "floor-again"
// NewPublishAgain is the value builder_publish_again writes for a name's fingerprint: the key of the name's module and
// fingerprint, and the value.
func NewPublishAgain(name, fp, why, by string, at time.Time) (string, []byte, error) {
module, artifact, ok := strings.Cut(strings.TrimSpace(name), "/")
if !ok || module == "" || artifact == "" {
return "", nil, fmt.Errorf("name %q is not a numbered name, <module>/<artifact>, as builder_versions lists it", name)
}
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return "", nil, err
}
if strings.TrimSpace(why) == "" {
return "", nil, errors.New("say why it is published again (why)")
}
data, err := json.Marshal(ReAsk{V: 1, Why: strings.TrimSpace(why), By: by, At: at.UTC(), Mode: ModePublishAgain, Module: module, FP: fp, Name: strings.TrimSpace(name)})
return key, data, err
}
// NewFloorAgain is the value builder_floor_again writes for a name: the key `floor.<name escaped>`, never a build key,
// and the value.
func NewFloorAgain(name, why, by string, at time.Time) (string, []byte, error) {
name = strings.TrimSpace(name)
if module, artifact, ok := strings.Cut(name, "/"); !ok || module == "" || artifact == "" {
return "", nil, fmt.Errorf("name %q is not a numbered name, <module>/<artifact>, as builder_versions lists it", name)
}
if strings.TrimSpace(why) == "" {
return "", nil, errors.New("say why the floor is asked again (why)")
}
data, err := json.Marshal(ReAsk{V: 1, Why: strings.TrimSpace(why), By: by, At: at.UTC(), Mode: ModeFloorAgain, Name: name})
return "floor." + versions.EscapeToken(name), data, err
}
// ModeOldFormDrop is the Mode of a reasks value of builder_old_form's drop (design 51 *Every build keeps the old
// reference form*): a retired node dropped from the count by hand; Name is the node.
const ModeOldFormDrop = "old-form-drop"
// NewOldFormDrop is the value builder_old_form writes to drop a retired node: the key `old-form.<node>`, never a build
// key, and the value.
func NewOldFormDrop(node, why, by string, at time.Time) (string, []byte, error) {
node = strings.TrimSpace(node)
if node == "" || strings.ContainsAny(node, ". *>") {
return "", nil, fmt.Errorf("node %q is not a node's name", node)
}
if strings.TrimSpace(why) == "" {
return "", nil, errors.New("say why the node leaves the count (why)")
}
data, err := json.Marshal(ReAsk{V: 1, Why: strings.TrimSpace(why), By: by, At: at.UTC(), Mode: ModeOldFormDrop, Name: node})
return "old-form." + node, data, err
}
// ModeBuild is the Mode of a reasks value that asks the key to be built again, not its waiting asks to be asked again.
@@ -0,0 +1,351 @@
package versions
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// What builder_publish_again does with one fingerprint of a name (design 51 *Tools of the builder*), after a holder failed.
const (
// AgainPublish says version.numbered again for the recorded version: the build module publishes the held digest.
AgainPublish = "publish"
// AgainBuild builds the fingerprint again under the same number: the build's file is gone and no published was said
// for the number, so the new digest replaces the recorded one (Numberer.Built).
AgainBuild = "build"
// AgainRenumber takes a new number: a published was said for the number, so other bytes never go under it.
AgainRenumber = "renumber"
)
// Again is what builder_publish_again does, with the entry it acts on and why.
type Again struct {
Do string `json:"do"`
Entry Entry `json:"entry"`
Why string `json:"why"`
}
// PublishAgain decides builder_publish_again for fingerprint among a name's entries; built says whether the builder still
// holds the build's file (its done key). It refuses a fingerprint not numbered, and a version refused for other bytes
// than the ones recorded under it.
func PublishAgain(entries []Entry, fingerprint string, built bool) (Again, error) {
fingerprint = strings.TrimSpace(fingerprint)
if fingerprint == "" {
return Again{}, errors.New("fp is required: the fingerprint whose version is published again")
}
for _, e := range latestFirst(entries) {
if e.Tag != "" || e.Fingerprint != fingerprint {
continue
}
switch {
case e.State == Refused && e.RefusedDigest != "" && e.Digest != "" && e.RefusedDigest != e.Digest:
return Again{}, fmt.Errorf("%s@%s was refused for other bytes (%s, recorded %s): it is not published again; a new build takes a new number",
e.Name, e.Version, e.RefusedDigest, e.Digest)
case e.State == Published || e.State == Removed:
return Again{Do: AgainRenumber, Entry: e, Why: fmt.Sprintf("%s@%s was %s, so the fingerprint takes a new number", e.Name, e.Version, e.State)}, nil
case !built:
return Again{Do: AgainBuild, Entry: e, Why: fmt.Sprintf("the build's file is gone and no published was said for %s@%s: built again under the same number", e.Name, e.Version)}, nil
}
return Again{Do: AgainPublish, Entry: e, Why: fmt.Sprintf("%s@%s (%s) is said numbered again, so its build module publishes the held digest", e.Name, e.Version, e.State)}, nil
}
return Again{}, fmt.Errorf("fingerprint %s is not numbered for this name: builder_versions lists what is", fingerprint)
}
// The builder's state of tag asks (design 51 *Tagged versions*): one key per repository and branch, removed when the
// branch is merged or deleted or its pull request closed.
const (
TagAsksState = "tag-asks"
TagAsksBucket = "mesh-delivery-builder_tag-asks"
)
// The states of a tag ask.
const (
TagAsked = "asked"
TagNumbered = "numbered"
TagRemoving = "removing"
TagRemoved = "removed"
)
// TagAsk is one value of tag-asks: an ask of a tagged version for a branch's head, from the checker's tag-wanted or by
// hand (builder_tag), or by hand the removal of the branch's tagged versions (builder_untag, Untag).
type TagAsk struct {
V int `json:"v"`
Repository string `json:"repository"`
Branch string `json:"branch"`
Head string `json:"head,omitempty"`
Tag string `json:"tag"`
By string `json:"by,omitempty"`
Why string `json:"why,omitempty"`
At time.Time `json:"at"`
Untag bool `json:"untag,omitempty"`
// State, Versions (name to version), Left (what a removal still waits on, name@version to its seat) and Answer are
// the builder's.
State string `json:"state,omitempty"`
Versions map[string]string `json:"versions,omitempty"`
Left map[string]string `json:"left,omitempty"`
Answer string `json:"answer,omitempty"`
Answered time.Time `json:"answered,omitzero"`
}
// TagAskKey is the key of a repository's branch in tag-asks.
func TagAskKey(repository, branch string) string {
return EscapeToken(repository) + "." + EscapeToken(branch)
}
// NewTagAsk is the value builder_tag writes: its key and the value. The tag is the branch's, cleaned (CleanTag).
func NewTagAsk(repository, branch, head, why, by string, at time.Time) (string, []byte, error) {
return newTagAsk(repository, branch, head, why, by, at, false)
}
// NewUntag is the value builder_untag writes: its key and the value.
func NewUntag(repository, branch, why, by string, at time.Time) (string, []byte, error) {
return newTagAsk(repository, branch, "", why, by, at, true)
}
func newTagAsk(repository, branch, head, why, by string, at time.Time, untag bool) (string, []byte, error) {
repository, branch, head, why = strings.TrimSpace(repository), strings.TrimSpace(branch), strings.TrimSpace(head), strings.TrimSpace(why)
switch {
case repository == "" || !strings.Contains(repository, "/"):
return "", nil, errors.New("repository is required: <owner>/<name>")
case branch == "":
return "", nil, errors.New("branch is required")
case !untag && head == "":
return "", nil, errors.New("head is required: the commit the tagged version is built from")
case why == "":
return "", nil, errors.New("say why (why): a hand act is recorded")
}
data, err := json.Marshal(TagAsk{V: 1, Repository: repository, Branch: branch, Head: head, Tag: CleanTag(branch), By: by, Why: why,
At: at.UTC(), Untag: untag, State: TagAsked})
return TagAskKey(repository, branch), data, err
}
// DecodeTagAsk reads a tag-asks value.
func DecodeTagAsk(data []byte) (TagAsk, error) {
var a TagAsk
if err := json.Unmarshal(data, &a); err != nil {
return a, err
}
if a.V != 1 {
return a, fmt.Errorf("a tag-asks value of version %d, not 1", a.V)
}
return a, nil
}
// TagAsksAnswer is what builder_tag_asks answers: the open asks, oldest first.
type TagAsksAnswer struct {
Asks []TagAsk `json:"asks"`
}
// AnswerTagAsks is builder_tag_asks over the state read, only repository's when it is given.
func AnswerTagAsks(r Reader, repository string) (TagAsksAnswer, error) {
keys, err := r.Keys()
if err != nil {
return TagAsksAnswer{}, fmt.Errorf("%s cannot be listed: %w", TagAsksState, err)
}
out := TagAsksAnswer{Asks: []TagAsk{}}
for _, k := range keys {
raw, ok, err := r.Get(k)
if err != nil {
return TagAsksAnswer{}, fmt.Errorf("%s.%s: %w", TagAsksState, k, err)
}
if !ok {
continue
}
a, err := DecodeTagAsk(raw)
if err != nil {
return TagAsksAnswer{}, fmt.Errorf("%s.%s holds what is not a tag ask: %w", TagAsksState, k, err)
}
if repository == "" || a.Repository == repository {
out.Asks = append(out.Asks, a)
}
}
sort.Slice(out.Asks, func(i, j int) bool { return out.Asks[i].At.Before(out.Asks[j].At) })
return out, nil
}
// Untagged are repository's tagged versions of tag that are not removed (a group's branch spans repositories, each
// ending apart): what a branch's end, or builder_untag, removes through
// each one's seat (design 51 *Tagged versions*). A version with no tag is never among them.
func Untagged(entries []Entry, repository, tag string) []Entry {
tag = CleanTag(tag)
var out []Entry
for _, e := range entries {
if e.Tag != "" && e.Repository == repository && CleanTag(e.Tag) == tag && e.State != Removed && e.State != Kept {
out = append(out, e)
}
}
return InOrder(out)
}
// The tools' descriptions.
const (
PublishAgainDescription = "Publish one fingerprint's version again after its holder failed (design 51): with the build's file held, the " +
"builder says version.numbered again and the build module publishes the held digest; with the file gone, it builds the fingerprint " +
"again under the same number, the new digest replacing the recorded one, since no published was said for it; after a published (or " +
"a removal) it takes a new number. It refuses a fingerprint not numbered and a version the holder refused for other bytes. A hand " +
"act, recorded with its caller and why in the builder's log."
FloorAgainDescription = "Ask the floor of one name again now, by hand (design 51 *The floor*): the seat's versions, the forge bridge's " +
"bridge_tags and the indexer's indexer_requirements, as the builder asks them by itself when a silent source says something. " +
"Answers which sources were silent and whether the floor is now known. A name the builder's versions already hold needs no floor: " +
"refused, saying so. A hand act."
TagDescription = "Ask a tagged version for a branch's head by hand, as the checker's tag-wanted asks it (design 51 *Tagged versions*): " +
"the builder keeps the ask in its state tag-asks until the branch is merged or deleted, or its pull request closed, and numbers " +
"each package the head builds below the next trunk version (0.1.15-<tag>.2). A hand act."
TagAsksDescription = "The open tag asks of the builder's state tag-asks, oldest first: repository, branch, head, tag, who asked and why, " +
"its state (asked, numbered, removing, removed), the versions numbered for it and, while its tagged versions are removed, the " +
"seats that have not yet said removed. With repository, that repository's only. Read-only."
UntagDescription = "Remove a branch's tagged versions by hand, as its merge, its pull request's close or its deletion does (design 51 " +
"*Tagged versions*): the builder calls each seat's remove with the reason tag-gone and marks each version removed once its seat says " +
"removed. Answers the versions it removes and the seats still waited on. A version with no tag is never removed. A hand act."
)
// Acts are the builder's hand acts on its versions, answered by its program (the tools bundle writes them to reasks).
type Acts struct {
Numberer *Numberer
// Done says whether the build key of module and fingerprint is done: the build's file is held.
Done func(ctx context.Context, module, fp string) (bool, error)
// BuildAgain builds a failed key again (builder_build_again's path): its answer, or why it refused.
BuildAgain func(ctx context.Context, module, fp, why, by string) (answer string, refused bool)
// Reconsider looks at every module again, so a claim asks a name's floor.
Reconsider func(ctx context.Context)
// Remove calls a seat's verb remove for one tagged version, reason tag-gone: what it answered, kept or asked.
Remove func(ctx context.Context, seat string, e Entry) (RemoveOutcome, error)
// DropOldForm drops a retired node from the old form's count by hand (builder_old_form): its answer, or why not.
DropOldForm func(ctx context.Context, node, by, why string, at time.Time) (string, error)
}
// TagAsked answers one unanswered value of tag-asks. An untag calls remove on each of the branch's tagged versions'
// seats and leaves each in Left until its seat says removed (Heard, on the seat's events); a call that fails stays in
// Left with its error, asked again by the branch's end. A tag ask is kept: its versions are numbered when the head's
// packages are claimed for the group's check.
func (a Acts) TagAsked(ctx context.Context, t TagAsk, now time.Time) TagAsk {
t.Answered = now.UTC()
if !t.Untag {
t.State = TagAsked
t.Answer = "kept: each package the head builds is numbered " + "<next>-" + t.Tag + ".<n> when its build is claimed for the check"
return t
}
all, err := a.Numberer.Versions.All(ctx)
if err != nil {
t.Answer = "nothing removed: " + err.Error()
return t
}
t.Left, t.Versions = map[string]string{}, map[string]string{}
for _, e := range Untagged(all, t.Repository, t.Tag) {
t.Versions[e.Name] = e.Version
at := e.Name + "@" + e.Version
t.Left[at] = e.Seat
if left, ok := a.removeOne(ctx, e); ok {
delete(t.Left, at)
} else if left != "" {
t.Left[at] = left
}
}
if len(t.Left) == 0 {
t.State, t.Left, t.Answer = TagRemoved, nil, "the branch holds no tagged version: nothing to remove"
return t
}
t.State = TagRemoving
t.Answer = fmt.Sprintf("remove asked of %d tagged versions; each is marked removed when its seat says removed", len(t.Left))
return t
}
// removeOne asks e's seat to remove it: done when the seat kept it whole (recorded Kept, no retry); otherwise left, with
// the error of a call that failed.
func (a Acts) removeOne(ctx context.Context, e Entry) (left string, done bool) {
got, err := a.Remove(ctx, e.Seat, e)
if err != nil {
return e.Seat + ": " + err.Error(), false
}
if !got.Kept {
return "", false
}
if err := a.Numberer.keep(ctx, e, got.Reason, got.Tags); err != nil {
return e.Seat + ": kept, and not recorded so: " + err.Error(), false
}
return "", true
}
// PublishAgain is builder_publish_again of name's fingerprint: what it did, the version, and its answer; an error refuses.
func (a Acts) PublishAgain(ctx context.Context, name, fp, why, by string, at time.Time) (Again, string, error) {
module, _, _ := strings.Cut(name, "/")
entries, err := a.Numberer.Versions.Of(ctx, name)
if err != nil {
return Again{}, "", err
}
done, err := a.Done(ctx, module, fp)
if err != nil {
return Again{}, "", fmt.Errorf("the build key of %s at %s cannot be read: %w", module, fp, err)
}
ag, err := PublishAgain(entries, fp, done)
if err != nil {
return Again{}, "", err
}
e := ag.Entry
if ag.Do == AgainRenumber {
_, ne, err := a.Numberer.Versions.Number(ctx, Claim{Name: e.Name, Kind: e.Kind, Series: e.Series, Module: e.Module, Fingerprint: e.Fingerprint,
Commit: e.Commit, Revision: e.Revision, Seat: e.Seat, Merge: e.Merge, Fresh: true})
if err != nil {
return Again{}, "", err
}
ag.Entry = ne
}
answer := ag.Why
if ag.Do == AgainBuild || (ag.Do == AgainRenumber && !done) {
said, refused := a.BuildAgain(ctx, module, fp, why, by)
if refused {
return Again{}, "", fmt.Errorf("%s; and the build again was refused: %s", ag.Why, said)
}
answer += "; " + said
}
if ag.Do == AgainPublish || (ag.Do == AgainRenumber && done) {
if a.Numberer.Said != nil {
b := NumberedOf(ag.Entry, "")
b.Again = at.UTC().Format(time.RFC3339Nano)
a.Numberer.Said(b)
}
answer += "; " + delivery.VersionNumberedEvent + " said for " + ag.Entry.Version
}
return ag, answer, nil
}
// FloorAgain is builder_floor_again of name: the floor counts only for a name the versions hold nothing of.
func (a Acts) FloorAgain(ctx context.Context, name string) (string, error) {
entries, err := a.Numberer.Versions.Of(ctx, name)
if err != nil {
return "", err
}
if len(entries) > 0 {
return "", fmt.Errorf("%s holds %d versions: its floor counts only for its first number, so nothing is asked", name, len(entries))
}
answer := name + " waits on no floor in this builder: the floor is asked at its next claim"
if a.Numberer.Waits != nil {
for _, w := range a.Numberer.Waits.Waiting() {
if w.Name == name {
answer = fmt.Sprintf("the floor of %s is asked again (%s were silent)", name, strings.Join(w.Silent, ", "))
}
}
a.Numberer.Waits.Wake(WakeByHand, "", "", name)
}
a.Reconsider(ctx)
return answer + "; builder_versions shows its version once numbered", nil
}
// TagFor is the cleaned tag a kept tag ask names for repository's head: the ask a check's build of that head is
// numbered under (design 51 *Tagged versions*). An untag, or an ask whose branch ended, names none.
func TagFor(asks []TagAsk, repository, head string) (string, bool) {
for _, t := range asks {
if t.Untag || t.State == TagRemoving || t.State == TagRemoved || t.Head == "" {
continue
}
if t.Repository == repository && t.Head == head {
return t.Tag, t.Tag != ""
}
}
return "", false
}
@@ -0,0 +1,92 @@
package versions
import (
"context"
"fmt"
"sort"
"strings"
"time"
)
// A branch's end removes its tagged versions (design 51 *Tagged versions*, DLV-B6): on the forge bridge's `merged`, a
// pull's `closed` or `branch-deleted`, heard on the builder's durable consumer of the stream `forge`, the builder calls
// each seat's `remove` and acknowledges the message only once every seat said `removed` (Heard marks the entry); until
// then the bus gives it again each minute (RemoveAgainAfter) and, past RemovalFindingAfter, the builder's finding names
// the seats still owing.
const (
RemoveAgainAfter = time.Minute
RemovalFindingAfter = 15 * time.Minute
// RemovalNotDone is the finding's kind.
RemovalNotDone = "tag-removal-not-done"
)
// BranchEnd is what one hearing of a branch's end did: the versions still not removed (name@version to its seat, with
// the error of a remove that failed), and whether all are gone so the message is acknowledged.
type BranchEnd struct {
Left map[string]string
Done bool
}
// Seats are the seats still owing a removal, sorted, each once.
func (b BranchEnd) Seats() []string {
seen := map[string]bool{}
var out []string
for _, s := range b.Left {
seat, _, _ := strings.Cut(s, ":")
if !seen[seat] {
seen[seat] = true
out = append(out, seat)
}
}
sort.Strings(out)
return out
}
// Finding is the removal finding's words: what is left, since when.
func (b BranchEnd) Finding(repository, branch string, since time.Time) string {
var at []string
for v, s := range b.Left {
at = append(at, v+" on "+s)
}
sort.Strings(at)
return fmt.Sprintf("the tagged versions of %s %s are not all removed since its end at %s: %s; the builder asks remove again each minute",
repository, branch, since.UTC().Format(time.RFC3339), strings.Join(at, ", "))
}
// BranchEnded removes repository's tagged versions of branch: every one not yet removed is asked of its seat again.
// A store that cannot be read is the error: nothing is acknowledged then.
func (a Acts) BranchEnded(ctx context.Context, repository, branch string) (BranchEnd, error) {
all, err := a.Numberer.Versions.All(ctx)
if err != nil {
return BranchEnd{}, err
}
left := map[string]string{}
for _, e := range Untagged(all, repository, branch) {
at := e.Name + "@" + e.Version
left[at] = e.Seat
if a.Remove == nil {
continue
}
// A seat that kept it whole (another tag names its digest) ends it here: recorded kept, never asked again.
if l, done := a.removeOne(ctx, e); done {
delete(left, at)
} else if l != "" {
left[at] = l
}
}
return BranchEnd{Left: left, Done: len(left) == 0}, nil
}
// TagWanted is the body of the checker's `tag-wanted` (design 51 *Tagged versions*): a group member's head to number
// tagged versions for.
type TagWanted struct {
Repository string `json:"repository"`
Branch string `json:"branch"`
Head string `json:"head"`
Tag string `json:"tag,omitempty"`
}
// TagWantedAsk is the tag-asks value a tag-wanted becomes, as builder_tag writes it by hand: by the checker.
func TagWantedAsk(t TagWanted, by string, at time.Time) (string, []byte, error) {
return NewTagAsk(t.Repository, t.Branch, t.Head, "the checker's tag-wanted for a group member's check", by, at)
}
@@ -0,0 +1,263 @@
package versions
import (
"context"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The registry seats a version is published to (design 51 *The seats*), by what is published.
const (
SeatImages = "container-image-registry"
SeatArtifacts = "oci-artifact-registry"
SeatGoPackages = "go-package-registry"
SeatNPM = "npm-package-registry"
)
// SeatOf is the registry seat an artifact of kind in language is published to, and the kind the builder numbers it as;
// ok is false for what is not numbered (an upstream copy is mirrored by its pinned digest, design 51 *Publishing*).
func SeatOf(kind, language string) (seat, numberedAs string, ok bool) {
switch kind {
case KindImage:
return SeatImages, KindImage, true
case KindBundle, KindArchive:
return SeatArtifacts, kind, true
case KindPackage:
switch language {
case "go":
return SeatGoPackages, KindPackage, true
case "typescript", "javascript", "node":
return SeatNPM, KindPackage, true
}
}
return "", "", false
}
// NameOf is the name a module's image, bundle or archive is numbered under: `<module>/<artifact>`, the name the seat
// answers it by (ADR 0605 §9's `<registry>/<module>/<artifact>@…`). A package is numbered under its name in its own
// ecosystem instead (Numberer.Names).
func NameOf(module, artifact string) string { return module + "/" + artifact }
// PackageNames answers a package's name in its own ecosystem (design 32: a Go package's module path, an npm package's
// package.json name) for the module and artifact it is built from, as the indexer's packages state holds it; known is
// false when the state names none for it. An error is a state that cannot be read.
type PackageNames func(ctx context.Context, module, artifact string) (name string, known bool, err error)
// TagPrefix is the git tag prefix of a package in the module's directory path: `go/v` for mesh-sdk's `go`, `v` at the
// root (design 51: the builder tags each Go version `go/v0.1.<n>`).
func TagPrefix(path string) string {
path = strings.Trim(path, "/")
if path == "" || path == "." {
return "v"
}
return path + "/v"
}
// ClaimArtifact is one artifact of a trunk build to number.
type ClaimArtifact struct {
Artifact, Kind, Language string
}
// BuildClaim is a trunk build at its claim: what Numberer.Number numbers.
type BuildClaim struct {
Module, Path, Repository, Fingerprint, Commit, Series string
// Revision is the source revision of the merge (a package is numbered per revision); Merge its sequence, for order.
Revision string
Merge uint64
Artifacts []ClaimArtifact
// Tag numbers a check's build of a head a tag ask names (design 51 *Tagged versions*): its packages only, each a
// tagged version kept with Repository for its branch's end; Head is the head, said in numbered. Empty for a trunk build.
Tag, Head string
// Done are the digests of the build's done key, by artifact, when the claim is a look at a build already done: what it
// left unnumbered (a package whose name was not known yet, a floor that was silent) is numbered with them, so
// version.numbered carries the digest its build module publishes late. Nil for a build still to make.
Done map[string]string
}
// Numbering is what the claim got.
type Numbering struct {
// Versions are each artifact's version, by artifact; an artifact built unnumbered (its floor not known) has none.
Versions map[string]string
// Unnumbered are the artifacts built and published by digest, numbered once their floor answers (ADR 0617 §1).
Unnumbered []string
// Wait is why the build waits: a package whose floor is not known (design 51 *The floor*). Nothing is claimed.
Wait string
// Said are the builder's narrations of each decision, for its process log and its build log (build.said).
Said []Narration
// Names are the names each numbered artifact was numbered under, by artifact (NameOf, or a package's name in its
// ecosystem).
Names map[string]string
}
// NameOf is the name artifact of module was numbered under in this numbering.
func (g Numbering) NameOf(module, artifact string) string {
if name := g.Names[artifact]; name != "" {
return name
}
return NameOf(module, artifact)
}
// Narration is one line the builder says in its build log (design 51: the build log says a version was kept, and
// why), and what it narrates.
type Narration struct{ Line, Why string }
// What a narration of the build log narrates (the build.said entry's why).
const (
SaidNumbered = "numbered"
SaidKept = "kept"
SaidWaiting = "waiting"
SaidUnnumbered = "unnumbered"
SaidUnanswered = "tag ask unanswered"
SaidOldFormClosed = "old-form-closed"
)
// narrationOf is a decision's narration.
func narrationOf(name string, d Decision) Narration {
why := SaidNumbered
switch {
case d.Keep:
why = SaidKept
case d.Wait:
why = SaidWaiting
case d.Unnumbered:
why = SaidUnnumbered
}
return Narration{Line: name + ": " + d.Why, Why: why}
}
// Numberer numbers trunk builds at their claim (design 51 *The versions*, DLV-B2, B3): it asks a name's floor only
// when versions holds nothing for it, records a silent floor as a wait (FloorWaits, woken by its source's event), and
// says each new number (Said, the event version.numbered).
type Numberer struct {
Versions Versions
Floor FloorSources
Waits *FloorWaits
// Holder is the module holding a seat, whose health condition's clear wakes a silent seat source; nil names none.
Holder func(seat string) string
// Said is told each version newly numbered; nil tells nobody.
Said func(delivery.VersionNumbered)
// Names answers a package's name in its ecosystem; nil numbers a package as NameOf names it.
Names PackageNames
// Publishing is told each version Built recorded that is still numbered, its holder's word not heard yet: the
// builder arms its publish bound (publish.go); nil arms none.
Publishing func(ctx context.Context, e Entry)
Now func() time.Time
}
// Number numbers every numbered artifact of c. An error is a store that cannot be read or written: the claim is then
// built as before, unnumbered, and said.
func (n *Numberer) Number(ctx context.Context, c BuildClaim) (Numbering, error) {
out := Numbering{Versions: map[string]string{}, Names: map[string]string{}}
arts := append([]ClaimArtifact(nil), c.Artifacts...)
sort.Slice(arts, func(i, j int) bool { return arts[i].Artifact < arts[j].Artifact })
for _, a := range arts {
seat, kind, ok := SeatOf(a.Kind, a.Language)
if !ok || (c.Tag != "" && kind != KindPackage) {
continue
}
name := NameOf(c.Module, a.Artifact)
if kind == KindPackage && n.Names != nil {
// A package is numbered, asked for its floor and published under its name in its ecosystem (design 32), the
// name the holder's builder_versions question and the seat's versions name it by.
named, known, err := n.Names(ctx, c.Module, a.Artifact)
if err != nil || !known {
why := "the indexer's packages state names no name of it in its ecosystem yet (it writes one at the module's next trunk merge)"
if err != nil {
why = "the indexer's packages state cannot be read: " + err.Error()
}
out.Unnumbered = append(out.Unnumbered, a.Artifact)
out.Said = append(out.Said, Narration{Line: name + ": built unnumbered: " + why, Why: SaidUnnumbered})
continue
}
name = named
}
out.Names[a.Artifact] = name
entries, err := n.Versions.Of(ctx, name)
if err != nil {
return out, err
}
var floor Floor
var why map[string]string
if len(entries) == 0 {
floor, why = n.askFloor(ctx, seat, name, c)
}
d, e, err := n.Versions.Number(ctx, Claim{Name: name, Kind: kind, Series: c.Series, Module: c.Module, Fingerprint: c.Fingerprint,
Commit: c.Commit, Revision: c.Revision, Digest: c.Done[a.Artifact], Seat: seat, Merge: c.Merge, Floor: floor, Tag: c.Tag, Repository: c.Repository})
if err != nil {
return out, err
}
out.Said = append(out.Said, narrationOf(name, d))
switch {
case d.Wait:
n.wait(name, seat, floor, why)
out.Wait = name + ": " + d.Why
return out, nil
case d.Unnumbered:
n.wait(name, seat, floor, why)
out.Unnumbered = append(out.Unnumbered, a.Artifact)
continue
}
if n.Waits != nil {
n.Waits.Answered(name)
}
out.Versions[a.Artifact] = d.Version
if !d.Keep && n.Said != nil {
n.Said(NumberedOf(e, c.Head))
}
}
return out, nil
}
func (n *Numberer) askFloor(ctx context.Context, seat, name string, c BuildClaim) (Floor, map[string]string) {
return n.Floor.Ask(ctx, FloorOf{Seat: seat, Name: name, Repository: c.Repository, Prefix: TagPrefix(c.Path)})
}
func (n *Numberer) wait(name, seat string, f Floor, why map[string]string) {
if n.Waits == nil {
return
}
holder := ""
if n.Holder != nil {
holder = n.Holder(seat)
}
now := time.Now()
if n.Now != nil {
now = n.Now()
}
n.Waits.Wait(name, seat, holder, f, why, now)
}
// Built records the digest each numbered artifact was built with, on an entry that holds none yet or was never published: what the holder's
// builder_versions question compares (ADR 0605 §4).
func (n *Numberer) Built(ctx context.Context, c BuildClaim, got Numbering, digests map[string]string) error {
var errs []string
for artifact, version := range got.Versions {
d := digests[artifact]
if d == "" {
continue
}
e, err := n.Versions.Mark(ctx, got.NameOf(c.Module, artifact), version, func(e *Entry) {
// The new digest replaces the recorded one only while no published was said for the number (design 51
// *Tools of the builder*): a rebuild of a gone file under the same number.
if e.Digest == "" || e.State == Numbered || e.State == Refused {
e.Digest = d
}
})
if err != nil {
errs = append(errs, err.Error())
continue
}
if e.State == Numbered && n.Publishing != nil {
n.Publishing(ctx, e)
}
}
if len(errs) > 0 {
sort.Strings(errs)
return fmt.Errorf("the built digests could not be recorded: %s", strings.Join(errs, "; "))
}
return nil
}

Some files were not shown because too many files have changed in this diff Show More