Compare commits

...
Author SHA1 Message Date
mesh-tools f45b4cb213 Merge pull request 'Issue the provisioner client its own account when its module declares no broker (issue 917)' (#359) from fix/917-provisioner-account into main 2026-10-16 14:11:26 +00:00
jschoubben 407dc86a74 Merge pull request 'The delivery seat promises mark-ready: a reviewer marks a head ready through one audited verb (issue 913)' (#357) from feat/913-mark-ready into main
Reviewed-on: #357
2026-10-16 13:47:28 +00:00
mesh-tools 611e2275cf Merge pull request 'The controller's catalogue tests judge the catalogue as the merge check does, so main passes again (issue 914)' (#358) from fix/914-catalogue-tests-judge-like-the-check into main 2026-10-16 13:09:43 +00:00
mesh-tools eea5866216 List the client's assign test among the store tests
mesh/merge-gate head eea58662163e merged into trunk 407dc86a74df — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head eea58662163e merged into trunk 407dc86a74df — pass: its merge-check.sh passed
mesh/delivery delivered
The merge check proves only listed store tests ran; unlisted, the one
test that shows assign issuing the client's account could skip unseen
(review of issue 917).
2026-10-16 13:03:06 +00:00
mesh-tools b5da0b2161 Issue the provisioner client its own account when its module declares no broker
mesh/merge-gate error: builds mesh-controller → novox; no bus step; the check could not run: killed by hand
mesh/repo-check error: the check could not run: killed by hand
mesh/delivery superseded: a newer head of the same pull request
The catalogue's mesh-provisioner-client declares only provisioner-broker:
the client speaks on the bus as itself. module issue refused it for
lacking a broker, assign minted nothing, and push refused it for lacking
an account, so no machine could run the client (issue 917).
2026-10-16 13:01:03 +00:00
jschoubben c860426c55 Narrow the bar test to what the controller owns for a moved bar; rendering is issue 915 (issue 914)
mesh/merge-gate head c860426c555a merged into trunk a20680dfe769 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head c860426c555a merged into trunk a20680dfe769 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 14:52:13 +02:00
jschoubben 65f24cd45d Judge catalogue tests as the check does: the predefined list and templates read from the tree (issue 914) 2026-10-16 14:50:40 +02:00
jschoubben 114c99e02a The delivery seat promises mark-ready: a reviewer marks a head ready through one audited verb (issue 913)
mesh/merge-gate head 114c99e02a52 merged into trunk 611e2275cf7c beside mesh-catalog#331@2d1b2036 — pass: builds build-agent, mesh-controller → ace, g14, n…
mesh/repo-check head 114c99e02a52 merged into trunk 611e2275cf7c beside mesh-catalog#331@2d1b2036 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/913-mark-ready delivered: every member is delivered
2026-10-16 14:31:04 +02:00
mesh-tools a20680dfe7 Merge pull request 'Seal core credentials by the controller while the vault does not hold them (issue 908, ADR 0396)' (#356) from fix/908-seal-core-until-vault-holds into main 2026-10-16 10:38:54 +00:00
mesh-tools ebdbba213e List the renamed store test of issue 908 in testdata/store-tests
mesh/merge-gate head ebdbba213e73 merged into trunk 18c58b4903ad — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head ebdbba213e73 merged into trunk 18c58b4903ad — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 10:33:45 +00:00
mesh-tools 5eb5ea1774 Seal core credentials by the controller while the vault does not hold them, so the bus is never left out of a declaration (issue 908, ADR 0396)
mesh/merge-gate head 5eb5ea177482 merged into trunk 18c58b4903ad — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 5eb5ea177482 merged into trunk 18c58b4903ad — fail: its merge-check.sh failed: 1 store test(s) listed in testdata/store-tests did not …
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 10:27:53 +00:00
mesh-tools 18c58b4903 Merge pull request 'Feature 57 (config and provisioning): release mesh-controller' (#355) from release/57-config-and-provisioning into main 2026-10-16 08:42:05 +00:00
mesh-tools 8c381172e7 Restore the blank last line the patch route dropped from seven golden files, so the branch holds the reviewed tree (issue 875)
mesh/merge-gate head 8c381172e7ea merged into trunk d70bb1f38867 beside mesh-host#116@bff1ac14 — pass: builds build-agent, mesh-controller, route-proxy → a…
mesh/repo-check head 8c381172e7ea merged into trunk d70bb1f38867 beside mesh-host#116@bff1ac14 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group release/57-config-and-provisioning delivered: every member is delivered
2026-10-16 08:29:21 +00:00
mesh-tools d984816844 Feature 57 (config and provisioning): release mesh-controller, the reviewed release branch merged with main, on top of main 2026-10-16 08:28:38 +00:00
mesh-tools d70bb1f388 Merge pull request 'Judge a delivery group's composed check by the group's own controller, release branch included (issue 867, ADR 0239 §14)' (#352) from fix/867-group-check-judged-by-its-controller into main 2026-10-16 01:04:00 +00:00
mesh-toolsandClaude 151faa10a1 Judge a delivery group's composed check by the group's own controller, release branch included (issue 867, ADR 0239 §14)
mesh/merge-gate head 151faa10a13b merged into trunk a07dd78c3e90 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 151faa10a13b merged into trunk a07dd78c3e90 — pass: its merge-check.sh passed
mesh/delivery delivered
The composed check of mesh-controller#346 and mesh-host#114 into feat/57-config-and-provisioning was judged by the release branch's controller, which cannot build the node-engine member's validator, so the group could never pass. ADR 0239 §14 has the group's own controller judge when a member moves it; ADR 0358 §2 changes the judge only for a pull request into a release branch. A group member in mesh-controller now judges, with the mesh-host member's validator, built in the isolated judge environment and kept by content; the statuses say the group's controller judged.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-16 00:59:27 +00:00
mesh-tools a07dd78c3e Merge pull request 'Ask mesh-delivery for a recheck's group on a bus connection of its own (issue 867, ADR 0382 §1)' (#351) from fix/867-recheck-asks-on-the-bus into main 2026-10-16 00:33:18 +00:00
mesh-toolsandClaude 3fdbea4204 Ask mesh-delivery for a recheck's group on a bus connection of its own (issue 867, ADR 0382 §1)
mesh/merge-gate head 3fdbea420440 merged into trunk af60d2719c0c — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 3fdbea420440 merged into trunk af60d2719c0c — pass: its merge-check.sh passed
mesh/delivery delivered
`delivery check` runs as a command of its own, where no serving connection exists, so a recheck said its siblings could not be learned ("this controller is not on the bus") and mesh-host#114 was judged without its sibling. groupsAsked now asks through onTheBus, which lends the serving connection or dials one, as the hand-act log does. TestTheRecheckVerbAsksMeshDeliveryOnABusOfItsOwn drives the verb against a real responder.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-16 00:27:57 +00:00
mesh-tools af60d2719c Merge pull request 'Find a rechecked pull request's group by its repository and number (issue 867, ADR 0382 §1)' (#350) from fix/867-recheck-finds-its-group into main 2026-10-16 00:17:06 +00:00
mesh-toolsandClaude de5292323f Find a rechecked pull request's group by its repository and number (issue 867, ADR 0382 §1)
mesh/merge-gate head de5292323f33 merged into trunk a47976be4e69 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head de5292323f33 merged into trunk a47976be4e69 — pass: its merge-check.sh passed
mesh/delivery delivered
A recheck rebuilds the pull request without its head branch name, so the sibling lookup found no group and said nothing: mesh-host#114's recheck was judged without its sibling. With no branch name, the group is the one whose member_details hold the pull request; an older mesh-delivery says the siblings could not be learned. TestReplay867Recheck replays it.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-16 00:13:06 +00:00
mesh-tools a47976be4e Merge pull request 'Check a delivery group's member beside its siblings' merges (issue 867, ADR 0382)' (#348) from fix/867-group-siblings into main 2026-10-15 23:53:01 +00:00
mesh-tools 4446027108 Merge pull request 'Compose the forge in its manifest tests with the operator's account (issue 869)' (#349) from fix/869-forge-tests-name-an-account into main 2026-10-15 23:46:21 +00:00
mesh-toolsandClaude e42d72a80a Compose the forge in its manifest tests with the operator's account, as the mesh does (issue 869)
mesh/merge-gate head e42d72a80a62 merged into trunk 7543f64911c5 — pass: the change touches no module of the mesh's graph
mesh/repo-check head e42d72a80a62 merged into trunk 7543f64911c5 — pass: its merge-check.sh passed
mesh/delivery delivered
The gitea manifest names its patch spool's owner ${machine:account} since mesh-catalog d4b8b247 (issue 852); these tests gave the machine no account, so main failed its own merge check.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-15 23:36:30 +00:00
mesh-toolsandClaude 37fd73f378 Check a delivery group's member beside its siblings' merges, judged by the group's controller (issue 867, ADR 0382)
mesh/merge-gate head 37fd73f3781e merged into trunk 4446027108fd — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 37fd73f3781e merged into trunk 4446027108fd — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/867-group-siblings delivered: every member is delivered
A mesh-host change and a mesh-controller change that need each other could not both pass their own checks: each read the other at its base. The check now asks mesh-delivery for the group's open siblings into a base of the same name, reads each at its head merged onto its base, builds the judge from a controller sibling's merge in the isolated judge environment keyed on its inputs, and names the siblings in the verdict. With no answer from mesh-delivery the verdict says it could not learn them; a pull request with no sibling is checked as before. TestReplay867 replays the failure.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-15 23:27:12 +00:00
mesh-tools 7543f64911 Merge pull request 'Mail seats: a declared seat carries needs; a secret given is told once (issues 802, 835)' (#345) from feat/mail-seats into main 2026-10-15 17:44:20 +00:00
jochen 5cf8938f55 Let the genesis test accept a degradable grant missing from the installer's list (issue 835)
mesh/merge-gate head 5cf8938f55b3 merged into trunk 0ab61d1be8c5 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 5cf8938f55b3 merged into trunk 0ab61d1be8c5 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/mail-seats delivered: every member is delivered
Forge check of mesh-controller#345: the merge check clones mesh-host at the commit the
machines run, not its main, so a change that only edits mesh-host's examples (the
operator-channel.tell grant) can never be seen there until mesh-host's own module next
builds. operator-channel.tell is the one grant with a fallback for a refusal (the
secret-given condition, ADR 0380 §9), so mark it degradable and let
TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose pass without it while every
other grant stays required.
2026-10-15 19:29:13 +02:00
jochen a18c17d3cc Name the machine in the secret-given message (issue 835, review round 2)
mesh/delivery-group group feat/mail-seats delivering: 1 of 3 delivered
mesh/merge-gate head a18c17d3cc5f merged into trunk 0ab61d1be8c5 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head a18c17d3cc5f merged into trunk 0ab61d1be8c5 — fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheContro…
mesh/delivery superseded: a newer head of the same pull request
ADR 0380 §1 says the message names the module and the machine it is
for; the words named no machine at the terminal and only the desk's
machine at the desk. The headline now ends "on <node>" and the tell
carries "subject": <node> in both cases.

Also covers two gaps review round 2 flagged: a tell failure other
than ErrNothingServes/ErrGrantRefused surfaces and raises no
condition, and the terminal's hand act now names who asked (not just
where), extracted into terminalGiveHandAct so it is testable on its
own.
2026-10-15 18:45:03 +02:00
jochen 5f24a47df6 Refresh captured mesh-host at c3b2fce, carrying the operator-channel.tell grant (issue 835)
mesh-host joined feat/mail-seats to add the controller's new
operator-channel.tell publish subject to its genesis user list. Moved
per testdata/beside/CAPTURED's own recipe (git archive, never by hand);
the refreshed snapshot also carries main's drift since the last capture
(ADR 0338's store placement). TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose
now passes; merge-check.sh green.
2026-10-15 17:55:30 +02:00
jochen 435468c151 A secret given tells the operator once, opens no condition (issue 835)
ADR 0380: giving a module's own secret now asks the operator channel's
tell over the bus, naming the secret, where, when and who asked, and
raises no secret-given condition. Until nothing serves tell yet, or the
controller's grant is refused (link.ErrNothingServes / the new
link.ErrGrantRefused), the condition is raised exactly as before, so the
change composes whichever side of issue 797 merges first. A successful
tell also retires every open secret-given condition. The terminal's own
secret accept now records a hand act too, as the desk path already did.

The controller's bus grant for operator-channel.tell is added to the
composed config (golden fixture updated).
2026-10-15 17:45:45 +02:00
jochen b5a441a978 Let a module-declared seat carry needs, so mail-notifier can need node-notifier (issue 802)
ADR 0220's dependency mechanism served compiled seats only and retired its one user with
node-resolver-config (commit f68521da). ADR 0378 S8 revives it for module-declared seats:
SeatDeclaration gains Needs, judged by the catalogue check (node scope only, a name that
resolves to a node seat somewhere in the catalogue, never itself), and DependsOn now takes
the catalogue so a claimed seat's needs are found wherever they are declared.
2026-10-15 17:18:23 +02:00
mesh-tools 0ab61d1be8 Merge pull request 'Mail and bus hardening: restricted streams, recorded moves that restart nothing, desk prompts, abandoned calls (issues 727, 728, 785, 789, 711)' (#344) from feat/mail-hardening into main 2026-10-15 14:34:24 +00:00
jochen 2347a4c73b Merge branch 'feat/mail-hardening' into mh-711-728-r1
mesh/merge-gate head 2347a4c73bd0 merged into trunk 20e58907b0f8 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 2347a4c73bd0 merged into trunk 20e58907b0f8 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/mail-hardening delivered: every member is delivered
2026-10-15 15:43:34 +02:00
jochen ceffbf500a Merge branch 'feat/mail-hardening' into mh-711-728-r1 2026-10-15 15:29:15 +02:00
jochen d55e148a1d Never wait NextMsg's bound for a dead letter held already overcounts (issue 728) 2026-10-15 15:28:56 +02:00
jochen ab74bbb56b Merge branch 'feat/mail-hardening' into mh-789-r1 2026-10-15 15:27:13 +02:00
jochen 320dde41d9 Count an own secret's prompt by character and refuse {member} on a single secret (issue 789) 2026-10-15 15:27:06 +02:00
jochen e603b2cae1 Bound MarkAbandoned's compare-and-set retries, giving up after 10 tries (issue 711) 2026-10-15 15:20:43 +02:00
jochen 87780f482b Merge branch 'feat/mail-hardening' into mh-785-r1 2026-10-15 15:15:55 +02:00
jochen 867a3f81a9 Hold a record-policy move that changes a non-file/directory resource (issue 785)
ADR 0377 §3's allowlist names only tools bundles, files no container mounts
and holds no data, and manifest fields that recreate nothing. Holds only
checked containers, files, directories and processes/services; any other
resource type (archive, package, user, network, ...) fell through unheld.
Add a default case that holds and names the resource type and id.
2026-10-15 14:59:06 +02:00
jochen 9aeb5f4493 Keep a restricted message past its stream's age as a record without its body, as ADR 0376 §3 says (issue 728)
Reverts 45b6c347, which kept nothing of such a message and raised an aged-out
condition: that departed from an accepted decision on a lead's word alone. The
concern behind it (the record keeps the subject, with no age) is its own issue.
2026-10-15 14:56:24 +02:00
jochen 15a374659e Merge branch 'feat/mail-hardening' into mh-711 2026-10-15 14:47:09 +02:00
jochen 868e3ae377 Never mark a call abandoned over its recorded finish when a controller starts (issue 711)
A starting controller's sweep read each call it saw running and wrote it back
abandoned unconditionally, so a finish written in between, with its kept answer,
was lost. The mark is now a compare-and-set on the call's record: a finish already
recorded, or racing the mark, stands.
2026-10-15 14:47:09 +02:00
jochen ce77a2caaa Merge branch 'feat/mail-hardening' into mh-785 2026-10-15 14:27:37 +02:00
jochen 0d1bb035cb Let a recorded module's move that restarts nothing roll out with its walk (issue 785)
Under a module's own record (its manifest's, or its irreplaceable data's), every
move waited for a person's push, even one that only changed a tools bundle: the
database server stayed behind on ace on 2026-10-15 although its walk said it
recreated none of its containers. ADR 0377 lets such a move roll out.

A move is judged per machine against the build that machine was last sent: it
goes when no container, mounted file, data path, data declaration, provider or
(for a module with data) long-running process changes. A person's record, the
bus and any move that cannot be judged still wait. The walk keeps per machine
which way each recorded module went and why, says it in its note and log, and
status names a machine behind with `push <node>` only for a move that waits.
2026-10-15 14:27:37 +02:00
jochen 45b6c347b2 Keep nothing of a restricted message past its stream's age, only count it (issue 728)
A record kept without its body still carried its subject, which names whose mail it was, with no age. The lead decided such a message is not kept at all: the notice is taken, the message counted, and the log and an aged-out condition say how many messages of the stream aged out undelivered. Unrestricted streams are unchanged.
2026-10-15 14:22:07 +02:00
jochen d03e24f423 Age a restricted stream's dead letters out with it and never show their body (issue 728)
DEAD_LETTERS kept a message given up on MAIL forever and showed it whole to any caller of dead-letters, outside the grant and past MAIL's week. Per ADR 0376 §3-§6: the kept copy carries a per-message time to live of what is left of its stream's age (a copy already past it is kept without its body), DEAD_LETTERS and EVENTS allow per-message TTL, the list and show give no body or own headers for it, delivering it again needs a minute left and the copy keeps the time left, and its max-deliveries condition clears saying how many aged out undelivered and from which stream.
2026-10-15 14:11:37 +02:00
jochen f75cd0239d Hear a restricted stream only from a module's own account, never the tool runner's (issue 727)
The tool runner's bus account carried the heard grants, consumer and bind word of every module it runs, and every agent session on that machine reaches the bus as that account, so any session could pull a granted module's consumer on MAIL. Per ADR 0376 §1 the controller now composes a restricted stream's consumer, grants and bind word only for a module that declares runs-as; registration refuses a consumes of a restricted stream without runs-as (§2), and a grant that composes nothing is said in doctor (D-heard) and in the controller's log on every raise and send.
2026-10-15 14:11:37 +02:00
jochen f7928a2bf7 Let an own secret's manifest declare a desk prompt (issue 789)
A family like mesh-mail's mail-credential-* names only the module and the secret's key at the desk, so the operator cannot tell which account is asked. An own secret (or a family) may now declare a one-line prompt in its manifest, with {member} filled by a family member's suffix; the module check refuses one that is not one line, over 120 characters, or holds another placeholder. desk_secret.go reads it from the controller's own catalogue and sends it to the desk as what, never from the call or the module's runtime; omitted when none is declared, so an old desk or module behaves as today.
2026-10-15 13:47:20 +02:00
mesh-tools 20e58907b0 Merge pull request 'Read the borrowed-words log without the logger's clock (issue 752)' (#343) from 752-log-without-clock into main 2026-10-15 00:43:53 +00:00
mesh-toolsandClaude Opus 5.5 f877c33c7c Read the borrowed-words log without the logger's clock, so the test passes at any time of day (issue 752)
mesh/merge-gate head f877c33c7c18 merged into trunk 0b7e740b299c — pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes …
mesh/repo-check head f877c33c7c18 merged into trunk 0b7e740b299c — pass: its merge-check.sh passed
mesh/delivery delivered
The test forbids "23:3" in the log line, and the logger's own timestamp held it from 23:30
to 23:39 every night, so the check went red for no reason in that window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:51:05 +00:00
mesh-tools 0b7e740b29 Merge pull request 'Let the tool runner's account hear the event families its manifest names (issue 745, ADR 0367)' (#342) from 745-mesh-wait into main 2026-10-14 23:14:41 +00:00
mesh-toolsandClaude Opus 5.5 755457cc37 Let the tool runner's account hear the event families its manifest names (issue 745, ADR 0367)
mesh/merge-gate head 755457cc379e merged into trunk 4cc6b6c566a5 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 755457cc379e merged into trunk 4cc6b6c566a5 — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792019685173332639 (novox/mesh-controller at 0b7e740b into ma…
A session on the mesh MCP server polls for every verdict, answer and delivery because the tool
runner's account hears no event. ADR 0367 lets the tool runner's module name, one by one, the
families a session may wait on: the manifest's `waits`, refused for any other module and for a
wildcard. The controller grants the runtime's account a plain subscription on those subjects
alone, makes no durable consumer for them, and issues the same list in the tool runner's
membership, so the grant and what `mesh_wait` offers come from one composition and no call or
setting widens it. A family on a seat whose events end in a kind or an asker must name it, and is
refused at composition without it, so it is never granted a subject nothing publishes on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:05:14 +00:00
mesh-admin 4cc6b6c566 Merge pull request 'Say a read-only mesh-cli call in no line of its own, and count them hourly (hq issue 762)' (#341) from fix/762-a-read-call-writes-no-journal-line into main 2026-10-14 22:33:08 +00:00
jschoubben 9e11790bd7 Say a read-only mesh-cli call in no line of its own, and count them hourly (hq issue 762)
mesh/merge-gate head 9e11790bd74a merged into trunk 3ed08c3464e3 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 9e11790bd74a merged into trunk 3ed08c3464e3 — pass: its merge-check.sh passed
mesh/delivery delivered
A page polling plans every few seconds wrote about 14,000 journal lines a day. A line is judged a
read by the generic command verb's reading forms (ADR 0266), on its every word, so plans go,
conditions silence and settings set are still said; a refused or failed read is said too.
2026-10-15 00:28:17 +02:00
mesh-admin 3ed08c3464 Merge pull request 'Take no rebuild into a waiting rollout, and ask what a tier left unasked (hq issue 754)' (#340) from fix/754-a-rebuild-in-the-wait-is-not-the-walks-build into main 2026-10-14 22:06:58 +00:00
jschoubben 9f072c110c Take no rebuild into a waiting walk, and ask what a tier left unasked (hq issue 754)
mesh/merge-gate head 9f072c110c14 merged into trunk 60cc9ec363df — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 9f072c110c14 merged into trunk 60cc9ec363df — pass: its merge-check.sh passed
mesh/delivery delivered
A rebuild that landed while a walk waited for mesh-delivery's word was taken as
the walk's build of that module; once let go, the walk asked a tier only when
every module of it was unasked, so it never asked the rest and stood still.

A walk now takes only the outcomes of builds it asked, and none while it waits;
a rebuild does not join a waiting walk and says it runs beside it; and a tier
asked in part is asked the rest, including a module held built that the walk
never asked for, which is somebody else's build and not sent as the walk's.
2026-10-15 00:02:28 +02:00
mesh-admin 60cc9ec363 Merge pull request 'End the check of a pull request's old head when a new head is checked (hq issue 726)' (#339) from fix/726-a-new-push-ends-the-old-check into main 2026-10-14 20:28:16 +00:00
mesh-admin 5d22d7df86 Merge pull request 'A module moves only when its program changes (hq issue 721)' (#336) from fix/a-module-moves-only-when-its-program-changes into main 2026-10-14 20:21:05 +00:00
mesh-admin 86057f4551 Merge pull request 'A module may declare a stream of its own, heard only by the consumers its trusted config value grants (hq issue 718, ADR 0363)' (#338) from fix-718-module-streams-and-grants into main 2026-10-14 20:13:33 +00:00
jschoubben e4acab32d6 End the check of a pull request's old head when a new head is checked
mesh/merge-gate head e4acab32d6a2 merged into trunk 60cc9ec363df — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head e4acab32d6a2 merged into trunk 60cc9ec363df — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792009731456564734 (novox/mesh-controller at 60cc9ec3 into ma…
A check judges one head; once the pull request has another, its verdict
answers nobody and its run holds a build machine the new head waits
behind. Asking a check of R#N at H now drops every waiting check of R#N
at another head and kills every one in flight, saying each in the log.
Announcements are unordered and a recheck asks a recorded head, so this
happens only when the forge confirms H is the pull request's current
head; otherwise nothing ends and that is said. A one-shot recheck ends
them within the call. Tries, walk builds, group checks, other pull
requests' checks and rechecks of the same head are left alone.

novox/hq issue 726
2026-10-14 22:10:15 +02:00
jochen 743e6478a7 Remove a restricted stream's consumer when the bus refuses to narrow it, so its wider filters deliver nothing more (ADR 0363 §6, review of PR 338)
mesh/merge-gate head 743e6478a76c merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 743e6478a76c merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 22:05:47 +02:00
jochen ede2772b3c Deliver a module stream's given-up message again only while its consumer is still granted it (ADR 0363 §7)
mesh/merge-gate head ede2772b3c90 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head ede2772b3c90 merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:55:55 +02:00
jschoubben b34055f580 Register the compared module after the fixture's own registration (review of #336)
mesh/merge-gate head b34055f580d4 merged into trunk 86057f455194 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head b34055f580d4 merged into trunk 86057f455194 — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792009731456564734 (novox/mesh-controller at 60cc9ec3 into ma…
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes delivered: every member is delivered
2026-10-14 21:53:50 +02:00
jochen ee13e75bf3 Let a module declare a stream of its own and restrict who hears it by a trusted config value (hq issue 718, ADR 0363)
mesh-mail's events must live on a stream of their own and reach only the modules the operator granted
an account (ADR 0359 §2, §4). The controller now creates a module's declared stream, publishes its events
on mesh.stream.<module>.event.<event>, and composes a consumer's durable consumer, filters and bus grants
on a restricted stream only where its consumes asks and the provider's trusted value names it; a consumer
that loses its grant has its consumer removed on the next assertion. Nothing changes while no manifest
declares a stream.
2026-10-14 21:52:45 +02:00
jschoubben 480657d95b Register the module a walk's store test compares with, as the mesh runs it (review of #336)
mesh/merge-gate head 480657d95b9b merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 480657d95b9b merged into trunk c2ec0a601acb — fail: its merge-check.sh failed: --- FAIL: TestAWalkWhoseEveryBuildCameOutTheSameIsDone …
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:43:20 +02:00
jschoubben 38c87a7e47 Compare a build with the one the mesh registered, not the newest, which may be one put back (review of #336, hq issue 721)
mesh/merge-gate head 38c87a7e4714 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 38c87a7e4714 merged into trunk c2ec0a601acb — fail: its merge-check.sh failed: --- FAIL: TestAWalkWhoseEveryBuildCameOutTheSameIsDone …
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:30:09 +02:00
jschoubben 6acce60f8b List the store test of a walk whose every build came out the same (hq issues 611, 721)
mesh/merge-gate head 6acce60f8bd3 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 6acce60f8bd3 merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:23:46 +02:00
jschoubben 4390589c61 Leave out a dependent of a base that came out the same, and end a walk with every build the same as done (ADR 0267 rule 3; hq issues 607, 611, 721)
mesh/merge-gate head 4390589c61a4 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 4390589c61a4 merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:19:14 +02:00
jschoubben e6a28a2d28 Say why the walk moves every module when the checks' builds cannot be read (lint, hq issue 721)
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes checking: 1 of 2 member(s) ready
mesh/merge-gate head e6a28a2d2814 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head e6a28a2d2814 merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:15:24 +02:00
jschoubben 32bd8b5d5f Leave a module out of a walk when its check built it the same as what runs (hq issue 721)
mesh/merge-gate head 32bd8b5d5fb2 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 32bd8b5d5fb2 merged into trunk c2ec0a601acb — fail: its merge-check.sh failed: --- FAIL: TestNoReaderAnswersEmptyForAnError (2.20s)
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
A one-sentence change in a package mesh-builder imports but never reaches moved
build-agent. A merge check now compares each module it builds with the build the
mesh holds, the whole image without file times, labels or creation time, and a
walk is cut without a module every check of its pull requests found the same,
while the mesh still holds that build. A comparison that fails moves the module
as before.
2026-10-14 21:07:25 +02:00
mesh-admin c2ec0a601a Merge pull request 'A one-machine walk says its report was its run (hq issue 720)' (#335) from fix/720-a-one-machine-walk-says-its-report into main 2026-10-14 18:59:22 +00:00
jschoubben 34050dc6a3 Say a one-machine walk's report was its run: no gate reads it since ADR 0360 (hq issue 720)
mesh/merge-gate head 34050dc6a3c5 merged into trunk a5500f732f77 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 34050dc6a3c5 merged into trunk a5500f732f77 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 20:55:04 +02:00
mesh-admin a5500f732f Merge pull request 'Run the slow test packages in shards and build judges from a warm Go cache (hq issue 717)' (#334) from fix/717-the-suite-runs-in-under-two-minutes into main 2026-10-14 18:46:47 +00:00
jschoubben 3ec4473b29 Run the slow test packages in shards and stop compiling the standard library per judge test (hq issue 717)
mesh/merge-gate head 3ec4473b29b2 merged into trunk df8c1ccf1fff — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 3ec4473b29b2 merged into trunk df8c1ccf1fff — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
The controller's check spent 3-5 minutes in cmd/mesh-controller, whose few hundred store tests run one
after another, while most packages end in seconds. merge-check.sh now compiles each slow package once and
deals its tests round-robin into processes that run beside each other and beside the rest of the suite;
every test still runs once, under the race detector, through its package's TestMain. The listed store
tests run beside the suite instead of after it.

internal/builder's judge tests ran a real go build with a fresh Go cache each, compiling the standard
library every time: half the suite's CPU. The fake runtime fills an empty cache from one warmed per
process. The lease tests run in parallel. The link test of slow work now sets its one-second window after
the serving loop asserted the controller's thirty-second one, so it proves the heartbeat (it passed
without it) in half the time.
2026-10-14 20:38:35 +02:00
mesh-admin df8c1ccf1f Merge pull request 'Build a check's builds where the host's engine sees them, a tier at once and beside the tests (hq issues 713, 716)' (#333) from fix/713-the-walk-reuses-its-checks-builds into main 2026-10-14 18:36:18 +00:00
jschoubben bbee20adf9 Build a merge's modules of a tier at once, and beside the check's tests where the judge cache allows (hq issue 716)
mesh/merge-gate head bbee20adf9b9 merged into trunk 622fb1e9cbac — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head bbee20adf9b9 merged into trunk 622fb1e9cbac — pass: its merge-check.sh passed
mesh/delivery delivered
A check built its images one after another, after merge-check.sh. With a judge cache the
builds run during the tests, out of their reach; a failed check lets go of them.
2026-10-14 20:20:49 +02:00
jschoubben afed887217 Build a check's builds for its merge where the host's container engine sees them (hq issue 713)
The check built in a temporary directory of the build holder's container, which the
host's engine mounted empty: every check build failed and every walk built again.
2026-10-14 20:14:20 +02:00
mesh-admin 622fb1e9cb Merge pull request 'Judge a pull request into a release branch by that branch's controller and repositories (issue 704, ADR 0358)' (#329) from fix/704-release-branch-judge into main 2026-10-14 18:09:03 +00:00
mesh-adminandClaude Opus 5.5 5d5bdf9e32 Refuse the replays without a judge cache rather than run them from a volume
mesh/merge-gate head 5d5bdf9e32d1 merged into trunk 4af9f1147de6 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 5d5bdf9e32d1 merged into trunk 4af9f1147de6 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/704-release-branch-judge delivered: every member is delivered
A check's own volume is no place for the replays: the store replay binds a
directory it makes there into other containers by path, which the container
runtime would create on the build node's own disk (review 12308). Without the
build agent's judge cache the replays are refused and the gate says so as an
error; replays the runtime cannot start are the check's error, not the
change's; and an id that cannot name a place in the judge cache is refused
(hq issue 709, issue 704).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:58:07 +00:00
mesh-admin 4af9f1147d Merge pull request 'Send a merge's builds to every machine at once, with no first-machine gate (hq ADR 0360)' (#332) from feat/lean-rollout into main 2026-10-14 17:53:34 +00:00
jschoubben 704b21b3c5 Carry the new controller's user list to the bus when its walk ends, as the gate's pass did (hq ADR 0236 §34, ADR 0360)
mesh/merge-gate head 704b21b3c547 merged into trunk 76304d9db817 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 704b21b3c547 merged into trunk 76304d9db817 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 19:36:28 +02:00
jschoubben 5b455b8a24 Keep the first-machine gate behind a switch for walks begun before hq ADR 0360, and test both paths
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 5b455b8a2438 merged into trunk 76304d9db817 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 5b455b8a2438 merged into trunk 76304d9db817 — fail: its merge-check.sh failed: --- FAIL: TestARenewalWrittenThoughItsAnswerWasLostIsSt…
2026-10-14 19:35:07 +02:00
jschoubben 0e97413875 Assert the backlog walk sends every machine at once and passes on the report (hq ADR 0360)
mesh/merge-gate head 0e974138751e merged into trunk 76304d9db817 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 0e974138751e merged into trunk 76304d9db817 — fail: its merge-check.sh failed: --- FAIL: TestAReusedCheckBuildIsRegisteredByItsDigestA…
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 19:21:42 +02:00
jschoubben da8a5e9bc0 Send a merge's builds to every machine at once, with no first-machine gate (hq ADR 0360)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head da8a5e9bc0f7 merged into trunk 76304d9db817 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head da8a5e9bc0f7 merged into trunk 76304d9db817 — fail: its merge-check.sh failed: --- FAIL: TestAReusedCheckBuildIsRegisteredByItsDigestA…
The operator chose the leanest rollout: no gate, no one machine first, failures loud and fixed forward.
A walk sends every rolled-out module to every machine running it in one step, built-by orders no tier,
and a walk ends on its machines' reports. The backlog walk sends every machine at once and passes each
on its applied report.
2026-10-14 19:19:49 +02:00
mesh-adminandClaude Opus 5.5 847ffd6d12 Never run the replays from the workspace, with or without a judge cache
mesh/merge-gate out of date: judged against trunk 76304d9db817, the trunk is now 4af9f1147de6; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 76304d9db817, the trunk is now 4af9f1147de6; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Without a judge cache the replays still ran mesh-lab's checkout in the workspace
with the container runtime's socket, so a self-judging pull request could write
a test into them (hq issue 709, review 12303). They now run from a copy in a
container volume of the check's own, made before anything of the change runs,
with a fresh home and throwaway caches, removed after the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 14:34:00 +00:00
mesh-adminandClaude Opus 5.5 0214b45469 Run mesh-lab's replays where no pull request reaches
mesh/merge-gate head 0214b45469ec merged into trunk 76304d9db817 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 0214b45469ec merged into trunk 76304d9db817 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The replays hold the container runtime's socket, yet took their HOME, Go
environment, build cache and module cache from the workspace a pull request's
own script writes, so a planted -toolexec hook ran as root on the build node
(hq issue 709). They now run from a copy of mesh-lab made before anything of
the change runs, with a fresh home and the judge cache's caches, as a judge is
built (ADR 0358 §6).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 14:15:19 +00:00
mesh-adminandClaude Opus 5.5 2bee290eea Keep judge builds warm in the build agent's judge cache, and leave nothing on the host
mesh/delivery-group group fix/704-release-branch-judge rejected: its order contradicts itself: novox/mesh-catalog@47ef2644aca1 and novox/mesh-controller@2bee29…
mesh/merge-gate head 2bee290eea56 merged into trunk 76304d9db817 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 2bee290eea56 merged into trunk 76304d9db817 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Review of #329: a judge's home made with a temporary directory inside the build
agent is resolved on the host by the container runtime, leaving a cold cache
behind on every check, and a kept judge was lost on every redeploy. Judge builds
now use the judge cache the build agent declares (MESH_JUDGE_CACHE), mounted into
judge builds only: one warm Go cache, the kept judges, and a fresh home per check
removed after it (hq ADR 0358 §6, issue 704, issue 482).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:53:05 +00:00
mesh-admin 76304d9db8 Merge pull request 'Keep the controller lease through a stalled renewal until its TTL passes (hq issue 477)' (#331) from fix-477-lease-survives-stall into main 2026-10-14 13:35:23 +00:00
mesh-adminandClaude Opus 5.5 b7084a19e9 Build every judge where no pull request can reach, and never read mesh-lab at a release
mesh/merge-gate out of date: judged against trunk c975dbf7b053, the trunk is now 76304d9db817; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk c975dbf7b053, the trunk is now 76304d9db817; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Review of #329: a judge built with HOME in the workspace runs git under a
configuration a pull request's script can plant, so a hook could rewrite the
judge's sources before they compile and the result be kept under an honest key
(issue 482, issue 704). Every judge now builds with its own HOME, configuration
and Go cache outside the workspace, no git configuration and no VCS stamping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:30:46 +00:00
mesh-admin c975dbf7b0 Merge pull request 'A module may declare its unhealthy urgent, with why (hq ADR 0355, issue 691)' (#330) from adr-355-urgent-unhealthy into main 2026-10-14 13:21:56 +00:00
jochen 98e8568749 Keep the controller lease through a renewal the bus did not answer, until its TTL passes
mesh/merge-gate head 98e856874913 merged into trunk c975dbf7b053 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 98e856874913 merged into trunk c975dbf7b053 — pass: its merge-check.sh passed
mesh/delivery delivered
A bus stall of a few seconds made a renewal time out, and the controller
called that a lost lease, exited and raised the urgent lease-lost alert,
though no other controller could take a key that still lived. Now only
another holder seen, or the key's TTL passed unrenewed, loses the lease;
new acts stop at half the TTL unrenewed and resume on the next renewal
(novox/hq issue 477).
2026-10-14 15:20:19 +02:00
jochen 58889235d1 Raise a module's unhealthy urgent when its manifest says why (hq ADR 0355, issue 691)
mesh/merge-gate head 58889235d163 merged into trunk 535c7641ac41 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 58889235d163 merged into trunk 535c7641ac41 — pass: its merge-check.sh passed
mesh/delivery delivered
A broken service people rely on, such as mail, was said only as a warning on
the desktop. A module may now declare unhealthy urgent with what people lose;
the module check refuses it without why or with another severity.
2026-10-14 15:13:57 +02:00
mesh-adminandClaude Opus 5.5 48b2906b5c Keep mesh-lab at its trunk and build the release judge with a cache of its own
mesh/merge-gate out of date: judged against trunk 535c7641ac41, the trunk is now c975dbf7b053; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 535c7641ac41, the trunk is now c975dbf7b053; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Review of #329: mesh-lab's replays run with the container runtime's socket, so
a same-named branch must never be what runs there; the shared Go cache and HOME
are writable by a pull request's own script, so the judge kept for a head must
not be built from them (issue 704, issue 482). A forge answer naming no branch
is an error, and a trunk is the branch a repository follows, not a name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:13:30 +00:00
mesh-adminandClaude Opus 5.5 ea7d91bff1 Say which validator judged a release-branch check, and prove clause 5
mesh/merge-gate head ea7d91bff190 merged into trunk 535c7641ac41 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head ea7d91bff190 merged into trunk 535c7641ac41 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
ADR 0358 version 4 asks the verdict to say whether mesh-host's release-branch
validator or the vendored one judged, and checks that a release branch's own
pull request into its trunk is judged as any trunk pull request (issue 704).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:56:55 +00:00
mesh-adminandClaude Opus 5.5 4a75cc51dd Judge a pull request into a release branch by that branch's controller and repositories
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 4a75cc51dda6 merged into trunk 535c7641ac41 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 4a75cc51dda6 merged into trunk 535c7641ac41 — pass: its merge-check.sh passed
A release branch could never carry a change that needs another repository's
release-branch change: its pull requests were judged by the running controller
and against the other repositories' main (hq issue 704, ADR 0358).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:48:44 +00:00
mesh-admin 535c7641ac Merge pull request 'A fresh check keeps a build whose key is unchanged instead of building it again (hq issue 703)' (#328) from fix/703-a-fresh-check-keeps-an-unchanged-build into main 2026-10-14 08:15:56 +00:00
jschoubben c707bf18d8 Say which check made a build the walk does not take, and test the merged heads
mesh/merge-gate head c707bf18d8bf merged into trunk 04730b54a8f2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head c707bf18d8bf merged into trunk 04730b54a8f2 — pass: its merge-check.sh passed
mesh/delivery delivered
A build of another pull request or of a head the forge did not merge is
in no verdict the walk is handed; the walk now names the check that made
it, so its "built:" line says which head it was (novox/hq issue 703).
2026-10-14 10:05:31 +02:00
jschoubben 060f2a9ca5 Build for the merge from a copy its tests never touch, and keep or take only the same head's build
mesh/merge-gate head 060f2a9ca5fd merged into trunk 04730b54a8f2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 060f2a9ca5fd merged into trunk 04730b54a8f2 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The gate and the repository's check mount the workspace read-write, so a
build from that tree could hold what the tests wrote under a key that
describes the merge. A check now clones the merge outside the workspace
before anything of the change runs, and builds from that. A check keeps,
and a walk takes, only a build an earlier check of the same pull request
at the same head made: another pull request's build, or an earlier
head's, is never registered. The walk reads the verdicts once
(novox/hq issue 703, review of #328).
2026-10-14 09:56:23 +02:00
jschoubben 1666747a22 Keep a check build whose key is unchanged instead of building it again
mesh/merge-gate head 1666747a2246 merged into trunk 04730b54a8f2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 1666747a2246 merged into trunk 04730b54a8f2 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A fresh check asked when only the trunk moved rebuilt every image, though
the reuse key said nothing it is built from changed: a controller check
took about 12 minutes instead of 5.5. The check now keeps a build kept
under its key when a verdict of the build seat names that key and every
digest, names it as the build's own check so a walk takes it, and the
cleanup keeps it while any check names it (novox/hq issue 703).
2026-10-14 09:41:39 +02:00
mesh-admin 04730b54a8 Merge pull request 'S9 raises every bus user's slow consumer and closed connection, from the bus module's events (hq issue 270, ADR 0357)' (#326) from fix/270-the-bus-module-says-every-users-slow-consumer into main 2026-10-14 07:14:33 +00:00
jochen 45870ab31c Bound the read of bus-module events, keep S9 seeing when it fails, count the controller's own slow consumer once (hq issue 270, review of #326)
mesh/merge-gate head 45870ab31cc5 merged into trunk 3479a35462b2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 45870ab31cc5 merged into trunk 3479a35462b2 — pass: its merge-check.sh passed
mesh/delivery delivered
A storm of closures must not spend a watchdog tick, so the read stops at 2000
events and says the count is a floor. A failed read is its own condition
rather than blinding S9, so the controller's own signals are still said. The
controller's fall behind, told to its client and closed by the server, is one
fault, counted once.
2026-10-14 09:02:47 +02:00
jochen fd9308d328 Raise every bus user's slow consumer and closed connection as S9, from the bus module's events (hq issue 270, ADR 0357)
mesh/delivery-group group fix/270-the-bus-module-says-every-users-slow-consumer rejected: a member's own check failed
mesh/merge-gate head fd9308d3287e merged into trunk 3479a35462b2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head fd9308d3287e merged into trunk 3479a35462b2 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The controller heard a slow consumer only on its own connection: the server
says another user's only in its record of closed connections. The bus module
now says that record as events; the watchdogs read the last hour back on every
tick and raise one condition per user and kind, naming the module and machine.
The module check lets the bus module emit, since its tool runner publishes.
2026-10-14 08:41:47 +02:00
mesh-admin 3479a35462 Merge pull request 'List the provider-hold tests among the store tests (hq issue 145, to-be 48 Phase C)' (#319) from fix/145-a-consumer-is-held-under-its-provider into main
mesh/merge-gate head 3479a35462b2 merged into trunk 221fe5a60d1f — fail: head 3479a35462b2 does not merge cleanly into trunk 221fe5a60d1f: in conflict test…
mesh/repo-check head 3479a35462b2 merged into trunk 221fe5a60d1f — fail: head 3479a35462b2 does not merge cleanly into trunk 221fe5a60d1f: in conflict test…
mesh/delivery stopped: the pull request closed unmerged
2026-10-14 05:18:00 +00:00
jochen cebc593627 List the provider-hold tests among the store tests, so a skip cannot pass unseen (hq issue 145)
mesh/merge-gate head cebc593627d3 merged into trunk 2959c8143536 — pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes …
mesh/repo-check head cebc593627d3 merged into trunk 2959c8143536 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/145-a-consumer-is-held-under-its-provider delivered: every member is delivered
The rule 5 test of ADR 0240 (one provider down, three consumers, one
condition) and its healthy-provider twin need the store and skip without
one; unlisted, the merge check passed them without running them.
2026-10-14 03:21:50 +02:00
mesh-admin 2959c81435 Merge pull request 'Link a build's bases into its repository before pushing it (issue 694)' (#318) from fix/694-a-push-holds-its-bases into main 2026-10-14 01:19:24 +00:00
jochen 8933864a49 Link a build's bases into its repository before pushing it (issue 694)
mesh/merge-gate head 8933864a49d0 merged into trunk df9e5728a925 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 8933864a49d0 merged into trunk df9e5728a925 — pass: its merge-check.sh passed
mesh/delivery delivered
A runtime on containerd's image store, the default on a fresh docker 29,
need not hold the compressed layers of a base it built FROM: a layer it
had unpacked already is never fetched. Its push then fails on that layer
although the registry holds it under the base's mirror. A push reads
locally only what the target repository lacks, so mount the base's blobs
there first.
2026-10-14 02:59:53 +02:00
mesh-admin df9e5728a9 Merge pull request 'Try a pushed branch on the build seat before its pull request (hq issue 686)' (#314) from fix/686-a-branch-is-tested-before-its-pull-request into main 2026-10-14 00:57:46 +00:00
mesh-admin 2d18f438c5 Merge pull request 'A no-move tier waits for no new report, so its walk asks the next tier (issue 648)' (#309) from fix/648-a-no-move-tier-waits-for-nothing into main 2026-10-14 00:32:47 +00:00
jschoubben 3ce5a0239b Keep a try off the forge and bound how many run (hq issue 686)
mesh/merge-gate head 3ce5a0239bb1 merged into trunk 2d18f438c516 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 3ce5a0239bb1 merged into trunk 2d18f438c516 — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
Review of #314: a try's start named its commit, which the forge's holder
matches to an open pull request's head and marks pending. Its start now
names no repository or commit. One try per commit and four in all, a try
never ages ahead of deliveries or pull request checks, and its id is
checked before it is a bus subject.
2026-10-14 02:29:28 +02:00
jochen 70885ab3c4 Share one prefix between the no-move Why's writer and its reader (review of #309, issue 648)
mesh/delivery delivered
mesh/merge-gate head 70885ab3c4a3 merged into trunk ca3f43c400ba — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 70885ab3c4a3 merged into trunk ca3f43c400ba — pass: its merge-check.sh passed
2026-10-14 02:22:25 +02:00
jschoubben 65508d9df9 Try a pushed branch on the build seat before its pull request (hq issue 686)
mesh/merge-gate head 65508d9df9f4 merged into trunk ca3f43c400ba — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 65508d9df9f4 merged into trunk ca3f43c400ba — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery superseded: a newer head of the same pull request
Store tests skip without a store and no agent may start one, so each failure
cost a 15-30 minute review round. The try verb asks the build seat for the
repository's merge-check.sh on the ref merged into the trunk, as a check in
the build queue, and reads its outcome from the build's log: no status on
the forge, nothing registered, nothing built for a merge.
2026-10-14 02:20:37 +02:00
jschoubben 8123ca5654 Test that a branch is tried before its pull request (hq issue 686)
A builder's store tests first ran in the pull request's merge check, a review
round late. These fail on main: a try asks the repository's own check alone,
its outcome sets no status and registers nothing, an unknown ref is refused
before anything is asked, and the answer carries the failing lines.
2026-10-14 02:20:37 +02:00
jochen d9b86376d3 A no-move tier waits for no new report, so its walk asks the next tier (issue 648)
A build made from the source every machine running it runs is marked sent
and sent nothing, yet the tier's wait asked each machine for a report after
that marking. None follows, and the walk waited for ever. Each machine is
now judged by whether it answered its last send, the one whose builds made
the build no move.
2026-10-14 02:18:53 +02:00
mesh-admin ca3f43c400 Merge pull request 'The rest go at once after the first passes, and a gate survives a new controller (hq ADR 0353)' (#312) from fix/the-rest-in-parallel-and-gates-survive-a-new-controller into main 2026-10-13 23:43:02 +00:00
jschoubben b482122d66 Say a read error of a silent rest, and keep the node-engine one at a time in a kept step (hq ADR 0353)
mesh/delivery delivered
mesh/merge-gate head b482122d6646 merged into trunk 1601cb17432e — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head b482122d6646 merged into trunk 1601cb17432e — pass: its merge-check.sh passed
An empty answer on a read error would clear release-silent unseen.
2026-10-14 01:26:00 +02:00
jschoubben 2f5cc146fe Keep a walk's sends to what was kept, and say a silent rest (hq ADR 0353 review)
mesh/merge-gate head 2f5cc146fe46 merged into trunk 1601cb17432e — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 2f5cc146fe46 merged into trunk 1601cb17432e — fail: its merge-check.sh failed: --- FAIL: TestAFirstMachineThatFailsStillStopsTheWalk (…
mesh/delivery superseded: a newer head of the same pull request
A release's own gates held its other sends back for ever; an adopted send
lost what the machine ran before; a kept send could grow a build no gate
passed; a take-over matched by commit alone; a rest that went silent ended
done. The node-engine and the resolver still go one machine at a time, since
a fault that cuts a machine off shows only as silence.
2026-10-14 01:18:51 +02:00
jschoubben c2ea9b4c51 Send the rest at once after the first passes, and resume a stored gate (hq ADR 0353)
mesh/repo-check head c2ea9b4c5187 merged into trunk 1601cb17432e — fail: its merge-check.sh failed: --- FAIL: TestAReleaseSendsTheRestAtOnceAfterTheFirstPa…
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head c2ea9b4c5187 merged into trunk 1601cb17432e — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
A release now sends every other machine in one step once its first machine
passed, each judged by its own gate; a failing one is put back without
stopping the others, a silent one is left. A send is kept before it is made,
and a machine already sent what a gate is for is not sent again: the gate
counts from that send and takes over another walk's readings of the same
build there.
2026-10-14 01:06:49 +02:00
jschoubben b03d257f81 Test that the rest go at once after the first passes and a gate survives a new controller (hq ADR 0353)
The release walked four machines one after another in eleven minutes, and a
new controller re-sent and re-judged a build its machine already ran.
2026-10-14 01:06:49 +02:00
mesh-admin 1601cb1743 Merge pull request 'A walk's first-machine gate waits for a busy container, and mesh-builder busy drains the agent (issue 594, hq ADR 0335)' (#283) from feat/335-a-busy-container-is-replaced-when-idle into main 2026-10-13 22:20:55 +00:00
jschoubben 873b19566a Read the delivery group's order again, now the catalogue and hq left it
mesh/delivery delivered
mesh/merge-gate head 873b19566a82 merged into trunk 72214ae989b3 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 873b19566a82 merged into trunk 72214ae989b3 — pass: its merge-check.sh passed
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle delivered: every member is delivered
2026-10-14 00:08:44 +02:00
jschoubben b0753e2aac Let a drain lapse, bound the gate's wait on the controller's clock, keep a busy wait out of apply time
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle checking: 1 of 2 member(s) ready
mesh/merge-gate head b0753e2aac95 merged into trunk 72214ae989b3 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head b0753e2aac95 merged into trunk 72214ae989b3 — pass: its merge-check.sh passed
Review of #283: a drain whose replacement was withdrawn held the agent for
ever with its health green; the gate trusted the engine's until; a machine
of the rest waiting at the bound read as silent and its apply time held the
wait; an unknown mesh-builder word exited 0; plan said nothing of a busy an
older engine is not sent; and a node with a report and no send broke the
reading of every last report (the store test's NULL).
2026-10-14 00:07:31 +02:00
jschoubben a69c0e50dc Merge origin/main into feat/335, and pin mesh-host at #98's new head 2026-10-14 00:01:21 +02:00
mesh-admin 72214ae989 Merge pull request 'One build per change: the check builds what its merge moves, the walk takes it by key (ADR 0350, issue 649)' (#311) from feat/350-one-build-per-change into main 2026-10-13 21:30:25 +00:00
jschoubben 5c606fd1d4 Take only a check build its verdict names, and keep a build another check still names
mesh/merge-gate head 5c606fd1d48b merged into trunk 372faebf2824 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 5c606fd1d48b merged into trunk 372faebf2824 — pass: its merge-check.sh passed (run 2)
mesh/delivery delivered
mesh/delivery-group group feat/350-one-build-per-change delivered: every member is delivered
The artifact store takes writes from anything on its network, so a record
planted under another merge's key could have been registered: the walk now
takes a scratch build only when the build seat's own verdict for one of its
pull requests names its key and digests. Deleting by digest also took builds
another check shared (review of #311; novox/hq ADR 0350).
2026-10-13 23:16:55 +02:00
mesh-admin 372faebf28 Merge pull request 'The controller passes its first-node gate on one reading (ADR 0351)' (#310) from feat/351-the-controller-passes-on-one-reading into main 2026-10-13 21:14:27 +00:00
jschoubben 675466a8c3 Hold the store tests' modules as the settle time did, and fault a dip after any healthy reading
mesh/delivery delivered
mesh/merge-gate head 675466a8c35f merged into trunk bff5b7f0e4f8 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 675466a8c35f merged into trunk bff5b7f0e4f8 — pass: its merge-check.sh passed
Two tier tests relied on the two-minute settle to keep a module unjudged; the
witness window and the gate's spacing now hold it. A healthy reading taken
before the controller's floor counted no pass, so a dip after it did not take
the two-minute pass away (novox/hq ADR 0352 decision 3).
2026-10-13 23:07:15 +02:00
jschoubben cd49b50c40 Build a change once: the check builds what its merge moves, the walk takes it by key
mesh/merge-gate out of date: judged against trunk bff5b7f0e4f8, the trunk is now 372faebf2824; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk bff5b7f0e4f8, the trunk is now 372faebf2824; judged again on a recheck or when it is merged
mesh/delivery-group group feat/350-one-build-per-change checking: composed check out of date: novox/mesh-controller@cd49b50c4075 was judged against trunk bff5b…
mesh/delivery superseded: a newer head of the same pull request
A merge built every moved module a second time although its check had judged
the same tree (issue 649). The check now builds into the scratch namespace
under a reuse key (source fingerprint plus the builder's own digest), and the
walk of the trunk commit copies a build of the same key by digest instead of
building, saying per module why when it cannot (novox/hq ADR 0350).
2026-10-13 23:04:40 +02:00
jschoubben 41eefc2f8c Test that a healthy node-engine passes at two minutes on one reading
mesh/merge-gate head 41eefc2f8c3e merged into trunk bff5b7f0e4f8 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 41eefc2f8c3e merged into trunk bff5b7f0e4f8 — fail: its merge-check.sh failed: --- FAIL: TestACoreBehindWhileTheNodeToolsSettleFailsNo…
mesh/delivery superseded: a newer head of the same pull request
2026-10-13 22:58:37 +02:00
jschoubben c3946e5d72 Pass the node-engine and resolver on their third reading, at most two minutes
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head c3946e5d72c2 merged into trunk bff5b7f0e4f8 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head c3946e5d72c2 merged into trunk bff5b7f0e4f8 — fail: its merge-check.sh failed: --- FAIL: TestACoreBehindWhileTheNodeToolsSettleFailsNo…
The operator wants these gates as fast as they can be: counting readings only after
two minutes added about 80 s to every node-engine rollout for no safety. Count
from the send, pass on the third healthy reading, and pass a module healthy at
every reading at two minutes on what it has (novox/hq ADR 0352).
2026-10-13 22:55:46 +02:00
jschoubben 524295926f Pass the controller's first-node gate on one reading after the witness window
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 524295926f76 merged into trunk bff5b7f0e4f8 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 524295926f76 merged into trunk bff5b7f0e4f8 — fail: its merge-check.sh failed: --- FAIL: TestACoreBehindWhileTheNodeToolsSettleFailsNo…
The controller runs on one machine, so three readings over two minutes guarded no
other machine and held every walk behind it. Judge each module by its own rule:
one reading for the controller and the tool runner once the node-engine's witness
window has ended, one for mesh-delivery, three for the node-engine and the
resolver with the first two minutes after the send (novox/hq ADR 0351). The
gate's verdict names the rule it applied.
2026-10-13 22:39:32 +02:00
mesh-admin bff5b7f0e4 Merge pull request 'A settings proposal reads at a glance, its values whole where the sender is proven (issues 383 and 639)' (#304) from fix/383-an-ask-reads-at-a-glance into main 2026-10-13 19:22:06 +00:00
mesh-admin cef22a2526 Merge pull request 'A provider serves its consumers its own composed names (ADR 0346, issue 624)' (#306) from feat/346-a-provider-serves-its-own-names into main 2026-10-13 19:11:16 +00:00
jochen deef9421a2 A change's line the router would refuse is named with its value withheld, or refused at propose (the review of #304)
mesh/merge-gate head deef9421a276 merged into trunk cef22a25264b — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head deef9421a276 merged into trunk cef22a25264b — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/383-an-ask-reads-at-a-glance delivered: every member is delivered
{a:1, b:2, c:3, password:hunter2} was proposed with "+ password: hunter2" in its Details: the router
withheld the line unnamed and Approve set a value the operator never saw (ADR 0347 §1, §3, §4). Each line is
now judged as the router judges it, masked under the content rule and whole under the secret rule; a line
refused is said "+ password (value withheld)", and any line the router would still refuse is refused at
propose in the controller's words, never echoing a secret-shaped key. SDK at mesh-sdk 5933eb7 (no two asks
digest the same).
2026-10-13 20:45:25 +02:00
jochen 3c19e9ee93 A settings proposal's summary says the layer is sent at once by a walk (ADR 0344 with ADR 0345)
Rebased on #305: Approve now sends the layer by a walk, so the summary's "takes it at its next push" was
false. The summary says it is set and sent at once by a walk to where it reaches, one machine at a time,
each judged; the rest of what Approve does (onApprove) leads the Details, beside every key whole.
2026-10-13 20:45:25 +02:00
jochen 3d94f87bb0 The glance's cut says only what the note holds there: whole beside the whole details, else in the note (ADR 0347) 2026-10-13 20:45:25 +02:00
jochen 4644f25edd A settings proposal carries every key with its whole value in its Details (issue 383)
The review of #304: with more than three keys the ask showed only the names of the rest, a long value was
cut without a word, and the Details held no key, so Approve bound values the operator never saw. The
Details now list every key uncut — masked, and whole in WholeDetails for a channel that proves who
answers — before who proposed it and the fingerprint; the glance says what it cut. Vendors mesh-sdk at
f474544 and captures its conformance events there.
2026-10-13 20:45:25 +02:00
jochen c58e0a571e A settings proposal reads at a glance, its values whole where the sender is proven (issues 383 and 639)
The operator could not read the settings proposals on the phone: the change, the fingerprint and a how-to
paragraph in one wall of plain text, and the values masked as ‹path› and ‹address›, so the approval was
blind.

The ask is now a summary of what changes where ("Changes 3 setting(s) of mounts on shanks; approved, the
layer is set at once and shanks takes it at its next push."), then at most three points, one key each (more
keys: the first two and the names of the rest). Who proposed it, the fingerprint and how to read it whole
are its Details, which the router notes after the answers. The change is carried twice: under the content
rule, and whole (asks.Ask.Whole), shown only on a channel that proves who answers, a value shaped like a
secret withheld in both (from the earlier work on issue 383, #202).

Vendors mesh-sdk at 9588871 (fix/639-an-ask-reads-at-a-glance: Message.Note, asks.Glance), and captures its
conformance events at that commit.
2026-10-13 20:45:25 +02:00
mesh-admin 332fbaa9b1 Merge pull request 'settings: no layer replaces a value the vault fills, at any depth (issue 610)' (#308) from fix/610-no-layer-replaces-a-vault-secret into main 2026-10-13 18:33:41 +00:00
jochen 7a0c219f96 List the provider-elsewhere store test, read a machine's domain as its profile, reuse the settings read once, and refuse a nested own name (hq ADR 0346, review 228)
mesh/merge-gate head 7a0c219f961e merged into trunk 332fbaa9b175 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 7a0c219f961e merged into trunk 332fbaa9b175 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-13 20:32:51 +02:00
jochen 985423ae61 Say that a provider's own name is not on its consumers' entries (hq ADR 0346) 2026-10-13 20:29:17 +02:00
jochen cfacda2625 Let a provider serve its consumers its own composed names (hq ADR 0346, issue 624)
The forge writes every package tarball at its public name, and its
consumers had no source for that name but a second copy kept by hand,
though the mesh composes it for the forge's route. A served value may now
name the provider's own names; a name its machine does not compose leaves
the key out, and the plan says so.
2026-10-13 20:29:17 +02:00
mesh-admin bf6beec465 settings: no layer replaces a value the vault fills, at any depth (issue 610)
mesh/merge-gate head bf6beec465c2 merged into trunk 9f3aa91e812b — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head bf6beec465c2 merged into trunk 9f3aa91e812b — pass: its merge-check.sh passed
mesh/delivery delivered
A mergeable file's protected keys were checked at the top level only, so a layer could still set searxng's server.secret_key under the server key its file takes. The merge now refuses any layer value that would replace a ${secret:} placeholder in the file's own content, at any depth, whatever the module's protected list says.
2026-10-13 17:43:24 +00:00
mesh-admin 9f3aa91e81 Merge pull request 'List the settings walk's store tests, and drop a refusal check that never matched (issue 641 review)' (#307) from fix/641-list-the-settings-store-tests into main 2026-10-13 17:35:00 +00:00
jochen 4bdcef3f73 List the settings walk's store tests, and drop a refusal check that never matched (issue 641 review)
mesh/merge-gate head 4bdcef3f73b5 merged into trunk 8ddf8b700558 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 4bdcef3f73b5 merged into trunk 8ddf8b700558 — pass: its merge-check.sh passed
mesh/delivery delivered
A store test not named in testdata/store-tests skips unseen without a store
(issue 459). The check after the send could not match: the send turns its
refusals into words; the composition before it already fails the walk.
2026-10-13 19:24:02 +02:00
mesh-admin 8ddf8b7005 Merge pull request 'Send an approved settings layer by a walk at once, so the operator's Approve takes effect (issue 641, ADR 0344)' (#305) from fix/641-an-approved-setting-is-sent into main 2026-10-13 17:22:20 +00:00
jochen 8bbb1b19ae A settings gate keeps the module's build, and a walk that cannot keep a build fails (issue 641 review)
mesh/merge-gate head 8bbb1b19ae63 merged into trunk 6f1a05366c1d — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 8bbb1b19ae63 merged into trunk 6f1a05366c1d — pass: its merge-check.sh passed
mesh/delivery delivered
Without a build on the gate, a merge walk sending the machine a new build of
the module was read as the settings send's own report. A refusal for a last
send not known, or a kept build gone, never clears, so retrying it every
tick held the walk of waiting builds; it now fails the walk with why. Every
failure names the machines passed over as away (ADR 0344 §6).
2026-10-13 17:33:07 +02:00
jochen e32ca00aed Test that a settings send moves no ungated build waiting on its machine (issue 641, ADR 0344 §3)
mesh/merge-gate head e32ca00aed9d merged into trunk 6f1a05366c1d — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head e32ca00aed9d merged into trunk 6f1a05366c1d — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-13 16:55:49 +02:00
jochen 1219ec8229 A settings walk sends the layer alone, every build kept, and stops before a data move (issue 641, ADR 0344)
mesh/merge-gate head 1219ec822943 merged into trunk 6f1a05366c1d — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 1219ec822943 merged into trunk 6f1a05366c1d — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A judged send carries what waits on its machine, which would have released
builds held after a failed walk without a person's word. The settings walk
now keeps every build of its machine, refuses a send that would move a
running module's data, and never says delivered when it reached no machine.
2026-10-13 16:50:10 +02:00
jochen e324f5d821 Send an approved settings layer by a walk at once, so the operator's Approve is the act that takes effect (issue 641, ADR 0344)
mesh/merge-gate head e324f5d82162 merged into trunk 6f1a05366c1d — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head e324f5d82162 merged into trunk 6f1a05366c1d — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The warrant set the layer and left the push as a second act nobody took: the
operator approved a betterbird change on Telegram and nothing reached g14.
Now the warrant opens a settings walk: the machines the layer reaches, one at
a time, the module judged on each at its gate, its end told once through
plan-moved. The ask and the proposer's answer say so instead of naming push.
2026-10-13 16:39:47 +02:00
mesh-admin 6f1a05366c Merge pull request 'Admit a port from the machine's own local network, and never a public subnet (issue 154, ADR 0340)' (#299) from fix/154-local-network into main 2026-10-13 13:07:11 +00:00
mesh-admin 0a1dc5f93f Merge pull request 'seat: a push to a named machine waits for the machine's answer, so its call says how it went (issue 629)' (#301) from fix/629-a-push-says-how-it-went into main 2026-10-13 12:44:51 +00:00
mesh-admin ae0b384497 push-wait is untrusted, so it is set in one call like any preference (issue 629)
mesh/merge-gate head ae0b384497c2 merged into trunk f3875a6b6019 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head ae0b384497c2 merged into trunk f3875a6b6019 — pass: its merge-check.sh passed
mesh/delivery delivered
The gate warned that the push file asks for a setting without saying whether it is trusted, which makes push-wait terminal-only.
2026-10-13 12:38:31 +00:00
mesh-admin f3875a6b60 Merge pull request 'catalogue: say in the comments that a package may be declared by several modules (issue 628)' (#300) from fix/628-packages-are-shared into main 2026-10-13 12:38:30 +00:00
mesh-admin f09c5ebdec push: the wait is the call's or the controller's setting push-wait, never a length in the code (issue 629)
mesh/merge-gate head f09c5ebdec6c merged into trunk 584aa2d7d7a5 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head f09c5ebdec6c merged into trunk 584aa2d7d7a5 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The operator's rule: a wait is never hard-coded, at most a default is configured. The push verb takes `wait`; without it the controller's setting `push-wait` (module default 2m) applies; a setting that cannot be read waits for nothing, as a push did before, and says so.
2026-10-13 12:09:43 +00:00
mesh-admin 6d2e4c299a seat: a push to a named machine waits for the machine's answer, so its call says how it went (issue 629)
mesh/merge-gate head 6d2e4c299a34 merged into trunk 584aa2d7d7a5 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 6d2e4c299a34 merged into trunk 584aa2d7d7a5 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The seat ran `push <node> --wait 0`, so a push's call ended at "sent" and nobody learned whether the machine applied it. The caller is answered before a push starts (answersFirst, issue 265), so waiting holds nobody: the call now waits up to two minutes for the machine's own report and keeps it, applied, refused with the machine's reason, or each failed resource with its error.
2026-10-13 12:03:53 +00:00
mesh-admin b654b320a1 catalogue: say in the comments that a package may be declared by several modules (issue 628)
mesh/merge-gate head b654b320a18a merged into trunk 584aa2d7d7a5 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head b654b320a18a merged into trunk 584aa2d7d7a5 — pass: its merge-check.sh passed
mesh/delivery delivered
The resolver follows ADR 0303, but four comments still stated ADR 0210's retired rule that two modules declaring one package collide, and an agent took them as the rule. What stays exclusive is what a module owns: a path, a unit, a name, and a package declared present by one module and absent by another.
2026-10-13 11:52:17 +00:00
jochen 4cc8588afb Test that a global IPv6 subnet is public and never counts as the local network (issue 154, ADR 0340)
mesh/merge-gate head 4cc8588afb87 merged into trunk 0a1dc5f93fcb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 4cc8588afb87 merged into trunk 0a1dc5f93fcb — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/154-local-network delivered: every member is delivered
The review asked for the IPv6 side of the rule that local-network cannot open a port to the internet.
2026-10-13 13:20:50 +02:00
mesh-admin 584aa2d7d7 Merge pull request 'Never make a credential only a person can give, and give it once per provider (hq issue 164, ADR 0337)' (#298) from fix/164-an-accepted-credential-is-never-minted into main 2026-10-13 09:52:07 +00:00
jochen c9dd07f919 Never make a credential only a person can give, and give it once per provider (hq issue 164, ADR 0337)
mesh/merge-gate head c9dd07f919a2 merged into trunk 404b4409e70e — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head c9dd07f919a2 merged into trunk 404b4409e70e — pass: its merge-check.sh passed (run 2)
mesh/delivery delivered
A Servarr app's API key, an indexer's key and a downloader's password are values the provider's
software holds; the pair path made a random value for every consumer of them, sealed it to both ends
and reported success, and the provider refused it. An offer may now say `"credential": {"issued-by":
"outside"}`: SecretFor never makes a value for it, the plan reads such a pair as given or not given
and sends the consumer no credential file until a person gives it, without refusing its machine;
remakeShared and SharedSecretFor never make a provider's own secret issued outside either.
`secret accept <provider's node> <provider module> <provision>` seals one value to every consumer that
binds the provision from that provider, the provider and the operator, under one generation; the
per-pair form is refused for such a provision, and rotate names the accept. A consumer may wait for a
provision's credential: the controller excuses the wait only when the store bears it out, and its
needs-operator condition names the one accept line.

Adds the replay of issue 164 (TestReplay164), written with only what the controller had before.
2026-10-13 11:37:31 +02:00
mesh-admin 404b4409e7 Merge pull request 'A built module keeps its build edges: a new SDK plans the toolchain and its TypeScript bundles (hq issue 614)' (#296) from fix/614-a-built-module-keeps-its-build-edges into main 2026-10-13 09:35:26 +00:00
jochen 6e6cd5e26d Admit a port from the machine's own local network, and never a public subnet (issue 154, ADR 0340)
mesh/delivery-group group fix/154-local-network checking: 0 of 2 member(s) ready
mesh/merge-gate out of date: judged against trunk 1826b68a22b5, the trunk is now 404b4409e70e; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 1826b68a22b5, the trunk is now 404b4409e70e; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
A local service could only be declared public, which says the internet. local-network is a source and a reach: the private network and the private networks the machine reported on a link facing outside, judged here, shown by plan and node, and said in the rule set when there are none.
2026-10-13 11:20:22 +02:00
jochen e658a14296 Read a built module's build edges from what is kept of it, so a new SDK rebuilds its toolchain and bundles (hq issue 614)
mesh/merge-gate head e658a14296f9 merged into trunk 1826b68a22b5 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head e658a14296f9 merged into trunk 1826b68a22b5 — pass: its merge-check.sh passed
mesh/delivery delivered
A registered manifest carries no build, so mesh-tools' build.on naming the
SDK's package and every bundle's toolchain were edges only before a module was
first built: mesh-sdk 0.1.14 planned "sent nowhere". The relation now reads a
package base from the recorded name@version against the package the mesh holds,
and the toolchain from the bundles the resolved manifest keeps. Only records the
mesh already keeps are read, so it holds for every module built before it and a
rollback reads the same records. Replay R614.
2026-10-13 11:08:17 +02:00
jschoubben b1546ed28a Hold a walk's first-machine gate while its busy container finishes its build
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle rejected: its order contradicts itself: novox/mesh-catalog@190608193764, novox/mesh-c…
mesh/merge-gate out of date: judged against trunk 56d6313262e8, the trunk is now cbac703e817d; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 56d6313262e8, the trunk is now cbac703e817d; judged again on a recheck or when it is merged
A merge into mesh-controller replaced every build agent and threw away the
checks in flight (issue 594). The manifest's container busy is checked and
sent only to an engine that reads it; a report's busy waits are kept, the
gate starts at the report that says the container replaced, and the wait is
a phase of the delivery time. mesh-builder busy drains the agent and answers
while a build is in hand (novox/hq ADR 0335).
2026-10-13 00:53:10 +02:00
457 changed files with 60401 additions and 1186 deletions
+6
View File
@@ -0,0 +1,6 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
+146
View File
@@ -0,0 +1,146 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
// **The build agent says whether it is busy, and being asked drains it** (novox/hq ADR 0335 §1, §5). Its
// container declares `busy` as `mesh-builder busy`: the node-engine runs it inside the container before it
// replaces it. It exits 0 while a build is in hand and 1 when none is; and from the first ask the agent takes
// no new build request, so its answer ends once the build in hand ends — a replaced builder finishes the check
// it runs, instead of the build queue handing it out again from the start (issue 594).
//
// Two marks in the container's own temporary directory, as the ready mark (ready.go): a new container starts
// without them, and a restarted one removes them before it takes work again.
//
// - in hand: written by the agent when it claims a request, before it reads whether it is drained; removed
// when the build ends.
// - drained: written by the ask, before it reads whether a build is in hand.
//
// Each side writes its own mark before it reads the other's, so an ask racing a fetch sees the build in hand,
// or the agent sees the drain and hands the request back; never neither.
func inHandMark() string { return filepath.Join(os.TempDir(), "mesh-builder.in-hand") }
func drainMark() string { return filepath.Join(os.TempDir(), "mesh-builder.drained") }
// The drain lapses when no ask has come for drainLapse: the node-engine asks every 20 seconds for as long as it
// really waits, so an agent whose replacement was withdrawn — the declaration put back, the walk stopped —
// takes work again rather than staying drained for ever (review of ADR 0335). And an agent drained for longer
// than the node-engine's bound and the slack says it is not ready: the replacement it was drained for never
// came, which is a fault to see.
const (
drainLapse = 2 * time.Minute
drainTooLong = 15*time.Minute + 5*time.Minute
)
// drainedAt is when the drain began and when it was last asked, from its mark; false when there is no mark.
func drainedAt() (began, asked time.Time, ok bool) {
info, err := os.Stat(drainMark())
if err != nil {
return time.Time{}, time.Time{}, false
}
asked = info.ModTime()
began = asked
if raw, err := os.ReadFile(drainMark()); err == nil {
if t, err := time.Parse(time.RFC3339, strings.TrimSpace(string(raw))); err == nil && !t.After(asked) {
began = t
}
}
return began, asked, true
}
// drained says the agent was asked whether it is busy within drainLapse: it takes no new build request.
func drained() bool { return drainedSince(time.Now()) }
func drainedSince(now time.Time) bool {
_, asked, ok := drainedAt()
return ok && now.Sub(asked) < drainLapse
}
// drainedTooLong is the health check's second half: nil unless the agent has been drained, asked all along, for
// longer than its node-engine's bound and the slack.
func drainedTooLong(now time.Time) error {
began, asked, ok := drainedAt()
if !ok || now.Sub(asked) >= drainLapse || now.Sub(began) <= drainTooLong {
return nil
}
return fmt.Errorf("drained since %s and still asked: the replacement its node-engine waits for has not come "+
"within %s (novox/hq ADR 0335)", began.UTC().Format(time.RFC3339), drainTooLong)
}
// claim marks a request in hand, unless the agent is drained: then nothing is in hand, and the request goes
// back to the queue.
func claim() bool {
// Made, never written over: one build at a time, and a mark already there is a build in hand.
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
// Unmarked, an ask would say idle while this build runs: the build is not taken.
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
_, err = f.WriteString(time.Now().UTC().Format(time.RFC3339) + "\n")
if closeErr := f.Close(); err == nil {
err = closeErr
}
if err != nil {
_ = os.Remove(inHandMark())
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
if drained() {
_ = os.Remove(inHandMark())
fmt.Fprintln(os.Stderr, "asked whether busy, so this agent takes no new build: handed back")
return false
}
return true
}
// holding says what is in hand, for the ask to print: written once the request is read.
func holding(what string) {
_ = os.WriteFile(inHandMark(), []byte(time.Now().UTC().Format(time.RFC3339)+" "+what+"\n"), 0o644)
}
// released ends the build in hand.
func released() {
if err := os.Remove(inHandMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
fmt.Fprintf(os.Stderr, "cannot remove the mark of the build in hand (%s): %v\n", inHandMark(), err)
}
}
// undrained clears both marks at the start of a holder: a container started again is asked afresh.
func undrained() error {
for _, mark := range []string{drainMark(), inHandMark()} {
if err := os.Remove(mark); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove %s: %w", mark, err)
}
}
return nil
}
// busy is the ask: it drains the agent, then says whether a build is in hand. Nil — exit 0 — while one is.
func busy() (string, error) {
// The mark keeps when the drain began while it is asked within drainLapse, and is written again on every
// ask: its time is the last ask.
began := time.Now().UTC()
if was, _, ok := drainedAt(); ok && drained() {
began = was.UTC()
}
if err := os.WriteFile(drainMark(), []byte(began.Format(time.RFC3339)+"\n"), 0o644); err != nil {
// Not drained, it could take the next build: said, and answered busy only for what is in hand now.
fmt.Fprintf(os.Stderr, "cannot mark this agent drained (%s): %v\n", drainMark(), err)
}
held, err := os.ReadFile(inHandMark())
if err != nil {
return "", errors.New("idle: no build in hand, and this agent takes no new one")
}
since, what, _ := strings.Cut(strings.TrimSpace(string(held)), " ")
if what == "" {
what = "a build"
}
return fmt.Sprintf("busy: %s, in hand since %s; taking no new build", what, since), nil
}
+162
View File
@@ -0,0 +1,162 @@
package main
import (
"os"
"strings"
"sync"
"testing"
"time"
)
// `mesh-builder busy` answers 0 while a build is in hand and non-zero when idle, and after its first ask the
// agent takes no new build request (novox/hq ADR 0335 §5).
func TestBusyAnswersWhileABuildIsInHandAndDrainsTheAgent(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("an agent never asked is drained")
}
// A build is claimed and runs.
if !claim() {
t.Fatal("an agent never asked refused a build")
}
holding("novox/mesh-controller (check-280)")
said, err := busy()
if err != nil || !strings.Contains(said, "novox/mesh-controller (check-280)") {
t.Fatalf("an agent with a build in hand did not answer busy with what it holds: %q %v", said, err)
}
// From that ask on it takes nothing new: the fetch is not made, and one fetched is handed back.
if !drained() {
t.Fatal("the ask did not drain the agent")
}
if claim() {
t.Fatal("a drained agent took a new build")
}
// The build in hand is still in hand after the refused claim, and busy goes on saying so.
if _, err := busy(); err != nil {
t.Fatalf("a refused claim ended the build in hand: %v", err)
}
// It ends: idle, and still drained.
released()
if _, err := busy(); err == nil {
t.Fatal("an agent with nothing in hand answered busy")
}
if claim() {
t.Fatal("a drained agent took a build after its last one ended")
}
// Started again, it is asked afresh.
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() || !claim() {
t.Fatal("a holder started again is still drained")
}
released()
}
// An ask racing a fetch sees the build in hand, or the agent sees the drain and hands the request back: never
// neither, which would replace the container under a build just taken.
func TestAnAskRacingAFetchNeverMissesTheBuild(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
for i := 0; i < 200; i++ {
if err := undrained(); err != nil {
t.Fatal(err)
}
var claimed bool
var askErr error
var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); claimed = claim() }()
go func() { defer wg.Done(); _, askErr = busy() }()
wg.Wait()
if claimed && askErr != nil {
t.Fatalf("round %d: the agent took a build and the ask said idle", i)
}
released()
}
}
// A drain no longer asked lapses: an agent whose replacement was withdrawn after the first ask (a put-back, a
// revert, a stopped walk) takes work again after drainLapse, rather than staying drained with its health green.
func TestADrainNoLongerAskedLapses(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if !drained() || claim() {
t.Fatal("an agent just asked takes work")
}
// The last ask was three minutes ago.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("a drain no longer asked for three minutes still holds")
}
if !claim() {
t.Fatal("an agent whose drain lapsed takes no work")
}
released()
}
// An agent drained, and asked all along, for longer than its node-engine's bound and the slack is not ready: the
// replacement it was drained for never came. One asked again after a lapse starts a new drain.
func TestAnAgentDrainedTooLongIsNotReady(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
taking("mesh-build-machine")
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
now := time.Now()
if err := drainedTooLong(now); err != nil {
t.Fatalf("an agent drained a moment ago is not ready: %v", err)
}
// The drain began 25 minutes ago, and the node-engine asked a moment ago.
began := now.Add(-25 * time.Minute).UTC().Format(time.RFC3339)
if err := os.WriteFile(drainMark(), []byte(began+"\n"), 0o644); err != nil {
t.Fatal(err)
}
err := drainedTooLong(time.Now())
if err == nil || !strings.Contains(err.Error(), "has not come") {
t.Fatalf("an agent drained for 25 minutes reads ready: %v", err)
}
// Each ask keeps when the drain began.
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if err := drainedTooLong(time.Now()); err == nil {
t.Fatal("an ask started the drain again")
}
// The asks stopped: the drain lapsed, and so did the fault.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if err := drainedTooLong(time.Now()); err != nil {
t.Fatalf("a lapsed drain still reads not ready: %v", err)
}
}
// An ask this build does not know exits non-zero: an older image asked `busy` reads idle, not busy for the
// node-engine's whole bound.
func TestAnUnknownCommandFails(t *testing.T) {
was := os.Args
t.Cleanup(func() { os.Args = was })
os.Args = []string{"mesh-builder", "some-later-word"}
if err := run(); err == nil {
t.Fatal("an unknown command exited 0")
}
os.Args = []string{"mesh-builder", "help"}
if err := run(); err != nil {
t.Fatalf("help failed: %v", err)
}
}
+31
View File
@@ -41,6 +41,9 @@ type holder struct {
say func(link.HolderState) error say func(link.HolderState) error
// remove runs what a kill needs outside the build: the containers left behind. // remove runs what a kill needs outside the build: the containers left behind.
remove builder.Runner remove builder.Runner
// scratch is where check builds are kept (novox/hq ADR 0350 §5): what the week's sweep lets go of. Nil where
// this machine publishes to no artifact store.
scratch builder.Scratch
mu sync.Mutex mu sync.Mutex
paused bool paused bool
@@ -131,6 +134,34 @@ func (h *holder) stateWhilePaused(ctx context.Context, every time.Duration) {
} }
} }
// expireChecks lets go of every check build older than a week (novox/hq ADR 0350 §5), at start and every while
// after. Every holder sweeps the one store; what one let go of, another finds gone.
func (h *holder) expireChecks(ctx context.Context, every time.Duration, now func() time.Time) {
sweep := func() {
if h.scratch == nil {
return
}
n, err := h.scratch.Expire(ctx, builder.CheckBuildsKept, now())
switch {
case err != nil:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d let go of, and not all: %v\n", builder.CheckBuildsKept, n, err)
case n > 0:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d check(s)' builds let go of\n", builder.CheckBuildsKept, n)
}
}
sweep()
tick := time.NewTicker(every)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
sweep()
}
}
}
// begin records the build this machine took, with the cancel that ends it. // begin records the build this machine took, with the cancel that ends it.
func (h *holder) begin(request link.BuildRequest, cancel context.CancelFunc) *running { func (h *holder) begin(request link.BuildRequest, cancel context.CancelFunc) *running {
r := &running{request: request, started: time.Now(), cancel: cancel, done: make(chan struct{})} r := &running{request: request, started: time.Now(), cancel: cancel, done: make(chan struct{})}
+182 -12
View File
@@ -26,6 +26,7 @@ import (
"os/signal" "os/signal"
"sort" "sort"
"strings" "strings"
"sync"
"syscall" "syscall"
"time" "time"
@@ -75,6 +76,12 @@ digest of their own configuration. The result is printed as JSON.
exits 0 once the agent running in this container is bound to its build work and pulls it, and exits 0 once the agent running in this container is bound to its build work and pulls it, and
1 before: its container's health check. 1 before: its container's health check.
mesh-builder busy
exits 0 while the agent running in this container has a build in hand, and 1 when it has none;
from the first ask the agent takes no new build: its container's busy command, asked before the
container is replaced (novox/hq ADR 0335).
` `
func run() error { func run() error {
@@ -86,17 +93,41 @@ func run() error {
case "build": case "build":
return buildOnce(context.Background(), os.Args[2:]) return buildOnce(context.Background(), os.Args[2:])
case "ready": case "ready":
// The container's health check (novox/hq issue 544): exits non-zero until the agent takes work. // The container's health check (novox/hq issue 544): exits non-zero until the agent takes work, and
return ready() // once it has been drained longer than its node-engine would wait to replace it (ADR 0335).
default: if err := ready(); err != nil {
return err
}
return drainedTooLong(time.Now())
case "busy":
// The container's busy command (novox/hq ADR 0335): drains the agent, and exits non-zero once
// nothing is in hand.
said, err := busy()
if err != nil {
return err
}
fmt.Println(said)
return nil
case "help", "-h", "--help":
fmt.Print(usage) fmt.Print(usage)
return nil return nil
default:
// **An ask this build does not know fails** (novox/hq ADR 0335): an image older than `busy`, asked
// it, must read as idle — exiting 0 would hold its container for the node-engine's whole bound.
fmt.Fprint(os.Stderr, usage)
return fmt.Errorf("%q is not a command this mesh-builder knows", os.Args[1])
} }
} }
if err := notReady(); err != nil { if err := notReady(); err != nil {
return err return err
} }
// **The builder's own version is part of every reuse key** (novox/hq ADR 0350 §1): a builder that builds
// differently never takes a build another one made.
builder.Version = builder.SelfVersion(version)
if err := undrained(); err != nil {
return err
}
credential, err := brokerFrom() credential, err := brokerFrom()
if err != nil { if err != nil {
return err return err
@@ -149,8 +180,13 @@ func run() error {
// **What an earlier holder here left goes before anything is taken** (novox/hq issue 306): a holder // **What an earlier holder here left goes before anything is taken** (novox/hq issue 306): a holder
// recreated mid-check left the check's store and bus running, and the ask went to another machine. // recreated mid-check left the check's store and bus running, and the ask went to another machine.
h.removeLeftBehind() h.removeLeftBehind()
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused, // Paused or drained, it takes nothing new (novox/hq ADR 0219, ADR 0335): a drained agent finishes the build
Taking: func() { taking(seat) }}) // in hand and then says idle, and its container is replaced between builds.
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{
Paused: func() bool { return h.Paused() || drained() },
Taking: func() { taking(seat) },
Claim: claim,
})
defer machine.Close() defer machine.Close()
// The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired // The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired
@@ -174,8 +210,13 @@ func run() error {
fmt.Fprintf(os.Stderr, "building for the mesh, publishing to %s\n", registry) fmt.Fprintf(os.Stderr, "building for the mesh, publishing to %s\n", registry)
publisher := builder.Registry{Address: registry, Run: builder.Command} publisher := builder.Registry{Address: registry, Run: builder.Command}
// Check builds are let go of after a week (novox/hq ADR 0350 §5).
h.scratch = publisher
go h.expireChecks(ctx, time.Hour, time.Now)
return machine.Take(ctx, func(ctx context.Context, work link.Build) { return machine.Take(ctx, func(ctx context.Context, work link.Build) {
defer released()
holding(work.Request().Repository + " (" + work.Request().ID + ")")
answer(ctx, publisher, on, workspace, work, h) answer(ctx, publisher, on, workspace, work, h)
}) })
} }
@@ -232,7 +273,12 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// role's own events under the build's id (novox/hq ADR 0157) — so whoever asked, and anybody // role's own events under the build's id (novox/hq ADR 0157) — so whoever asked, and anybody
// watching, reads the same lines this container's log holds, live, and after the fact from the // watching, reads the same lines this container's log holds, live, and after the fact from the
// stream. Said first, before anything runs, so a build that hangs is one that visibly started. // stream. Said first, before anything runs, so a build that hangs is one that visibly started.
// One line at a time: a check's builds for its merge say their lines while its tests say theirs (novox/hq
// issue 716).
var saying sync.Mutex
say := func(step, message string) { say := func(step, message string) {
saying.Lock()
defer saying.Unlock()
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message) fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
work.Say(step, message) work.Say(step, message)
if mine != nil && step != "run" && step != "output" { if mine != nil && step != "run" && step != "output" {
@@ -262,6 +308,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if err == nil { if err == nil {
say("packages", saysRegistry(npmrc)) say("packages", saysRegistry(npmrc))
} }
registryHere, isRegistry := publisher.(builder.Registry)
var built builder.Result var built builder.Result
if request.Check != nil { if request.Check != nil {
// **A pull request's merge check, not a build** (novox/hq to-be 45 §9): nothing is built, // **A pull request's merge check, not a build** (novox/hq to-be 45 §9): nothing is built,
@@ -271,17 +318,38 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if r, ok := publisher.(builder.Registry); ok { if r, ok := publisher.(builder.Registry); ok {
registry = r.Address registry = r.Address
} }
spec := checkSpecOf(request)
// The judge cache the build agent declares as its own data, never in the workspace: only judge builds and
// mesh-lab's replays mount it, never a check's containers, so no pull request's script can write what a judge
// is built with or kept as, or what the replays run (novox/hq ADR 0358 §6, issue 709).
spec.JudgeCache = judgeCache()
// What the merge would move, built by the check for the walk to take (novox/hq ADR 0350 §1): into the
// scratch namespace. **Never with the package registry's credential**: this is code nobody has approved
// yet, and a recipe could read the credential it is handed (issue 462). A module that resolves packages
// at build time has no reuse key anyway, so nothing is lost by refusing it here.
if isRegistry && len(spec.Builds) > 0 {
spec.BuildWith = &builder.CheckBuilding{Publish: publisher, Scratch: registryHere, Held: request.Held,
Running: request.Check.Running, Seats: request.Seats, Named: namedOf(request.Check.Named)}
}
var v builder.CheckVerdict var v builder.CheckVerdict
v, err = builder.Check(building, builder.Command, checkSpecOf(request), workspace, registry, forgeFrom(), say) v, err = builder.Check(building, builder.Command, spec, workspace, registry, forgeFrom(), say)
if err == nil { if err == nil {
result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report, result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report,
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo), Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo),
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside} Head: v.Head, Trunk: v.Trunk, Beside: v.Beside, Built: checkBuiltOf(v.Built),
Release: v.Release, JudgedBy: v.JudgedBy, BesideAt: v.BesideAt, Validator: v.Validator,
Siblings: siblingsOut(v.Siblings), SiblingsUnknown: v.SiblingsUnknown, JudgeFrom: v.JudgeFrom}
// What each of a group's heads was judged as (novox/hq ADR 0323 §6). // What each of a group's heads was judged as (novox/hq ADR 0323 §6).
for _, m := range v.Members { for _, m := range v.Members {
result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo, result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo,
Commit: m.Head, Trunk: m.Trunk}) Commit: m.Head, Trunk: m.Trunk})
} }
// A branch tried before its pull request reads its outcome from this log, whole (novox/hq issue 686).
if request.Check.Try {
if line, err := json.Marshal(triedOf(v)); err == nil {
say(link.StepTried, string(line))
}
}
} }
} else if err == nil { } else if err == nil {
// The package-registry credential is a build input, so it is resolved before the clone: a // The package-registry credential is a build input, so it is resolved before the clone: a
@@ -289,9 +357,22 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// after a clone that then fails at npm ci. // after a clone that then fails at npm ci.
// Every container it starts is labelled with its id, so a kill finds what outlived the // Every container it starts is labelled with its id, so a kill finds what outlived the
// docker client (ADR 0219). // docker client (ADR 0219).
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher, if request.Reuse != nil && isRegistry && !request.DryRun {
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc, // **A walk's build takes the check's build of the same key** (novox/hq ADR 0350 §2).
forgeFrom(), say, request.Seats) reuse := builder.Reuse{Scratch: registryHere}
for _, p := range request.Reuse.Pulls {
reuse.Pulls = append(reuse.Pulls, builder.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number})
}
// Only what a verdict names is taken (ADR 0350 §2).
reuse.Named = namedOf(request.Reuse.Builds)
built, err = builder.BuildReusing(building, builder.Labelled(builder.Command, request.ID), publisher, reuse,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
} else {
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
} }
// What it copied into the store, whatever became of the build (novox/hq ADR 0257). // What it copied into the store, whatever became of the build (novox/hq ADR 0257).
result.Mirrored = built.Mirrored result.Mirrored = built.Mirrored
@@ -327,6 +408,19 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
} }
result.Against = built.Against result.Against = built.Against
result.SourceFingerprint = built.Source result.SourceFingerprint = built.Source
// What the walk did instead of building, or why it built (novox/hq ADR 0350 §3, §4).
result.ReuseKey = built.ReuseKey
if built.Reused != nil {
f := built.Reused.From
result.Reused = built.Reused.Words()
result.ReusedFrom = &link.ReusedFrom{Pull: link.PullRef{Owner: f.Pull.Owner, Repo: f.Pull.Repo,
Number: f.Pull.Number}, Head: f.Head, Trunk: f.Trunk, Check: f.Check, Key: f.Key}
} else if request.Reuse != nil {
result.BuiltWhy = built.Why
if result.BuiltWhy == "" {
result.BuiltWhy = "it has no artifact to build"
}
}
result.Trunk, result.OnTrunk, result.Branches = built.Trunk, built.OnTrunk, built.Branches result.Trunk, result.OnTrunk, result.Branches = built.Trunk, built.OnTrunk, built.Branches
for _, r := range built.Read { for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref}) result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
@@ -334,6 +428,11 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
for _, s := range built.Sources { for _, s := range built.Sources {
result.Sources = append(result.Sources, link.BuildSource{Repository: s.Repository, Ref: s.Ref, Paths: s.Paths}) result.Sources = append(result.Sources, link.BuildSource{Repository: s.Repository, Ref: s.Ref, Paths: s.Paths})
} }
// The list of predefined keys the manifest was judged against, for the controller to publish
// (novox/hq ADR 0348 §7).
if p := built.Predefined; p != nil {
result.Predefined = &link.PredefinedList{Keys: p.Keys, Generation: p.Generation}
}
say("built", built.Manifest.Module+" from "+short(built.Commit)) say("built", built.Manifest.Module+" from "+short(built.Commit))
} }
} }
@@ -373,9 +472,18 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
spec := builder.CheckSpec{ID: request.ID, Repository: request.Repository, Ref: request.Ref, spec := builder.CheckSpec{ID: request.ID, Repository: request.Repository, Ref: request.Ref,
Owner: c.Owner, Repo: c.Repo, Number: c.Number, Paths: c.Paths, Beside: map[string]builder.Beside{}, Owner: c.Owner, Repo: c.Repo, Number: c.Number, Paths: c.Paths, Beside: map[string]builder.Beside{},
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base, Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held)} Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, Release: c.Release, SiblingsUnknown: c.SiblingsUnknown}
for _, s := range c.Siblings {
spec.Siblings = append(spec.Siblings, builder.Sibling{Repository: s.Repository, Number: s.Number, Head: s.Head,
Base: s.Base, Dir: s.Dir})
}
for dir, b := range c.Beside { for dir, b := range c.Beside {
spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref} spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref, Else: b.Else}
}
for _, b := range c.Builds {
spec.Builds = append(spec.Builds, builder.CheckBuildSpec{Module: b.Module, Repository: b.Repository, Path: b.Path,
Ref: b.Ref, Merged: b.Merged, Tier: b.Tier, Skip: b.Skip})
} }
for _, m := range c.Members { for _, m := range c.Members {
spec.Group = append(spec.Group, builder.GroupHead{Owner: m.Owner, Repo: m.Repo, Repository: m.Repository, spec.Group = append(spec.Group, builder.GroupHead{Owner: m.Owner, Repo: m.Repo, Repository: m.Repository,
@@ -384,6 +492,68 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
return spec return spec
} }
// siblingsOut is the siblings a check read as the outcome carries them (novox/hq ADR 0382 §4).
func siblingsOut(read []builder.JudgedSibling) []link.CheckSibling {
var out []link.CheckSibling
for _, s := range read {
out = append(out, link.CheckSibling{Repository: s.Repository, Number: s.Number, Head: s.Head, Base: s.Base,
BaseCommit: s.BaseCommit})
}
return out
}
// judgeCache is the judge cache the build agent's manifest declares (MESH_JUDGE_CACHE, novox/hq ADR 0358 §6): a
// directory mounted at the same path on the host, so a judge build's mounts name what the agent sees. Empty when the
// agent declares none, and every judge is then built cold under its check's root.
func judgeCache() string { return os.Getenv("MESH_JUDGE_CACHE") }
// triedOf is a try's outcome as its log line says it (novox/hq issue 686): the repository's own check's verdict,
// or the check's where that did not run — a head that does not merge — and the lines naming what failed.
func triedOf(v builder.CheckVerdict) link.Tried {
t := link.Tried{Verdict: v.Verdict, Summary: v.Summary, Took: v.Took.Round(time.Second).String(),
Head: v.Head, Trunk: v.Trunk}
if v.Repo != nil && (v.Gate == nil || v.Gate.Verdict == "pass") {
t.Verdict, t.Summary = v.Repo.Verdict, v.Repo.Summary
}
if v.Repo != nil && v.Repo.Failed != "" {
t.Failed = strings.Split(v.Repo.Failed, "\n")
}
if t.Verdict == "" {
t.Verdict = "error"
}
return t
}
// namedOf are the builds the seat's verdicts name, as the builder believes a scratch record by them (novox/hq ADR
// 0350 §2, issue 703): each module's key with every artifact's digest.
func namedOf(builds []link.CheckedBuild) []builder.NamedBuild {
var out []builder.NamedBuild
for _, b := range builds {
named := builder.NamedBuild{Pull: builder.PullRef{Owner: b.Pull.Owner, Repo: b.Pull.Repo,
Number: b.Pull.Number}, Check: b.Check, Module: b.Built.Module, Key: b.Built.Key,
Digests: map[string]string{}}
for _, m := range b.Built.Made {
named.Digests[m.Name] = builder.DigestOf(m.Reference)
}
out = append(out, named)
}
return out
}
// checkBuiltOf is a check's builds for its merge as the outcome carries them (novox/hq ADR 0350 §1).
func checkBuiltOf(built []builder.CheckBuilt) []link.CheckBuilt {
var out []link.CheckBuilt
for _, b := range built {
c := link.CheckBuilt{Module: b.Module, Key: b.Key, NotBuilt: b.NotBuilt, Took: b.Took.Round(time.Second).String(),
Kept: b.Kept, Same: b.Same, SameAs: b.SameAs}
for _, m := range b.Made {
c.Made = append(c.Made, link.MadeArtifact{Name: m.Name, Kind: m.Kind, Reference: m.Reference})
}
out = append(out, c)
}
return out
}
// layerOf is one layer of a check as the outcome carries it. // layerOf is one layer of a check as the outcome carries it.
func layerOf(l *builder.Layer) *link.CheckLayer { func layerOf(l *builder.Layer) *link.CheckLayer {
if l == nil { if l == nil {
+46
View File
@@ -0,0 +1,46 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
)
// **A pull request into a release branch reaches the check as one** (novox/hq ADR 0358): the build seat hands the
// check the release branch and, beside it, the trunk each repository is read at when it has no branch of that name;
// and it hands the check the judge cache the build agent declares (MESH_JUDGE_CACHE), never a path of its own.
func TestTheBuildSeatHandsTheCheckItsReleaseBranch(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{ID: "b", Repository: "r", Ref: "abc", Check: &link.CheckRequest{
Owner: "novox", Repo: "mesh-catalog", Base: "feat/57", Release: "feat/57",
Beside: map[string]link.CheckedOut{"mesh-host": {Repository: "h", Ref: "feat/57", Else: "main"}}}})
if spec.Release != "feat/57" || spec.Beside["mesh-host"].Ref != "feat/57" || spec.Beside["mesh-host"].Else != "main" {
t.Fatalf("the check was handed release %q and mesh-host %+v", spec.Release, spec.Beside["mesh-host"])
}
t.Setenv("MESH_JUDGE_CACHE", "/var/lib/mesh/build-agent/judge-cache")
if got := judgeCache(); got != "/var/lib/mesh/build-agent/judge-cache" {
t.Fatalf("the judge cache the manifest declares is read as %q", got)
}
t.Setenv("MESH_JUDGE_CACHE", "")
if got := judgeCache(); got != "" {
t.Fatalf("with no judge cache declared, %q is used", got)
}
}
// The siblings a check is asked to read reach the build seat's spec and, as read, the outcome (novox/hq ADR 0382).
func TestTheSiblingsAnAskNamesReachTheSpecAndTheOutcome(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main",
Siblings: []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", Dir: "mesh-controller"}},
SiblingsUnknown: "why"}})
if len(spec.Siblings) != 1 || spec.Siblings[0] != (builder.Sibling{Repository: "novox/mesh-controller", Number: 346,
Head: "a", Base: "main", Dir: "mesh-controller"}) || spec.SiblingsUnknown != "why" {
t.Fatalf("the spec reads %+v", spec)
}
out := siblingsOut([]builder.JudgedSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", BaseCommit: "b"}})
if len(out) != 1 || out[0].BaseCommit != "b" || out[0].Number != 346 {
t.Fatalf("the outcome reads %+v", out)
}
if plain := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "o", Repo: "r"}}); plain.Siblings != nil {
t.Fatalf("a request that names none has siblings %+v", plain.Siblings)
}
}
+1
View File
@@ -16,6 +16,7 @@ func TestBuilderDiagnosticsStayOffStdout(t *testing.T) {
"version)": true, // --version "version)": true, // --version
`"stopping")`: true, // the loop.s shutdown line `"stopping")`: true, // the loop.s shutdown line
"usage)": true, // --help text, for a human "usage)": true, // --help text, for a human
"said)": true, // busy: the answer its node-engine reads (novox/hq ADR 0335)
} }
for _, file := range []string{"once.go", "main.go"} { for _, file := range []string{"once.go", "main.go"} {
src, err := os.ReadFile(file) src, err := os.ReadFile(file)
+31
View File
@@ -0,0 +1,31 @@
package main
import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// A try's outcome, as the build seat says it in the build's log (novox/hq issue 686): the repository's own
// check's verdict, how long it took, and each line naming what failed.
func TestATrysOutcomeNamesWhatFailed(t *testing.T) {
tried := triedOf(builder.CheckVerdict{Verdict: "pass", Summary: "the change touches no module", Took: 303 * time.Second,
Head: "c0ffee", Trunk: "ca3f43c", Gate: &builder.Layer{Verdict: "pass"},
Repo: &builder.Layer{Verdict: "fail", Summary: "its merge-check.sh failed: TestX",
Failed: "--- FAIL: TestX (0.01s)\n x_test.go:12: got 1, want 2"}})
if tried.Verdict != "fail" || tried.Took != "5m3s" || len(tried.Failed) != 2 || tried.Trunk != "ca3f43c" ||
tried.Summary != "its merge-check.sh failed: TestX" {
t.Fatalf("a failed try: %+v", tried)
}
// A head that does not merge fails both layers: the check's own word.
conflict := triedOf(builder.CheckVerdict{Verdict: "fail", Summary: "does not merge", Gate: &builder.Layer{Verdict: "fail"},
Repo: &builder.Layer{Verdict: "fail", Summary: "does not merge"}})
if conflict.Verdict != "fail" || conflict.Summary != "does not merge" {
t.Fatalf("a try that does not merge: %+v", conflict)
}
// No verdict is an error, never a pass.
if none := triedOf(builder.CheckVerdict{}); none.Verdict != "error" {
t.Fatalf("a try with no verdict: %+v", none)
}
}
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A provision whose credential only a person outside the mesh can give (novox/hq ADR 0337, issue 164, issue 165):
// given once for its provider, at the controller's terminal, and sealed in one act to every consumer that binds the
// provision from that provider now. The value is never read back, printed or kept.
// outsideOffer is the provider module's own name when `secret accept <node> <module> <name>` names a provision that
// module offers with a credential issued outside the mesh, and "" when it names anything else (an own secret, or a
// pair with --provider).
func outsideOffer(ctx context.Context, module, name, provider string) (string, error) {
if provider != "" {
return "", nil
}
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
if m, known := shelf[module]; known && m.IssuedOutsideOf(name) {
return module, nil
}
return "", nil
}
// offeredOutsideBy is a module that offers the provision with a credential issued outside the mesh, or "".
func offeredOutsideBy(ctx context.Context, provision string) (string, error) {
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
module, _ := catalogue.OffersIssuedOutside(shelf, provision)
return module, nil
}
// consumersFrom is every consumer that binds a provision from one provider now, found by resolving every machine as
// its next send would, and the machines that could not be resolved, with why.
func consumersFrom(ctx context.Context, open *stores, provider, providerModule, provision string) (
[]inventory.PairEnd, []string, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, nil, err
}
var out []inventory.PairEnd
var unresolved []string
for _, n := range nodes {
plan, _, err := planFor(ctx, open, n.Name)
if err != nil {
unresolved = append(unresolved, fmt.Sprintf("%s: %s", n.Name, oneLine(err.Error())))
continue
}
out = append(out, bindsFrom(plan, n.Name, provider, providerModule, provision)...)
}
return out, unresolved, nil
}
// bindsFrom is every consumer on one machine whose resolution answers the provision from that provider. Pure.
func bindsFrom(plan catalogue.Resolution, machine, provider, providerModule, provision string) []inventory.PairEnd {
var out []inventory.PairEnd
for _, need := range plan.Needs {
if need.ByRecord || need.Name != provision || need.From != provider {
continue
}
if need.Module != "" && need.Module != providerModule {
continue
}
out = append(out, inventory.PairEnd{Consumer: machine, ConsumerModule: need.For, Local: need.Local})
}
sort.Slice(out, func(i, j int) bool {
return out[i].ConsumerModule+"\x00"+out[i].Local < out[j].ConsumerModule+"\x00"+out[j].Local
})
return out
}
// acceptForProvision gives a provision's credential issued outside the mesh, once for its provider (ADR 0337 §3).
func acceptForProvision(ctx context.Context, provider, providerModule, provision, value string) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
consumers, unresolved, err := consumersFrom(ctx, open, provider, providerModule, provision)
if err != nil {
return err
}
accepted, err := open.inventory.AcceptSecretForProvision(ctx, provision, provider, providerModule, value, consumers)
value = ""
if err != nil {
return err
}
fmt.Print(acceptedSaid(provider, providerModule, provision, accepted, unresolved))
if len(accepted.Sealed) == 0 {
return nil
}
if err := sendTo(ctx, open, accepted.Machines); err != nil {
return fmt.Errorf("%s is sealed and could not be sent: %w; `push %s` sends it", provision, err,
strings.Join(accepted.Machines, "`, `push "))
}
fmt.Printf(" sent %s; each consumer starts again on it\n", strings.Join(accepted.Machines, ", "))
return nil
}
// acceptedSaid is what one accept for a provider did, in words that never carry the value. Pure.
func acceptedSaid(provider, providerModule, provision string, a inventory.ProvisionAccepted, unresolved []string) string {
var b strings.Builder
if len(a.Sealed) == 0 {
fmt.Fprintf(&b, "%s from %s on %s: no consumer binds it now, so nothing was sealed; a consumer that binds "+
"later waits for it, and this line given again seals it to every consumer then\n", provision,
providerModule, provider)
} else {
var names []string
for _, c := range a.Sealed {
names = append(names, fmt.Sprintf("%s on %s%s", c.ConsumerModule, c.Consumer, asLocal(c.Local)))
}
fmt.Fprintf(&b, "%s from %s on %s is given, sealed to %s and to %s\n", provision, providerModule, provider,
strings.Join(names, ", "), provider)
fmt.Fprintf(&b, " the mesh cannot read it back, never makes one of its own for it, and will not rotate it\n")
}
for _, c := range a.Unsealable {
fmt.Fprintf(&b, " not sealed to %s on %s: no sealing key on that machine; give it again once it has one\n",
c.ConsumerModule, c.Consumer)
}
for _, u := range unresolved {
fmt.Fprintf(&b, " not known whether a consumer on %s binds it, which did not resolve; give it again once it does\n", u)
}
return b.String()
}
+5 -2
View File
@@ -26,8 +26,11 @@ import (
// //
// - **The serving controller** takes it before it does anything else — before it asserts the bus's // - **The serving controller** takes it before it does anything else — before it asserts the bus's
// objects, which are the controller's to write — waiting while another holds it, and renews it. // objects, which are the controller's to write — waiting while another holds it, and renews it.
// A renewal refused or failed is the lease lost: the gate closes at once and the process exits, so // Another holder seen, or the key's age passed with no renewal made, is the lease lost: the gate
// its service manager restarts it as a candidate (serve, in push.go). // closes at once and the process exits, so its service manager restarts it as a candidate (serve,
// in push.go). A renewal the bus does not answer in time is not a loss (novox/hq issue 477): the
// lease is kept, no new act starts once half the key's age passed unrenewed, and the next renewal
// made opens the gate again.
// - **A command run at a shell** — `push` in the installer, the lab, a person repairing a mesh whose // - **A command run at a shell** — `push` in the installer, the lab, a person repairing a mesh whose
// controller is down (issue 201) — acts **under the holder's epoch** when a controller holds the // controller is down (issue 201) — acts **under the holder's epoch** when a controller holds the
// lease: it is the same mesh's word, composed and sent under the store's hold of each machine like // lease: it is the same mesh's word, composed and sent under the store's hold of each machine like
+11 -6
View File
@@ -203,8 +203,9 @@ func seatDependenciesOnAssign(ctx context.Context, open *stores, node string, mo
if _, err := catalogue.AssignRefusal(shelf, node, assigned, adding); err != nil { if _, err := catalogue.AssignRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err return nil, nil, err
} }
// Two modules declaring one package, path or unit is refused before anything is recorded // Two modules declaring one path or unit, or one package present and absent, is refused before anything
// (novox/hq ADR 0210, 04-ISSUES/235): kept, the node would not resolve until one came off again. // is recorded (novox/hq ADR 0210, ADR 0303, 04-ISSUES/235): kept, the node would not resolve until one came
// off again. Two modules declaring one package present is not a collision (ADR 0303, issue 628).
if err := catalogue.CollisionRefusal(shelf, node, assigned, adding); err != nil { if err := catalogue.CollisionRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err return nil, nil, err
} }
@@ -390,10 +391,10 @@ func blockedElsewhere(ctx context.Context, open *stores, except string) string {
} }
// issueOnAssign gives a newly assigned module its bus credential, the way `module issue` does, and // issueOnAssign gives a newly assigned module its bus credential, the way `module issue` does, and
// says what it did in one line. Nothing for a module that declares no broker secret; nothing for one // says what it did in one line. Nothing for a module that declares no broker secret and runs no
// whose user is already minted (a credential is rotated on purpose, never by re-assigning); and when // provisioner client; nothing for one whose user is already minted (a credential is rotated on
// the bus cannot be reached from here, the line names the verb and the push that would refuse the // purpose, never by re-assigning); and when the bus cannot be reached from here, the line names the
// module until it is run — never a silent placeholder (novox/hq issue 203). // verb and the push that would refuse the module until it is run — never a silent placeholder (novox/hq issue 203).
func issueOnAssign(ctx context.Context, open *stores, node, module string) string { func issueOnAssign(ctx context.Context, open *stores, node, module string) string {
inv := open.inventory inv := open.inventory
shelf, err := inv.Catalogue(ctx) shelf, err := inv.Catalogue(ctx)
@@ -405,6 +406,10 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return "" return ""
} }
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username() user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, reads := m.OwnSecrets["broker"]; !reads {
// A module running the provisioner client alone is minted the client's account (issue 917).
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
}
if _, minted, err := inv.BusUserHash(ctx, user); err != nil || minted { if _, minted, err := inv.BusUserHash(ctx, user); err != nil || minted {
return "" return ""
} }
+1 -1
View File
@@ -140,7 +140,7 @@ func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testin
} }
// And a replay to the catalogue says where the store is now. // And a replay to the catalogue says where the store is now.
announced, err := following{open}.Announceable(ctx) announced, err := following{open: open}.Announceable(ctx)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+18 -3
View File
@@ -424,8 +424,9 @@ func comparisonOf(module string, c comparison, opts takeOptions) (preview string
} }
// And its settings on this machine, composed against its definition (rule 1, rule 6). // And its settings on this machine, composed against its definition (rule 1, rule 6).
for _, layer := range c.layers { for _, layer := range c.layers {
keys := make([]string, 0, len(layer.Values)) said := layer.Whole()
for k := range layer.Values { keys := make([]string, 0, len(said))
for k := range said {
keys = append(keys, k) keys = append(keys, k)
} }
sort.Strings(keys) sort.Strings(keys)
@@ -722,7 +723,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
return "", err return "", err
} }
derived := derivedFilter{rules: rules, foundation: with.Foundation, mesh: with.Mesh, derived := derivedFilter{rules: rules, foundation: with.Foundation, mesh: with.Mesh,
outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks} outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks, local: with.LocalNetworks}
filtering, err := inv.FilteringOf(ctx, node) filtering, err := inv.FilteringOf(ctx, node)
if err != nil { if err != nil {
return "", err return "", err
@@ -930,6 +931,9 @@ type derivedFilter struct {
// The filter constrains what arrives on them; everything arriving elsewhere is this machine's // The filter constrains what arrives on them; everything arriving elsewhere is this machine's
// own guest and is not filtered. // own guest and is not filtered.
outwardLinks []string outwardLinks []string
// local is the machine's local networks, which a port from `local-network` stays open to (novox/hq
// ADR 0340).
local []string
} }
// closesOutside is what a narrowing from everywhere to the private network is called: it closes. // closesOutside is what a narrowing from everywhere to the private network is called: it closes.
@@ -977,6 +981,17 @@ func (d derivedFilter) fate(r inventory.Reach) string {
if !onMesh { if !onMesh {
return fmt.Sprintf("%s — declared by %s from the mesh only", closesOutside, by) return fmt.Sprintf("%s — declared by %s from the mesh only", closesOutside, by)
} }
case catalogue.FromLocalNetwork:
// The private network and this machine's local networks (novox/hq ADR 0340): it closes to
// everything else outside, and says which local networks stay.
if !onMesh {
if len(d.local) == 0 {
return fmt.Sprintf("%s — declared by %s from the local network, and this machine "+
"reported none", closesOutside, by)
}
return fmt.Sprintf("WILL CLOSE to everything outside the private network and %s — declared "+
"by %s from the local network", strings.Join(d.local, ", "), by)
}
} }
return fmt.Sprintf("declared by %s (from %s)", by, rule.From) return fmt.Sprintf("declared by %s (from %s)", by, rule.From)
} }
+46
View File
@@ -38,15 +38,61 @@ func watchApplyFailing(f *signalFacts) []conditions.Observation {
} }
} }
var out []conditions.Observation var out []conditions.Observation
said := map[string]bool{}
for _, d := range f.doing { for _, d := range f.doing {
if !d.Wrong() || d.Since == nil || f.now.Sub(*d.Since) <= applyEvery || !heard[d.Node] { if !d.Wrong() || d.Since == nil || f.now.Sub(*d.Since) <= applyEvery || !heard[d.Node] {
continue continue
} }
out = append(out, applyFailing(d, f.now)) out = append(out, applyFailing(d, f.now))
said[d.Node] = true
}
// **A module's files the provisioner failed to place are a failed apply too** (novox/hq ADR 0362 §3): its client's
// newest word, judged by the same bound — the client says a failure once, and fills again on every change —
// on a machine whose node-engine applied what it was sent.
failed := map[string][]inventory.ProvReport{}
for _, r := range f.provFailed {
if r.Status != inventory.ProvFailed || f.now.Sub(r.At) <= applyEvery || !heard[r.Node] || said[r.Node] {
continue
}
failed[r.Node] = append(failed[r.Node], r)
}
for _, node := range sortedNames(failed) {
out = append(out, provisioningFailing(node, failed[node], f.now))
} }
return out return out
} }
// provisioningFailing is one machine's S21 condition for the modules whose files its provisioner client failed to
// place.
func provisioningFailing(m string, failed []inventory.ProvReport, now time.Time) conditions.Observation {
sort.Slice(failed, func(a, b int) bool { return failed[a].Module < failed[b].Module })
var names, words []string
since := failed[0].At
for _, r := range failed {
names = append(names, r.Module)
words = append(words, r.Module+": "+oneLine(orNoReason(r.Reason)))
if r.At.Before(since) {
since = r.At
}
}
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: m, Kind: kindApplyFailing, Machine: m,
Severity: conditions.Warning, Said: strings.Join(words, "; "),
Resolved: m + " applies what it is sent again"}
o.Summary = fmt.Sprintf("%s's provisioner has failed to place the files of %s since %s (bound %s)", m,
strings.Join(names, ", "), since.UTC().Format("2006-01-02 15:04 MST"), ago(applyEvery))
o.Headline = m + " is not applying its update"
o.Explanation = fmt.Sprintf("The provisioner on %s has not been able to write the files of %s for %s. The "+
"details say why.", m, strings.Join(names, ", "), humanDuration(now.Sub(since)))
return o
}
func orNoReason(s string) string {
if s == "" {
return "no reason given"
}
return s
}
// applyFailing is one machine's S21 condition. // applyFailing is one machine's S21 condition.
func applyFailing(d inventory.Doing, now time.Time) conditions.Observation { func applyFailing(d inventory.Doing, now time.Time) conditions.Observation {
m := d.Node m := d.Node
+8 -3
View File
@@ -688,7 +688,7 @@ func spelledOf(m inventory.BatchedMerge) string {
func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge { func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge {
k := announcedOf(m) k := announcedOf(m)
return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves, return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves,
Merged: m.Merged, Heard: m.Heard} Head: k.HeadSHA, Merged: m.Merged, Heard: m.Heard}
} }
// laterMerge says a was merged after b: by the forge's merge time, then by when each was heard. // laterMerge says a was merged after b: by the forge's merge time, then by when each was heard.
@@ -1000,10 +1000,15 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
plan.Delivery.Merges = named plan.Delivery.Merges = named
plan.Delivery.Alone = batch.Delivery != nil && batch.Delivery.Alone plan.Delivery.Alone = batch.Delivery != nil && batch.Delivery.Alone
plan.Delivery.Withheld = withheld plan.Delivery.Withheld = withheld
// **Left where it is, a module whose program did not change** (novox/hq issue 721): its check found it the same as
// the build the mesh holds.
for n, why := range leaveSameAsRunning(ctx, inv, &plan, moved, append(append([]inventory.Edge{}, edges...), ordered...)) {
unchanged[n] = why
}
plan.Delivery.Unchanged = unchangedLeft(unchanged, plan.Modules) plan.Delivery.Unchanged = unchangedLeft(unchanged, plan.Modules)
// **Its moments and its class** (novox/hq ADR 0282 decision 6): measured, never acted on. // **Its moments and its class** (novox/hq ADR 0282 decision 6): measured, never acted on.
plan.Times = walkTimesAtCut(*batch, plan, entries, now) plan.Times = walkTimesAtCut(*batch, plan, entries, now)
if len(moved) == 0 { if len(moved) == 0 || len(plan.Modules) == 0 {
plan.State = inventory.PlanDone plan.State = inventory.PlanDone
plan.Tiers = [][]string{} plan.Tiers = [][]string{}
plan.Note = "moved nothing the mesh holds: nothing to walk" plan.Note = "moved nothing the mesh holds: nothing to walk"
@@ -1044,7 +1049,7 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil return nil
} }
// Written and its first tier asked as one act on the plans (novox/hq issue 213), under the caller's hold. // Written and its first tier asked as one act on the plans (novox/hq issue 213), under the caller's hold.
if err := askTier(ctx, inv, &plan); err != nil { if err := askTier(ctx, inv, &plan, plan.Tiers[plan.Tier]); err != nil {
return err return err
} }
if err := inv.SavePlan(ctx, &plan); err != nil { if err := inv.SavePlan(ctx, &plan); err != nil {
+1 -1
View File
@@ -758,7 +758,7 @@ func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
return entries, read, err return entries, read, err
} }
if err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announcedList(lost), open.inventory}, catalogued, if err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announcedList(lost), open.inventory}, catalogued,
(following{open}).SourceMoved, t.Logf); err != nil { (following{open: open}).SourceMoved, t.Logf); err != nil {
t.Fatal(err) t.Fatal(err)
} }
got, _ := open.inventory.PlanByID(ctx, w.ID) got, _ := open.inventory.PlanByID(ctx, w.ID)
+84 -3
View File
@@ -15,6 +15,7 @@ import (
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker" "github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue" "github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory" "github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link" "github.com/novox/mesh-controller/internal/link"
@@ -182,8 +183,10 @@ func buildFrom(result link.BuildResult) inventory.Build {
kept.Manifest = recordedManifest(result.Manifest, announced) kept.Manifest = recordedManifest(result.Manifest, announced)
// The module name comes from the manifest, which only exists when the build got that far. // The module name comes from the manifest, which only exists when the build got that far.
if len(kept.Manifest) > 0 { if len(kept.Manifest) > 0 {
if m, err := catalogue.ParseManifest(kept.Manifest); err == nil { if list, err := judgedWith(result); err == nil {
kept.Module = m.Module if m, err := catalogue.ParseManifestWith(kept.Manifest, list); err == nil {
kept.Module = m.Module
}
} }
} }
return kept return kept
@@ -483,6 +486,7 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
Seats: seatBases(ctx), Seats: seatBases(ctx),
DryRun: dryRun, DryRun: dryRun,
} }
request.Reuse = reuseAsked(ctx, dryRun)
fmt.Printf("asked for %s", source) fmt.Printf("asked for %s", source)
if source.Seat != "" { if source.Seat != "" {
fmt.Printf(" (%s)", repository) fmt.Printf(" (%s)", repository)
@@ -601,6 +605,11 @@ func saysWhenThePolicyActs(ctx context.Context, inv *inventory.Inventory, module
// written with the same values. // written with the same values.
func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResult) ( func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResult) (
catalogue.Manifest, inventory.Build, error) { catalogue.Manifest, inventory.Build, error) {
// The list of predefined keys as the store keeps it now: another controller process may have kept a newer one.
if err := loadPredefined(ctx, inv); err != nil {
return catalogue.Manifest{}, inventory.Build{}, fmt.Errorf("the list of predefined keys kept cannot be "+
"read: %w", err)
}
kept := buildFrom(result) kept := buildFrom(result)
if err := inv.RecordBuild(ctx, kept); err != nil { if err := inv.RecordBuild(ctx, kept); err != nil {
return catalogue.Manifest{}, kept, err return catalogue.Manifest{}, kept, err
@@ -611,7 +620,15 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return catalogue.Manifest{}, kept, fmt.Errorf("%s could not build %s:\n%s", return catalogue.Manifest{}, kept, fmt.Errorf("%s could not build %s:\n%s",
result.On, result.Repository, result.Failed) result.On, result.Repository, result.Failed)
} }
manifest, err := catalogue.ParseManifest(kept.Manifest) // Judged first against the list the build was judged against: the catalogue's at the commit built (novox/hq
// ADR 0348 §7), so a module naming a key its own commit adds is read; and again below against the list the
// mesh publishes, before it is registered.
list, err := judgedWith(result)
if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
result.Repository, short(result.Commit), err)
}
manifest, err := catalogue.ParseManifestWith(kept.Manifest, list)
if err != nil { if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s and what came back is not a manifest: %w", return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s and what came back is not a manifest: %w",
result.On, result.Repository, err) result.On, result.Repository, err)
@@ -669,6 +686,12 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On, return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
manifest.Module, short(result.Commit), err) manifest.Module, short(result.Commit), err)
} }
// **Never a build in the scratch namespace** (novox/hq ADR 0239 §9, ADR 0350 §2): what a check builds is
// registered only as the walk's copy in the module's own namespace.
if scratch := inScratch(result.Made); scratch != "" {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %s is in the scratch "+
"namespace, which only a walk's copy leaves (ADR 0350)", result.On, manifest.Module, short(result.Commit), scratch)
}
// **A build that failed its gate is never registered again** (novox/hq ADR 0236): an outcome heard // **A build that failed its gate is never registered again** (novox/hq ADR 0236): an outcome heard
// twice, or replayed, would otherwise make the build a rollback put back what the module is again, // twice, or replayed, would otherwise make the build a rollback put back what the module is again,
// and the next push would send it. // and the next push would send it.
@@ -679,6 +702,23 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
"back: it is recorded and not registered again — a newer build is", result.On, manifest.Module, "back: it is recorded and not registered again — a newer build is", result.On, manifest.Module,
short(result.Commit)) short(result.Commit))
} }
// **The newest trunk build's list of predefined keys is published** (novox/hq ADR 0348 §7): a build of the
// catalogue that may be registered keeps its list when its commit is newer than the one kept. Then the module
// is judged against the list the mesh publishes, as every manifest read again is: one naming a key that list
// does not hold, or of another type, is refused here and never enters the catalogue.
if carried, err := carriedList(result); err != nil {
return manifest, kept, err
} else if carried != nil {
if err := publishPredefined(ctx, inv, result, carried); err != nil {
fmt.Printf("%s: the list of predefined keys it carried is not published: %v\n", result.ID, err)
}
}
if manifest.NamesPredefined() {
if _, err := catalogue.ParseManifest(kept.Manifest); err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: judged against the "+
"list of predefined keys the mesh publishes, %w", result.On, manifest.Module, short(result.Commit), err)
}
}
// **A build whose source is unchanged is never a move** (novox/hq issue 280): a rebuild made from // **A build whose source is unchanged is never a move** (novox/hq issue 280): a rebuild made from
// what the build the mesh stands on was made from registers that build's artifacts at the new // what the build the mesh stands on was made from registers that build's artifacts at the new
// commit, so no machine is sent a new digest for a source nobody changed — an image is not // commit, so no machine is sent a new digest for a source nobody changed — an image is not
@@ -714,6 +754,16 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return manifest, kept, nil return manifest, kept, nil
} }
// inScratch is the first artifact a build made into the scratch namespace; empty when none is.
func inScratch(made []link.MadeArtifact) string {
for _, m := range made {
if path, kept := catalogue.InArtifactStore(catalogue.Recorded(m.Reference)); kept && strings.HasPrefix(path, builder.ScratchPrefix) {
return m.Reference
}
}
return ""
}
// errOffTheTrunk is a build of a commit off its repository's trunk, which is never published. // errOffTheTrunk is a build of a commit off its repository's trunk, which is never published.
var errOffTheTrunk = errors.New("the commit is not on its repository's trunk: a commit off the trunk is checked, " + var errOffTheTrunk = errors.New("the commit is not on its repository's trunk: a commit off the trunk is checked, " +
"never published (ADR 0238) — merge it, and the merge builds it") "never published (ADR 0238) — merge it, and the merge builds it")
@@ -802,6 +852,9 @@ type answers struct {
sources map[string]inventory.Source sources map[string]inventory.Source
// waiting is every machine not running what the mesh would send it. // waiting is every machine not running what the mesh would send it.
waiting []inventory.Machine waiting []inventory.Machine
// recordedWaits is, per machine, every move of a recorded module that waits there for a person (novox/hq ADR
// 0377 §6): the machine is behind for these, and `push <node>` sends them.
recordedWaits map[string][]string
// reported is every machine's last word beside when it was last sent a declaration — the // reported is every machine's last word beside when it was last sent a declaration — the
// pair that answers "has it caught up", which waiting alone cannot (the sent digest is // pair that answers "has it caught up", which waiting alone cannot (the sent digest is
// recorded at send, not at apply). // recorded at send, not at apply).
@@ -881,6 +934,34 @@ type answers struct {
// //
// Routed through the artifact store as the network reaches it now (novox/hq 04-ISSUES/102): a // Routed through the artifact store as the network reaches it now (novox/hq 04-ISSUES/102): a
// base is recorded by digest and path, and a build machine needs something it can pull. // base is recorded by digest and path, and a build machine needs something it can pull.
// runningBy is, keyed "<module>/<artifact>", each artifact of the build the mesh has registered for its module, as a
// builder reaches the artifact store (novox/hq issue 721): what a check compares its builds with. Empty when it cannot
// be read, said, and then nothing is said unchanged.
func runningBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
defer open.Close()
running, err := open.inventory.RegisteredHeld(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
address, err := whereABuilderReachesTheStore(ctx, open.inventory)
if err != nil || address == "" {
return running
}
routed := make(map[string]string, len(running))
for key, reference := range running {
routed[key] = catalogue.Rerouted(reference, address)
}
return routed
}
func heldBy(ctx context.Context) map[string]string { func heldBy(ctx context.Context) map[string]string {
open, err := openStores(ctx) open, err := openStores(ctx)
if err != nil { if err != nil {
@@ -0,0 +1,92 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **Every user's slow consumer is S9's, not the controller's alone** (novox/hq ADR 0357, issue 270). On
// 2026-10-07 the bus closed two clients on the laptop as slow consumers and nothing raised a condition:
// the controller heard only its own connection. Said by the bus module and read back, a user the bus
// closes is one condition naming the module and its machine, raised at the first look and cleared an hour
// after the last closure.
func TestAUserTheBusClosedAsASlowConsumerIsOneConditionNamingItsModuleAndMachine(t *testing.T) {
closed := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
now := closed.Add(20 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
advisories := link.BusClosures([]link.BusClosure{
{User: "laptop.claude-code", CID: 10, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Hour),
Closed: closed.Add(-10 * time.Minute)},
{User: "laptop.claude-code", CID: 44, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Minute),
Closed: closed},
{User: "anchor.desk", CID: 7, Reason: "Maximum Message Payload Exceeded", Opened: closed.Add(-time.Hour),
Closed: closed},
}, []string{link.AdvisorySlowConsumer, link.AdvisorySlowConsumer, link.AdvisoryConnectionClosed}, closed.Add(-time.Hour))
look := func() {
t.Helper()
f := &signalFacts{now: now, host: "anchor", advisories: advisories}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
}
look()
slow, found, err := conditions.ReadOne(t.Context(), store, "bus.laptop.claude-code.slow-consumer")
if err != nil || !found {
t.Fatalf("no slow-consumer condition for the user: %v %v", found, err)
}
if slow.Subject.Machine != "laptop" || slow.Observations != 2 || !slow.LastObserved.Equal(closed) {
t.Fatalf("the slow consumer is said on %q, %d closure(s), last %s: %+v", slow.Subject.Machine, slow.Observations,
slow.LastObserved, slow)
}
if !strings.Contains(slow.Summary, "claude-code on laptop") || strings.Contains(slow.Summary, "laptop.claude-code") {
t.Fatalf("the summary does not name the module in words, or carries the dotted user: %q", slow.Summary)
}
if !strings.Contains(slow.Headline, "Claude-code on laptop") {
t.Fatalf("the headline is %q", slow.Headline)
}
cut, found, err := conditions.ReadOne(t.Context(), store, "bus.anchor.desk.connection-closed")
if err != nil || !found || cut.Subject.Machine != "anchor" || !strings.Contains(cut.Summary, "maximum message payload exceeded") {
t.Fatalf("the user cut off for breaking the bus's rules: %v %v %+v", found, err, cut)
}
// An hour after the newest closure without another, both clear.
now = closed.Add(time.Hour + time.Minute)
look()
for _, key := range []string{"bus.laptop.claude-code.slow-consumer", "bus.anchor.desk.connection-closed"} {
if _, found, _ := k.Get(t.Context(), key); found {
t.Fatalf("%s is still open an hour after the last closure", key)
}
}
}
// **A read of the bus module's events that fails does not blind S9** (review of #326): the controller's own
// signals are still said, and the failed read is a condition of its own.
func TestABusModuleReadThatFailsKeepsTheControllersOwnSignals(t *testing.T) {
at := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
f := &signalFacts{now: at.Add(time.Minute), host: "anchor",
advisories: []link.Advisory{{Kind: link.AdvisorySlowConsumer, ID: "controller", Said: "the controller fell behind",
First: at, Last: at, Count: 1}},
busFaultsErr: errors.New("nats: timeout")}
keys := map[string]bool{}
for _, o := range watchAdvisories(f) {
keys[o.Key()] = true
}
if !keys["bus.controller.slow-consumer"] || !keys["bus.bus-module.bus-faults-unread"] {
t.Fatalf("observed %v: the controller's own slow consumer and the unread events, both", keys)
}
for _, row := range signalsTable {
if row.Row == "S9" && row.needs(f) != nil {
t.Fatal("the S9 row went blind for a read of the bus module's events")
}
}
}
+42 -5
View File
@@ -49,10 +49,12 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
// composed from, so a module the mesh grants a consumer's subjects has that consumer waiting. // composed from, so a module the mesh grants a consumer's subjects has that consumer waiting.
// Done on every raise, not only when a credential is issued: every module moved onto this bus // Done on every raise, not only when a credential is issued: every module moved onto this bus
// by the rollout was issued on the old one, and came up with nothing to bind to (2026-09-28). // by the rollout was issued on the old one, and came up with nothing to bind to (2026-09-28).
consumers, err := moduleConsumers(ctx, inv) consumers, records, err := moduleConsumersAndRecords(ctx, inv)
if err != nil { if err != nil {
return nil, err return nil, err
} }
// And a grant of a restricted stream that composes nothing, said on every raise and send (novox/hq ADR 0376 §2).
sayWithheld(records, " ")
// And the work queues of seats that name their caller or their kind, with each holder's worker // And the work queues of seats that name their caller or their kind, with each holder's worker
// (novox/hq ADR 0259 §3): an ask queues until the router takes it, a channel's work until that kind // (novox/hq ADR 0259 §3): an ask queues until the router takes it, a channel's work until that kind
// takes it. // takes it.
@@ -60,9 +62,19 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
if err != nil { if err != nil {
return nil, err return nil, err
} }
// And every module's own stream (novox/hq ADR 0363), from the catalogue, before the consumers on it.
modStreams, err := inv.DeclaredStreams(ctx)
if err != nil {
return nil, err
}
// Every one tried, and every failure named: one module's consumer the bus refuses is no reason // Every one tried, and every failure named: one module's consumer the bus refuses is no reason
// the modules after it in the list hear nothing (novox/hq issue 208, where this runs on each send). // the modules after it in the list hear nothing (novox/hq issue 208, where this runs on each send).
var failed []error var failed []error
for _, s := range broker.ModuleStreams(modStreams) {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the module stream %s: %w", s.Name, err))
}
}
for _, s := range trafficStreams { for _, s := range trafficStreams {
if err := r.EnsureStream(s); err != nil { if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the work queue %s: %w", s.Name, err)) failed = append(failed, fmt.Errorf("the work queue %s: %w", s.Name, err))
@@ -73,11 +85,30 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
failed = append(failed, fmt.Errorf("the worker %s on %s: %w", c.Name, c.Stream, err)) failed = append(failed, fmt.Errorf("the worker %s on %s: %w", c.Name, c.Stream, err))
} }
} }
restricted := map[string]bool{}
for _, s := range modStreams {
restricted[s.Name] = s.Restricted
}
for _, c := range consumers { for _, c := range consumers {
if err := r.EnsureConsumer(c.Consumer); err != nil { if err := broker.EnsureHeard(r, c.Consumer, restricted[c.Consumer.Stream]); err != nil {
failed = append(failed, fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err)) failed = append(failed, fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err))
} }
} }
// **And a consumer no longer granted loses what it reads through** (novox/hq ADR 0363 §6): every consumer
// on a module's stream the composition no longer asks for is removed, on this very send.
if p, prunes := r.(broker.Pruner); prunes && len(modStreams) > 0 {
var wanted []broker.Consumer
for _, c := range consumers {
wanted = append(wanted, c.Consumer)
}
removed, err := broker.PruneHeard(p, modStreams, wanted)
if err != nil {
failed = append(failed, err)
}
for _, gone := range removed {
fmt.Printf(" removed the consumer %s: its module is no longer granted that stream (novox/hq ADR 0363)\n", gone)
}
}
if len(failed) > 0 { if len(failed) > 0 {
return nil, errors.Join(failed...) return nil, errors.Join(failed...)
} }
@@ -136,15 +167,21 @@ func unassertedObservation(err error) conditions.Observation {
// moduleConsumers is every module's durable consumer, from the records the user list is composed from. // moduleConsumers is every module's durable consumer, from the records the user list is composed from.
func moduleConsumers(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, error) { func moduleConsumers(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, error) {
consumers, _, err := moduleConsumersAndRecords(ctx, inv)
return consumers, err
}
// moduleConsumersAndRecords is moduleConsumers with the records they were composed from.
func moduleConsumersAndRecords(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, broker.Records, error) {
records, err := inv.BusRecords(ctx) records, err := inv.BusRecords(ctx)
if err != nil { if err != nil {
return nil, err return nil, broker.Records{}, err
} }
users, err := broker.Users(records) users, err := broker.Users(records)
if err != nil { if err != nil {
return nil, err return nil, broker.Records{}, err
} }
return broker.ConsumersOf(users), nil return broker.ConsumersOf(users), records, nil
} }
// seatTrafficObjects is the work queues and workers of seats that name their caller or their kind, from // seatTrafficObjects is the work queues and workers of seats that name their caller or their kind, from
+103
View File
@@ -0,0 +1,103 @@
package main
import (
"fmt"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A first-node gate waits for a busy container (novox/hq ADR 0335 §4).
//
// When a machine reports, on what the gate sent it, that a container of a judged module was left running
// because it said busy, the judging has not started: the build is not yet what runs there. The walk says it
// waits for a busy container on that machine, counts no pass and no bound, and starts the judging — its
// Since — at the report that no longer lists the container waiting. The wait is the node-engine's, bounded
// at 15 minutes; the controller's own bound (busyGateBound past the engine's) is for an engine that stopped
// reporting while it waited, so no walk waits for ever. An engine older than the field never lists one, and
// its gate starts as before.
// busyGateSlack is how long past the engine's bound the controller still waits for the report that says the
// container replaced: a pass of the engine (its asks every 20 seconds) and the report's way here.
var busyGateSlack = 5 * time.Minute
// busyEngineBound is the node-engine's bound on a wait (novox/hq ADR 0335 §3), counted again here on this
// controller's clock from when it first read the wait.
const busyEngineBound = 15 * time.Minute
// busyGateEnds is when the gate stops waiting: the engine's own word on its bound, or this controller's count of
// it from when it first read the wait, whichever is sooner, and the slack — so neither a clock ahead on the
// machine nor a wrong word from it holds a walk longer.
func busyGateEnds(b *inventory.GateBusy) time.Time {
until := b.Seen.Add(busyEngineBound)
if !b.Until.IsZero() && b.Until.Before(until) {
until = b.Until
}
return until.Add(busyGateSlack)
}
// busyWait decides, before a judging, whether the gate waits for a busy container on a machine it judges:
// true while it waits, the gate saying so. False when nothing waits, or no longer: a wait that ended moves the
// gate's Since to its end. Pure but for the gate it is handed.
func busyWait(g *inventory.PlanGate, pairs []judged, f gateFacts, now time.Time) bool {
if g.Busy != nil && g.Busy.Ended != nil {
return false
}
for _, j := range pairs {
b, waits := busyOn(j, f)
if !waits {
continue
}
if g.Busy == nil {
g.Busy = &inventory.GateBusy{Machine: j.node, Module: j.module, Container: b.Container, Since: b.Since,
Seen: now}
}
if g.Busy.Seen.IsZero() {
g.Busy.Seen = now
}
g.Busy.Until, g.Busy.Said = b.Until, b.Said
if now.After(busyGateEnds(g.Busy)) {
// The engine said it would have replaced it by now: the judging starts, and finds the module not
// yet healthy while the machine still lists it waiting (busyStill), until the gate's own bound.
end := now
g.Busy.Ended, g.Busy.PastBound = &end, true
g.Since, g.Passes, g.LastPass, g.Healthy, g.HealthyAt, g.Applied = &end, 0, nil, nil, nil, nil
return false
}
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, g.Busy.Words(), nil
g.Read(now, false, g.Busy.Words())
return true
}
if g.Busy == nil {
return false
}
// Replaced: the judging starts at the report that said so.
end := now
if r, said := f.reports[g.Busy.Machine]; said && r.At != nil && r.At.After(g.Busy.Since) && r.At.Before(now) {
end = r.At.UTC()
}
g.Busy.Ended = &end
g.Since, g.Passes, g.LastPass, g.Last, g.Healthy, g.HealthyAt, g.Applied = &end, 0, nil, "", nil, nil, nil
return false
}
// busyOn is the container of a judged module its machine last reported, on what the gate sent it, as left
// running because it said busy.
func busyOn(j judged, f gateFacts) (inventory.BusyWait, bool) {
r, said := f.reports[j.node]
if !said || r.At == nil || !f.reportedOn(j.node, r) {
return inventory.BusyWait{}, false
}
return inventory.WaitingFor(r.Busy, j.module)
}
// busyStill is a judged module whose container its machine still lists waiting once the gate stopped waiting
// for it: not yet what was sent, so not healthy, whatever its health says of the container still running.
func busyStill(j judged, f gateFacts) (string, bool) {
b, waits := busyOn(j, f)
if !waits {
return "", false
}
return fmt.Sprintf("%s still runs its busy container %s, past its node-engine's bound of %s",
j.node, b.Container, b.Until.UTC().Format(time.RFC3339)), true
}
+148
View File
@@ -0,0 +1,148 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A walk whose module's container waits as busy on its first machine says so, and starts that machine's
// first-machine gate only once the machine reports the container replaced (novox/hq ADR 0335 §4).
func TestAGateWaitsForABusyContainerAndStartsWhenItIsReplaced(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent,
Sent: map[string]inventory.SentDeclaration{"novox": {Digest: "d1", Sequence: 7}}}
pairs := []judged{{module: "build-agent", node: "novox"}, {module: "mesh-controller", node: "novox"}}
reported := sent.Add(20 * time.Second)
waiting := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: reported, Until: reported.Add(15 * time.Minute),
Said: "busy: novox/mesh-controller (check of #280), in hand since 00:58:00Z"}}
facts := func(at time.Time, busy []inventory.BusyWait) gateFacts {
// As judgeMoves hands it: the report held against the gate's own send.
return gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"novox": {Node: "novox",
Outcome: inventory.OutcomeApplied, At: &at, Declared: "d1", ReportedSequence: 7, Busy: busy}}}
}
now := reported.Add(time.Minute)
if !busyWait(g, pairs, facts(reported, waiting), now) {
t.Fatal("a gate whose module's container waits as busy judged it")
}
if !g.Since.Equal(sent) || g.Busy == nil || g.Busy.Machine != "novox" || g.Busy.Ended != nil {
t.Fatalf("the gate does not keep the wait: %+v %+v", g.Since, g.Busy)
}
for _, said := range []string{gateLine(g), g.Last} {
if !strings.Contains(said, "waits for a busy container on novox") {
t.Errorf("the walk does not say it waits for a busy container there: %s", said)
}
}
if !strings.Contains(g.Last, "check of #280") {
t.Errorf("the wait does not say what the container holds: %s", g.Last)
}
// Ten minutes on, still busy, within the engine's bound: still waiting, nothing judged, no bound run.
now = reported.Add(10 * time.Minute)
if !busyWait(g, pairs, facts(reported.Add(9*time.Minute), waiting), now) || g.Passes != 0 {
t.Fatalf("the gate stopped waiting within the bound: %+v", g)
}
// The machine reports it replaced: the judging starts at that report.
replacedAt := reported.Add(12 * time.Minute)
now = replacedAt.Add(5 * time.Second)
replaced := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: "replaced-when-idle", Since: reported, Until: reported.Add(15 * time.Minute)}}
if busyWait(g, pairs, facts(replacedAt, replaced), now) {
t.Fatal("the gate still waits after the container was replaced")
}
if g.Since == nil || !g.Since.Equal(replacedAt) || g.Busy.Ended == nil || !g.Busy.Ended.Equal(replacedAt) {
t.Fatalf("the gate does not start at the report that said it replaced: since %v, %+v", g.Since, g.Busy)
}
if strings.Contains(gateLine(g), "waits for a busy container") {
t.Errorf("a gate judging still says it waits: %s", gateLine(g))
}
// And a later report listing it waiting again does not hold a gate that started.
if busyWait(g, pairs, facts(now, waiting), now.Add(time.Minute)) {
t.Error("a gate that started waits again")
}
}
// An engine that stops reporting while it waits holds no walk for ever: past the engine's bound and the slack,
// the judging starts, and finds the module not yet healthy while its container is still listed waiting.
func TestAGateWaitsNoLongerThanTheEnginesBound(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent}
pairs := []judged{{module: "build-agent", node: "novox"}}
waiting := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: sent, Until: sent.Add(15 * time.Minute)}}
f := gateFacts{reports: map[string]inventory.Reported{"novox": {Node: "novox", Outcome: inventory.OutcomeApplied,
At: &sent, Current: true, Busy: waiting}}}
now := sent.Add(15*time.Minute + busyGateSlack + time.Second)
if busyWait(g, pairs, f, now) {
t.Fatal("a gate waited past the engine's bound and the slack")
}
if !g.Busy.PastBound || !g.Since.Equal(now) {
t.Fatalf("the gate does not start, said past the bound: %+v %v", g.Busy, g.Since)
}
if why, still := busyStill(pairs[0], f); !still || !strings.Contains(why, "past its node-engine's bound") {
t.Fatalf("a module still running its busy container reads as what was sent: %q", why)
}
}
// A report from an engine older than the field lists nothing: the gate starts at the send, as before; and a
// report not about the gate's send says nothing of it.
func TestAGateOfAnEngineThatNeverWaitsStartsAsBefore(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"ace"}, Since: &sent,
Sent: map[string]inventory.SentDeclaration{"ace": {Digest: "d2", Sequence: 9}}}
pairs := []judged{{module: "build-agent", node: "ace"}}
at := sent.Add(time.Minute)
older := gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"ace": {Node: "ace", Outcome: inventory.OutcomeApplied,
At: &at, Declared: "d2", ReportedSequence: 9}}}
if busyWait(g, pairs, older, at) || g.Busy != nil || !g.Since.Equal(sent) {
t.Fatalf("a gate on an engine that never waits did not start at the send: %+v %v", g.Busy, g.Since)
}
elsewhere := gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"ace": {Node: "ace", Outcome: inventory.OutcomeApplied,
At: &at, Declared: "d1", ReportedSequence: 8, Busy: []inventory.BusyWait{{ID: "build-agent.server",
State: inventory.BusyWaiting, Until: at.Add(time.Hour)}}}}}
if busyWait(g, pairs, elsewhere, at) {
t.Fatal("a wait reported about an earlier declaration held the gate")
}
}
// The wait is bounded on this controller's clock: an engine whose clock runs an hour ahead, saying it waits until
// then, holds the gate no longer than 15 minutes and the slack from when the controller first read the wait.
func TestAGateWaitIsBoundedOnTheControllersClock(t *testing.T) {
seen := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &seen}
pairs := []judged{{module: "build-agent", node: "novox"}}
ahead := seen.Add(time.Hour)
f := gateFacts{reports: map[string]inventory.Reported{"novox": {Node: "novox", Outcome: inventory.OutcomeApplied,
At: &seen, Current: true, Busy: []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: ahead, Until: ahead.Add(15 * time.Minute)}}}}}
if !busyWait(g, pairs, f, seen) || !g.Busy.Seen.Equal(seen) {
t.Fatalf("the gate did not start waiting, seen now: %+v", g.Busy)
}
if !busyWait(g, pairs, f, seen.Add(15*time.Minute+busyGateSlack-time.Second)) {
t.Fatal("the gate stopped waiting within the bound on its own clock")
}
if busyWait(g, pairs, f, seen.Add(15*time.Minute+busyGateSlack+time.Second)) || !g.Busy.PastBound {
t.Fatalf("an engine's later word held the gate past the controller's bound: %+v", g.Busy)
}
}
// `plan` says, for a machine whose node-engine is older than the field, each container whose busy is not sent.
func TestAPlanSaysABusyAnOlderEngineIsNotSent(t *testing.T) {
modules := []catalogue.Manifest{{Module: "build-agent", Resources: []map[string]any{
{"id": "server", "type": "container", "busy": map[string]any{"command": "mesh-builder busy"}},
{"id": "agent-env", "type": "file"}}}}
lines := busyNotSent(modules, false)
if len(lines) != 1 || !strings.Contains(lines[0], "build-agent.server declares busy") ||
!strings.Contains(lines[0], "replaced at once") {
t.Fatalf("the plan does not say the busy not sent: %v", lines)
}
if lines := busyNotSent(modules, true); len(lines) != 0 {
t.Fatalf("a busy sent is said as not sent: %v", lines)
}
}
+5
View File
@@ -73,6 +73,11 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
"(issue 280)", busUpgradeNeeded, name, busUpgradeVerb, strings.Join(e.On, ", "))) "(issue 280)", busUpgradeNeeded, name, busUpgradeVerb, strings.Join(e.On, ", ")))
case name == "nats" || catalogue.ProvidesBus(e.Manifest): case name == "nats" || catalogue.ProvidesBus(e.Manifest):
p.Steps = append(p.Steps, "a planned bus step: the bus is upgraded by `bus upgrade`, never by an ordinary send") p.Steps = append(p.Steps, "a planned bus step: the bus is upgraded by `bus upgrade`, never by an ordinary send")
case waits && len(e.On) > 0 && (u.From == catalogue.FromModule || u.From == catalogue.FromData):
// Judged per machine by the walk, against the build each was last sent (novox/hq ADR 0377).
p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s) — unless on a machine "+
"its move recreates none of its containers and moves none of its data, which the walk judges and then sends",
name, orNone(u.From)))
case waits && len(e.On) > 0: case waits && len(e.On) > 0:
p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s)", name, p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s)", name,
orNone(u.From))) orNone(u.From)))
+36 -1
View File
@@ -47,6 +47,24 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
shelf := catalogue.Shelf{} shelf := catalogue.Shelf{}
faulted := map[string]bool{} faulted := map[string]bool{}
failed := 0 failed := 0
// The list of predefined keys at the root of the repository above each manifest (novox/hq ADR 0348 §7), read
// once per repository: a manifest naming a key is judged against the list its own tree holds.
lists := map[string]catalogue.PredefinedList{}
listOf := func(path string) (catalogue.PredefinedList, error) {
root := catalogue.RepositoryRoot(filepath.Dir(path))
if root == "" {
return nil, nil
}
if list, read := lists[root]; read {
return list, nil
}
list, err := catalogue.ReadPredefined(root)
if err != nil {
return nil, err
}
lists[root] = list
return list, nil
}
for _, path := range paths { for _, path := range paths {
raw, err := os.ReadFile(path) raw, err := os.ReadFile(path)
if err != nil { if err != nil {
@@ -54,7 +72,18 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed++ failed++
continue continue
} }
m, err := catalogue.ParseManifest(raw) list, err := listOf(path)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
continue
}
m, err := catalogue.ParseManifestWith(raw, list)
if err == nil {
// Each template's text read from the tree beside the manifest, and judged as inline content is
// (novox/hq ADR 0349 §2): a missing template is refused by name.
m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(path))
}
if err != nil { if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err) fmt.Fprintf(out, "%s: %v\n", path, err)
failed++ failed++
@@ -215,6 +244,12 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
checks = append(checks, fmt.Sprintf("%v by %s", r["id"], catalogue.HealthWords(h))) checks = append(checks, fmt.Sprintf("%v by %s", r["id"], catalogue.HealthWords(h)))
} }
} }
// And its own runtime's, where it runs as its own account (novox/hq ADR 0366).
if own, has := catalogue.OwnRuntimeHealth(m); has {
if h, _, _ := catalogue.ReadHealth(own); h.Kind != "" {
checks = append(checks, fmt.Sprintf("%v by %s", own["id"], catalogue.HealthWords(h)))
}
}
if len(checks) > 0 { if len(checks) > 0 {
fmt.Fprintf(out, ", ready: %s", strings.Join(checks, "; ")) fmt.Fprintf(out, ", ready: %s", strings.Join(checks, "; "))
} }
+87
View File
@@ -0,0 +1,87 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"strconv"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
)
// A pull request's check builds go when it closes (novox/hq ADR 0350 §5): the build seat keeps them in the artifact
// store's scratch namespace for the walk of its merge to take. **Let go of where the moment is heard**: the
// controller ends the walk answering a merge — after that walk took what it reuses — and the delivery's owner, which
// hears the forge say a pull request closed unmerged, asks the `forget-checks` verb; both let go through the store,
// as the controller's own collection does. The build seat's holders let go of anything a week old on their own,
// so a word lost here leaves a week's builds, never more.
// checkBuildsIn is the scratch namespace as the controller reaches the artifact store.
func checkBuildsIn(ctx context.Context, inv *inventory.Inventory) (builder.Scratch, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, errors.New("the artifact store is not on the mesh's network")
}
return builder.Registry{Address: address}, nil
}
// forgetChecks lets go of a pull request's check builds and answers what it did.
var forgetChecks = func(ctx context.Context, inv *inventory.Inventory, pull builder.PullRef) (string, error) {
scratch, err := checkBuildsIn(ctx, inv)
if err != nil {
return "", err
}
n, err := scratch.Forget(ctx, pull)
if err != nil {
return "", fmt.Errorf("%s: %d check(s)' builds let go of, and not all: %w", pull, n, err)
}
return fmt.Sprintf("%s: %d check(s)' builds let go of", pull, n), nil
}
// forgetChecksOfWalk lets go, on the save that ends a walk, of the check builds of every pull request whose merge
// it answers: the walk took what it reuses. Said, never fatal.
func forgetChecksOfWalk(ctx context.Context, inv *inventory.Inventory, p inventory.Plan) {
if p.WalkSaid != inventory.WalkEnded || p.Delivery == nil {
return
}
for _, pull := range reuseAskOf(&p).Pulls {
said, err := forgetChecks(ctx, inv, builder.PullRef{Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number})
if err != nil {
fmt.Fprintf(os.Stderr, "%s: its check builds stay until the week's sweep: %v\n", p.ID, err)
continue
}
fmt.Printf("%s: %s\n", p.ID, said)
}
}
// forgetChecksCommand is `forget-checks <owner> <repo> <number>`: the verb the delivery's owner asks when the forge
// says a pull request closed unmerged (novox/hq ADR 0350 §5).
func forgetChecksCommand(ctx context.Context, args []string) error {
if len(args) != 3 {
return errors.New("forget-checks <owner> <repo> <number>")
}
number, err := strconv.Atoi(args[2])
if err != nil || number <= 0 {
return fmt.Errorf("%q is not a pull request's number", args[2])
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
said, err := forgetChecks(ctx, open.inventory, builder.PullRef{Owner: args[0], Repo: args[1], Number: number})
if err != nil {
return err
}
fmt.Println(said)
return nil
}
+280
View File
@@ -0,0 +1,280 @@
package main
import (
"context"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// One build per change (novox/hq ADR 0350), as the controller asks it and takes it in.
// The check of a change is asked to build what a merge of it would move, in the walk's order: a module of the
// repository checked from the merge, one standing on it from its own repository as the walk builds it, and one
// that reads the repository through a context not at all. A change that moves no module builds nothing.
func TestACheckIsAskedToBuildWhatTheMergeWouldMove(t *testing.T) {
entries := []inventory.Entry{
{Manifest: catalogue.Manifest{Module: "app"}, Source: inventory.Source{Repository: "novox/shop", Seat: "git", Path: "modules/app"}},
{Manifest: catalogue.Manifest{Module: "web"}, Source: inventory.Source{Repository: "novox/web", Seat: "git", Ref: "main"}},
{Manifest: catalogue.Manifest{Module: "docs"}, Source: inventory.Source{Repository: "novox/docs", Seat: "git"}},
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Base: "main", Commit: "abc"}
scope := checkScope{Reach: mergeReach{Packaging: entries[2:], Plan: inventory.Plan{Tiers: [][]string{{"app", "docs"}, {"web"}}}}}
clone := func(s inventory.Source) (string, error) { return "http://forge.invalid/" + s.Repository + ".git", nil }
got := checkBuildsOf(p, scope, entries, clone)
want := []link.CheckBuild{
{Module: "app", Path: "modules/app", Merged: true, Tier: 0},
{Module: "docs", Tier: 0, Skip: "its build reads novox/shop through a context, which a check cannot hand the merge"},
{Module: "web", Repository: "http://forge.invalid/novox/web.git", Ref: "main", Tier: 1},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("the check is asked to build\n%+v\nwant\n%+v", got, want)
}
if got := checkBuildsOf(p, checkScope{}, entries, clone); got != nil {
t.Fatalf("a change that moves no module asks for builds: %+v", got)
}
}
// A walk's ask carries its leave to take a check's build, naming the pull requests of the merges it answers; a dry
// run never does, and neither does an ask that is not a walk's.
func TestOnlyAWalksAskTakesAChecksBuildAndADryRunNever(t *testing.T) {
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}, {Repository: "http://forge.invalid/novox/web.git", Number: 3},
{Repository: "novox/shop", Number: 7}, {Repository: "novox/shop"}}}}
asked := reuseAskOf(p)
if want := []link.PullRef{{Owner: "novox", Repo: "shop", Number: 7}, {Owner: "novox", Repo: "web", Number: 3}}; !reflect.DeepEqual(asked.Pulls, want) {
t.Fatalf("the walk names %+v", asked.Pulls)
}
walk := withReuse(t.Context(), asked)
if reuseAsked(walk, false) != asked {
t.Fatal("a walk's ask does not carry its leave")
}
if reuseAsked(walk, true) != nil {
t.Fatal("a dry run carries a leave to take a check's build")
}
if reuseAsked(t.Context(), false) != nil {
t.Fatal("an ask that is not a walk's carries a leave")
}
}
// A walk ended lets go of its merges' check builds — only once it ended, after it took what it reuses.
func TestAWalkEndedLetsGoOfItsMergesCheckBuilds(t *testing.T) {
var forgot []builder.PullRef
was := forgetChecks
forgetChecks = func(_ context.Context, _ *inventory.Inventory, pull builder.PullRef) (string, error) {
forgot = append(forgot, pull)
return "", nil
}
t.Cleanup(func() { forgetChecks = was })
p := inventory.Plan{ID: "plan-1", Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}}}}
p.WalkSaid = inventory.WalkStarted
forgetChecksOfWalk(t.Context(), nil, p)
p.WalkSaid = ""
forgetChecksOfWalk(t.Context(), nil, p)
if len(forgot) != 0 {
t.Fatalf("a walk not ended let go of %v, which it may still take", forgot)
}
p.WalkSaid = inventory.WalkEnded
forgetChecksOfWalk(t.Context(), nil, p)
if want := []builder.PullRef{{Owner: "novox", Repo: "shop", Number: 7}}; !reflect.DeepEqual(forgot, want) {
t.Fatalf("the walk ended let go of %v", forgot)
}
}
// A build in the scratch namespace is never registered: only the walk's copy in the module's own namespace is.
func TestABuildInTheScratchNamespaceIsNeverRegistered(t *testing.T) {
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: "store.internal:5000/app/server@sha256:aa"}}); got != "" {
t.Fatalf("a module's own build reads as a scratch build: %s", got)
}
scratch := "store.internal:5000/scratch/app/server@sha256:aa"
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: scratch}}); got != scratch {
t.Fatalf("a scratch build was not found: %q", got)
}
if got := inScratch([]link.MadeArtifact{{Name: "look", Reference: "http://store.internal:5000/v2/scratch/app/look/blobs/sha256:bb"}}); got == "" {
t.Fatal("a scratch archive was not found")
}
}
// The walk says per module what it did: "reused from …" or "built: <why>" — and a build seat from before says
// neither, which is said.
func TestAWalksBuildSaysWhetherItReusedOrBuilt(t *testing.T) {
if got := (link.BuildResult{Reused: "reused from the check of novox/shop#7"}).MadeWords(); got != "reused from the check of novox/shop#7" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{BuiltWhy: "no check built it"}).MadeWords(); got != "built: no check built it" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{}).MadeWords(); !strings.Contains(got, "predates ADR 0350") {
t.Fatalf("said %q", got)
}
}
// A walk's module the check's build stood in for is registered by the digest the check built — the copy in the
// module's namespace — and the walk keeps "reused from …" for it; its first-node gate judges that registered
// build as it judges one the walk built. A scratch reference is refused. Needs the store (MESH_TEST_POSTGRES), and
// is named in testdata/store-tests.
func TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
gateEvery = 0
digest := "sha256:" + strings.Repeat("c", 64)
reused := func(id, reference string) link.BuildResult {
manifest := mustJSON(t, catalogue.Manifest{Module: "app", Version: "c3", Resources: []map[string]any{
{"id": "svc", "type": "container", "name": "app", "image": reference}}})
return link.BuildResult{ID: id, Repository: "novox/mesh-catalog", Path: "modules/app", Ref: "main", On: "anchor",
Commit: "c3", Manifest: manifest, Trunk: "main", OnTrunk: true, Branches: []string{"main"},
Made: []link.MadeArtifact{{Name: "server", Kind: "image", Reference: reference}},
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"},
Reused: "reused from the check of novox/mesh-catalog#9 at head abcdefabcdef (merged into trunk 123456123456): server " + digest,
ReusedFrom: &link.ReusedFrom{Pull: link.PullRef{Owner: "novox", Repo: "mesh-catalog", Number: 9},
Head: "abcdefabcdef", Trunk: "123456123456", Key: "r1:" + strings.Repeat("1", 64)}}
}
if _, _, err := takeIn(ctx, inv, reused(link.NewBuildID(time.Now()), "store.internal:5000/scratch/app/server@"+digest)); err == nil ||
!strings.Contains(err.Error(), "scratch") {
t.Fatalf("a build in the scratch namespace was registered: %v", err)
}
id := link.NewBuildID(time.Now().Add(time.Second))
result := reused(id, "store.internal:5000/app/server@"+digest)
if _, _, err := takeIn(ctx, inv, result); err != nil {
t.Fatal(err)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil || current["app"].Commit != "c3" {
t.Fatalf("the reused build is not what app is registered at: %+v %v", current["app"], err)
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(shelf["app"].Resources)
if !strings.Contains(string(raw), "app/server@"+digest) {
t.Fatalf("app is registered as %s, not by the check's digest in its own namespace", raw)
}
// The walk asked it: the walk keeps what its build did, and its gate judges the registered build.
asked := time.Now().UTC()
p := g.plan(t)
p.Commit = "c3"
p.Modules["app"] = &inventory.PlanModule{State: "asked", AskedAt: &asked, Build: id}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
planBuiltSaying(ctx, g.open, "app", "c3", "", asked, id, result.MadeWords())
p = g.plan(t)
if m := p.Modules["app"]; m.State != "built" || !strings.HasPrefix(m.Made, "reused from the check of novox/mesh-catalog#9") {
t.Fatalf("the walk keeps %+v", m)
}
for i := 0; i < 6; i++ {
advancePlans(ctx, g.open)
}
p = g.plan(t)
gate := p.Modules["app"].Gate
if gate == nil || gate.To != "c3" || gate.Verdict != inventory.GatePassed {
t.Fatalf("the gate judged %+v, not the reused build", gate)
}
if sent, _, _ := inv.SentBuilds(ctx, "anchor"); sent["app"] != "c3" {
t.Fatalf("the first machine was sent %v, not the reused build", sent)
}
}
// A walk's leave names the builds its own pull requests' verdicts made at the head each merge merged — never an
// earlier head's — read once per walk, and none when they cannot be read: the builder takes nothing a verdict does
// not name (novox/hq ADR 0350 §2, issue 703).
func TestAWalksLeaveNamesTheBuildsItsPullRequestsVerdictsMade(t *testing.T) {
was := checkedBuildsOf
forgetWalkVerdicts()
t.Cleanup(func() { checkedBuildsOf = was; forgetWalkVerdicts() })
var asked [][]link.PullRef
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
atHead := link.CheckedBuild{Pull: pull, Check: "build-2", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:2"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{atHead, earlier}, nil
}
pulls := []link.PullRef{pull}
heads := map[link.PullRef]string{pull: "bbbbbbbbbbbb0123456789"}
for range 3 {
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, heads); !reflect.DeepEqual(got.Builds,
[]link.CheckedBuild{atHead}) {
t.Fatalf("the leave names %+v, not only the merged head's build", got.Builds)
}
}
if !reflect.DeepEqual(asked, [][]link.PullRef{pulls}) {
t.Fatalf("the verdicts were read %d times for one walk: %v", len(asked), asked)
}
// A merge whose head was not said takes nothing.
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, nil); got.Builds != nil {
t.Fatalf("a merge of no said head names %+v", got.Builds)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := reuseWithVerdicts(t.Context(), "plan-2", &link.ReuseAsk{Pulls: pulls}, heads); got.Builds != nil {
t.Fatalf("verdicts that could not be read named %+v", got.Builds)
}
}
// A check is handed only the builds earlier checks of its own pull request at its own head made: another pull
// request's build, or an earlier head's, is never kept (novox/hq issue 703). None when they cannot be read.
func TestACheckIsHandedOnlyItsOwnHeadsBuilds(t *testing.T) {
was := checkedBuildsOf
t.Cleanup(func() { checkedBuildsOf = was })
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
mine := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-0", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
another := link.CheckedBuild{Pull: link.PullRef{Owner: "novox", Repo: "shop", Number: 8}, Check: "build-9",
Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
var asked [][]link.PullRef
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{mine, earlier, another}, nil
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Commit: "bbbbbbbbbbbb0123456789"}
builds := []link.CheckBuild{{Module: "app"}}
if got := namedForTheCheck(t.Context(), p, builds); !reflect.DeepEqual(got, []link.CheckedBuild{mine}) ||
!reflect.DeepEqual(asked, [][]link.PullRef{{pull}}) {
t.Fatalf("the check is handed %+v, read for %v", got, asked)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := namedForTheCheck(t.Context(), p, builds); got != nil {
t.Fatalf("verdicts that could not be read named %+v", got)
}
}
func forgetWalkVerdicts() {
walkVerdicts.Lock()
defer walkVerdicts.Unlock()
walkVerdicts.read = map[string]verdictsRead{}
}
// A walk's merged heads are the forge's word for each merge; a merge announced without its head, or a pull request
// merged at two heads in one walk, has none, and its walk takes no check's build (novox/hq issue 703).
func TestAWalksMergedHeadsAreTheForgesWord(t *testing.T) {
merge := func(number int, head string) inventory.PlanMerge {
return inventory.PlanMerge{Repository: "novox/shop", Number: number, Head: head}
}
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
merge(7, "aaaaaaaaaaaa1"), merge(8, ""), merge(9, "bbbbbbbbbbbb"), merge(9, "cccccccccccc")}}}
heads := mergedHeadsOf(p)
shop := func(n int) link.PullRef { return link.PullRef{Owner: "novox", Repo: "shop", Number: n} }
if heads[shop(7)] != "aaaaaaaaaaaa1" || heads[shop(8)] != "" || heads[shop(9)] != "" {
t.Fatalf("merged heads %v", heads)
}
}
+140
View File
@@ -0,0 +1,140 @@
package main
import (
"context"
"fmt"
"regexp"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382, issue 867).
//
// A mesh-host change and a mesh-controller change that need each other could not both pass their own merge checks:
// each was read beside the other's repository at the commit the mesh runs, which holds neither change. So the
// controller asks mesh-delivery's `groups` once for every pull request's check it asks (§1), and names, in the ask,
// the group's other open pull requests into a base of the same name, in a repository the build seat reads beside the
// check — mesh-lab never — each with its whole head. The build seat reads those repositories at the sibling's merge (§2).
// A pull request for which mesh-delivery answered and named no sibling is asked exactly as before (§7).
// wholeCommit is a commit as the forge says a head: forty hexadecimal characters.
var wholeCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
// besideRepositories are the repositories (lower case, owner/repo) a check reads beside its own, by the directory each
// is found under — the core's, as besideOf clones them, and mesh-sdk beside the controller; never mesh-lab, which is
// always read at its trunk.
func besideRepositories(entries []inventory.Entry) map[string]string {
read := map[string]string{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
read[nameOf(e.Source.Repository)] = dir
if dir == "mesh-controller" && e.Source.Seat != "" {
read[nameOf(siblingOf(e.Source.Repository, "mesh-sdk"))] = "mesh-sdk"
}
}
delete(read, "")
return read
}
// nameOf is a recorded repository as owner/repo in lower case, empty when it names no owner.
func nameOf(repository string) string {
owner, repo := sourceOwner(repository), sourceName(repository)
if owner == "" || repo == "" {
return ""
}
return strings.ToLower(owner + "/" + repo)
}
// siblingsFor is the pull request's siblings, as mesh-delivery's groups say them (ADR 0382 §1): the other members of
// the group its head branch names that are open, unmerged and into a base of its own base's name, in a repository the
// check reads beside it (beside: the directories the ask already names). The second answer is why they could not be
// learned, empty when they could: the group's members carry no head or base — a mesh-delivery from before the rule.
func siblingsFor(p link.PullUpdated, entries []inventory.Entry, beside map[string]link.CheckedOut,
lines []groupsLine) ([]link.CheckSibling, string) {
own := strings.ToLower(p.Owner + "/" + p.Repo)
var group *groupsLine
if p.Head == "" {
// A recheck (mesh-delivery's `recheck` through `delivery check`) carries no head branch name: the group is the one
// whose members hold this very pull request, by repository and number.
learnable := false
for i := range lines {
for _, m := range lines[i].Details {
learnable = true
if p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
group = &lines[i]
}
}
}
if group == nil {
for _, l := range lines {
if len(l.Details) == 0 && len(l.Members) > 0 && !learnable {
return nil, "mesh-delivery named the groups' members without their pull requests (a mesh-delivery from before novox/hq ADR 0382): " +
"the group of a pull request announced without its head branch cannot be found"
}
}
return nil, "" // a lone pull request
}
} else {
for i := range lines {
if lines[i].ID == p.Head {
group = &lines[i]
}
}
if group == nil {
return nil, ""
}
}
if len(group.Details) == 0 && len(group.Members) > 0 {
return nil, "mesh-delivery named the group's members without their heads and bases (a mesh-delivery from before novox/hq ADR 0382)"
}
read := besideRepositories(entries)
var siblings []link.CheckSibling
for _, m := range group.Details {
repository := strings.ToLower(m.Repository)
if repository == own {
continue
}
if !m.Open {
continue
}
if m.Head == "" || m.Base == "" {
return nil, "mesh-delivery named " + m.ID + " without its head or base (a mesh-delivery from before novox/hq ADR 0382)"
}
dir, readBeside := read[repository]
if m.Base != p.Base || !readBeside || dir == "mesh-lab" {
continue
}
if _, there := beside[dir]; !there {
continue
}
head := strings.ToLower(m.Head)
if !wholeCommit.MatchString(head) {
return nil, fmt.Sprintf("mesh-delivery named %s with a head that is not a whole commit (%q)", m.ID, m.Head)
}
siblings = append(siblings, link.CheckSibling{Repository: m.Repository, Number: m.Number, Head: head, Base: m.Base, Dir: dir})
}
sort.Slice(siblings, func(i, j int) bool { return siblings[i].Dir < siblings[j].Dir })
return siblings, ""
}
// withSiblings asks mesh-delivery's `groups` once and says in the request the pull request's siblings, or why they
// could not be learned (ADR 0382 §1): a check never waits on mesh-delivery past groupAskWithin and never fails for it,
// so a fix to mesh-delivery is still checked without it.
func withSiblings(ctx context.Context, request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry) {
if request.Check == nil {
return
}
lines, err := groupsAsked(ctx, deliverySeatHeld(entries))
if err != nil {
request.Check.SiblingsUnknown = fmt.Sprintf("mesh-delivery did not say the group's pull requests (%v): "+
"the check reads no siblings", err)
return
}
request.Check.Siblings, request.Check.SiblingsUnknown = siblingsFor(p, entries, request.Check.Beside, lines)
}
+248
View File
@@ -0,0 +1,248 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382 §1, §7, issue 867): the
// controller asks mesh-delivery's `groups` once for every check it asks, and names in the ask the group's other open
// pull requests into a base of the same name, in a repository the build seat reads beside it — mesh-lab never.
const (
headHost = "1111111111111111111111111111111111111111"
headController = "2222222222222222222222222222222222222222"
headSDK = "3333333333333333333333333333333333333333"
headLab = "4444444444444444444444444444444444444444"
headHq = "5555555555555555555555555555555555555555"
)
func theCoreOnTheGitSeat() []inventory.Entry {
onSeat := func(module, repository string) inventory.Entry {
e := fromRepo(module, repository, "")
e.Source.Seat = "git"
return e
}
nats := onSeat("nats", "novox/mesh-catalog")
nats.Source.Path = "modules/nats"
held := catalogue.Manifest{Module: "mesh-delivery", Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}}
return []inventory.Entry{onSeat("mesh-controller", "novox/mesh-controller"), onSeat("mesh-host", "novox/mesh-host"),
onSeat("node-tools", "novox/mesh-tools"), nats, {Manifest: held, On: []string{"anchor"}}}
}
func besideTheCore() map[string]link.CheckedOut {
return map[string]link.CheckedOut{"mesh-controller": {Repository: "c"}, "mesh-host": {Repository: "h"},
"mesh-tools": {Repository: "t"}, "mesh-catalog": {Repository: "k"}, "mesh-sdk": {Repository: "s"}, "mesh-lab": {Repository: "l"}}
}
func member(repository string, number int, head, base string, open bool) memberLine {
return memberLine{ID: repository + "@" + head[:12], Repository: repository, State: "checked", Number: number, Head: head,
Base: base, Open: open}
}
func group(id string, members ...memberLine) groupsLine {
line := groupsLine{ID: id, Details: members}
for _, m := range members {
line.Members = append(line.Members, m.ID+" ("+m.State+")")
}
return line
}
func TestASiblingIsTheGroupsOtherOpenPullRequestIntoABaseOfTheSameName(t *testing.T) {
entries := theCoreOnTheGitSeat()
pull := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
for _, c := range []struct {
what string
lines []groupsLine
want []string
unknown string
}{
{"the other member, open, same base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
[]string{"mesh-controller=novox/mesh-controller#346@" + headController + " onto main"}, ""},
{"a member into a base of another name is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "release", true))},
nil, ""},
{"a member not open (merged or closed) is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", false))},
nil, ""},
{"mesh-lab is never one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-lab", 5, headLab, "main", true))},
nil, ""},
{"a repository the check reads nothing of is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 9, headHq, "main", true))},
nil, ""},
{"mesh-sdk, read beside the controller, is one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-sdk", 7, headSDK, "main", true))},
[]string{"mesh-sdk=novox/mesh-sdk#7@" + headSDK + " onto main"}, ""},
{"another group is none of its business", []groupsLine{group("feat/y",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
nil, ""},
{"a group of its own alone has none", []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))},
nil, ""},
{"an older mesh-delivery names no heads", []groupsLine{{ID: "feat/x",
Members: []string{"novox/mesh-host@111111111111 (checked)", "novox/mesh-controller@222222222222 (checked)"}}},
nil, "without their heads and bases"},
{"a member without its base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "", true))},
nil, "without its head or base"},
{"a head that is no commit", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController[:12]+"zzzzzzzzzzzzzzzzzzzzzzzzzzzz", "main", true))},
nil, "not a whole commit"},
} {
got, unknown := siblingsFor(pull, entries, besideTheCore(), c.lines)
var said []string
for _, s := range got {
said = append(said, s.Dir+"="+s.Repository+"#"+itoa(s.Number)+"@"+s.Head+" onto "+s.Base)
}
if strings.Join(said, ";") != strings.Join(c.want, ";") {
t.Errorf("%s: siblings %v, wanted %v", c.what, said, c.want)
}
if (c.unknown == "") != (unknown == "") || !strings.Contains(unknown, c.unknown) {
t.Errorf("%s: could not learn %q, wanted %q", c.what, unknown, c.unknown)
}
}
// A pull request into a release branch has siblings into the release branch of the same name (ADR 0358 §1).
release := pull
release.Base = "feat/57"
lines := []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "feat/57", true),
member("novox/mesh-controller", 346, headController, "feat/57", true), member("novox/mesh-tools", 8, headLab, "main", true))}
if got, _ := siblingsFor(release, entries, besideTheCore(), lines); len(got) != 1 || got[0].Dir != "mesh-controller" {
t.Errorf("into a release branch: %+v", got)
}
// A recheck announces the pull request without its head branch: its group is the one holding it (repository, number).
pull.Head = ""
got, unknown := siblingsFor(pull, entries, besideTheCore(), lines)
if len(got) != 1 || got[0].Dir != "mesh-tools" || unknown != "" {
t.Errorf("a recheck of a group member: %+v %q", got, unknown)
}
// A pull request in no group, rechecked, is alone: no sibling, and nothing said.
pull.Number = 999
if got, unknown := siblingsFor(pull, entries, besideTheCore(), lines); got != nil || unknown != "" {
t.Errorf("a lone pull request: %+v %q", got, unknown)
}
if got, unknown := siblingsFor(pull, entries, besideTheCore(), nil); got != nil || unknown != "" {
t.Errorf("no groups at all: %+v %q", got, unknown)
}
// A mesh-delivery that names members without their pull requests cannot find the group of a recheck: it says so.
older := []groupsLine{{ID: "feat/x", Members: []string{"novox/mesh-host@111111111111 (checked)"}}}
pull.Number = 114
if got, unknown := siblingsFor(pull, entries, besideTheCore(), older); got != nil || unknown == "" {
t.Errorf("an older mesh-delivery: %+v %q", got, unknown)
}
}
func itoa(n int) string { b, _ := json.Marshal(n); return string(b) }
// Every check asks mesh-delivery's `groups` once; the ask names the sibling's repository, head and base.
func TestEveryCheckAsksMeshDeliveryOnceAndNamesItsSiblings(t *testing.T) {
entries := theCoreOnTheGitSeat()
asked := 0
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
asked++
if !held {
t.Error("the seat is held, yet it was asked as not held")
}
return []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true),
member("novox/mesh-controller", 346, headController, "main", true))}, nil
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if asked != 1 {
t.Fatalf("mesh-delivery was asked %d times for one check", asked)
}
want := link.CheckSibling{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", Dir: "mesh-controller"}
if got := request.Check.Siblings; len(got) != 1 || got[0] != want || request.Check.SiblingsUnknown != "" {
t.Fatalf("the ask names siblings %+v, unknown %q", got, request.Check.SiblingsUnknown)
}
body, _ := json.Marshal(request.Check)
if !strings.Contains(string(body), `"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"`+headController+
`","base":"main","dir":"mesh-controller"}]`) {
t.Fatalf("on the wire: %s", body)
}
}
// When mesh-delivery does not answer in time (or is not held, or answered without the fields), the check reads no
// sibling and its request says it could not learn them, so a fix to mesh-delivery is still checked without it.
func TestWhenMeshDeliveryDoesNotAnswerTheCheckReadsNoSiblingAndSaysSo(t *testing.T) {
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, c := range []struct {
what string
err error
held bool
}{{"it did not answer", context.DeadlineExceeded, true}, {"nothing serves it", errors.New("nothing serves mesh-delivery.groups"), true},
{"no holder on record", errors.New("no holder of mesh-delivery is on record to say it"), false}} {
entries := theCoreOnTheGitSeat()
if !c.held {
entries = entries[:4]
}
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
if held != c.held {
t.Errorf("%s: held %v", c.what, held)
}
return nil, c.err
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if len(request.Check.Siblings) != 0 || !strings.Contains(request.Check.SiblingsUnknown, c.err.Error()) ||
!strings.Contains(request.Check.SiblingsUnknown, "reads no siblings") {
t.Errorf("%s: siblings %+v, unknown %q", c.what, request.Check.Siblings, request.Check.SiblingsUnknown)
}
}
}
// **A pull request for which mesh-delivery answered and named no sibling is asked exactly as before** (ADR 0382 §7):
// byte for byte the request it was without the rule.
func TestAPullRequestWithNoSiblingIsAskedByteForByteAsBefore(t *testing.T) {
entries := theCoreOnTheGitSeat()
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, answer := range [][]groupsLine{nil, {group("feat/other", member("novox/mesh-controller", 346, headController, "main", true))},
{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 3, headHq, "main", true))}} {
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, nil }
build := func() link.BuildRequest {
return link.BuildRequest{ID: "build-1", Repository: "r", Ref: headHost, Check: &link.CheckRequest{Owner: "novox",
Repo: "mesh-host", Number: 114, Base: "main", Beside: besideTheCore(), Modules: []string{"mesh-host"}, Judge: "validator"}}
}
before, _ := json.Marshal(build())
after := build()
withSiblings(t.Context(), &after, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
got, _ := json.Marshal(after)
if string(before) != string(got) {
t.Fatalf("the request changed:\nbefore %s\nafter %s", before, got)
}
}
}
// What the build seat's verdict says of its siblings reaches the forge's holder in the controller's `checked`.
func TestTheVerdictCarriesTheSiblingsAndWhereTheJudgeCameFrom(t *testing.T) {
read := []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", BaseCommit: headHost}}
c := checkedOf(link.BuildResult{ID: "b", Ref: headHost, Checked: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114},
Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok", Siblings: read, JudgeFrom: "sibling novox/mesh-controller#346 merge abc",
Validator: "the change's own validator"}})
if len(c.Siblings) != 1 || c.Siblings[0] != read[0] || c.JudgeFrom == "" || c.Validator == "" {
t.Fatalf("the verdict says %+v", c)
}
body, _ := json.Marshal(c)
for _, want := range []string{`"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"` + headController +
`","base":"main","base_commit":"` + headHost + `"}]`, `"judge_from":"sibling novox/mesh-controller#346 merge abc"`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the verdict lacks %s:\n%s", want, body)
}
}
unknown := checkedOf(link.BuildResult{ID: "b", Check: &link.CheckOutcome{Verdict: "pass", SiblingsUnknown: "mesh-delivery did not answer"}})
if body, _ := json.Marshal(unknown); !strings.Contains(string(body), `"siblings_unknown":"mesh-delivery did not answer"`) ||
strings.Contains(string(body), `"siblings":`) {
t.Fatalf("the verdict that could not learn them says %s", body)
}
}
+246 -25
View File
@@ -3,6 +3,7 @@ package main
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"path" "path"
"regexp" "regexp"
@@ -179,6 +180,8 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
return err return err
} }
request.Check.Plan = &plan request.Check.Plan = &plan
// **The group's other open pull requests are read beside it** (novox/hq ADR 0382 §1).
withSiblings(ctx, &request, p, entries)
seat := buildSeatHeld(ctx) seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat) ask, err := askOverOn(seat)
if err != nil { if err != nil {
@@ -188,6 +191,8 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
if err := ask.Ask(ctx, request); err != nil { if err := ask.Ask(ctx, request); err != nil {
return err return err
} }
// A new head ends the check of the old one (novox/hq issue 726).
endingSupersededChecks(ctx, seat, p, request.ID, f.oneShot)
what := "its own merge-check.sh alone: " + noModuleTouched what := "its own merge-check.sh alone: " + noModuleTouched
if scope.gated() { if scope.gated() {
what = fmt.Sprintf("the gate over %s (a merge would build %d module(s) in %d tier(s))", what = fmt.Sprintf("the gate over %s (a merge would build %d module(s) in %d tier(s))",
@@ -248,31 +253,16 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
if err != nil { if err != nil {
return link.BuildRequest{}, err return link.BuildRequest{}, err
} }
// Beside it, at what the mesh runs: each core repository by the module the mesh builds from it. // **A pull request into a release branch is judged by that branch** (novox/hq ADR 0358): asked of the forge once.
beside := map[string]link.CheckedOut{} release, err := releaseBranchOf(ctx, p, scope, entries)
for _, e := range entries { if err != nil {
dir, core := coreModules[e.Manifest.Module] return link.BuildRequest{}, err
if !core || e.Provided || e.Source.Repository == "" {
continue
}
url, err := clone(e.Source)
if err != nil {
return link.BuildRequest{}, err
}
refs := besideRefs(dir, current[e.Manifest.Module].Commit)
beside[dir] = link.CheckedOut{Repository: url, Ref: refs[dir]}
if dir == "mesh-controller" {
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
if url, err := clone(inventory.Source{Seat: e.Source.Seat, Repository: siblingOf(e.Source.Repository,
sibling)}); err == nil {
beside[sibling] = link.CheckedOut{Repository: url, Ref: refs[sibling]}
}
}
}
}
} }
beside, err := besideOf(entries, func(module string) string { return current[module].Commit }, clone, release)
if err != nil {
return link.BuildRequest{}, err
}
builds := checkBuildsOf(p, scope, entries, clone)
return link.BuildRequest{ return link.BuildRequest{
ID: link.NewBuildID(time.Now()), ID: link.NewBuildID(time.Now()),
Repository: repository, Repository: repository,
@@ -282,10 +272,237 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
Source: sourceOnSeat(source), Source: sourceOnSeat(source),
Check: &link.CheckRequest{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base, Check: &link.CheckRequest{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base,
Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents, New: scope.New, Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents, New: scope.New,
Manifests: scope.Manifests, Judge: scope.Judge}, Manifests: scope.Manifests, Judge: scope.Judge, Builds: builds, Named: namedForTheCheck(ctx, p, builds),
Release: release, Running: runningBy(ctx)},
}, nil }, nil
} }
// besideOf is what a check reads beside its own repository, by the directory it is found under: each core repository
// by the module the mesh builds from it, at besideRefs — or, for a pull request into a release branch (novox/hq ADR
// 0358 §3), at its branch of that name where it has one and at its trunk otherwise, the controller's main kept for
// its judge, and mesh-lab at its trunk whatever the base: its replays run with the container runtime's socket. running
// is the commit the mesh runs of a module.
func besideOf(entries []inventory.Entry, running func(module string) string,
clone func(inventory.Source) (string, error), release string) (map[string]link.CheckedOut, error) {
beside := map[string]link.CheckedOut{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
url, err := clone(e.Source)
if err != nil {
return nil, err
}
refs := besideRefs(dir, running(e.Manifest.Module))
at := func(dir, trunk string) link.CheckedOut {
if release == "" {
return link.CheckedOut{Repository: url, Ref: refs[dir]}
}
if trunk == "" {
trunk = "main"
}
return link.CheckedOut{Repository: url, Ref: release, Else: trunk}
}
beside[dir] = at(dir, followedBranch(e.Source.Ref))
if dir == "mesh-controller" {
// The controller's main stays its main: it judges only a check the running controller predates.
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if release != "" {
// mesh-controller has the release branch: that is what makes it one (ADR 0358 §1).
beside[dir] = link.CheckedOut{Repository: url, Ref: release}
}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
if url, err := clone(inventory.Source{Seat: e.Source.Seat, Repository: siblingOf(e.Source.Repository,
sibling)}); err == nil {
b := link.CheckedOut{Repository: url, Ref: refs[sibling]}
// **mesh-lab is read at its trunk, always** (ADR 0358): its replays run with the container
// runtime's socket, so they are only ever reviewed code — never a branch anyone may create.
if release != "" && sibling != "mesh-lab" {
b = link.CheckedOut{Repository: url, Ref: release, Else: "main"}
}
beside[sibling] = b
}
}
}
}
}
return beside, nil
}
// releaseBranchOf is the release branch a pull request merges into (novox/hq ADR 0358 §1): its base, when that is not
// its repository's trunk and mesh-controller has a branch of that name; empty otherwise, and the pull request is
// checked as before it. **A base that is a trunk asks nothing**: the branch a module built from the repository follows
// (main when it records none), or mesh-controller's own trunk. Any other base is asked of the forge once, and an
// answer it cannot give is an error, never read as "no release branch".
func releaseBranchOf(ctx context.Context, p link.PullUpdated, scope checkScope, entries []inventory.Entry) (string, error) {
base := p.Base
if base == "" {
return "", nil
}
if scope.From != nil && base == trunkOf(scope.From.Source) {
return "", nil
}
var controller *inventory.Entry
for i, e := range entries {
if e.Manifest.Module == "mesh-controller" && !e.Provided && e.Source.Repository != "" {
controller = &entries[i]
break
}
}
if controller == nil || base == trunkOf(controller.Source) {
return "", nil
}
if !tryRef.MatchString(base) || strings.Contains(base, "..") {
return "", nil // no branch the forge could hold: checked as before
}
owner, repo := sourceOwner(controller.Source.Repository), sourceName(controller.Source.Repository)
if owner == "" || repo == "" {
return "", nil
}
has, err := forgeHasBranch(ctx, owner, repo, base)
if err != nil {
return "", fmt.Errorf("whether %s/%s#%d merges into a release branch cannot be told — the forge did not say "+
"whether %s/%s has a branch %s (novox/hq ADR 0358): %w", p.Owner, p.Repo, p.Number, owner, repo, base, err)
}
if !has {
return "", nil
}
return base, nil
}
// trunkOf is the branch a module's source follows: main when it records none, or records a commit.
func trunkOf(s inventory.Source) string {
if t := strings.TrimSpace(followedBranch(s.Ref)); t != "" {
return t
}
return "main"
}
// sourceName is the name of a recorded repository, a path on the git seat or a URL: novox/mesh-host → mesh-host.
func sourceName(repository string) string {
parts := strings.Split(strings.Trim(strings.TrimSuffix(repository, ".git"), "/"), "/")
if len(parts) < 2 {
return ""
}
return parts[len(parts)-1]
}
// forgeHasBranch asks the forge, through its module's tools on the bus, whether a repository has a branch. A
// variable so a test needs no forge.
var forgeHasBranch = func(ctx context.Context, owner, repo, branch string) (bool, error) {
js, err := aBus()
if err != nil {
return false, err
}
defer js.Close()
raw, _ := json.Marshal(map[string]any{"path": "/repos/" + owner + "/" + repo + "/branches/" + branch})
answer, err := link.Ask(ctx, link.OverNATS{Conn: js.Conn()}, "gitea", "gitea_api", raw, 30*time.Second)
if err != nil {
return false, err
}
return branchAnswered(answer, branch)
}
// branchAnswered reads the forge's answer about a branch: a 404 is "no such branch"; an answer naming the branch is
// "it has it"; anything else — another error, an unreadable answer, one naming no branch or another — is an error,
// never read as "no release branch".
func branchAnswered(answer link.Answer, branch string) (bool, error) {
if answer.Error != "" {
if strings.Contains(answer.Error, " 404 ") {
return false, nil
}
return false, errors.New(answer.Error)
}
var found struct {
Result struct {
Name string `json:"name"`
} `json:"result"`
}
if err := json.Unmarshal(answer.Result, &found); err != nil {
return false, fmt.Errorf("the forge's answer about branch %s cannot be read: %w", branch, err)
}
if found.Result.Name != branch {
return false, fmt.Errorf("the forge answered about branch %s naming %q", branch, found.Result.Name)
}
return true, nil
}
// namedForTheCheck are the builds earlier checks of the same pull request at the same head named, read once for the
// check (novox/hq issue 703): asked again because only the trunk moved, the check keeps what they built of a key
// still the same. Another pull request's build, or one of an earlier head, is never kept. None when they cannot be
// read, and the check builds, as before.
func namedForTheCheck(ctx context.Context, p link.PullUpdated, builds []link.CheckBuild) []link.CheckedBuild {
if len(builds) == 0 || p.Commit == "" {
return nil
}
pull := link.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number}
read, err := checkedBuildsOf(ctx, []link.PullRef{pull})
if err != nil {
fmt.Printf(" the earlier checks' builds of %s/%s#%d could not be read, so the check builds: %v\n",
p.Owner, p.Repo, p.Number, err)
// empty-on-error: said above; a check handed no named build builds every module, as before issue 703
return nil
}
var named []link.CheckedBuild
for _, b := range read {
if b.Pull == pull && link.SameHead(b.Head, p.Commit) {
named = append(named, b)
}
}
return named
}
// checkBuildsOf are the modules a merge of the change would build, in the walk's order, as the check builds them
// for the walk to take (novox/hq ADR 0350 §1): a module built from the repository checked, at the branch it merges
// into, from the merge the check makes; any other — a module standing on one — from its own repository at the
// branch it follows, as the walk builds it. A module whose build reads the repository checked through a context,
// or that the merge deletes, cannot be built as the walk would: said, and nothing after it is built. A change that
// moves no module builds nothing.
func checkBuildsOf(p link.PullUpdated, scope checkScope, entries []inventory.Entry,
clone func(inventory.Source) (string, error)) []link.CheckBuild {
if len(scope.Reach.Plan.Tiers) == 0 {
return nil
}
m := link.SourceMoved{Owner: p.Owner, Repo: p.Repo, Base: p.Base, CloneURL: p.CloneURL}
byName := map[string]inventory.Entry{}
for _, e := range entries {
byName[e.Manifest.Module] = e
}
skip := map[string]string{}
for _, e := range scope.Reach.Packaging {
skip[e.Manifest.Module] = "its build reads " + p.Owner + "/" + p.Repo + " through a context, which a check " +
"cannot hand the merge"
}
for _, e := range scope.Reach.Deleted {
skip[e.Manifest.Module] = "the merge deletes it"
}
var out []link.CheckBuild
for tier, names := range scope.Reach.Plan.Tiers {
for _, name := range names {
e, known := byName[name]
b := link.CheckBuild{Module: name, Tier: tier, Skip: skip[name]}
switch {
case !known:
b.Skip = "the catalogue does not hold it"
case b.Skip != "":
case sourceIs(e.Source, m):
b.Merged, b.Path = true, e.Source.Path
default:
url, err := clone(e.Source)
if err != nil {
b.Skip = "its repository cannot be cloned: " + err.Error()
break
}
b.Repository, b.Path, b.Ref = url, e.Source.Path, followedBranch(e.Source.Ref)
}
out = append(out, b)
}
}
return out
}
// besideRefs is the ref each repository is cloned at beside a check, by the directory it is found under: // besideRefs is the ref each repository is cloned at beside a check, by the directory it is found under:
// a core repository at the commit the mesh runs of the module built from it (`running`) — but the // a core repository at the commit the mesh runs of the module built from it (`running`) — but the
// catalogue, whose checkout beside is what tests read its files from, at its main, what the next merge // catalogue, whose checkout beside is what tests read its files from, at its main, what the next merge
@@ -382,6 +599,10 @@ func checkedOf(result link.BuildResult) link.Checked {
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and the commits of the // What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and the commits of the
// repositories read beside it. // repositories read beside it.
c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took
c.Built = result.Check.Built
c.Release, c.JudgedBy, c.BesideAt = result.Check.Release, result.Check.JudgedBy, result.Check.BesideAt
c.Validator = result.Check.Validator
c.Siblings, c.SiblingsUnknown, c.JudgeFrom = result.Check.Siblings, result.Check.SiblingsUnknown, result.Check.JudgeFrom
if c.Gate == nil { if c.Gate == nil {
// A build seat from before the layers: its verdict is the gate's. // A build seat from before the layers: its verdict is the gate's.
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary} c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
+616
View File
@@ -0,0 +1,616 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"reflect"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **The move of config** (novox/hq ADR 0348 §9 and its consequences, design 57 K5, issue 664): every config value a
// settings layer holds is set in mesh-config, one of the moves of data a release runs once after its delivery, after
// the move of placement. A module's layer on a node becomes its own value on that node (`module.<node>.<module>.<key>`).
// A whole-mesh layer has no module value for the whole mesh (ADR 0348 §3): its value becomes mesh config for the
// predefined key it falls back to where that module alone falls back to the key (claude-code's managed settings
// become `agent-managed-settings`), and otherwise the module's own value on each node it is assigned to, laid under
// that node's layer as composition lays it. A key a layer still says by its file's name (nodered's `timeZone`) is
// set under its config key (`time-zone`, issue 751).
//
// **Every node composes byte for byte as before.** The move sets the values, reads every module's resolved values
// back through mesh-config and compares them with what the layers lay today; only when they agree does it empty the
// layers it moved (each kept whole in its history first) and turn on the switch (inventory.FinishConfigMove), after
// which the controller reads mesh-config where it read the layers. A module still in the settings form keeps its
// layers and is said so. Anything that cannot move as it composes refuses the whole move, by name, before anything
// is set. Asked again, it sets nothing that is set already.
// configLevel is where mesh-config keeps a value: mesh config, node config, or a module's own value on a node.
type configLevel struct {
Level string `json:"level"`
Node string `json:"node,omitempty"`
Module string `json:"module,omitempty"`
}
func (l configLevel) valueKey(key string) string {
switch l.Level {
case "node":
return "node." + l.Node + "." + key
case "module":
return "module." + l.Node + "." + l.Module + "." + key
}
return "mesh." + key
}
func (l configLevel) String() string {
switch l.Level {
case "node":
return "node config on " + l.Node
case "module":
return l.Module + "'s own value on " + l.Node
}
return "mesh config"
}
// configSet is one value the move sets in mesh-config, and the layers it comes from.
type configSet struct {
configLevel
Key string `json:"key"`
Value any `json:"value"`
Trusted bool `json:"trusted,omitempty"`
// From is the layers it comes from, as `<module> on <where>: <key as the layer says it>`.
From []string `json:"from"`
// Already is true when mesh-config holds this value at this level already: nothing is set.
Already bool `json:"already,omitempty"`
}
// configMoveAnswer is what the move of config says it did, or would do.
type configMoveAnswer struct {
// DryRun is true when nothing was set: asked without --confirm.
DryRun bool `json:"dry-run"`
Sets []configSet `json:"sets"`
// Moved is each layer whose values the move sets, emptied once they read back.
Moved []string `json:"moved"`
// Stays is each layer of a module still in the settings form: kept, and composed from as before.
Stays []string `json:"stays,omitempty"`
// Unread is each layer that composes nothing (its module is not assigned where it says): kept in the history,
// emptied, and set nowhere.
Unread []string `json:"unread,omitempty"`
// Problems is why the move refuses, each by name; nothing is set while there is one.
Problems []string `json:"problems,omitempty"`
// Switched is true once the controller reads mesh-config where it read the layers.
Switched bool `json:"switched"`
}
// configMovePlan is the move worked out from the layers, before anything is asked of mesh-config.
type configMovePlan struct {
answer configMoveAnswer
// moved is the layers the move empties once the values read back.
moved []inventory.LayerToMove
// expected is, for each module the move reaches, keyed `<node>.<module>`, every value composition lays over
// its files today: what mesh-config must resolve for it from mesh config, node config or its own value.
expected map[string]map[string]any
// modules is each module in the config form, by the nodes it is assigned to: whose resolved values are read back.
modules map[string][]string
}
func whereOf(node string) string {
if node == "" {
return "the whole mesh"
}
return node
}
// planConfigMove works out the move from every layer that holds a value, the catalogue, and where each module is
// assigned.
func planConfigMove(layers []inventory.LayerToMove, shelf map[string]catalogue.Manifest,
on map[string][]string) configMovePlan {
p := configMovePlan{answer: configMoveAnswer{Sets: []configSet{}, Moved: []string{}},
expected: map[string]map[string]any{}, modules: map[string][]string{}}
problem := func(format string, a ...any) {
p.answer.Problems = append(p.answer.Problems, fmt.Sprintf(format, a...))
}
for name, m := range shelf {
if m.DeclaresConfig() {
p.modules[name] = on[name]
}
}
byModule := map[string]*layersOfModule{}
var names []string
for _, l := range layers {
said := fmt.Sprintf("%s on %s", l.Module, whereOf(l.Node))
m, known := shelf[l.Module]
if !known || !m.DeclaresConfig() {
p.answer.Stays = append(p.answer.Stays, said+" (its module is in the settings form)")
continue
}
if _, placement := catalogue.SplitPlacement(l.Values); len(placement) > 0 {
problem("%s still holds placement (%s): run move-placement first", said, strings.Join(movedKeys(placement), ", "))
continue
}
assigned := on[l.Module]
if (l.Node == "" && len(assigned) == 0) || (l.Node != "" && !containsString(assigned, l.Node)) {
p.answer.Unread = append(p.answer.Unread, said+" (the module is not assigned there, so it composes nothing)")
p.moved = append(p.moved, l)
continue
}
// The layer's keys as config keys: a key the file names otherwise is set under its config key.
as := catalogue.TakenAs(m)
values := map[string]any{}
from := map[string]string{}
for _, k := range movedKeys(l.Values) {
key := k
if _, declared := m.Settings[k]; !declared {
if renamed, ok := as[k]; ok {
key = renamed
} else {
problem("%s says %q, which %s does not declare: declare it, or clear it from the layer", said, k, l.Module)
continue
}
}
if _, both := values[key]; both {
// The layer says the key and the file's name for it: the config key wins, as composition has it (issue 751).
if key != k {
continue
}
}
values[key] = l.Values[k]
from[key] = said + ": " + k
}
om := byModule[l.Module]
if om == nil {
om = &layersOfModule{nodes: map[string]map[string]any{}, saidAt: map[string]map[string]string{}}
byModule[l.Module] = om
names = append(names, l.Module)
}
om.saidAt[l.Node] = from
if l.Node == "" {
om.mesh = values
} else {
om.nodes[l.Node] = values
}
p.moved = append(p.moved, l)
p.answer.Moved = append(p.answer.Moved, said)
}
sort.Strings(names)
// Which predefined key each module falls back to, so a whole-mesh value goes to mesh config only where no other
// module's value moves with it.
fallingBack := map[string][]string{}
for name, m := range shelf {
for _, d := range m.Settings {
if d.Predefined != "" && !containsString(fallingBack[d.Predefined], name) {
fallingBack[d.Predefined] = append(fallingBack[d.Predefined], name)
}
}
}
for _, name := range names {
m, om := shelf[name], byModule[name]
asked := catalogue.AskedKeys(m)
keys := map[string]bool{}
for k := range om.mesh {
keys[k] = true
}
for _, values := range om.nodes {
for k := range values {
keys[k] = true
}
}
for _, key := range movedKeys(keys) {
d := m.Settings[key]
meshValue, meshSays := om.mesh[key]
alone := d.Predefined != "" && len(fallingBack[d.Predefined]) == 1
if meshSays && alone {
p.set(configLevel{Level: "mesh"}, d.Predefined, meshValue, d, om.said("", key))
}
for _, node := range on[name] {
nodeValue, nodeSays := om.nodes[node][key]
value := meshValue
switch {
case nodeSays && meshSays:
_, under := meshValue.(map[string]any)
_, over := nodeValue.(map[string]any)
if under && over && asked[key] {
problem("%s on %s lays an object for %q over the whole mesh's, and a placeholder takes it whole "+
"where its file merges the two: say the value on %s whole, then move", name, node, key, node)
continue
}
value = catalogue.LaidOver(meshValue, nodeValue)
case nodeSays:
value = nodeValue
case !meshSays:
continue
}
p.expect(node, name, key, value)
if nodeSays || !alone {
saidBy := om.said(node, key)
if meshSays {
saidBy = append(om.said("", key), saidBy...)
}
p.set(configLevel{Level: "module", Node: node, Module: name}, key, value, d, saidBy)
}
}
}
}
sort.Strings(p.answer.Problems)
return p
}
// layersOfModule is one module's layers as the move reads them: its whole-mesh layer's config values, each node's,
// and where each key was said, by node ("" for the whole mesh).
type layersOfModule struct {
mesh map[string]any
nodes map[string]map[string]any
saidAt map[string]map[string]string
}
func (om *layersOfModule) said(node, key string) []string {
if s, ok := om.saidAt[node][key]; ok {
return []string{s}
}
return nil
}
// set is one value the move sets, judged against its declaration as mesh-config will judge it.
func (p *configMovePlan) set(level configLevel, key string, value any, d catalogue.SettingDeclaration, from []string) {
p.answer.Sets = append(p.answer.Sets, configSet{configLevel: level, Key: key, Value: value, Trusted: d.Trusted,
From: from})
for _, wrong := range catalogue.ValueProblems(d, value) {
p.answer.Problems = append(p.answer.Problems, fmt.Sprintf("%s at %s: %s", key, level, wrong))
}
}
func (p *configMovePlan) expect(node, module, key string, value any) {
k := node + "." + module
if p.expected[k] == nil {
p.expected[k] = map[string]any{}
}
p.expected[k][key] = value
}
func movedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// configService is mesh-config as the move asks it: what it holds at each level, what it resolved, and a value set
// as the controller carries it from its terminal.
type configService interface {
// Values is every value set, by its key in mesh-config_values (`mesh.<key>`, `node.<node>.<key>`,
// `module.<node>.<module>.<key>`).
Values(ctx context.Context) (map[string]json.RawMessage, error)
// Resolved is every resolved value.
Resolved(ctx context.Context) ([]resolvedEntry, error)
// Set sets one value at its level, by whom and why.
Set(ctx context.Context, level configLevel, key string, value any, by, why string) error
}
// runConfigMove is the move against a store and mesh-config: a dry run says what it would set.
func runConfigMove(ctx context.Context, inv *inventory.Inventory, config configService, apply bool, by,
why string) (configMoveAnswer, error) {
layers, holding, err := inv.LayersToMove(ctx)
if err != nil {
return configMoveAnswer{}, err
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return configMoveAnswer{}, err
}
shelf := map[string]catalogue.Manifest{}
on := map[string][]string{}
for _, e := range entries {
shelf[e.Manifest.Module] = e.Manifest
on[e.Manifest.Module] = e.On
}
p := planConfigMove(layers, shelf, on)
if len(holding) > 0 && len(p.answer.Problems) == 0 {
p.answer.Problems = append(p.answer.Problems, "a layer still holds placement ("+strings.Join(holding, "; ")+
"): run move-placement first")
}
moved, err := inv.ConfigMoved(ctx)
if err != nil {
return configMoveAnswer{}, err
}
p.answer.Switched = moved.Moved
if config == nil {
return p.answer, errors.New("mesh-config cannot be asked, so nothing was moved")
}
held, err := config.Values(ctx)
if err != nil {
return p.answer, fmt.Errorf("what mesh-config holds cannot be read, so nothing was moved: %w", err)
}
for i := range p.answer.Sets {
s := &p.answer.Sets[i]
if was, ok := held[s.valueKey(s.Key)]; ok && sameJSON(was, s.Value) {
s.Already = true
}
}
p.answer.DryRun = !apply
if !apply || len(p.answer.Problems) > 0 {
if apply {
return p.answer, fmt.Errorf("the move of config refuses, and nothing was set: %s",
strings.Join(p.answer.Problems, "; "))
}
return p.answer, nil
}
if len(p.moved) == 0 && moved.Moved {
return p.answer, nil
}
for _, s := range p.answer.Sets {
if s.Already {
continue
}
said := "moved from the settings layer of " + strings.Join(s.From, " and ") + " (issue 664): " + why
if err := config.Set(ctx, s.configLevel, s.Key, s.Value, by, said); err != nil {
return p.answer, fmt.Errorf("%s at %s was not set, and no layer was emptied; ask the move again: %w",
s.Key, s.configLevel, err)
}
}
// Read back through mesh-config before anything points at it (design 57): every module the move reaches, and
// every other in the config form, must resolve exactly what its layers lay today.
if differs, err := readsBack(ctx, config, p); err != nil {
return p.answer, err
} else if len(differs) > 0 {
return p.answer, fmt.Errorf("mesh-config does not resolve what the layers lay today, so no layer was emptied and "+
"the controller still composes from the layers: %s", strings.Join(differs, "; "))
}
if err := inv.FinishConfigMove(ctx, p.moved, by, why); err != nil {
return p.answer, err
}
p.answer.Switched = true
return p.answer, nil
}
// readsBack compares what mesh-config resolved with what the layers lay, for every module in the config form on
// every node it is assigned to, and says each difference. A value the layers do not lay and mesh-config resolves from
// a level would be laid once the switch is on: a difference too.
func readsBack(ctx context.Context, config configService, p configMovePlan) ([]string, error) {
var resolved []resolvedEntry
var err error
// mesh-config writes the resolved values with each set; a pass that missed one is put right by its next
// resolution, so a difference is read again for a little while before it is said.
for wait := 0; ; wait++ {
if resolved, err = config.Resolved(ctx); err != nil {
return nil, fmt.Errorf("mesh-config's resolved values cannot be read back, so no layer was emptied: %w", err)
}
differs, err := differences(resolved, p)
if err != nil || len(differs) == 0 || wait >= readBackTries {
return differs, err
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(readBackEvery):
}
}
}
// How often, and how many times, the move reads mesh-config's resolved values back before it says they differ.
var (
readBackEvery = 2 * time.Second
readBackTries = 15
)
func differences(resolved []resolvedEntry, p configMovePlan) ([]string, error) {
var differs []string
for _, module := range movedKeys(p.modules) {
for _, node := range p.modules[module] {
laid, err := configLaid(resolved, node, module)
if err != nil {
return nil, err
}
want := p.expected[node+"."+module]
if want == nil {
want = map[string]any{}
}
for _, k := range movedKeys(mergedKeys(laid, want)) {
got, has := laid[k]
w, wants := want[k]
switch {
case !wants:
differs = append(differs, fmt.Sprintf("%s on %s resolves %s from a level no layer says; clear it "+
"there, or set it in a layer first", module, node, k))
case !has:
differs = append(differs, fmt.Sprintf("%s on %s does not resolve %s from the value moved", module, node, k))
case !reflect.DeepEqual(normalJSON(got), normalJSON(w)):
differs = append(differs, fmt.Sprintf("%s on %s resolves %s as %s, and its layers lay %s", module,
node, k, compactJSON(got), compactJSON(w)))
}
}
}
}
return differs, nil
}
func mergedKeys(a, b map[string]any) map[string]bool {
out := map[string]bool{}
for k := range a {
out[k] = true
}
for k := range b {
out[k] = true
}
return out
}
func normalJSON(v any) any {
raw, _ := json.Marshal(v)
var out any
_ = json.Unmarshal(raw, &out)
return out
}
func compactJSON(v any) string {
raw, _ := json.Marshal(v)
return string(raw)
}
func sameJSON(raw json.RawMessage, v any) bool {
var held any
if err := json.Unmarshal(raw, &held); err != nil {
return false
}
return reflect.DeepEqual(held, normalJSON(v))
}
// busConfig is mesh-config on this controller's bus: its buckets read directly, a value set through its verb in
// the controller's own name, so mesh-config takes a trusted value as given at the controller's terminal (ADR 0348 §8).
type busConfig struct{}
func (busConfig) Values(ctx context.Context) (map[string]json.RawMessage, error) {
out := map[string]json.RawMessage{}
err := onTheBus(func(conn *nats.Conn) error {
js, err := jetstream.New(conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
kv, err := js.KeyValue(ctx, broker.BucketName("mesh-config", "values"))
if err != nil {
return err
}
w, err := kv.WatchAll(ctx, jetstream.IgnoreDeletes())
if err != nil {
return err
}
defer func() { _ = w.Stop() }()
for e := range w.Updates() {
if e == nil {
return nil
}
var v struct {
Value json.RawMessage `json:"value"`
}
if err := json.Unmarshal(e.Value(), &v); err != nil {
return fmt.Errorf("mesh-config holds %q, which is not a value: %w", e.Key(), err)
}
out[e.Key()] = v.Value
}
return errors.New("mesh-config's values stopped answering")
})
return out, err
}
func (busConfig) Resolved(ctx context.Context) ([]resolvedEntry, error) { return readResolved(ctx) }
func (busConfig) Set(ctx context.Context, level configLevel, key string, value any, by, why string) error {
args := map[string]any{"action": "set", "key": key, "value": value, "why": why, "by": by}
if level.Node != "" {
args["node"] = level.Node
}
if level.Module != "" {
args["module"] = level.Module
}
return onTheBus(func(conn *nats.Conn) error {
answer, err := link.AskMeshSeatToolAs(ctx, conn, broker.ControllerName, "mesh-config", "config", args, 30*time.Second)
if err != nil {
return err
}
if answer.Error != "" {
return errors.New(answer.Error)
}
return nil
})
}
// moveConfigCommand is `move-config`: the move of config (issue 664).
func moveConfigCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("move-config", flag.ContinueOnError)
asJSON := set.Bool("json", false, "answer as JSON")
confirm := set.Bool("confirm", false, "set the values and switch, rather than only say what would move")
f := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 0 {
return errors.New("move-config [--json] [--confirm --why <text>]")
}
if !*confirm && f.given() {
return errors.New("move-config takes --why only with --confirm: without it nothing is moved. Nothing was done")
}
if *confirm {
if err := f.require("move-config"); err != nil {
return err
}
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
if *confirm {
f.record(ctx, "move-config", nil)
}
why := ""
if *confirm {
why = strings.TrimSpace(*f.why)
}
answer, err := runConfigMove(ctx, open.inventory, busConfig{}, *confirm, "the move of config, run by "+link.Caller(), why)
if *asJSON {
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
if encErr := encoder.Encode(answer); encErr != nil {
return encErr
}
return err
}
fmt.Print(describeConfigMove(answer))
return err
}
// describeConfigMove says the answer as a person reads it.
func describeConfigMove(a configMoveAnswer) string {
var b strings.Builder
verb := "set"
if a.DryRun {
verb = "would set"
}
for _, s := range a.Sets {
state := verb
if s.Already {
state = "holds already"
}
fmt.Fprintf(&b, "%s %s = %s at %s (from %s)\n", state, s.Key, compactJSON(s.Value), s.configLevel,
strings.Join(s.From, " and "))
}
for _, s := range a.Unread {
fmt.Fprintf(&b, "emptied, set nowhere: %s\n", s)
}
for _, s := range a.Stays {
fmt.Fprintf(&b, "stays: %s\n", s)
}
for _, s := range a.Problems {
fmt.Fprintf(&b, "refused: %s\n", s)
}
switch {
case len(a.Problems) > 0:
b.WriteString("nothing was set: the move refuses until each is put right\n")
case a.DryRun:
fmt.Fprintf(&b, "%d layer(s), %d value(s); nothing was set — with --confirm --why <text> they are\n",
len(a.Moved), len(a.Sets))
case a.Switched:
fmt.Fprintf(&b, "%d layer(s) moved into mesh-config, each kept whole in its history first; the controller "+
"reads mesh-config where it read them\n", len(a.Moved))
}
return b.String()
}
+785
View File
@@ -0,0 +1,785 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/overlay"
)
// configuredMesh is every live settings layer (issue 664) — read with the settings verb on 2026-10-15, on the four
// machines as anchor, laptop, workstation and desktop, the whole mesh's with no node; personal values stood in for,
// and the placement words left out, which move-placement moves first — with each module in the config form shaped
// as its definition on the feature branch: its config, its files (templates inlined) and directories, placeholders
// of kinds this test's mesh does not hold stood in for. A module still in the settings form (the media catalogue's,
// and kernel) stands in as a mergeable file that takes every key. `on` is where each is assigned, as live.
type configuredMesh struct {
Modules []json.RawMessage `json:"modules"`
Layers []configuredLayer `json:"layers"`
On map[string][]string `json:"on"`
// Predefined is the catalogue's list of predefined keys, which a module falls back to.
Predefined json.RawMessage `json:"predefined"`
}
type configuredLayer struct {
Node string `json:"node"`
Module string `json:"module"`
Values map[string]any `json:"values"`
}
var fourMachines = []string{"anchor", "laptop", "workstation", "desktop"}
// aConfiguredMesh is aMesh with two machines more, every module of the configured mesh registered and assigned, and
// its layers held as the store holds them once move-placement has run.
func aConfiguredMesh(t *testing.T) (*stores, configuredMesh, map[string]map[string]any) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for i, name := range []string{"workstation", "desktop"} {
record, err := inv.AddNode(ctx, name)
if err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(ctx, name, name+".example:51820", "here", false, fmt.Sprintf("10.77.0.%d", i+3)); err != nil {
t.Fatal(err)
}
if err := inv.RecordProfile(ctx, record.ID, map[string]any{"capabilities": []any{
map[string]any{"name": "container-runtime", "present": true},
map[string]any{"name": "systemd", "present": true}}}); err != nil {
t.Fatal(err)
}
if err := inv.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, name, overlay.Name); err != nil {
t.Fatal(err)
}
}
raw, err := os.ReadFile(filepath.Join("testdata", "config-moved", "mesh.json"))
if err != nil {
t.Fatal(err)
}
var w configuredMesh
if err := json.Unmarshal(raw, &w); err != nil {
t.Fatal(err)
}
list, err := catalogue.ParsePredefined(w.Predefined)
if err != nil {
t.Fatal(err)
}
catalogue.UsePredefined(list)
served := map[string]map[string]any{}
for _, text := range w.Modules {
var shape map[string]json.RawMessage
if err := json.Unmarshal(text, &shape); err != nil {
t.Fatal(err)
}
var name string
_ = json.Unmarshal(shape["module"], &name)
if s, ok := shape["serves-checked"]; ok {
var serves map[string]any
if err := json.Unmarshal(s, &serves); err != nil {
t.Fatal(err)
}
served[name] = serves
delete(shape, "serves-checked")
}
text, err := json.Marshal(shape)
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(text)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
register(t, open, m)
}
for _, l := range w.Layers {
values, err := json.Marshal(l.Values)
if err != nil {
t.Fatal(err)
}
if l.Node == "" {
inv.ExecForTest(t, `insert into settings (node, module, values) values (null, $1, $2)`, l.Module, string(values))
continue
}
inv.ExecForTest(t, `insert into settings (node, module, values) select id, $2, $3 from node where name = $1`,
l.Node, l.Module, string(values))
}
for _, module := range movedKeys(w.On) {
for _, node := range w.On[module] {
if _, err := assign(ctx, open, node, module); err != nil {
t.Fatalf("assign %s %s: %v", node, module, err)
}
}
}
return open, w, served
}
// standInMeshConfig is mesh-config as this test holds it: each value at its level, and resolved from the catalogue
// the store holds as mesh-config resolves (novox/hq ADR 0348 §2): the module's own value on the node, else node
// config then mesh config for the predefined key it falls back to, else its default; a required value nothing sets is
// unset.
type standInMeshConfig struct {
open *stores
values map[string]any
sets []string
// asked is each call of its verb: the action and the level's key.
asked []string
}
func (c *standInMeshConfig) Values(context.Context) (map[string]json.RawMessage, error) {
out := map[string]json.RawMessage{}
for k, v := range c.values {
raw, err := json.Marshal(v)
if err != nil {
return nil, err
}
out[k] = raw
}
return out, nil
}
func (c *standInMeshConfig) Set(_ context.Context, level configLevel, key string, value any, by, why string) error {
if by == "" || why == "" {
return fmt.Errorf("%s at %s is set without who or why", key, level)
}
c.values[level.valueKey(key)] = normalJSON(value)
c.sets = append(c.sets, level.valueKey(key))
return nil
}
func (c *standInMeshConfig) Resolved(ctx context.Context) ([]resolvedEntry, error) {
entries, err := c.open.inventory.Catalogued(ctx)
if err != nil {
return nil, err
}
var out []resolvedEntry
for _, e := range entries {
m := e.Manifest
if !m.DeclaresConfig() {
continue
}
for _, node := range e.On {
for _, key := range movedKeys(m.Settings) {
d := m.Settings[key]
r := resolvedEntry{Node: node, Module: m.Module, Key: key, Source: "default"}
value, has := c.values["module."+node+"."+m.Module+"."+key]
switch {
case has:
r.Source = "module"
case d.Predefined != "" && c.values["node."+node+"."+d.Predefined] != nil:
value, r.Source = c.values["node."+node+"."+d.Predefined], "node"
case d.Predefined != "" && c.values["mesh."+d.Predefined] != nil:
value, r.Source = c.values["mesh."+d.Predefined], "mesh"
case d.Required:
r.Source = "unset"
default:
value = d.Default
}
if r.Source != "unset" {
if r.Value, err = json.Marshal(value); err != nil {
return nil, err
}
}
out = append(out, r)
}
}
}
return out, nil
}
// laid is the stand-in as composition reads it once the switch is on (inventory.ConfigValues).
func (c *standInMeshConfig) laid(ctx context.Context, node, module string) (map[string]any, error) {
resolved, err := c.Resolved(ctx)
if err != nil {
return nil, err
}
return configLaid(resolved, node, module)
}
// snapshotFacts is the facts snapshot as gatherFacts gives it.
type snapshotFacts = snapshot.Facts
// toldEveryMachine is what each of the four machines is told — every resource and the filter's rules — and what
// each module that serves something serves, settled from its layers, as JSON a person can diff.
func toldEveryMachine(t *testing.T, open *stores, served map[string]map[string]any, on map[string][]string) map[string]string {
t.Helper()
ctx := t.Context()
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
told := map[string]string{}
for _, node := range fourMachines {
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Allocating)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
composed, err := plan.Compose(with)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
rules, err := plan.Rules(with)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
settles := map[string]any{}
for module, serves := range served {
if !containsString(on[module], node) {
continue
}
layers, err := open.inventory.SettingsFor(ctx, node, module)
if err != nil {
t.Fatal(err)
}
for name, s := range serves {
fact, _ := s.(map[string]any)
settled, err := catalogue.Settle(fact, catalogue.WithDefaults(shelf[module], layers))
if err != nil {
settles[module+" serves "+name] = "error: " + err.Error()
continue
}
settles[module+" serves "+name] = settled
}
}
got, err := json.MarshalIndent(map[string]any{"resources": composed.Resources, "left-out": composed.LeftOut,
"rules": rules, "served": settles}, "", " ")
if err != nil {
t.Fatal(err)
}
told[node] = string(got)
}
return told
}
// effectiveEverywhere is each module's every value on each machine it is assigned to, as `settings show` and
// `settings preferences` say them (catalogue.Effective): the value, never where it came from, which the move
// changes by design; a key a layer says by its file's name is read as its config key.
func effectiveEverywhere(t *testing.T, open *stores, w configuredMesh) map[string]any {
t.Helper()
ctx := t.Context()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
out := map[string]any{}
for _, module := range movedKeys(w.On) {
if !shelf[module].DeclaresConfig() {
continue
}
for _, node := range w.On[module] {
layers, err := open.inventory.SettingsFor(ctx, node, module)
if err != nil {
t.Fatal(err)
}
// A layer's old spelling of a key (nodered's timeZone) is listed beside the key it is the file's name
// for, and reaches the file as that key does (issue 751): read as that key, which the move sets.
as := catalogue.TakenAs(shelf[module])
for _, v := range catalogue.Effective(shelf[module], layers) {
if key, old := as[v.Key]; old {
out[node+"."+module+"."+key] = normalJSON(v.Value)
continue
}
if _, said := out[node+"."+module+"."+v.Key]; said && v.FromDefault {
continue
}
out[node+"."+module+"."+v.Key] = normalJSON(v.Value)
}
}
}
return out
}
func setAt(a configMoveAnswer, key string) []configSet {
var out []configSet
for _, s := range a.Sets {
if s.valueKey(s.Key) == key {
out = append(out, s)
}
}
return out
}
// **The settings layers move into mesh-config and no machine is told anything new** (novox/hq ADR 0348 §9, design
// 57 K5, issue 664). Every live layer is seeded as the store holds it once move-placement has run, on modules shaped
// as their definitions, and every machine composed. The move, asked without confirming, says each value it would set
// and where from, and sets nothing; confirmed, it sets them in mesh-config, reads them back, empties the layers it
// moved and switches. Each machine is then told byte for byte what it was told before, every module serves the same,
// and `settings` lists the same values `config` resolves. Asked again, it sets nothing.
func TestEveryMachineIsToldTheSameAfterTheMoveOfConfig(t *testing.T) {
open, w, served := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
before := toldEveryMachine(t, open, served, w.On)
listedBefore := effectiveEverywhere(t, open, w)
factsBefore, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
asked, err := runConfigMove(ctx, inv, config, false, "the move of config, run by a test", "")
if err != nil || !asked.DryRun || asked.Switched || len(asked.Problems) > 0 || len(config.sets) > 0 {
t.Fatalf("the move, not confirmed, says %+v (%v) and set %v", asked, err, config.sets)
}
// nodered's live layer says the file's name for its time zone: set under the config key (issue 751).
if s := setAt(asked, "module.anchor.nodered.time-zone"); len(s) != 1 || s[0].Value != "Europe/Brussels" ||
!strings.HasSuffix(s[0].From[0], ": timeZone") {
t.Errorf("nodered's time zone moves as %+v", s)
}
for _, s := range asked.Sets {
if s.Key == "timeZone" {
t.Errorf("a key is set under the file's name: %+v", s)
}
}
// claude-code's whole-mesh managed settings become mesh config for agent-managed-settings (ADR 0348's
// consequences), and its whole-mesh layer sets no module value.
if s := setAt(asked, "mesh.agent-managed-settings"); len(s) != 1 || s[0].From[0] != "claude-code on the whole mesh: managed_settings" {
t.Errorf("claude-code's managed settings move as %+v", s)
}
for _, node := range fourMachines {
if s := setAt(asked, "module."+node+".claude-code.managed_settings"); len(s) > 0 {
t.Errorf("claude-code's managed settings are also set on %s: %+v", node, s)
}
}
// A whole-mesh layer of a value no predefined key holds goes to each machine it is assigned to, under that
// machine's layer: power's on the workstation is the workstation's, its own elsewhere.
if s := setAt(asked, "module.workstation.power.handle-power-key"); len(s) != 1 || s[0].Value != "suspend" {
t.Errorf("power's key on the workstation moves as %+v", s)
}
if s := setAt(asked, "module.anchor.power.handle-power-key"); len(s) != 1 || s[0].Value != "poweroff" {
t.Errorf("power's key on the anchor moves as %+v", s)
}
for _, s := range asked.Sets {
if s.Level == "node" {
t.Errorf("a machine's layer became node config: %+v", s)
}
}
var stays []string
for _, s := range asked.Stays {
stays = append(stays, strings.Fields(s)[0])
}
sort.Strings(stays)
if want := []string{"bazarr", "bazarr", "bookshelf", "jackett", "kernel", "kometa", "lidarr", "nzbget", "ombi",
"plex", "qbittorrent", "radarr", "sonarr", "tautulli"}; len(stays) < 10 || !containsString(stays, "kernel") {
t.Errorf("the layers that stay are %v, want those of %v", stays, want)
}
if after := toldEveryMachine(t, open, served, w.On); !reflect.DeepEqual(after, before) {
t.Fatal("a dry run changed what a machine is told")
}
moved, err := runConfigMove(ctx, inv, config, true, "the move of config, run by a test", "release 57 is delivered")
if err != nil || moved.DryRun || !moved.Switched {
t.Fatalf("the move says %+v (%v)", moved, err)
}
if len(config.sets) != len(asked.Sets) {
t.Fatalf("the move set %d values, and said it would set %d", len(config.sets), len(asked.Sets))
}
if held := inv.QueryForTest(t, `select module from settings s where values <> '{}'
and module in (select name from module where manifest ? 'config')`); len(held) != 0 {
t.Fatalf("a layer of a module in the config form still holds values: %v", held)
}
after := toldEveryMachine(t, open, served, w.On)
for _, node := range fourMachines {
if after[node] != before[node] {
t.Errorf("%s is told something else after the move:\n%s\nwas:\n%s", node, after[node], before[node])
}
}
if listedAfter := effectiveEverywhere(t, open, w); !reflect.DeepEqual(listedAfter, listedBefore) {
t.Errorf("settings lists other values after the move:\n%v\nwas:\n%v", listedAfter, listedBefore)
}
// And config lists what settings listed: each module's resolved value on each machine is the value settings
// said before the move.
resolved, err := config.Resolved(ctx)
if err != nil {
t.Fatal(err)
}
for _, r := range resolved {
said, listed := listedBefore[r.Node+"."+r.Module+"."+r.Key]
if r.Source == "unset" {
if listed && said != nil {
t.Errorf("%s of %s on %s is unset in config, and settings said %v", r.Key, r.Module, r.Node, said)
}
continue
}
var v any
_ = json.Unmarshal(r.Value, &v)
if !listed || !reflect.DeepEqual(v, said) {
t.Errorf("config resolves %s of %s on %s as %v, and settings said %v", r.Key, r.Module, r.Node, v, said)
}
}
// The facts snapshot carries mesh-config's values for the merge gate, as each machine's layer of each module, and
// each machine's declaration composes to the digest it did before the move.
factsAfter, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
// Machines are named in a snapshot by stand-ins of its own; each is found by what it composes.
digests := func(f snapshotFacts) ([]string, string) {
var out []string
zone := ""
for _, m := range f.Machines {
out = append(out, m.Declaration.Digest)
for _, l := range m.Settings {
if l.Module == "nodered" {
zone = compactJSON(l.Values["time-zone"])
}
}
}
sort.Strings(out)
return out, zone
}
was, _ := digests(factsBefore)
now, zone := digests(factsAfter)
if zone != `"Europe/Brussels"` {
t.Errorf("the facts snapshot does not carry nodered's time zone from mesh-config: %s", zone)
}
if len(now) != len(fourMachines) || now[0] == "" || !reflect.DeepEqual(now, was) {
t.Errorf("the declarations in the facts snapshot are %v after the move, %v before", now, was)
}
for _, s := range factsAfter.Settings {
if _, isConfig := map[string]bool{"betterbird": true, "claude-code": true, "power": true, "xdg": true}[s.Module]; isConfig &&
len(s.Values) > 0 {
t.Errorf("the facts snapshot still carries the whole mesh's layer of %s: %v", s.Module, s.Values)
}
}
config.sets = nil
again, err := runConfigMove(ctx, inv, config, true, "the move of config, run by a test", "again")
if err != nil || len(config.sets) != 0 || len(again.Moved) != 0 {
t.Fatalf("asked again, the move set %v and says %+v (%v)", config.sets, again, err)
}
if after := toldEveryMachine(t, open, served, w.On); !reflect.DeepEqual(after, before) {
t.Fatal("the move asked again changed what a machine is told")
}
}
// **The move refuses what it cannot move as it composes, and sets nothing** (issue 664): a layer that still holds
// placement (move-placement has not run), a key its module does not declare, and a value its declaration refuses —
// each said by name; and a value mesh-config resolves from a level no layer says (it would be laid once the switch
// is on) leaves every layer and the switch as they were.
func TestTheMoveOfConfigRefusesWhatWouldChangeAMachine(t *testing.T) {
open, _, _ := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
inv.ExecForTest(t, `update settings set values = values || '{"endpoints": {"web": {"reach": "public"}}}'
where module = 'nodered'`)
inv.ExecForTest(t, `update settings set values = values || '{"colour": "blue"}' where module = 'unifi'`)
inv.ExecForTest(t, `update settings set values = '{"font-size": 900}' where module = 'dunst'`)
answer, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered")
if err == nil || answer.Switched || len(config.sets) > 0 {
t.Fatalf("the move went ahead: %+v (%v), set %v", answer, err, config.sets)
}
for _, want := range []string{"nodered on anchor still holds placement (endpoints): run move-placement first",
`unifi on anchor says "colour", which unifi does not declare`, "font-size at dunst's own value on workstation"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
inv.ExecForTest(t, `update settings set values = values - 'endpoints' where module = 'nodered'`)
inv.ExecForTest(t, `update settings set values = values - 'colour' where module = 'unifi'`)
inv.ExecForTest(t, `update settings set values = '{"font-size": 13, "width": 500}' where module = 'dunst'`)
// Set in mesh-config before the move, on a module whose layers say nothing of it: the switch would lay it.
config.values["module.anchor.grafana."+movedKeys(func() map[string]catalogue.SettingDeclaration {
shelf, _ := inv.Catalogue(ctx)
return shelf["grafana"].Settings
}())[0]] = "set before the move"
answer, err = runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered")
if err == nil || !strings.Contains(err.Error(), "grafana on anchor resolves") || answer.Switched {
t.Fatalf("a value no layer says was laid by the switch: %+v (%v)", answer, err)
}
if moved, _ := inv.ConfigMoved(ctx); moved.Moved {
t.Fatal("the switch is on after a refused move")
}
if held := inv.QueryForTest(t, `select count(*) from settings where values <> '{}'`); held[0][0] == "0" {
t.Fatal("a refused move emptied the layers")
}
}
// The verb's command line: a dry run unless confirmed, and a confirmed move says why, recorded as a person's step of
// the release, never a repair (issue 664).
func TestTheMoveOfConfigIsADryRunUnlessConfirmed(t *testing.T) {
for _, c := range []struct {
args map[string]any
want string
}{
{map[string]any{}, "move-config --json"},
{map[string]any{"confirm": "true", "why": "release 57 is delivered"}, "move-config --json --confirm --why release 57 is delivered"},
} {
argv, err := argvFor("move-config", c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%v: %q %v, want %q", c.args, argv, err, c.want)
}
}
for _, args := range []map[string]any{{"confirm": "true"}, {"why": "tidy"}, {"node": "anchor"}} {
if argv, err := argvFor("move-config", args); err == nil {
t.Errorf("%v was composed as %q", args, argv)
}
}
if repairingCommand([]string{"move-config", "--json", "--confirm", "--why", "x"}) != "move-config" {
t.Error("a confirmed move through the generic verb would go unrecorded")
}
if !personsDecision(link.HandAct{Verb: "move-config"}) {
t.Error("a move of the release would count toward a healer the mesh lacks")
}
}
// ask is mesh-config's verb as the stand-in answers it: list, set, clear and propose.
func (c *standInMeshConfig) ask(ctx context.Context, args map[string]any) (json.RawMessage, error) {
str := func(k string) string { s, _ := args[k].(string); return s }
level := configLevel{Level: "mesh"}
switch {
case str("module") != "" && str("node") != "":
level = configLevel{Level: "module", Node: str("node"), Module: str("module")}
case str("node") != "":
level = configLevel{Level: "node", Node: str("node")}
}
c.asked = append(c.asked, str("action")+" "+level.valueKey(str("key")))
switch str("action") {
case "list":
resolved, err := c.Resolved(ctx)
if err != nil {
return nil, err
}
out := map[string]any{"resolved": []resolvedEntry{}, "set": []any{}}
var listed []resolvedEntry
for _, r := range resolved {
if r.Module == str("module") && (str("node") == "" || r.Node == str("node")) {
listed = append(listed, r)
}
}
out["resolved"] = listed
var set []any
for _, k := range movedKeys(c.values) {
parts := strings.SplitN(k, ".", 4)
if len(parts) == 4 && parts[0] == "module" && parts[2] == str("module") {
set = append(set, map[string]any{"level": configLevel{Level: "module", Node: parts[1], Module: parts[2]},
"key": parts[3], "value": c.values[k], "by": "a test"})
}
}
out["set"] = set
return json.Marshal(out)
case "set":
if str("why") == "" || str("by") == "" {
return nil, fmt.Errorf("a set without why or by: %v", args)
}
return nil, c.Set(ctx, level, str("key"), args["value"], str("by"), str("why"))
case "clear":
if _, had := c.values[level.valueKey(str("key"))]; !had {
return nil, fmt.Errorf("nothing is set for %s", str("key"))
}
delete(c.values, level.valueKey(str("key")))
return json.RawMessage(`{}`), nil
case "propose":
return json.RawMessage(`{"proposal":"p-1"}`), nil
}
return nil, fmt.Errorf("the stand-in does not %s", str("action"))
}
// **`settings` answers by forwarding to `config`** (novox/hq ADR 0348 §1, design 57 K5, issue 664): once the move
// has run, a module in the config form is shown from mesh-config, its values set and cleared there as its own on the
// machine (placement still beside the assignment), and proposed through it; a trusted value is refused through a
// verb, a module value for the whole mesh is refused, and a module still in the settings form keeps today's verb.
func TestTheSettingsVerbForwardsToConfigOnceTheValuesMoved(t *testing.T) {
open, w, served := aConfiguredMesh(t)
ctx := t.Context()
inv := open.inventory
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
savedRead, savedAsk := readConfigValues, configAsk
readConfigValues, configAsk = config.laid, config.ask
t.Cleanup(func() { readConfigValues, configAsk = savedRead, savedAsk })
if _, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered"); err != nil {
t.Fatal(err)
}
shown := stdoutOf(t, func() error { return settingsCommand(ctx, []string{"show", "nodered", "--node", "anchor"}) })
for _, want := range []string{`"time-zone": "Europe/Brussels"`, `time-zone = "Europe/Brussels" (module)`,
"mesh-config.config", `"mqtt": {`} {
if !strings.Contains(shown, want) {
t.Errorf("settings show does not say %q:\n%s", want, shown)
}
}
if strings.Contains(shown, "timeZone") {
t.Errorf("settings show still says the file's name:\n%s", shown)
}
config.asked = nil
said := stdoutOf(t, func() error {
return settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12, "width": 500}`, "--node", "workstation"})
})
if config.values["module.workstation.dunst.font-size"] != float64(12) || !strings.Contains(said, "~ font-size") ||
containsString(config.asked, "set module.workstation.dunst.width") {
t.Fatalf("settings set reached mesh-config as %v (%v):\n%s", config.values, config.asked, said)
}
told := toldEveryMachine(t, open, served, w.On)["workstation"]
if !strings.Contains(told, "12") {
t.Error("the workstation is not told the font size set through settings")
}
if err := settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12}`, "--node", "workstation"}); err == nil ||
!strings.Contains(err.Error(), "would no longer set width") {
t.Fatalf("a set dropping a value without --replace: %v", err)
}
if err := settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12}`}); err == nil ||
!strings.Contains(err.Error(), "no own value for the whole mesh") {
t.Fatalf("a module value for the whole mesh: %v", err)
}
t.Setenv(verbVar, "settings")
for _, c := range []struct{ module, node, values, key, held, why string }{
// A file root reads: refused as before the move (novox/hq issue 339).
{"hostname", "anchor", `{"hostname": "elsewhere"}`, "module.anchor.hostname.hostname", "Ace", "issue 339"},
{"messenger", "laptop", `{"desktop-machines": ["g14", "shanks"], "time-zone": "UTC"}`,
"module.laptop.messenger.time-zone", "Europe/Brussels", "issue 339"},
} {
err := settingsCommand(ctx, []string{"set", c.module, c.values, "--node", c.node})
if err == nil || !strings.Contains(err.Error(), c.why) || config.values[c.key] != c.held {
t.Fatalf("a trusted value of %s through a verb: %v, holds %v", c.module, err, config.values[c.key])
}
}
// And a trusted value no file makes the terminal's is refused as trusted, since mesh-config takes the controller's
// word for it (ADR 0348 §8).
if err := refuseTrustedThroughAVerb(ctx, inv, "messenger", "laptop", []string{"time-zone"}); err == nil ||
!strings.Contains(err.Error(), "ADR 0348 §8") {
t.Fatalf("a trusted value through a verb: %v", err)
}
if err := refuseTrustedThroughAVerb(ctx, inv, "dunst", "workstation", []string{"width"}); err != nil {
t.Fatalf("a value nobody trusts through a verb: %v", err)
}
stdoutOf(t, func() error {
return settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12, "width": 600}`, "--node", "workstation"})
})
if config.values["module.workstation.dunst.width"] != float64(600) {
t.Fatalf("a value nobody trusts was not set through a verb: %v", config.values["module.workstation.dunst.width"])
}
t.Setenv(verbVar, "")
stdoutOf(t, func() error { return settingsCommand(ctx, []string{"clear", "dunst", "--node", "workstation"}) })
for _, k := range []string{"font-size", "width"} {
if _, held := config.values["module.workstation.dunst."+k]; held {
t.Errorf("settings clear left dunst's %s on the workstation", k)
}
}
config.asked = nil
proposed := stdoutOf(t, func() error {
return settingsCommand(ctx, []string{"propose", "hostname", `{"hostname": "elsewhere"}`, "--node", "anchor"})
})
if !containsString(config.asked, "propose module.anchor.hostname.hostname") || !strings.Contains(proposed, "p-1") {
t.Fatalf("settings propose reached mesh-config as %v:\n%s", config.asked, proposed)
}
config.asked = nil
kept := stdoutOf(t, func() error { return settingsCommand(ctx, []string{"show", "kernel", "--node", "laptop"}) })
if !strings.Contains(kept, "transparent-huge-pages") || len(config.asked) > 0 {
t.Fatalf("a module in the settings form is shown as %q, asking mesh-config %v", kept, config.asked)
}
}
// **An approval for a module whose config has moved is refused, and writes nothing** (issue 830, novox/hq ADR 0348):
// a proposal raised while grafana read its layer (it had none, so the layer is as the operator was shown it) and
// approved after the move would put its values back into the layer, ReadsConfig would turn false, and the module
// would compose from emptied layers everywhere.
func TestAnApprovalRaisedBeforeTheMoveOfConfigIsRefusedAfterIt(t *testing.T) {
open, _, _ := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
key := movedKeys(func() map[string]catalogue.SettingDeclaration { return shelf["grafana"].Settings }())[0]
values := map[string]any{key: "raised before the move"}
p := settingsProposal{Module: "grafana", Node: "anchor", Values: values, From: "g14/claude-code", At: time.Now(),
Digest: layerDigest(values), BeforeDigest: layerDigest(nil)}
if _, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || has {
t.Fatalf("grafana has a layer on anchor before the test: %v %v", has, err)
}
if moved, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered"); err != nil || !moved.Switched {
t.Fatalf("the move says %+v (%v)", moved, err)
}
if reads, err := inv.ReadsConfig(ctx, "grafana"); err != nil || !reads {
t.Fatalf("grafana does not read config after the move: %v %v", reads, err)
}
_, err = setLayerIn(open)(ctx, p, "s30", "approved by the operator")
if err == nil || !strings.Contains(err.Error(), "moved to mesh-config since the operator was shown the change") ||
!strings.Contains(err.Error(), "propose it again") {
t.Fatalf("the stale approval: %v", err)
}
if layer, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || has && len(layer) > 0 {
t.Errorf("the refused approval wrote the layer: %v %v %v", layer, has, err)
}
if reads, err := inv.ReadsConfig(ctx, "grafana"); err != nil || !reads {
t.Errorf("the refused approval turned ReadsConfig off: %v %v", reads, err)
}
}
// **A proposal of placement only is set after the move, as before** (issue 830): forwardPropose sends it down the
// old path on purpose, so refusing it would loop the operator (refused, proposed again, refused).
func TestAPlacementOnlyApprovalRaisedBeforeTheMoveIsSetAfterIt(t *testing.T) {
open, _, _ := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
values := map[string]any{catalogue.PlacesSetting: map[string]any{"data": "/srv/grafana-data"}}
p := settingsProposal{Module: "grafana", Node: "anchor", Values: values, From: "g14/claude-code", At: time.Now(),
Digest: layerDigest(values), BeforeDigest: layerDigest(nil)}
if _, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || has {
t.Fatalf("grafana has a layer on anchor before the test: %v %v", has, err)
}
if moved, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered"); err != nil || !moved.Switched {
t.Fatalf("the move says %+v (%v)", moved, err)
}
if _, err := setLayerIn(open)(ctx, p, "s31", "approved by the operator"); err != nil {
t.Fatalf("the placement-only approval after the move: %v", err)
}
if _, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || !has {
t.Errorf("the approval did not set the layer: %v %v", has, err)
}
if reads, err := inv.ReadsConfig(ctx, "grafana"); err != nil || !reads {
t.Errorf("the approval turned ReadsConfig off: %v %v", reads, err)
}
}
+128
View File
@@ -0,0 +1,128 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// **Where the controller reads config values once the move has run** (novox/hq ADR 0348 §9, design 57 K5, issue 664):
// mesh-config_resolved, the bucket mesh-config alone writes, keyed `<node>.<module>.<key>`, each value with the
// source it came from. Read only for a module the switch reaches (inventory.ReadsConfig), so a controller before the
// move never asks the bus for config. The exception to ADR 0348 §9 ("the controller holds no config value") that
// ADR 0349 clause 17's old way needs, until design 57's R1 hands every module to the provisioner.
// resolvedEntry is one key of mesh-config_resolved as the controller reads it: the value and where it came from.
type resolvedEntry struct {
Node string `json:"node"`
Module string `json:"module"`
Key string `json:"key"`
Value json.RawMessage `json:"value,omitempty"`
Source string `json:"source"`
}
// The sources a resolved value is laid over a module's files from: mesh config, node config and the module's own
// value. Its manifest's default is composition's already, and a taken key's default is never laid over its file;
// a required value nothing sets (`unset`) is refused by composition by name, as it is today.
var laidSources = map[string]bool{"mesh": true, "node": true, "module": true}
// configLaid is the values of one module on one node that are laid over its files, from the resolved entries.
func configLaid(entries []resolvedEntry, node, module string) (map[string]any, error) {
out := map[string]any{}
for _, e := range entries {
if e.Node != node || e.Module != module || !laidSources[e.Source] {
continue
}
var v any
if err := json.Unmarshal(e.Value, &v); err != nil {
return nil, fmt.Errorf("mesh-config's value %s of %s on %s cannot be read: %w", e.Key, module, node, err)
}
out[e.Key] = v
}
return out, nil
}
// resolvedRead is what one read of the whole bucket held, and when it was read.
type resolvedRead struct {
mu sync.Mutex
at time.Time
entries []resolvedEntry
}
// resolvedFresh is how long one read of the bucket answers: a plan reads every module of a machine, and one read
// serves them all; a value set since is read by the next composition.
const resolvedFresh = 5 * time.Second
var theResolved resolvedRead
// readConfigValues is where an opened inventory reads config values once the switch is on; a test replaces it.
var readConfigValues inventory.ConfigValues = busConfigValues
// busConfigValues reads one module's config values on one node from mesh-config_resolved (inventory.ConfigValues).
func busConfigValues(ctx context.Context, node, module string) (map[string]any, error) {
theResolved.mu.Lock()
defer theResolved.mu.Unlock()
if theResolved.entries == nil || time.Since(theResolved.at) > resolvedFresh {
entries, err := readResolved(ctx)
if err != nil {
return nil, err
}
theResolved.entries, theResolved.at = entries, time.Now()
}
return configLaid(theResolved.entries, node, module)
}
// readResolved is every entry of mesh-config_resolved, read on the bus.
func readResolved(ctx context.Context) ([]resolvedEntry, error) {
entries := []resolvedEntry{}
err := onTheBus(func(conn *nats.Conn) error {
js, err := jetstream.New(conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
kv, err := js.KeyValue(ctx, broker.ConfigResolvedBucket)
if err != nil {
return fmt.Errorf("%s cannot be opened: %w", broker.ConfigResolvedBucket, err)
}
w, err := kv.WatchAll(ctx, jetstream.IgnoreDeletes())
if err != nil {
return err
}
defer func() { _ = w.Stop() }()
for {
select {
case <-ctx.Done():
return fmt.Errorf("%s was not read whole: %w", broker.ConfigResolvedBucket, ctx.Err())
case e, open := <-w.Updates():
if !open {
return errors.New(broker.ConfigResolvedBucket + " stopped answering")
}
if e == nil {
return nil
}
var r resolvedEntry
if err := json.Unmarshal(e.Value(), &r); err != nil {
return fmt.Errorf("%s holds %q, which is not a resolved value: %w", broker.ConfigResolvedBucket,
e.Key(), err)
}
if r.Node == "" || r.Module == "" || r.Key == "" || !strings.HasPrefix(e.Key(), r.Node+"."+r.Module+".") {
return fmt.Errorf("%s holds %q, whose value names another key", broker.ConfigResolvedBucket, e.Key())
}
entries = append(entries, r)
}
}
})
return entries, err
}
+305
View File
@@ -0,0 +1,305 @@
package main
import (
"context"
"crypto/ecdh"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"slices"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The core's own credentials, sealed by the vault (novox/hq ADR 0349 §16, issue 755; the lead's decision on issue 755,
// notes 4 and 5).
//
// For a machine whose engine opens the vault's box (link.VaultBoxContract), each own secret of a module on the core's
// list (catalogue.CoreCredentialModules) travels as the vault sealed it to that machine's key, which the controller
// asks for on the send path (`get`, answered to the controller alone and sealed only to the key that machine's engine
// stated: mesh-vault's core.go) and carries without being able to open it. A machine below that contract is sent the
// controller's own sealed value, exactly as before.
//
// **The vault seals every answer afresh**, so the box last got is kept (inventory.CoreBox) and carried again while the
// vault's generation and the machine's key are the same: a declaration that changed with every answer would never be
// current. Every question but a send reads the kept box and asks nothing.
//
// **Until the vault holds a machine's core credentials, the controller seals them itself** (novox/hq ADR 0396, issue
// 908; it replaces the ADR 0349 §16 rule that the controller never does). When the vault does not answer, a credential
// with a kept box keeps it — the machine keeps the file it has — and one with none has its module composed as for an
// engine below the contract: with the controller's own sealed value, never left out (a core module left out took the
// bus's port from the packet filter and cut the remote machines off). Either way the send raises a condition naming
// the vault and the machine, cleared by the first send the vault answers whole. Only a machine with no sealing key, or
// one whose key cannot be read, leaves its module out.
// engineOpensVaultBox says whether a machine's node-engine opens the vault's box, by its own newest statement.
func engineOpensVaultBox(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.VaultBoxContract, nil
}
// coreAnswer is the vault's `get` answer as the controller reads it: the name, the generation and the box, never
// opened.
type coreAnswer struct {
Name string `json:"name"`
Generation int `json:"generation"`
Sealed json.RawMessage `json:"sealed"`
}
// coreAsk asks the vault for one secret sealed to the given key.
type coreAsk func(name, publicKey string) (coreAnswer, error)
// coreBoxes is what the vault's boxes make of one machine's composition.
type coreBoxes struct {
// sealed is each core credential's box, by module and name (catalogue.Rendering.VaultSealed).
sealed map[string]map[string]json.RawMessage
// notPlaced is each core module left out, with why (catalogue.Rendering.NotPlaced).
notPlaced map[string]string
// keep is each box newly answered, to keep.
keep []inventory.CoreBox
// byController is each core module the vault did not answer for and no box was kept for, which is composed with
// the controller's own seal until the vault holds it (ADR 0396).
byController []string
// held is each credential the vault did not answer for whose kept box is carried again; silent is why the vault
// did not answer, per credential.
held, silent []string
}
// coreModulesOf is every module of the plan on the core's list whose own secrets this machine's declaration carries,
// with their plain own secret names, sorted: one handed to the provisioner has its secrets from the vault through the
// client, and one issued outside the mesh is never the vault's to seal here.
func coreModulesOf(modules []catalogue.Manifest, handed func(catalogue.Manifest) bool) map[string][]string {
out := map[string][]string{}
for _, m := range modules {
if !catalogue.CarriesVaultSealed(m.Module) || (handed != nil && handed(m)) {
continue
}
var names []string
for name, own := range m.OwnSecrets.Plain() {
if own.IssuedBy != catalogue.IssuedOutside {
names = append(names, name)
}
}
if len(names) > 0 {
sort.Strings(names)
out[m.Module] = names
}
}
return out
}
// sealingKeyPEM is a machine's sealing key, recorded raw in base64, as the SPKI PEM the vault seals to.
func sealingKeyPEM(raw string) (string, error) {
b, err := base64.StdEncoding.DecodeString(strings.TrimSpace(raw))
if err != nil || len(b) != 32 {
return "", errors.New("the machine's sealing key is not a raw X25519 key")
}
pub, err := ecdh.X25519().NewPublicKey(b)
if err != nil {
return "", err
}
der, err := x509.MarshalPKIXPublicKey(pub)
if err != nil {
return "", err
}
return string(pem.EncodeToMemory(&pem.Block{Type: "PUBLIC KEY", Bytes: der})), nil
}
// coreBoxesOf decides each core credential of a machine: on a send, the vault's answer (the kept box while its
// generation and key are the same), the kept box where the vault did not answer, or the module left to the controller's
// own seal where nothing was kept; on any other question, the kept box, or a stand-in the next send asks for. Pure: what to keep and what to
// say is answered, not done.
func coreBoxesOf(node, sealingKey string, core map[string][]string, kept map[string]map[string]inventory.CoreBox,
choosing Choosing, ask coreAsk) coreBoxes {
out := coreBoxes{sealed: map[string]map[string]json.RawMessage{}, notPlaced: map[string]string{}}
modules := make([]string, 0, len(core))
for m := range core {
modules = append(modules, m)
}
sort.Strings(modules)
for _, module := range modules {
if sealingKey == "" {
out.notPlaced[module] = fmt.Sprintf("%s has no sealing key, so the vault can seal %s's credentials to "+
"nothing there (novox/hq ADR 0349 §16)", node, module)
continue
}
key, err := sealingKeyPEM(sealingKey)
if err != nil {
out.notPlaced[module] = fmt.Sprintf("%s's credentials cannot be asked of the vault: %v", module, err)
continue
}
boxes := map[string]json.RawMessage{}
for _, name := range core[module] {
k, has := kept[module][name]
valid := has && k.SealedTo == sealingKey && len(k.Box) > 0
if choosing != Allocating {
if valid {
boxes[name] = k.Box
} else {
boxes[name] = catalogue.ForeseenVaultBox
}
continue
}
secret := "own/" + node + "/" + module + "/" + name
got, err := ask(secret, key)
if err == nil && (got.Generation <= 0 || canonicalBox(got.Sealed) == nil) {
err = errors.New("it answered no box")
}
if err != nil {
out.silent = append(out.silent, fmt.Sprintf("%s: %v", secret, err))
if valid {
boxes[name] = k.Box
out.held = append(out.held, module+"/"+name)
continue
}
out.byController = append(out.byController, module)
break
}
if valid && k.Generation == got.Generation {
boxes[name] = k.Box
continue
}
box := canonicalBox(got.Sealed)
boxes[name] = box
out.keep = append(out.keep, inventory.CoreBox{Module: module, Name: name, Generation: got.Generation,
SealedTo: sealingKey, Box: box})
}
if _, left := out.notPlaced[module]; !left && !slices.Contains(out.byController, module) {
out.sealed[module] = boxes
}
}
return out
}
// canonicalBox is the vault's box in the one form the declaration carries — the sealed box's fields, in their order,
// and nothing else — or nil when it is not one.
func canonicalBox(raw json.RawMessage) json.RawMessage {
var b struct {
V int `json:"v"`
Eph string `json:"eph"`
IV string `json:"iv"`
Tag string `json:"tag"`
Ct string `json:"ct"`
}
if err := json.Unmarshal(raw, &b); err != nil || b.V <= 0 || b.Eph == "" || b.Ct == "" {
return nil
}
out, _ := json.Marshal(b)
return out
}
// coreVault is how a send asks the vault for a core credential: its `get`, in the controller's name. A test stands
// in for it.
var coreVault = func(ctx context.Context) coreAsk {
vault := theVault(ctx)
return func(name, publicKey string) (coreAnswer, error) {
raw, err := vault("get", map[string]any{"name": name, "public_key": publicKey})
if err != nil {
return coreAnswer{}, err
}
var a coreAnswer
if err := json.Unmarshal(raw, &a); err != nil {
return coreAnswer{}, fmt.Errorf("the vault's answer is not one: %w", err)
}
return a, nil
}
}
// coreBoxesFor is coreBoxesOf for one machine, read from and kept in the store, and said on a send.
func coreBoxesFor(ctx context.Context, inv *inventory.Inventory, node string, modules []catalogue.Manifest,
handed func(catalogue.Manifest) bool, choosing Choosing) (coreBoxes, error) {
core := coreModulesOf(modules, handed)
if len(core) == 0 {
if choosing == Allocating {
settleVaultSilent(ctx, node)
}
return coreBoxes{}, nil
}
key, err := inv.SealingKeyOf(ctx, node)
if err != nil {
return coreBoxes{}, err
}
kept, err := inv.KeptCoreBoxes(ctx, node)
if err != nil {
return coreBoxes{}, err
}
var ask coreAsk
if choosing == Allocating {
ask = coreVault(ctx)
}
out := coreBoxesOf(node, key, core, kept, choosing, ask)
for _, b := range out.keep {
if err := inv.KeepCoreBox(ctx, node, b); err != nil {
return coreBoxes{}, err
}
}
if choosing == Allocating {
sayVaultSilent(ctx, node, out)
}
return out, nil
}
// kindVaultSilent is a send that did not have the vault's box for a core credential of a machine.
const kindVaultSilent = "vault-silent"
// vaultSilentObservation is that, as a condition: `machine.<machine>.vault-silent`.
func vaultSilentObservation(node string, b coreBoxes) conditions.Observation {
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: node, Kind: kindVaultSilent, Token: kindVaultSilent,
Machine: node, Severity: conditions.Warning, Source: coreDowngradeSource}
var said []string
if len(b.notPlaced) > 0 {
o.Severity = conditions.Urgent
var left []string
for m := range b.notPlaced {
left = append(left, m)
}
sort.Strings(left)
said = append(said, "not placed: "+strings.Join(left, ", "))
}
if len(b.byController) > 0 {
said = append(said, "sealed by the controller until the vault holds it: "+strings.Join(b.byController, ", "))
}
if len(b.held) > 0 {
said = append(said, "kept as last sent: "+strings.Join(b.held, ", "))
}
o.Summary = fmt.Sprintf("the vault did not answer for the core's credentials on %s; %s", node, strings.Join(said, "; "))
o.Said = o.Summary + " — " + strings.Join(b.silent, "; ")
return o
}
// sayVaultSilent raises the condition for a send the vault did not answer whole, and clears it for one it did.
func sayVaultSilent(ctx context.Context, node string, b coreBoxes) {
if len(b.silent) == 0 && len(b.notPlaced) == 0 {
settleVaultSilent(ctx, node)
return
}
fmt.Printf("HELD: %s\n", vaultSilentObservation(node, b).Said)
if conditionsFrom == nil {
return
}
if _, err := conditionsFrom.Observe(ctx, vaultSilentObservation(node, b)); err != nil {
fmt.Printf("the condition for the vault's silence on %s could not be raised: %v\n", node, err)
}
}
// settleVaultSilent clears the condition once a send had every core credential of the machine from the vault.
func settleVaultSilent(ctx context.Context, node string) {
if conditionsFrom == nil {
return
}
key := vaultSilentObservation(node, coreBoxes{}).Key()
if _, err := conditionsFrom.Clear(ctx, key, "the vault answered for every core credential of "+node); err != nil {
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
}
}
+244
View File
@@ -0,0 +1,244 @@
package main
import (
"context"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The core's own credentials travel sealed by the vault to their machine (novox/hq ADR 0349 §16, issue 755; the lead's
// decision, issue 755 notes 4 and 5): to an engine that opens the vault's box, the send carries the box the vault
// answered the controller and never the controller's own seal; when the vault does not answer, a kept box is carried
// again and a module with none is sealed by the controller (ADR 0396, issue 908), with a condition naming the vault and the machine. An engine below the
// contract composes exactly as before, and the vault is not asked.
// coreStandIn answers `get` with a box per name and generation, or its silence.
type coreStandIn struct {
asked []string
keys []string
generation int
silent error
}
func (v *coreStandIn) ask(name, key string) (coreAnswer, error) {
v.asked, v.keys = append(v.asked, name), append(v.keys, key)
if v.silent != nil {
return coreAnswer{}, v.silent
}
box := `{"v":1,"eph":"ZQ==","iv":"aQ==","tag":"dA==","ct":"` + strings.Repeat("c", len(v.asked)) + `"}`
return coreAnswer{Name: name, Generation: v.generation, Sealed: json.RawMessage(box)}, nil
}
func withVault(t *testing.T, v *coreStandIn) {
t.Helper()
before := coreVault
coreVault = func(context.Context) coreAsk { return v.ask }
t.Cleanup(func() { coreVault = before })
}
func aCoreMesh(t *testing.T, contract int) (*stores, *coreStandIn) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
client := aClientModule()
register(t, open, client)
if _, err := assign(ctx, open, "laptop", client.Module); err != nil {
t.Fatal(err)
}
if err := issueProvisionerClient(ctx, open.inventory, client, "laptop", broker.Broker{Fingerprint: "ab"},
"10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.RecordHealth(ctx, inventory.NodeHealth{Node: "laptop", Contract: contract,
SaidAt: time.Now()}); err != nil {
t.Fatal(err)
}
v := &coreStandIn{generation: 1}
withVault(t, v)
return open, v
}
func clientsBroker(resources []map[string]any) map[string]any {
for _, r := range resources {
if r["id"] == "mesh-provisioner-client.needs-"+broker.ProvisionerSecret {
return r
}
}
return nil
}
func vaultSilence(t *testing.T, node string) (conditions.Condition, bool) {
t.Helper()
c, found, err := conditionsFrom.Get(t.Context(), vaultSilentObservation(node, coreBoxes{}).Key())
if err != nil {
t.Fatal(err)
}
return c, found
}
func TestTheSendCarriesTheVaultsBoxForACoreCredentialAndKeepsIt(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
key, err := open.inventory.SealingKeyOf(t.Context(), "laptop")
if err != nil {
t.Fatal(err)
}
f := clientsBroker(sentTo(t, open, "laptop"))
if f == nil || f["sealed"] != nil || f[catalogue.VaultSealedField] == nil {
t.Fatalf("the core credential is sent as %v: the vault's box, never the controller's seal", f)
}
if want := []string{"own/laptop/mesh-provisioner-client/" + broker.ProvisionerSecret}; !reflect.DeepEqual(vault.asked, want) {
t.Fatalf("the vault was asked %v", vault.asked)
}
if want, _ := sealingKeyPEM(key); vault.keys[0] != want {
t.Fatalf("the vault was asked to seal to %q, not the machine's key", vault.keys[0])
}
first, _ := json.Marshal(f[catalogue.VaultSealedField])
// The vault answers afresh with the same generation: the kept box is carried again, so the declaration is stable.
again, _ := json.Marshal(clientsBroker(sentTo(t, open, "laptop"))[catalogue.VaultSealedField])
if string(again) != string(first) {
t.Fatalf("a fresh box of the same generation changed the declaration: %s, then %s", first, again)
}
// A rotation in the vault (a newer generation) is carried.
vault.generation = 2
rotated, _ := json.Marshal(clientsBroker(sentTo(t, open, "laptop"))[catalogue.VaultSealedField])
if string(rotated) == string(first) {
t.Fatal("a newer generation was not carried")
}
if _, found := vaultSilence(t, "laptop"); found {
t.Fatal("a send the vault answered raised its silence")
}
}
func TestASilentVaultHoldsTheKeptBoxAndSaysSo(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
first, _ := json.Marshal(clientsBroker(sentTo(t, open, "laptop"))[catalogue.VaultSealedField])
vault.silent = errors.New("no responders")
held := clientsBroker(sentTo(t, open, "laptop"))
got, _ := json.Marshal(held[catalogue.VaultSealedField])
if held == nil || held["sealed"] != nil || string(got) != string(first) {
t.Fatalf("a silent vault's credential is sent as %v: the box kept, never the controller's seal", held)
}
c, found := vaultSilence(t, "laptop")
if !found || !strings.Contains(c.Summary, "vault") || !strings.Contains(c.Summary, "laptop") {
t.Fatalf("no condition names the vault and the machine: %+v", c)
}
// The vault answers again: the condition clears.
vault.silent = nil
sentTo(t, open, "laptop")
if _, found := vaultSilence(t, "laptop"); found {
t.Fatal("the condition stayed after the vault answered")
}
}
func TestACoreModuleFirstPlacedWhileTheVaultIsSilentIsSealedByTheController(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
vault.silent = errors.New("no responders")
plan, settings, err := planFor(t.Context(), open, "laptop")
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(t.Context(), open, "laptop", plan, settings)
if err != nil {
t.Fatal(err)
}
f := clientsBroker(declared.Resources)
if f == nil || f["sealed"] == nil || f[catalogue.VaultSealedField] != nil {
t.Fatalf("the core credential is composed as %v: the controller's own seal, until the vault holds it", f)
}
if why, left := declared.leftOutWhy["mesh-provisioner-client"]; left {
t.Fatalf("the core module was left out: %v", why)
}
c, found := vaultSilence(t, "laptop")
if !found || !strings.Contains(c.Summary, "sealed by the controller until the vault holds it") {
t.Fatalf("the condition does not say why: %+v", c)
}
}
// Issue 908: a machine at the vault-box contract, the vault silent, no kept box: nats is composed with the
// controller's own seal, never left out, and the vault-silent condition names it.
func TestReplay908(t *testing.T) {
key := "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8="
core := map[string][]string{"nats": {"password"}}
ask := func(name, publicKey string) (coreAnswer, error) { return coreAnswer{}, errors.New("no responders") }
got := coreBoxesOf("novox", key, core, nil, Allocating, ask)
if _, sealed := got.sealed["nats"]; sealed || len(got.notPlaced) != 0 {
t.Fatalf("nats is vault-sealed %v or not placed %v", got.sealed, got.notPlaced)
}
if !reflect.DeepEqual(got.byController, []string{"nats"}) {
t.Fatalf("nats is not left to the controller's seal: %+v", got)
}
o := vaultSilentObservation("novox", got)
if o.Severity != conditions.Warning || !strings.Contains(o.Summary, "sealed by the controller until the vault holds it: nats") {
t.Fatalf("the condition does not name nats: %+v", o)
}
// A kept box is still carried while the vault is silent.
kept := map[string]map[string]inventory.CoreBox{"nats": {"password": {SealedTo: key, Box: json.RawMessage(`{"v":1}`)}}}
got = coreBoxesOf("novox", key, core, kept, Allocating, ask)
if string(got.sealed["nats"]["password"]) != `{"v":1}` || len(got.byController) != 0 {
t.Fatalf("the kept box was not held: %+v", got)
}
}
// An engine below VaultBoxContract is sent the controller's own seal, as before, and the vault is not asked.
func TestAnEngineBelowTheContractIsSentTheControllersSealAsBefore(t *testing.T) {
open, vault := aCoreMesh(t, link.HandedContract)
f := clientsBroker(sentTo(t, open, "laptop"))
if f == nil || f["sealed"] == nil || f[catalogue.VaultSealedField] != nil {
t.Fatalf("an older engine is sent %v", f)
}
if len(vault.asked) != 0 {
t.Fatalf("the vault was asked %v for an older engine", vault.asked)
}
}
// Every question but a send reads the kept box and asks nothing; with none kept, a stand-in the next send asks for.
func TestOnlyASendAsksTheVault(t *testing.T) {
core := map[string][]string{"nats": {"broker"}}
key := aPublicKey(t)
kept := map[string]map[string]inventory.CoreBox{"nats": {"broker": {Module: "nats", Name: "broker",
Generation: 1, SealedTo: key, Box: json.RawMessage(`{"v":1}`)}}}
never := func(string, string) (coreAnswer, error) {
t.Fatal("the vault was asked by a question that is not a send")
return coreAnswer{}, nil
}
for _, choosing := range []Choosing{Reading, Foreseeing} {
if got := coreBoxesOf("anchor", key, core, kept, choosing, never); string(got.sealed["nats"]["broker"]) != `{"v":1}` {
t.Errorf("read %s", got.sealed["nats"]["broker"])
}
if got := coreBoxesOf("anchor", key, core, nil, choosing, never); string(got.sealed["nats"]["broker"]) !=
string(catalogue.ForeseenVaultBox) {
t.Errorf("with none kept, read %s", got.sealed["nats"]["broker"])
}
}
// A box kept for another key of the machine is never carried.
other := coreBoxesOf("anchor", aPublicKey(t), core, kept, Allocating,
func(string, string) (coreAnswer, error) { return coreAnswer{}, errors.New("silent") })
if _, vaultSealed := other.sealed["nats"]; vaultSealed || !reflect.DeepEqual(other.byController, []string{"nats"}) {
t.Fatalf("a box sealed to the machine's old key was carried: %+v", other)
}
}
// The core's modules on a machine: on the list, not handed, with their own secrets but one issued outside.
func TestTheCoresModulesAreTheListsWithTheirOwnSecrets(t *testing.T) {
modules := []catalogue.Manifest{
{Module: "nats", OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/a"}, "given": {Path: "/b",
IssuedBy: catalogue.IssuedOutside}}},
{Module: "postgres", OwnSecrets: catalogue.OwnSecrets{"superuser": {Path: "/c"}}},
{Module: "mesh-provisioner", OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/d"}}},
}
got := coreModulesOf(modules, func(m catalogue.Manifest) bool { return m.Module == "mesh-provisioner" })
if want := map[string][]string{"nats": {"broker"}}; !reflect.DeepEqual(got, want) {
t.Fatalf("got %v, want %v", got, want)
}
}
+117
View File
@@ -0,0 +1,117 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// A dead letter from a restricted stream ages out with its stream (novox/hq ADR 0376 §3, §6).
//
// DEAD_LETTERS lets such a message go when its stream's maximum age runs out, so a consumer's `max-deliveries`
// condition can clear with nobody having delivered it again or dropped it. That is said: the watchdogs remember,
// from one look to the next, when each restricted dead letter ages out, count those gone once their time passed,
// say each in the controller's log as it goes, and clear the condition with words saying how many aged out
// undelivered and from which stream.
// agingTally is what the watchdogs remember of restricted dead letters between looks.
type agingTally struct {
// seen is each consumer's restricted dead letters at the last look, by id, with when each ages out.
seen map[string]map[uint64]time.Time
// agedOut is how many of each consumer's aged out undelivered while its condition was open.
agedOut map[string]int
}
// look compares this look's restricted dead letters with the last: newly is how many of each consumer's aged out
// since, and cleared how many aged out in all for each consumer DEAD_LETTERS no longer holds anything of, when any
// did. held is every consumer's count this look; aging its restricted dead letters.
func (t *agingTally) look(now time.Time, held map[string]int, aging map[string][]link.Aging) (newly, cleared map[string]int) {
if t.seen == nil {
t.seen, t.agedOut = map[string]map[uint64]time.Time{}, map[string]int{}
}
newly, cleared = map[string]int{}, map[string]int{}
for key, before := range t.seen {
still := map[uint64]bool{}
for _, a := range aging[key] {
still[a.ID] = true
}
for id, agesOut := range before {
if !still[id] && !agesOut.After(now) {
newly[key]++
}
}
if newly[key] == 0 {
delete(newly, key)
}
t.agedOut[key] += newly[key]
}
for key, n := range t.agedOut {
if held[key] > 0 {
continue
}
if n > 0 {
cleared[key] = n
}
delete(t.agedOut, key)
}
t.seen = map[string]map[uint64]time.Time{}
for key, letters := range aging {
t.seen[key] = map[uint64]time.Time{}
for _, a := range letters {
t.seen[key][a.ID] = a.AgesOut
}
}
return newly, cleared
}
// agedOutResolved is the line a consumer's condition clears with when messages it gave up on aged out undelivered.
func agedOutResolved(key string, n int) string {
stream, _, _ := strings.Cut(key, ".")
messages := "a message"
if n > 1 {
messages = fmt.Sprintf("%d messages", n)
}
return fmt.Sprintf("Resolved: %s of the stream %s aged out undelivered", messages, stream)
}
// sayAgedOut says, in the log, what aged out since the last look, and clears the condition of every consumer whose
// dead letters are all gone with the words saying how many aged out. Before the rows are reconciled, so the
// condition is cleared with these words and not its own.
func (w *watchdogs) sayAgedOut(ctx context.Context, f *signalFacts) []string {
if f.advisoriesErr != nil || f.deadLettersAging == nil {
return nil
}
newly, cleared := w.aging.look(f.now, f.deadLetters, f.deadLettersAging)
var problems []string
for _, key := range countedKeys(newly) {
stream, consumer, _ := strings.Cut(key, ".")
fmt.Printf("%d message(s) %s gave up on aged out of %s undelivered: %s keeps them only for its maximum age "+
"(novox/hq ADR 0376)\n", newly[key], link.ConsumerInWords(stream, consumer), "DEAD_LETTERS", stream)
}
for _, key := range countedKeys(cleared) {
stream, consumer, _ := strings.Cut(key, ".")
fmt.Printf("%s no longer has a message kept: %d aged out of %s undelivered, so its condition clears "+
"(novox/hq ADR 0376)\n", link.ConsumerInWords(stream, consumer), cleared[key], stream)
if _, err := w.keeper.ClearSaying(ctx, conditions.Key(conditions.ScopeBus, key, link.AdvisoryMaxDeliveries),
fmt.Sprintf("its %d dead letter(s) of %s aged out undelivered", cleared[key], stream),
agedOutResolved(key, cleared[key])); err != nil {
problems = append(problems, err.Error())
}
}
return problems
}
// countedKeys is a count's keys in order.
func countedKeys(m map[string]int) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,69 @@
package main
import (
"context"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **A dead letter that ages out clears its condition and says so** (novox/hq ADR 0376 §6), on a fake clock: while
// DEAD_LETTERS holds a restricted stream's messages for a consumer its max-deliveries condition is open; once they
// have aged out it clears, with words saying how many aged out undelivered and from which stream. One dropped before
// its age is not counted as aged out, and a consumer of EVENTS clears as before.
func TestADeadLetterThatAgesOutClearsItsConditionAndSaysSo(t *testing.T) {
now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
store := conditions.NewInMemory()
told := &conditions.Told{}
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store, Teller: told,
Now: func() time.Time { return now }})
w := &watchdogs{keeper: k, started: now.Add(-time.Hour)}
look := func(at time.Time, held map[string]int, aging map[string][]link.Aging) {
f := calm(at)
f.deadLetters, f.deadLettersAging = held, aging
w.see(t.Context(), f)
}
mail := "MAIL.ace_mailer"
look(now, map[string]int{mail: 3, "EVENTS.media_sonarr": 1}, map[string][]link.Aging{mail: {
{ID: 1, AgesOut: now.Add(time.Hour)}, {ID: 2, AgesOut: now.Add(2 * time.Hour)}, {ID: 3, AgesOut: now.Add(3 * time.Hour)}}})
if open, _ := k.Open(t.Context()); len(open) != 2 {
t.Fatalf("open: %+v", open)
}
// One dropped by a person before its age: not aged out.
look(now.Add(30*time.Minute), map[string]int{mail: 2, "EVENTS.media_sonarr": 1}, map[string][]link.Aging{mail: {
{ID: 2, AgesOut: now.Add(2 * time.Hour)}, {ID: 3, AgesOut: now.Add(3 * time.Hour)}}})
look(now.Add(150*time.Minute), map[string]int{mail: 1}, map[string][]link.Aging{mail: {{ID: 3, AgesOut: now.Add(3 * time.Hour)}}})
open, _ := k.Open(t.Context())
if len(open) != 1 || open[0].Key != conditions.Key(conditions.ScopeBus, mail, link.AdvisoryMaxDeliveries) {
t.Fatalf("after one aged out: %+v", open)
}
look(now.Add(4*time.Hour), map[string]int{}, map[string][]link.Aging{})
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("after all aged out: %+v", open)
}
k.Close(context.Background()) // everything said is said once it closes
var mailCleared, sonarrCleared string
for _, e := range told.Said() {
if e.Change != conditions.ChangeCleared {
continue
}
switch e.Condition.Key {
case conditions.Key(conditions.ScopeBus, mail, link.AdvisoryMaxDeliveries):
mailCleared = e.Condition.Resolved
case conditions.Key(conditions.ScopeBus, "EVENTS.media_sonarr", link.AdvisoryMaxDeliveries):
sonarrCleared = e.Condition.Resolved
}
}
if mailCleared != "Resolved: 2 messages of the stream MAIL aged out undelivered" {
t.Fatalf("MAIL's condition cleared with %q", mailCleared)
}
if sonarrCleared != "Resolved: the messages it gave up on were delivered again or dropped" {
t.Fatalf("an EVENTS consumer's condition cleared with %q", sonarrCleared)
}
if why, ok := conditions.Plain(mailCleared); !ok {
t.Fatalf("not plain: %s", why)
}
}
+4 -2
View File
@@ -952,7 +952,7 @@ func deliveryCommand(ctx context.Context, args []string) error {
if *group == "" && len(members) == 1 { if *group == "" && len(members) == 1 {
// One head, checked again as the forge's announcement would have it (a recheck): the controller's // One head, checked again as the forge's announcement would have it (a recheck): the controller's
// own path for a pull request, run because the delivery's owner asked. // own path for a pull request, run because the delivery's owner asked.
if err := sayingOnTheBus(ctx, inv, func() error { return (following{open}).PullUpdated(ctx, members[0].pull()) }); err != nil { if err := sayingOnTheBus(ctx, inv, func() error { return (following{open: open, oneShot: true}).PullUpdated(ctx, members[0].pull()) }); err != nil {
return err return err
} }
return answer(map[string]any{"rechecked": members[0].ID}) return answer(map[string]any{"rechecked": members[0].ID})
@@ -1051,7 +1051,9 @@ func appliedFrom(ctx context.Context, inv *inventory.Inventory) inventory.Applie
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, walkSent.Format(time.RFC3339), err) fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, walkSent.Format(time.RFC3339), err)
return inventory.AppliedReport{}, false return inventory.AppliedReport{}, false
} }
return r, ok // And, for each module the send handed to the provisioner, its client's report on that build (novox/hq ADR
// 0362): applied only once both said it, at the later; failed if the client failed.
return inventory.ProvisionedSend(node, send, r, ok, provLookup(ctx))
} }
} }
+123 -11
View File
@@ -5,6 +5,7 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"log"
"strings" "strings"
"time" "time"
@@ -54,6 +55,10 @@ type deskGive struct {
trusted func(module string) (bool, error) trusted func(module string) (bool, error)
// ask asks one machine's node-launcher.secret and answers its result, or the holder's refusal. // ask asks one machine's node-launcher.secret and answers its result, or the holder's refusal.
ask func(machine string, args map[string]any) (json.RawMessage, error) ask func(machine string, args map[string]any) (json.RawMessage, error)
// promptFor is the one line the secret's declaration asks for it as (novox/hq issue 789): read from the
// controller's own catalogue, never from the call or the module's runtime. Nil, or an empty answer, asks by
// name alone, as the desk does today.
promptFor func(module, name string) (string, error)
// accept seals the value as `secret accept` does, and says whether it lives until the module's start. // accept seals the value as `secret accept` does, and says whether it lives until the module's start.
accept func(value string) (untilStart bool, err error) accept func(value string) (untilStart bool, err error)
// record writes the act in the hand-act log. // record writes the act in the hand-act log.
@@ -67,6 +72,9 @@ type deskGive struct {
// nil keeps no record (a test that does not look). end closes it with how it ended. // nil keeps no record (a test that does not look). end closes it with how it ended.
open func(node, module, name, desk string) (int64, error) open func(node, module, name, desk string) (int64, error)
end func(id int64, outcome string) error end func(id int64, outcome string) error
// vault, when set, is the path of a module whose secrets are filled the new way (vault_secrets.go): the vault
// records the ask, holds its bounds and opens what is typed; open, end and accept are not used.
vault vaultAsker
} }
// askedByName is the caller as the prompt names it: the first clause of what the bus said, in the characters // askedByName is the caller as the prompt names it: the first clause of what the bus said, in the characters
@@ -133,6 +141,9 @@ func (d deskGive) give(node, module, name, desk string) (string, error) {
"bus, where an agent may answer first (novox/hq ADR 0259 §10)", module, name, node, module, name) "bus, where an agent may answer first (novox/hq ADR 0259 §10)", module, name, node, module, name)
} }
} }
if d.vault != nil {
return d.giveToVault(node, module, name, desk)
}
// The bounds, read and kept before anybody is asked to type (novox/hq ADR 0277): one open prompt per secret, // The bounds, read and kept before anybody is asked to type (novox/hq ADR 0277): one open prompt per secret,
// few an hour. How the ask ends is recorded whatever happens below. // few an hour. How the ask ends is recorded whatever happens below.
outcome := "failed" outcome := "failed"
@@ -153,15 +164,28 @@ func (d deskGive) give(node, module, name, desk string) (string, error) {
} }
// By name, never by words: the holder writes the prompt from these, and says the controller asks, which // By name, never by words: the holder writes the prompt from these, and says the controller asks, which
// the bus alone makes true (broker.ControllerOnly). Who asked is the bus's word on the caller, cut to a // the bus alone makes true (broker.ControllerOnly). Who asked is the bus's word on the caller, cut to a
// name's characters — never an argument of the call. // name's characters — never an argument of the call. `what` is the catalogue's own words for the secret
raw, err := d.ask(desk, map[string]any{ // (novox/hq issue 789), never a word the caller chose; omitted when the module declares none, so an old
// desk and an old module behave as today.
args := map[string]any{
"module": module, "module": module,
"secret": name, "secret": name,
"node": node, "node": node,
"asked_by": askedByName(d.askedBy), "asked_by": askedByName(d.askedBy),
"seal_to": public, "seal_to": public,
"timeout_seconds": deskPromptWithin, "timeout_seconds": deskPromptWithin,
}) }
if d.promptFor != nil {
what, err := d.promptFor(module, name)
if err != nil {
outcome = "refused"
return "", fmt.Errorf("nobody was asked to type anything: %w", err)
}
if what != "" {
args["what"] = what
}
}
raw, err := d.ask(desk, args)
if err != nil { if err != nil {
outcome = "refused" outcome = "refused"
return "", fmt.Errorf("the desk on %s could not be asked: %w", desk, err) return "", fmt.Errorf("the desk on %s could not be asked: %w", desk, err)
@@ -236,8 +260,9 @@ func askAtDesk(ctx context.Context, node, module, name, desk string) error {
return err return err
} }
defer open.Close() defer open.Close()
askedBy := link.Caller()
d := deskGive{ d := deskGive{
askedBy: link.Caller(), askedBy: askedBy,
open: func(node, module, name, desk string) (int64, error) { open: func(node, module, name, desk string) (int64, error) {
return open.inventory.OpenSecretAsk(ctx, node, module, name, askedByName(link.Caller()), desk) return open.inventory.OpenSecretAsk(ctx, node, module, name, askedByName(link.Caller()), desk)
}, },
@@ -247,7 +272,8 @@ func askAtDesk(ctx context.Context, node, module, name, desk string) error {
_, err := open.inventory.NodeByName(ctx, machine) _, err := open.inventory.NodeByName(ctx, machine)
return err return err
}, },
trusted: func(module string) (bool, error) { return open.inventory.RunsAsItsOwnAccount(ctx, module) }, trusted: func(module string) (bool, error) { return open.inventory.RunsAsItsOwnAccount(ctx, module) },
promptFor: func(module, name string) (string, error) { return open.inventory.OwnSecretPrompt(ctx, module, name) },
ask: func(machine string, args map[string]any) (json.RawMessage, error) { ask: func(machine string, args map[string]any) (json.RawMessage, error) {
var result json.RawMessage var result json.RawMessage
err := onTheBus(func(conn *nats.Conn) error { err := onTheBus(func(conn *nats.Conn) error {
@@ -273,7 +299,18 @@ func askAtDesk(ctx context.Context, node, module, name, desk string) error {
return err return err
}) })
}, },
announce: func(node, module, name, how string) error { return announceSecretGiven(ctx, node, module, name, how) }, announce: func(node, module, name, how string) error {
return announceSecretGiven(ctx, node, module, name, how, askedBy)
},
}
// A module whose secrets are filled the new way is asked for by the vault (ADR 0349 clause 9); every other
// keeps today's path (clause 17).
ours, err := filledTheNewWay(module)
if err != nil {
return fmt.Errorf("nothing was asked: %w", err)
}
if ours {
d.vault = theVault(ctx)
} }
words, err := d.give(node, module, name, desk) words, err := d.give(node, module, name, desk)
if err != nil { if err != nil {
@@ -324,10 +361,85 @@ func secretNameWords(name string) string {
return strings.Join(strings.FieldsFunc(name, func(r rune) bool { return r == '-' || r == '_' || r == '.' }), " ") return strings.Join(strings.FieldsFunc(name, func(r rune) bool { return r == '-' || r == '_' || r == '.' }), " ")
} }
// announceSecretGiven raises it on this controller's keeper. // tellWithin is how long the operator channel's `tell` is given to answer: well under a human's patience,
func announceSecretGiven(ctx context.Context, node, module, name, how string) error { // and short because nothing in the call waits on the operator — it asks nothing (novox/hq ADR 0380).
return withKeeper(ctx, func(k *conditions.Keeper) error { const tellWithin = 10 * time.Second
_, err := k.Observe(ctx, secretGivenObservation(node, module, name, how, time.Now()))
return err // askOperatorChannel asks the operator channel's seat (the holder issue 797 gives it) its `tell` verb,
// through the controller's grant (novox/hq ADR 0380 §8), over this controller's own bus connection; a seam
// a test replaces whole. The holder's own refusal is an error naming it; an absent holder or a refused
// grant answer as link.ErrNothingServes or link.ErrGrantRefused (checked with errors.Is), so the caller can
// tell the two apart from any other failure.
var askOperatorChannel = func(ctx context.Context, args map[string]any) (json.RawMessage, error) {
var result json.RawMessage
err := onTheBus(func(conn *nats.Conn) error {
answer, err := link.AskMeshSeatTool(ctx, conn, "operator-channel", "tell", args, tellWithin)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("operator-channel.tell refused: %s", answer.Error)
}
result = answer.Result
return nil
}) })
return result, err
}
// announceSecretGiven tells the operator, once, that a module's own secret was given (novox/hq ADR 0380):
// one plain message through the operator channel's `tell`, naming the secret, the module, the machine,
// where and when it was given and who asked — asking nothing, and opening no condition. **The controller's
// change waits for that verb at run time, not at merge** (clause 9): while nothing on the bus yet serves it,
// or the controller's own grant is refused, the secret-given condition is raised exactly as before ADR 0380
// — any other failure (the holder itself refusing, or answering something unreadable) is the announcement's
// own failure, said to the caller as it always was.
func announceSecretGiven(ctx context.Context, node, module, name, how, askedBy string) error {
at := time.Now()
body := map[string]any{
"headline": fmt.Sprintf("A password was entered for %s's %s on %s", module, secretNameWords(name), node),
"subject": node,
"lines": []string{
fmt.Sprintf("%s at %s, asked by %s.", how, at.Local().Format("15:04"), askedByName(askedBy)),
"If that was not you, give it again yourself at the controller's terminal.",
},
}
_, told := askOperatorChannel(ctx, body)
switch {
case told == nil:
clearSecretGivenConditions(ctx)
return nil
case errors.Is(told, link.ErrNothingServes), errors.Is(told, link.ErrGrantRefused):
log.Printf("secret-given: %s's %s on %s could not be told through the operator channel (%v); the "+
"secret-given condition is raised as it was before novox/hq ADR 0380 (its clause 9)", module, name,
node, told)
return withKeeper(ctx, func(k *conditions.Keeper) error {
_, err := k.Observe(ctx, secretGivenObservation(node, module, name, how, at))
return err
})
default:
return told
}
}
// clearSecretGivenConditions retires every open secret-given condition once a plain message has replaced
// it (novox/hq ADR 0380 §10): called after the first time a secret-given message is told successfully. No
// resolved line is sent — the condition is simply gone — and why is kept in the keeper's own history.
func clearSecretGivenConditions(ctx context.Context) {
if err := withKeeper(ctx, func(k *conditions.Keeper) error {
open, err := k.Open(ctx)
if err != nil {
return err
}
for _, c := range open {
if c.Kind != kindSecretGiven {
continue
}
if _, err := k.ClearSaying(ctx, c.Key, "retired by novox/hq ADR 0380", ""); err != nil {
return err
}
}
return nil
}); err != nil {
log.Printf("secret-given: the open secret-given conditions, retired by ADR 0380, could not be cleared: %v", err)
}
} }
+215
View File
@@ -456,6 +456,52 @@ func TestThePromptNamesWhoAskedFromTheBussWordAlone(t *testing.T) {
} }
} }
// The desk's prompt names which account of a family is asked for, from the catalogue's own words (novox/hq
// issue 789): a family declaring `prompt` with `{member}` sends `what` with the member's suffix filled in.
func TestThePromptNamesWhichAccountOfAFamilyIsAsked(t *testing.T) {
d, _, _, asked := aDesk(t, sealedTo(t, typed))
d.declares = func(module, name string) error {
if module != "mesh-mail" || name != "mail-credential-gmail" {
return errors.New(module + " does not declare " + name + " as an own secret")
}
return nil
}
d.promptFor = func(module, name string) (string, error) {
if module == "mesh-mail" && name == "mail-credential-gmail" {
return "the password of the mail account gmail", nil
}
return "", nil
}
if _, err := d.give("anchor", "mesh-mail", "mail-credential-gmail", "laptop"); err != nil {
t.Fatal(err)
}
if got := (*asked)[0]["what"]; got != "the password of the mail account gmail" {
t.Errorf(`"what" = %v`, got)
}
}
// Without a declared prompt, no `what` is sent: an old module, or one that declares none, is asked by name alone,
// as the desk does today (novox/hq issue 789).
func TestWithNoDeclaredPromptNoWhatIsSent(t *testing.T) {
d, _, _, asked := aDesk(t, sealedTo(t, typed))
d.promptFor = func(module, name string) (string, error) { return "", nil }
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
if _, there := (*asked)[0]["what"]; there {
t.Errorf(`"what" was sent with no declared prompt: %v`, (*asked)[0])
}
// And with no promptFor wired at all (an older caller of deskGive), same thing: `give` still works, asking
// nothing more than it does today.
d2, _, _, asked2 := aDesk(t, sealedTo(t, typed))
if _, err := d2.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
if _, there := (*asked2)[0]["what"]; there {
t.Errorf(`"what" was sent with promptFor nil: %v`, (*asked2)[0])
}
}
// An ask for a secret is bounded before anybody is asked to type (ADR 0277): the record refuses it, nothing is // An ask for a secret is bounded before anybody is asked to type (ADR 0277): the record refuses it, nothing is
// asked; and how every ask ends is recorded. // asked; and how every ask ends is recorded.
func TestASecretAskIsBoundedAndItsEndRecorded(t *testing.T) { func TestASecretAskIsBoundedAndItsEndRecorded(t *testing.T) {
@@ -510,3 +556,172 @@ func TestASecretAskIsNeverASecretRead(t *testing.T) {
} }
} }
} }
// askOperatorChannelFake replaces askOperatorChannel for the test and records every call it was given.
func askOperatorChannelFake(t *testing.T, answer func(args map[string]any) (json.RawMessage, error)) *[]map[string]any {
t.Helper()
var calls []map[string]any
was := askOperatorChannel
askOperatorChannel = func(_ context.Context, args map[string]any) (json.RawMessage, error) {
calls = append(calls, args)
return answer(args)
}
t.Cleanup(func() { askOperatorChannel = was })
return &calls
}
// announceWithRealTell wires a deskGive's announce to the real announceSecretGiven, as askAtDesk does,
// rather than the fakes the tests above use to look at what `give` does around it.
func announceWithRealTell(ctx context.Context, askedBy string) func(node, module, name, how string) error {
return func(node, module, name, how string) error {
return announceSecretGiven(ctx, node, module, name, how, askedBy)
}
}
// novox/hq ADR 0380 clauses 1-2: a secret given at the desk is told to the operator once, in a plain
// message through the operator channel's `tell`, naming the secret, the module, where and when and who
// asked, and opens no condition — the kind the review of 2026-10-09 (M4) raised is retired.
func TestASecretGivenAtTheDeskIsToldOnceAndOpensNoCondition(t *testing.T) {
ctx := context.Background()
k, _ := withConditionsInMemory(t)
calls := askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, nil })
d, _, _, _ := aDesk(t, sealedTo(t, typed))
d.announce = announceWithRealTell(ctx, "mail-seats-835@novox")
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
if len(*calls) != 1 {
t.Fatalf("told the operator %d times, want exactly 1: %v", len(*calls), *calls)
}
call := (*calls)[0]
if call["headline"] != "A password was entered for telegram's telegram token on anchor" {
t.Errorf("headline %v", call["headline"])
}
if call["subject"] != "anchor" {
t.Errorf("subject %v", call["subject"])
}
lines, _ := call["lines"].([]string)
if len(lines) != 2 || !strings.Contains(lines[0], "at the desk on laptop at") ||
!strings.Contains(lines[0], "asked by mail-seats-835@novox") ||
lines[1] != "If that was not you, give it again yourself at the controller's terminal." {
t.Errorf("lines %v", lines)
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
if len(open) != 0 {
t.Errorf("a condition was raised though the operator was told: %+v", open)
}
}
// novox/hq ADR 0380 §9: until the operator channel's `tell` is delivered — nothing yet serves it, or the
// controller's own grant is refused — the secret-given condition is raised exactly as before.
func TestASecretGivenAtTheDeskRaisesTheConditionWhileNothingServesTell(t *testing.T) {
ctx := context.Background()
for _, failure := range []error{link.ErrNothingServes, link.ErrGrantRefused} {
k, _ := withConditionsInMemory(t)
askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, failure })
d, _, _, _ := aDesk(t, sealedTo(t, typed))
d.announce = announceWithRealTell(ctx, "mesh-mail-835, novox")
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
if len(open) != 1 || open[0].Kind != kindSecretGiven {
t.Fatalf("%v: the condition was not raised as before: %+v", failure, open)
}
}
}
// novox/hq ADR 0380 §10: once the operator channel is told successfully, every secret-given condition
// still open from before this change is retired, without a resolved line of its own.
func TestASuccessfulTellRetiresEveryOpenSecretGivenCondition(t *testing.T) {
ctx := context.Background()
k, _ := withConditionsInMemory(t)
for _, module := range []string{"telegram", "mesh-mail"} {
if _, err := k.Observe(ctx, secretGivenObservation("anchor", module, "a-secret", "at the desk on laptop",
time.Now())); err != nil {
t.Fatal(err)
}
}
askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, nil })
if err := announceSecretGiven(ctx, "anchor", "telegram", "telegram-token", "at the desk on laptop", "me"); err != nil {
t.Fatal(err)
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
if len(open) != 0 {
t.Fatalf("the open secret-given conditions were not retired: %+v", open)
}
}
// novox/hq ADR 0380 clause 7: a trusted party's secret given at the controller's terminal is announced
// before it is kept, and not kept when the announcement fails. It still fails when telling the operator
// AND raising the fallback condition both fail — nothing silently keeps a trusted party's value unheard.
func TestATrustedPartysSecretAtTheTerminalIsNotKeptWhenTellAndTheConditionBothFail(t *testing.T) {
ctx := context.Background()
askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, link.ErrNothingServes })
// conditionsFrom is left nil and there is no bus to dial here, so the fallback condition cannot be
// raised either: both of ADR 0380's ways to tell the operator fail.
kept := false
if _, _, err := keepGiven(true,
func() error {
return announceSecretGiven(ctx, "anchor", "telegram", "telegram-token", "at the controller's terminal", "me")
},
func() (bool, error) { kept = true; return false, nil }); err == nil {
t.Error("a trusted party's secret was kept though telling the operator and raising the condition both failed")
}
if kept {
t.Error("the value was kept despite the failure")
}
}
// Review round 2 of mesh-controller (feat/mail-seats, head 5f24a47d): ADR 0380 §9's fallback — raising the
// secret-given condition — answers only link.ErrNothingServes and link.ErrGrantRefused. Any other failure of
// the operator channel's `tell` (the holder itself refusing, or answering something unreadable) is the
// announcement's own failure, surfaced to the caller, and opens no condition.
func TestAnOperatorChannelFailureOtherThanNothingServesOrGrantRefusedSurfacesAndRaisesNoCondition(t *testing.T) {
ctx := context.Background()
k, _ := withConditionsInMemory(t)
refused := errors.New("operator-channel.tell refused: too many lines")
askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, refused })
if err := announceSecretGiven(ctx, "anchor", "telegram", "telegram-token", "at the desk on laptop", "me"); err != refused {
t.Fatalf("announceSecretGiven = %v, want the refusal itself surfaced", err)
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
if len(open) != 0 {
t.Errorf("a condition was raised though the failure was neither ErrNothingServes nor ErrGrantRefused: %+v", open)
}
}
// Review round 2 of mesh-controller: a secret accepted at the controller's terminal writes a hand act naming
// who asked, where and when it was given — and never the value (novox/hq ADR 0380 §6).
func TestASecretAcceptedAtTheTerminalWritesAHandActNamingWhoWhereAndNeverTheValue(t *testing.T) {
act := terminalGiveHandAct("anchor", "telegram", "telegram-token", "mail-seats-835@novox")
if act.Verb != "secret accept" || act.Cause != "given-at-the-terminal" {
t.Fatalf("act %+v", act)
}
if !strings.Contains(act.Why, "at the controller's terminal") {
t.Errorf("the act does not name where: %q", act.Why)
}
if !strings.Contains(act.Why, "asked by mail-seats-835@novox") {
t.Errorf("the act does not name who: %q", act.Why)
}
if strings.Contains(act.Why, typed) {
t.Error("the act carries the value")
}
// When is RecordHandAct's own job (it stamps At with time.Now when the act leaves none, and By with the
// caller when the act leaves that empty too): this act leaves both for it to fill.
if act.By != "" || !act.At.IsZero() {
t.Errorf("the act sets its own who/when instead of leaving them to RecordHandAct: %+v", act)
}
}
+5
View File
@@ -146,6 +146,11 @@ var probeRegistry = []probe{
{ID: probeFillID, Asserts: "every module's bucket and log holds less than three quarters of its cap; one " + {ID: probeFillID, Asserts: "every module's bucket and log holds less than three quarters of its cap; one " +
"said filling is cleared once it holds less than seven tenths", From: "ADR 0297, issue 501", "said filling is cleared once it holds less than seven tenths", From: "ADR 0297, issue 501",
Kind: kindBucketOrLogFilling, Raises: []string{kindBucketOrLogUnread}, Phase: 1, run: probeFill}, Kind: kindBucketOrLogFilling, Raises: []string{kindBucketOrLogUnread}, Phase: 1, run: probeFill},
// A grant of a restricted stream that composes nothing (novox/hq ADR 0376 §2): the module is carried by the
// machine's tool runner, which never hears a restricted stream, so it hears nothing it was granted.
{ID: probeWithheldID, Asserts: "no grant of a restricted stream names a module that composes nothing for it: every " +
"module granted one runs as an account of its own", From: "ADR 0376 §2", Kind: kindGrantComposesNothing,
Phase: 1, run: probeWithheldHearing},
{ID: "DW", Asserts: "the watchdogs of the signals table ran within three of their intervals", {ID: "DW", Asserts: "the watchdogs of the signals table ran within three of their intervals",
From: "ADR 0227 rule 6: the watchers are watched", Kind: "watchdogs-silent", Phase: 1, run: probeWatchdogs}, From: "ADR 0227 rule 6: the watchers are watched", Kind: "watchdogs-silent", Phase: 1, run: probeWatchdogs},
// The core's health definitions (novox/hq to-be 45 §8, ADR 0236): what a core component's new build is // The core's health definitions (novox/hq to-be 45 §8, ADR 0236): what a core component's new build is
+2 -2
View File
@@ -348,11 +348,11 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
if layer.From == catalogue.MeshWideLayer { if layer.From == catalogue.MeshWideLayer {
if !meshWide[module] { if !meshWide[module] {
meshWide[module] = true meshWide[module] = true
f.Settings = append(f.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Values)}) f.Settings = append(f.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Whole())})
} }
continue continue
} }
m.Settings = append(m.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Values)}) m.Settings = append(m.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Whole())})
} }
} }
m.Declaration = declarationFacts(ctx, open, n.Name, gens, gensErr, scrub) m.Declaration = declarationFacts(ctx, open, n.Name, gens, gensErr, scrub)
+10 -3
View File
@@ -22,6 +22,7 @@ func foundDirectory(module string, since time.Time) inventory.ResourceHealth {
} }
func TestADirectoryFoundBeforeTheSendIsAWaitForAPerson(t *testing.T) { func TestADirectoryFoundBeforeTheSendIsAWaitForAPerson(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
now := time.Now() now := time.Now()
sent := now.Add(-time.Minute) sent := now.Add(-time.Minute)
f := gateFacts{now: now, health: map[string]inventory.NodeHealth{"laptop": {Node: "laptop", HeardAt: now, f := gateFacts{now: now, health: map[string]inventory.NodeHealth{"laptop": {Node: "laptop", HeardAt: now,
@@ -48,6 +49,7 @@ func TestADirectoryFoundBeforeTheSendIsAWaitForAPerson(t *testing.T) {
// The whole walk: a module whose directory was used as found long before still gets its fix to every machine, // The whole walk: a module whose directory was used as found long before still gets its fix to every machine,
// its pass kept and the wait said; a directory this very send found holds it and puts it back. // its pass kept and the wait said; a directory this very send found holds it and puts it back.
func TestAFixGoesThroughPastADirectoryFoundBefore(t *testing.T) { func TestAFixGoesThroughPastADirectoryFoundBefore(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
for _, c := range []struct { for _, c := range []struct {
name string name string
found time.Duration // when the directory was found, against now found time.Duration // when the directory was found, against now
@@ -81,9 +83,13 @@ func TestAFixGoesThroughPastADirectoryFoundBefore(t *testing.T) {
} }
return f, err return f, err
} }
wasSettle, wasEvery, wasBound := gateSettle, gateEvery, gateBound wasWithin, wasEvery, wasBound := gateWithin, gateEvery, gateBound
t.Cleanup(func() { gateSettle, gateEvery, gateBound = wasSettle, wasEvery, wasBound }) // The witness window is the node-engine's, from a start no test makes (novox/hq ADR 0351).
gateSettle, gateEvery, gateBound = 0, 0, 300*time.Millisecond wasWindow := gateWitnessWindow
gateWitnessWindow = 0
t.Cleanup(func() { gateWitnessWindow = wasWindow })
t.Cleanup(func() { gateWithin, gateEvery, gateBound = wasWithin, wasEvery, wasBound })
gateWithin, gateEvery, gateBound = time.Hour, 0, 300*time.Millisecond
deadline := time.Now().Add(5 * time.Second) deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) { for time.Now().Before(deadline) {
advancePlans(ctx, b.open) advancePlans(ctx, b.open)
@@ -112,6 +118,7 @@ func TestAFixGoesThroughPastADirectoryFoundBefore(t *testing.T) {
// The condition says the wait in its own kind: the operator's, never urgent, and cleared once handed over. // The condition says the wait in its own kind: the operator's, never urgent, and cleared once handed over.
func TestADirectoryUsedAsFoundIsItsOwnCondition(t *testing.T) { func TestADirectoryUsedAsFoundIsItsOwnCondition(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
k, _ := withConditionsInMemory(t) k, _ := withConditionsInMemory(t)
ctx := t.Context() ctx := t.Context()
rs := map[string][]inventory.ResourceHealth{"notes": {foundDirectory("notes", time.Now().Add(-time.Hour))}} rs := map[string][]inventory.ResourceHealth{"notes": {foundDirectory("notes", time.Now().Add(-time.Hour))}}
+248 -37
View File
@@ -28,9 +28,12 @@ import (
// that machine reported it applied. A build that applies and then does nothing, crashes, serves no // that machine reported it applied. A build that applies and then does nothing, crashes, serves no
// tools, or breaks the machine's word to the mesh passed that test. Now the first machine is judged by // tools, or breaks the machine's word to the mesh passed that test. Now the first machine is judged by
// the component's health — the core's health definitions, or a module's own — within ten minutes of the // the component's health — the core's health definitions, or a module's own — within ten minutes of the
// apply: a core module passing three times over at least two minutes, a leaf module once, as soon as the // apply, each module by its rule (gateRuleOf): the node-engine and the mesh's resolver passing on their third
// node-engine states it healthy (novox/hq ADR 0282 decision 4 as ADR 0304 decision 3 changed it). Only then // healthy reading, and never later than two minutes after the send while healthy; the controller and the tool runner once, no earlier than the end of the
// are the rest sent. // node-engine's witness window for it; mesh-delivery and a leaf module once, as soon as the node-engine states
// it healthy (novox/hq ADR 0351 and ADR 0352, changing ADR 0236 decision 3, ADR 0282 decision 4 and ADR 0304
// decision 3).
// Only then are the rest sent.
// //
// **A failing gate stops the plan and puts the previous build back there.** The build is marked failed // **A failing gate stops the plan and puts the previous build back there.** The build is marked failed
// at its gate — registration refuses it and nothing sends it again on its own — the module's registered // at its gate — registration refuses it and nothing sends it again on its own — the module's registered
@@ -41,22 +44,30 @@ import (
// The gate's bounds (to-be 45 §8). Variables so a test can judge in a second, not in minutes. // The gate's bounds (to-be 45 §8). Variables so a test can judge in a second, not in minutes.
var ( var (
// gateSettle is how long the first machine must stay healthy, at the least. // gateWithin is the longest a module under the three-reading rule that stays healthy is held: at it, the
gateSettle = 2 * time.Minute // gate passes on the healthy readings it has, the newest healthy and none since the send not (novox/hq ADR
// 0352). Readings gateEvery apart, the first at the apply, put three inside it.
gateWithin = 2 * time.Minute
// gateBound is how long after the apply the build may take to become healthy. // gateBound is how long after the apply the build may take to become healthy.
gateBound = 10 * time.Minute gateBound = 10 * time.Minute
// gatePasses is how many consecutive judgings must find a core module healthy, gateEvery apart at the // gatePasses is how many consecutive judgings must find such a core module healthy, gateEvery apart at
// least, after gateSettle. gateLeafPasses is a leaf module's: one, taken once the node-engine states every // the least, counted from the send: it passes on the third (novox/hq ADR 0352). gateLeafPasses is a leaf module's: one, taken once the
// resource of it healthy by its own check, with no settle (novox/hq ADR 0282 decision 4 as ADR 0304 // node-engine states every resource of it healthy by its own check, with no settle (novox/hq ADR 0282
// decision 3 changed it, issue 524). // decision 4 as ADR 0304 decision 3 changed it, issue 524).
gatePasses = 3 gatePasses = 3
gateLeafPasses = 1 gateLeafPasses = 1
gateLeafSettles = false gateLeafSettles = false
gateEvery = 40 * time.Second gateEvery = 40 * time.Second
// gateWitnessWindow is the node-engine's witness window for a new controller or tool runner, from its
// start (ADR 0236 decision 27, lease.ControllerWithin and lease.NodeToolsWithin): a reading of either
// counts only after it has ended, so the build is judged by the witness and the gate both (novox/hq ADR
// 0351 decisions 1 and 4).
gateWitnessWindow = max(lease.ControllerWithin, lease.NodeToolsWithin)
) )
// coreModule is whether a module is judged as the core (novox/hq ADR 0282 decision 1): one on the controller's // coreModule is whether a module is judged as the core (novox/hq ADR 0282 decision 1): one on the controller's
// own path, or one holding the mesh's resolver. Every other module is a leaf. // own path, or one holding the mesh's resolver. Every other module is a leaf. Its class sets the module's
// delivery budget; how many readings its gate takes is gateRuleOf's.
func coreModule(module string, m catalogue.Manifest) bool { func coreModule(module string, m catalogue.Manifest) bool {
if _, own := onTheControllersPath[module]; own { if _, own := onTheControllersPath[module]; own {
return true return true
@@ -69,12 +80,72 @@ func coreModule(module string, m catalogue.Manifest) bool {
return coreComponent(module) != "" return coreComponent(module) != ""
} }
// gateWants is how many passes a module needs, and whether it must stay healthy gateSettle first. // gateRule is how a module's first-node gate judges it: passes healthy readings, gateEvery apart at the
func gateWants(module string, m catalogue.Manifest) (passes int, settles bool) { // least, counted from the send or, where witnessed, from the end of the node-engine's witness window for it;
if coreModule(module, m) { // and, where within is set, passed at within after the send on fewer, if it was healthy at every reading
return gatePasses, true // (novox/hq ADR 0352). said names the rule in the gate's text.
type gateRule struct {
passes int
within time.Duration
witnessed bool
said string
}
// gateRuleOf is a module's gate rule (novox/hq ADR 0351): a core module keeps three readings only where its
// build, broken on every machine, could not receive its own fix — the node-engine, which applies every send,
// and the mesh's resolver, which gives every machine the bus's name (decisions 2, 3 and 6); the bus is never
// walked through a gate but kept so (decision 7). The controller, run on one machine, and the tool runner,
// whose fix the node-engine applies, are judged by one reading after the witness window (decisions 1 and 4);
// mesh-delivery, whose fix goes the controller's own path, by one (decision 5); a leaf module by one (ADR 0304
// decision 3).
func gateRuleOf(module string, m catalogue.Manifest) gateRule {
switch {
case coreComponent(module) == lease.ComponentController:
return gateRule{passes: 1, witnessed: true, said: "the controller's rule: one reading once it is ready, " +
"after the node-engine's witness window (novox/hq ADR 0351)"}
case coreComponent(module) == lease.ComponentNodeTools:
return gateRule{passes: 1, witnessed: true, said: "the tool runner's rule: one reading once it answers, " +
"after the node-engine's witness window (novox/hq ADR 0351)"}
case module == catalogue.DeliverySeat:
return gateRule{passes: 1, said: "mesh-delivery's rule: one reading (novox/hq ADR 0351)"}
case coreModule(module, m):
return gateRule{passes: gatePasses, within: gateWithin, said: fmt.Sprintf("the core's rule: %d readings, "+
"or healthy throughout %s after the send (novox/hq ADR 0352)", gatePasses, gateWithin)}
} }
return gateLeafPasses, gateLeafSettles r := gateRule{passes: gateLeafPasses, said: "a leaf module's rule: one reading (novox/hq ADR 0304)"}
if gateLeafSettles {
r.within = gateWithin
r.said = fmt.Sprintf("a leaf module's rule, judged as the core: %d readings, or healthy throughout %s after the send",
gateLeafPasses, gateWithin)
}
return r
}
// gateFloor is the earliest a module's reading counts by its rule: the send and, for a witnessed module, the
// end of the node-engine's witness window on every machine judged (novox/hq ADR 0351 decisions 1 and 4). A
// module under the three-reading rule counts from the send (novox/hq ADR 0352). The window opens when the node-engine starts the build, which is before the machine reports it
// applied (applied, as the gate first read it) and, for the controller, before the new controller took the
// lease (holder): the later of those, the send's time at the least, plus the window, is never before its end.
// A controller that starts again takes the lease again, and so moves its floor on. Pure.
func gateFloor(module string, rule gateRule, since time.Time, machines []string, applied map[string]time.Time,
holder *lease.Holder) time.Time {
floor := since
if !rule.witnessed {
return floor
}
for _, n := range machines {
start := since
if at, read := applied[n]; read && at.After(start) {
start = at
}
if coreComponent(module) == lease.ComponentController && holder != nil && holder.Taken.After(start) {
start = holder.Taken
}
if end := start.Add(gateWitnessWindow); end.After(floor) {
floor = end
}
}
return floor
} }
// gateProbe is the registry's row for the gate's verdicts and the witnesses' rollbacks: its conditions // gateProbe is the registry's row for the gate's verdicts and the witnesses' rollbacks: its conditions
@@ -164,6 +235,10 @@ type gateFacts struct {
sent map[string]inventory.SentDeclaration sent map[string]inventory.SentDeclaration
sentBuilds map[string]map[string]string sentBuilds map[string]map[string]string
commits map[string]string commits map[string]string
// provisioned answers each machine's provisioner client's newest report of a module (novox/hq ADR 0349 §15):
// a module the gate's send handed to the provisioner is healthy only once it reported that build placed. Nil
// reads as nothing said.
provisioned inventory.ProvLookup
} }
// heldReading is the provider a module's findings are held under, as "<module> on <machine>", and, when that // heldReading is the provider a module's findings are held under, as "<module> on <machine>", and, when that
@@ -220,7 +295,7 @@ func (f gateFacts) supersededOn(module, machine string) (string, bool) {
var gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) { var gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
inv := open.inventory inv := open.inventory
f := gateFacts{now: time.Now(), reports: map[string]inventory.Reported{}, engines: map[string]string{}, f := gateFacts{now: time.Now(), reports: map[string]inventory.Reported{}, engines: map[string]string{},
rolledBack: witnessed.all()} rolledBack: witnessed.all(), provisioned: provLookup(ctx)}
reports, err := inv.LastReports(ctx) reports, err := inv.LastReports(ctx)
if err != nil { if err != nil {
return f, err return f, err
@@ -297,6 +372,12 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
case r.Outcome != inventory.OutcomeApplied: case r.Outcome != inventory.OutcomeApplied:
return healthNotYet, fmt.Sprintf("%s reported %q", machine, r.Outcome) return healthNotYet, fmt.Sprintf("%s reported %q", machine, r.Outcome)
} }
// **And its files placed** (novox/hq ADR 0349 §15): a module the send handed to the provisioner is judged only
// once that machine's client reported the build's templates placed, so a module whose files are not yet written
// is never judged; a failure there is the build's on that machine.
if h, why := judgeProvisioned(module, machine, r, f); h != healthGood {
return h, why
}
// **No new condition about it**: about the machine itself, or naming the module on that machine, // **No new condition about it**: about the machine itself, or naming the module on that machine,
// raised since the judging began. The gate's own are not evidence about the build. A fault that was // raised since the judging began. The gate's own are not evidence about the build. A fault that was
// there at the send and reopened since is not new; one that had cleared before the send and came back // there at the send and reopened since is not new; one that had cleared before the send and came back
@@ -378,6 +459,32 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
return healthGood, "" return healthGood, ""
} }
// judgeProvisioned is whether a module's files are placed on a machine, for a module the gate's send handed to the
// provisioner there: healthy for any other.
func judgeProvisioned(module, machine string, r inventory.Reported, f gateFacts) (health, string) {
sent, kept := f.sent[machine]
digest := sent.Provisioned[module]
if !kept || digest == "" {
return healthGood, ""
}
at := time.Time{}
if r.At != nil {
at = *r.At
}
one := inventory.SentDeclaration{Digest: sent.Digest, At: sent.At, Provisioned: map[string]string{module: digest}}
got, ok := inventory.ProvisionedSend(machine, one, inventory.AppliedReport{At: at,
Outcome: inventory.OutcomeApplied}, true, f.provisioned)
switch {
case !ok:
return healthNotYet, fmt.Sprintf("the provisioner on %s has not reported %s's files placed for the new "+
"build", machine, module)
case got.Outcome != inventory.OutcomeApplied:
return healthBroken, fmt.Sprintf("the provisioner on %s did not place %s's files: %s", machine, module,
got.Outcome)
}
return healthGood, ""
}
// machineWord is what one judging found wrong with a machine itself, apart from its modules: facts is // machineWord is what one judging found wrong with a machine itself, apart from its modules: facts is
// the judging's facts without those conditions, on what each names among the modules the send moved, // the judging's facts without those conditions, on what each names among the modules the send moved,
// and whole what holds the machine back as a whole. // and whole what holds the machine back as a whole.
@@ -501,6 +608,13 @@ func judgeGate(ctx context.Context, open *stores, p *inventory.Plan, module stri
return judgeMoves(ctx, open, g, pairs, now) return judgeMoves(ctx, open, g, pairs, now)
} }
// dueAtTheMost says a gate healthy since its send has reached gateWithin and not been judged since: the one
// judging that may come sooner than gateEvery after the last pass (novox/hq ADR 0352).
func dueAtTheMost(g *inventory.PlanGate, now time.Time) bool {
most := g.Since.Add(gateWithin)
return !g.Faulted && g.LastPass != nil && !now.Before(most) && g.LastPass.Before(most)
}
// judged is one module on one machine, as a gate judges it. // judged is one module on one machine, as a gate judges it.
type judged struct{ module, node string } type judged struct{ module, node string }
@@ -511,7 +625,9 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
if g.Verdict != "" { if g.Verdict != "" {
return g.Verdict, nil return g.Verdict, nil
} }
if g.LastPass != nil && now.Sub(*g.LastPass) < gateEvery { // A judging comes gateEvery after the last pass, and once at gateWithin after the send whatever the spacing:
// a module healthy throughout is never held past it (novox/hq ADR 0352).
if g.LastPass != nil && now.Sub(*g.LastPass) < gateEvery && !dueAtTheMost(g, now) {
return "", nil return "", nil
} }
for _, c := range g.Carried { for _, c := range g.Carried {
@@ -544,6 +660,11 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
facts.sentBuilds[j.node] = builds facts.sentBuilds[j.node] = builds
} }
} }
// **A busy container on a judged machine holds the judging before it starts** (novox/hq ADR 0335 §4): the
// walk says it waits there, and its gate starts at the report that says the container replaced.
if busyWait(g, pairs, facts, now) {
return "", nil
}
// **What is wrong with a machine itself is the machine's** (novox/hq issue 281): read once for each // **What is wrong with a machine itself is the machine's** (novox/hq issue 281): read once for each
// machine judged, apart from what is wrong with a module there, and never pinned on the module the // machine judged, apart from what is wrong with a module there, and never pinned on the module the
// gate happens to be kept on. // gate happens to be kept on.
@@ -568,6 +689,9 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
for _, j := range pairs { for _, j := range pairs {
w := words[j.node] w := words[j.node]
h, said := judgeHealth(j.module, coreComponent(j.module), shelf[j.module], j.node, *g.Since, w.facts) h, said := judgeHealth(j.module, coreComponent(j.module), shelf[j.module], j.node, *g.Since, w.facts)
if why, still := busyStill(j, facts); still && (h == healthGood || h == healthPerson) {
h, said = healthNotYet, why
}
if h == healthGood || h == healthPerson { if h == healthGood || h == healthPerson {
if on, named := w.on[j.module]; named { if on, named := w.on[j.module]; named {
h, said = healthNotYet, on h, said = healthNotYet, on
@@ -608,6 +732,51 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
why = said why = said
} }
} }
// **Each module counts its readings by its own rule** (novox/hq ADR 0351): from when the gate first read
// each machine report its send applied, and who holds the controller lease, the earliest its reading counts.
if g.Applied == nil {
g.Applied = map[string]time.Time{}
}
for node := range byMachine {
r, said := facts.reports[node]
if _, read := g.Applied[node]; !read && said && r.At != nil && r.Outcome == inventory.OutcomeApplied &&
facts.reportedOn(node, r) {
g.Applied[node] = *r.At
}
}
rules, floors := map[string]gateRule{}, map[string]time.Time{}
for _, m := range modules {
var on []string
for _, j := range pairs {
if j.module == m {
on = append(on, j.node)
}
}
rules[m] = gateRuleOf(m, shelf[m])
floors[m] = gateFloor(m, rules[m], *g.Since, on, g.Applied, facts.holder)
}
tallyGate(g, gateJudging{modules: modules, reading: reading, waits: waits, worst: worst, why: why,
rules: rules, floors: floors}, now)
return g.Verdict, nil
}
// gateJudging is one judging's readings, as tallyGate counts them: the modules judged, each one's reading
// (the worst of its machines'), its waits for a person, the send's worst reading and why, and each module's
// rule and the earliest its reading counts.
type gateJudging struct {
modules []string
reading map[string]health
waits map[string]string
worst health
why string
rules map[string]gateRule
floors map[string]time.Time
}
// tallyGate counts one judging into the gate: a pass counted, a pass missed (and why), or the verdict. Pure but
// for the gate it writes.
func tallyGate(g *inventory.PlanGate, j gateJudging, now time.Time) {
modules, reading, waits, worst, why := j.modules, j.reading, j.waits, j.worst, j.why
if g.Healthy == nil { if g.Healthy == nil {
g.Healthy = map[string]int{} g.Healthy = map[string]int{}
} }
@@ -623,38 +792,53 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
delete(g.HealthyAt, m) delete(g.HealthyAt, m)
continue continue
} }
// **A pass is counted only gateEvery after the one before** (issue 318 review): a send judged more
// often while another module beside it is not yet healthy counts no faster.
if at, counted := g.HealthyAt[m]; !counted || now.Sub(at) >= gateEvery {
g.Healthy[m]++
g.HealthyAt[m] = now
}
if w := waits[m]; w != "" { if w := waits[m]; w != "" {
if g.Waits == nil { if g.Waits == nil {
g.Waits = map[string]string{} g.Waits = map[string]string{}
} }
g.Waits[m] = w g.Waits[m] = w
} }
// **A reading counts only from the module's floor** (novox/hq ADR 0351 decisions 1 and 4): the
// controller's and the tool runner's after the witness window, any other module's from the send.
if now.Before(j.floors[m]) {
continue
}
// **A pass is counted only gateEvery after the one before** (issue 318 review): a send judged more
// often while another module beside it is not yet healthy counts no faster.
if at, counted := g.HealthyAt[m]; !counted || now.Sub(at) >= gateEvery {
g.Healthy[m]++
g.HealthyAt[m] = now
}
} }
settled := now.Sub(*g.Since) >= gateSettle // What the send needs is what its most demanding module needs — the most readings, then the latest
// What the send needs is what its most demanding module needs: a send carrying a core module is judged // floor: a send carrying the node-engine is judged by its three readings, or by being healthy throughout
// as the core. // its two minutes (novox/hq ADR 0352).
wants, settles := 0, false wants, floor, within, said := 0, time.Time{}, time.Duration(0), ""
for _, m := range modules { for _, m := range modules {
p, s := gateWants(m, shelf[m]) r, f := j.rules[m], j.floors[m]
wants, settles = max(wants, p), settles || s if r.passes > wants || (r.passes == wants && f.After(floor)) {
wants, within, said = r.passes, r.within, r.said
}
if f.After(floor) {
floor = f
}
} }
if len(modules) == 0 { if len(modules) == 0 {
// A send judged on no machine of its modules (its first machine not running them) has no class to // A send judged on no machine of its modules (its first machine not running them) has no class to
// be judged by: it is judged as the core, never passed at once without a reading. // be judged by: it is judged as the core, never passed at once without a reading.
wants, settles = gatePasses, true wants, floor, within = gatePasses, *g.Since, gateWithin
said = fmt.Sprintf("the core's rule: %d readings, or healthy throughout %s after the send", gatePasses, gateWithin)
}
// heldLongEnough is a rule's most: within after the send, healthy at every reading since it.
heldLongEnough := func(within time.Duration) bool {
return within > 0 && !g.Faulted && !now.Before(g.Since.Add(within))
} }
g.Wants = wants g.Wants = wants
// A module's own pass is its own class's: a leaf beside a core module that fails keeps its build after // A module's own pass is its own rule's: a leaf beside a core module that fails keeps its build after
// its one reading, and only what did not pass is put back (issue 318's rule, by class since issue 524). // its one reading, and only what did not pass is put back (issue 318's rule, by class since issue 524).
passedAlone := func(m string) bool { passedAlone := func(m string) bool {
p, s := gateWants(m, shelf[m]) r := j.rules[m]
return (settled || !s) && reading[m] <= healthPerson && g.Healthy[m] >= p return reading[m] <= healthPerson && (g.Healthy[m] >= r.passes || (g.Healthy[m] >= 1 && heldLongEnough(r.within)))
} }
// fail decides the send failed: what passed on its own keeps its pass, everything else is put back. // fail decides the send failed: what passed on its own keeps its pass, everything else is put back.
fail := func(why string) { fail := func(why string) {
@@ -671,6 +855,7 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
pastBound := now.Sub(*g.Since) > gateBound pastBound := now.Sub(*g.Since) > gateBound
switch { switch {
case worst == healthBroken: case worst == healthBroken:
g.Faulted = true
g.Read(now, false, g.BrokenWhy) g.Read(now, false, g.BrokenWhy)
var judging []string var judging []string
for _, m := range modules { for _, m := range modules {
@@ -689,25 +874,36 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
case worst == healthWaiting: case worst == healthWaiting:
// Waiting on a provider that is unhealthy: not a pass, and not a failure at the bound either — // Waiting on a provider that is unhealthy: not a pass, and not a failure at the bound either —
// the provider's own condition says what is wrong (ADR 0240 rule 5). // the provider's own condition says what is wrong (ADR 0240 rule 5).
g.Faulted = g.Faulted || g.HealthySeen // not healthy after a healthy reading; before one, still starting
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
g.Read(now, false, why) g.Read(now, false, why)
case worst == healthNotYet: case worst == healthNotYet:
g.Faulted = g.Faulted || g.HealthySeen // not healthy after a healthy reading; before one, still starting
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
g.Read(now, false, why) g.Read(now, false, why)
if pastBound { if pastBound {
fail(fmt.Sprintf("not healthy within %s of its apply: %s", gateBound, why)) fail(fmt.Sprintf("not healthy within %s of its apply: %s", gateBound, why))
} }
case now.Before(floor):
// Healthy before its rule lets a reading count: not a pass yet, and not a fault (novox/hq ADR 0351).
g.HealthySeen = true
// The bound still holds: a module that cannot be read healthy after its floor fails at it.
g.Last, g.Failing = fmt.Sprintf("healthy, and a reading counts from %s by %s",
floor.UTC().Format(time.RFC3339), said), nil
if pastBound {
fail(fmt.Sprintf("not judged within %s of its apply: %s", gateBound, g.Last))
}
default: default:
// Healthy, or waiting for a person (ADR 0254): a pass, the wait carried along in the verdict. // Healthy, or waiting for a person (ADR 0254): a pass, the wait carried along in the verdict.
g.HealthySeen = true
g.Passes++ g.Passes++
g.LastPass, g.Last, g.Failing = &now, "", nil g.LastPass, g.Last, g.Failing = &now, "", nil
g.Read(now, true, "") g.Read(now, true, "")
if g.Passes >= wants && (settled || !settles) { if g.Passes >= wants || heldLongEnough(within) {
decide(g, inventory.GatePassed, fmt.Sprintf("healthy %d times over %s", g.Passes, decide(g, inventory.GatePassed, fmt.Sprintf("healthy %d times over %s, by %s", g.Passes,
now.Sub(*g.Since).Round(time.Second))+waitsSaid(g.Waits), now) now.Sub(*g.Since).Round(time.Second), said)+waitsSaid(g.Waits), now)
} }
} }
return g.Verdict, nil
} }
// judgedCommits is the commit of each module a gate judges: the gate's own To for its module, and each // judgedCommits is the commit of each module a gate judges: the gate's own To for its module, and each
@@ -1283,6 +1479,12 @@ func probeGates(ctx context.Context, d *doctor) ([]conditions.Observation, error
} }
// A release held after a failed one, while builds still wait for a gate (ADR 0236). // A release held after a failed one, while builds still wait for a gate (ADR 0236).
out = append(out, backlogObservation()...) out = append(out, backlogObservation()...)
// And a rest that went silent (novox/hq ADR 0353 §4).
silent, err := silentObservation(ctx, d.open.inventory)
if err != nil {
return nil, err
}
out = append(out, silent...)
return sortedFound(dedupeObservations(out)), nil return sortedFound(dedupeObservations(out)), nil
} }
@@ -1372,6 +1574,15 @@ func gateLine(g *inventory.PlanGate) string {
case inventory.GateFailed: case inventory.GateFailed:
what = "FAILED" what = "FAILED"
} }
switch {
case g.Verdict == "" && g.Pending:
what = "kept, not yet sent" // novox/hq ADR 0353 §8
case g.Verdict == "" && g.Silent:
what = "silent on its send, left" // ADR 0353 §4
}
if g.Verdict == "" && g.Waiting() {
what = "waits for a busy container on " + g.Busy.Machine
}
line := fmt.Sprintf("gate on %s: %s", strings.Join(g.Machines, ", "), what) line := fmt.Sprintf("gate on %s: %s", strings.Join(g.Machines, ", "), what)
if g.Component != "" { if g.Component != "" {
line += " (core: " + g.Component + ")" line += " (core: " + g.Component + ")"
+18 -7
View File
@@ -21,11 +21,15 @@ import (
// wait, and a provider waiting for a login says who waits on it. // wait, and a provider waiting for a login says who waits on it.
// withGateBounds sets the gate's bounds for one test. // withGateBounds sets the gate's bounds for one test.
func withGateBounds(t *testing.T, settle, every, bound time.Duration) { func withGateBounds(t *testing.T, within, every, bound time.Duration) {
t.Helper() t.Helper()
wasSettle, wasEvery, wasBound := gateSettle, gateEvery, gateBound wasWithin, wasEvery, wasBound := gateWithin, gateEvery, gateBound
gateSettle, gateEvery, gateBound = settle, every, bound // The witness window is the node-engine's, from a start no test makes (novox/hq ADR 0351).
t.Cleanup(func() { gateSettle, gateEvery, gateBound = wasSettle, wasEvery, wasBound }) wasWindow := gateWitnessWindow
gateWitnessWindow = 0
t.Cleanup(func() { gateWitnessWindow = wasWindow })
gateWithin, gateEvery, gateBound = within, every, bound
t.Cleanup(func() { gateWithin, gateEvery, gateBound = wasWithin, wasEvery, wasBound })
} }
// factsOn is a judging's facts for one machine that applied its send: the node tools answer, and what it says // factsOn is a judging's facts for one machine that applied its send: the node tools answer, and what it says
@@ -53,10 +57,11 @@ func healthyContainer(module string) inventory.ResourceHealth {
// tools' witness put its build back at once, and the send waits for the healthy module's own verdict — a pass, // tools' witness put its build back at once, and the send waits for the healthy module's own verdict — a pass,
// counted over the gate's spacing — before it says its own. The broken one alone is put back. // counted over the gate's spacing — before it says its own. The broken one alone is put back.
func TestAFaultBeforeTheSettleTimeWaitsForTheModuleBesideIt(t *testing.T) { func TestAFaultBeforeTheSettleTimeWaitsForTheModuleBesideIt(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t) judgeLeavesAsTheCore(t)
open := aMesh(t) open := aMesh(t)
ctx := t.Context() ctx := t.Context()
withGateBounds(t, 150*time.Millisecond, 20*time.Millisecond, 10*time.Second) withGateBounds(t, time.Hour, 20*time.Millisecond, 10*time.Second)
since := time.Now().Add(-time.Second) since := time.Now().Add(-time.Second)
base := factsOn(t, "laptop", since, healthyContainer("app")) base := factsOn(t, "laptop", since, healthyContainer("app"))
wasGather := gatherGateFacts wasGather := gatherGateFacts
@@ -95,9 +100,10 @@ const broker_runtime = "node-tools"
// **A pass is counted only the gate's spacing after the one before** (review (c)): a send judged every tick // **A pass is counted only the gate's spacing after the one before** (review (c)): a send judged every tick
// while a module beside it is not yet healthy counts the healthy one once. // while a module beside it is not yet healthy counts the healthy one once.
func TestAPassIsCountedNoFasterThanTheGatesSpacing(t *testing.T) { func TestAPassIsCountedNoFasterThanTheGatesSpacing(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t) open := aMesh(t)
ctx := t.Context() ctx := t.Context()
withGateBounds(t, 0, time.Hour, time.Hour) withGateBounds(t, time.Hour, time.Hour, time.Hour)
since := time.Now().Add(-time.Minute) since := time.Now().Add(-time.Minute)
base := factsOn(t, "laptop", since, healthyContainer("app"), inventory.ResourceHealth{Module: "late", base := factsOn(t, "laptop", since, healthyContainer("app"), inventory.ResourceHealth{Module: "late",
Resource: "late.web", Kind: "container", Target: "late", State: link.StateUnhealthy, Reason: "down"}) Resource: "late.web", Kind: "container", Target: "late", State: link.StateUnhealthy, Reason: "down"})
@@ -119,6 +125,7 @@ func TestAPassIsCountedNoFasterThanTheGatesSpacing(t *testing.T) {
// **A carried build's pass carries its own wait, never another's** (review (e)). // **A carried build's pass carries its own wait, never another's** (review (e)).
func TestACarriedPassCarriesOnlyItsOwnWait(t *testing.T) { func TestACarriedPassCarriesOnlyItsOwnWait(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
b := aBacklog(t) b := aBacklog(t)
ctx := t.Context() ctx := t.Context()
inv := b.open.inventory inv := b.open.inventory
@@ -143,6 +150,7 @@ func TestACarriedPassCarriesOnlyItsOwnWait(t *testing.T) {
// carried its own module, healthy, and a build waiting on that machine that never became healthy. At the // carried its own module, healthy, and a build waiting on that machine that never became healthy. At the
// bound the waiting one is put back and marked; the plan's own module keeps its pass, so no walk waits on it. // bound the waiting one is put back and marked; the plan's own module keeps its pass, so no walk waits on it.
func TestThePlansOwnModuleKeepsItsPassWhenItsSendFails(t *testing.T) { func TestThePlansOwnModuleKeepsItsPassWhenItsSendFails(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
g := aGateMesh(t) g := aGateMesh(t)
ctx := t.Context() ctx := t.Context()
inv := g.open.inventory inv := g.open.inventory
@@ -223,6 +231,7 @@ func TestThePlansOwnModuleKeepsItsPassWhenItsSendFails(t *testing.T) {
// **A provider waiting for a login says who waits on it** (review (g)): its consumer, failing a check that // **A provider waiting for a login says who waits on it** (review (g)): its consumer, failing a check that
// needs it, is held under it, and the provider's relogin-needed condition lists it and is urgent. // needs it, is held under it, and the provider's relogin-needed condition lists it and is urgent.
func TestAProviderWaitingForALoginSaysWhoWaitsOnIt(t *testing.T) { func TestAProviderWaitingForALoginSaysWhoWaitsOnIt(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t) open := aMesh(t)
ctx := t.Context() ctx := t.Context()
inv := open.inventory inv := open.inventory
@@ -285,6 +294,7 @@ func TestAProviderWaitingForALoginSaysWhoWaitsOnIt(t *testing.T) {
// tools are marked and put back in the very judging that found them broken — their registered build is the // tools are marked and put back in the very judging that found them broken — their registered build is the
// one before, and the laptop is sent it — while `app` goes on being judged, recovers, and keeps its own pass. // one before, and the laptop is sent it — while `app` goes on being judged, recovers, and keeps its own pass.
func TestABrokenModuleIsPutBackAtOnceAndTheOneBesideItGetsItsOwnVerdict(t *testing.T) { func TestABrokenModuleIsPutBackAtOnceAndTheOneBesideItGetsItsOwnVerdict(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t) judgeLeavesAsTheCore(t)
open := aMesh(t) open := aMesh(t)
ctx := t.Context() ctx := t.Context()
@@ -376,7 +386,7 @@ func TestABrokenModuleIsPutBackAtOnceAndTheOneBesideItGetsItsOwnVerdict(t *testi
f.health = map[string]inventory.NodeHealth{"laptop": {Node: "laptop", HeardAt: now, Resources: []inventory.ResourceHealth{app}}} f.health = map[string]inventory.NodeHealth{"laptop": {Node: "laptop", HeardAt: now, Resources: []inventory.ResourceHealth{app}}}
return f, nil return f, nil
} }
withGateBounds(t, 100*time.Millisecond, 0, 10*time.Second) withGateBounds(t, time.Hour, 0, 10*time.Second)
release := func() inventory.Plan { release := func() inventory.Plan {
t.Helper() t.Helper()
@@ -427,6 +437,7 @@ func TestABrokenModuleIsPutBackAtOnceAndTheOneBesideItGetsItsOwnVerdict(t *testi
// **A move from a build not known excuses no wait** (issue 318 review): a plan's own module whose previous // **A move from a build not known excuses no wait** (issue 318 review): a plan's own module whose previous
// build was not recorded, or whose previous manifest is not kept, cannot be shown to have added the group. // build was not recorded, or whose previous manifest is not kept, cannot be shown to have added the group.
func TestAMoveFromAnUnknownBuildExcusesNoWait(t *testing.T) { func TestAMoveFromAnUnknownBuildExcusesNoWait(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t) open := aMesh(t)
ctx := t.Context() ctx := t.Context()
inv := open.inventory inv := open.inventory
+338
View File
@@ -0,0 +1,338 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
)
// The first-node gate's readings by module (novox/hq ADR 0351, ADR 0352): the controller and the tool runner pass
// on one reading after the node-engine's witness window, mesh-delivery on one, the node-engine and the mesh's
// resolver on their third healthy reading and never later than two minutes while healthy; the bus is kept so.
// withTheGatesTimes sets the gate's production bounds for a test, whatever another test left.
func withTheGatesTimes(t *testing.T) {
t.Helper()
wasWithin, wasEvery, wasBound, wasWindow := gateWithin, gateEvery, gateBound, gateWitnessWindow
wasPasses, wasLeafPasses, wasLeafSettles := gatePasses, gateLeafPasses, gateLeafSettles
gateWithin, gateEvery, gateBound, gateWitnessWindow = 2*time.Minute, 40*time.Second, 10*time.Minute, 60*time.Second
gatePasses, gateLeafPasses, gateLeafSettles = 3, 1, false
t.Cleanup(func() {
gateWithin, gateEvery, gateBound, gateWitnessWindow = wasWithin, wasEvery, wasBound, wasWindow
gatePasses, gateLeafPasses, gateLeafSettles = wasPasses, wasLeafPasses, wasLeafSettles
})
}
func TestEachModuleHasItsGateRule(t *testing.T) {
withTheGatesTimes(t)
for _, c := range []struct {
module string
claims []catalogue.Claim
passes int
within time.Duration
witnessed bool
said string
}{
{"mesh-controller", nil, 1, 0, true, "the controller's rule"},
{"node-tools", nil, 1, 0, true, "the tool runner's rule"},
{catalogue.DeliverySeat, nil, 1, 0, false, "mesh-delivery's rule"},
{"mesh-host", nil, 3, 2 * time.Minute, false, "the core's rule"},
{"dnsmasq", []catalogue.Claim{{Name: resolverSeat}}, 3, 2 * time.Minute, false, "the core's rule"},
{"nats", nil, 3, 2 * time.Minute, false, "the core's rule"},
{"app", nil, 1, 0, false, "a leaf module's rule"},
} {
r := gateRuleOf(c.module, catalogue.Manifest{Module: c.module, Claims: c.claims})
if r.passes != c.passes || r.within != c.within || r.witnessed != c.witnessed || !strings.HasPrefix(r.said, c.said) {
t.Errorf("%s: %+v; want %d readings, within %s, witnessed %v, said %q", c.module, r, c.passes, c.within,
c.witnessed, c.said)
}
}
}
// Every module that was core stays core for its delivery budget and its walk: only its readings change.
func TestTheCoreStaysTheCore(t *testing.T) {
for _, m := range []string{"mesh-controller", "mesh-host", "node-tools", "nats", catalogue.DeliverySeat} {
if !coreModule(m, catalogue.Manifest{Module: m}) {
t.Errorf("%s is no longer judged as the core for its class", m)
}
}
if !coreModule("dnsmasq", catalogue.Manifest{Module: "dnsmasq", Claims: []catalogue.Claim{{Name: resolverSeat}}}) {
t.Error("the resolver's holder is no longer judged as the core for its class")
}
}
// gateOn judges one module on one machine, healthy at every reading.
type gateOn struct {
module, node string
g *inventory.PlanGate
holder *lease.Holder
}
func aGateOn(module, node string, since time.Time) *gateOn {
return &gateOn{module: module, node: node, g: &inventory.PlanGate{Machines: []string{node}, Since: &since,
Applied: map[string]time.Time{}}}
}
func (o *gateOn) read(at time.Time, h health, why string) {
r := gateRuleOf(o.module, catalogue.Manifest{Module: o.module})
floor := gateFloor(o.module, r, *o.g.Since, []string{o.node}, o.g.Applied, o.holder)
tallyGate(o.g, gateJudging{modules: []string{o.module}, reading: map[string]health{o.module: h},
worst: h, why: why, rules: map[string]gateRule{o.module: r}, floors: map[string]time.Time{o.module: floor}}, at)
}
func TestTheControllerPassesOnOneReadingAfterTheWitnessWindow(t *testing.T) {
withTheGatesTimes(t)
sent := time.Date(2026, 10, 13, 20, 0, 0, 0, time.UTC)
o := aGateOn("mesh-controller", "novox", sent)
o.g.Applied["novox"] = sent.Add(15 * time.Second)
o.holder = &lease.Holder{Instance: "new", Taken: sent.Add(20 * time.Second)}
// Ready, but inside the witness window, which opened before it took the lease: not counted.
for _, after := range []time.Duration{30 * time.Second, 60 * time.Second, 79 * time.Second} {
o.read(sent.Add(after), healthGood, "")
if o.g.Verdict != "" || o.g.Passes != 0 || !strings.Contains(o.g.Last, "counts from") {
t.Fatalf("at %s: %+v; want no reading counted inside the witness window", after, o.g)
}
}
o.read(sent.Add(80*time.Second), healthGood, "")
if o.g.Verdict != inventory.GatePassed || o.g.Passes != 1 || o.g.Wants != 1 {
t.Fatalf("after the witness window: %+v; want passed on one reading", o.g)
}
if !strings.Contains(o.g.Why, "healthy 1 times") || !strings.Contains(o.g.Why, "the controller's rule") ||
!strings.Contains(o.g.Why, "ADR 0351") {
t.Errorf("the verdict says %q; want the rule it applied", o.g.Why)
}
}
// A controller that starts again takes the lease again: its witness window starts again, and so does the
// gate's wait.
func TestAControllerThatStartsAgainWaitsOutItsNewWindow(t *testing.T) {
withTheGatesTimes(t)
sent := time.Now()
o := aGateOn("mesh-controller", "novox", sent)
o.g.Applied["novox"] = sent.Add(10 * time.Second)
o.holder = &lease.Holder{Instance: "again", Taken: sent.Add(70 * time.Second)}
o.read(sent.Add(90*time.Second), healthGood, "")
if o.g.Verdict != "" {
t.Fatalf("passed %s after a start the witness still judges: %+v", 20*time.Second, o.g)
}
o.read(sent.Add(130*time.Second), healthGood, "")
if o.g.Verdict != inventory.GatePassed {
t.Fatalf("not passed after the new window: %+v", o.g)
}
}
// Without the lease's word (a controller carried by another gate), the window counts from the applied report,
// and from the send at the least.
func TestTheWitnessWindowCountsFromTheSendAtTheLeast(t *testing.T) {
withTheGatesTimes(t)
sent := time.Now()
r := gateRuleOf("mesh-controller", catalogue.Manifest{})
if f := gateFloor("mesh-controller", r, sent, []string{"novox"}, nil, nil); !f.Equal(sent.Add(time.Minute)) {
t.Errorf("floor %s after the send; want 1m", f.Sub(sent))
}
if f := gateFloor("node-tools", gateRuleOf("node-tools", catalogue.Manifest{}), sent, []string{"ace"},
map[string]time.Time{"ace": sent.Add(12 * time.Second)}, nil); !f.Equal(sent.Add(72 * time.Second)) {
t.Errorf("the tool runner's floor %s after the send; want 1m12s", f.Sub(sent))
}
}
// A controller that a witness put back, or that never becomes ready, fails its gate as before.
func TestAControllerThatFailsIsStillFailed(t *testing.T) {
withTheGatesTimes(t)
sent := time.Now()
o := aGateOn("mesh-controller", "novox", sent)
o.read(sent.Add(30*time.Second), healthBroken, "the witness on novox judged the controller and rolled-back")
if o.g.Verdict != inventory.GateFailed {
t.Fatalf("a controller put back by its witness: %+v; want failed", o.g)
}
o = aGateOn("mesh-controller", "novox", sent)
o.read(sent.Add(5*time.Minute), healthNotYet, "the controller holding the lease does not say it is ready")
if o.g.Verdict != "" {
t.Fatalf("failed before the bound: %+v", o.g)
}
o.read(sent.Add(10*time.Minute+time.Second), healthNotYet, "the controller holding the lease does not say it is ready")
if o.g.Verdict != inventory.GateFailed || !strings.Contains(o.g.Why, "not healthy within") {
t.Fatalf("never ready: %+v; want failed at the bound", o.g)
}
}
// threeReadings judges a module under the three-reading rule on one machine, the send at sent.
func threeReadings(t *testing.T, module string, sent time.Time) (*inventory.PlanGate, func(time.Duration, health)) {
t.Helper()
g := &inventory.PlanGate{Machines: []string{"ace"}, Since: &sent}
rule := gateRuleOf(module, catalogue.Manifest{Module: module, Claims: []catalogue.Claim{{Name: resolverSeat}}})
floor := gateFloor(module, rule, sent, []string{"ace"}, nil, nil)
return g, func(after time.Duration, h health) {
why := ""
if h != healthGood {
why = "the node-engine reports another build"
}
tallyGate(g, gateJudging{modules: []string{module}, reading: map[string]health{module: h}, worst: h, why: why,
rules: map[string]gateRule{module: rule}, floors: map[string]time.Time{module: floor}}, sent.Add(after))
}
}
// A healthy node-engine or resolver passes at its third healthy reading, 40 s apart from the apply: before two
// minutes (novox/hq ADR 0352).
func TestTheNodeEngineAndTheResolverPassAtTheirThirdReading(t *testing.T) {
withTheGatesTimes(t)
for _, module := range []string{"mesh-host", "dnsmasq"} {
g, judge := threeReadings(t, module, time.Now())
judge(15*time.Second, healthGood)
judge(55*time.Second, healthGood)
if g.Verdict != "" || g.Passes != 2 {
t.Fatalf("%s: after two readings %+v; want still judging", module, g)
}
judge(95*time.Second, healthGood)
if g.Verdict != inventory.GatePassed || g.Passes != 3 || g.Wants != 3 ||
!strings.Contains(g.Why, "over 1m35s") || !strings.Contains(g.Why, "the core's rule: 3 readings") {
t.Fatalf("%s: after three readings %+v; want passed at 1m35s by the core's rule", module, g)
}
}
}
// One healthy throughout whose readings came late passes at two minutes on the readings it has; the judging at
// two minutes comes sooner than the spacing after the last pass.
func TestAHealthyNodeEngineIsNeverHeldPastTwoMinutes(t *testing.T) {
withTheGatesTimes(t)
sent := time.Now()
last := sent.Add(100 * time.Second)
held := &inventory.PlanGate{Since: &sent, LastPass: &last}
if dueAtTheMost(held, sent.Add(119*time.Second)) || !dueAtTheMost(held, sent.Add(2*time.Minute)) {
t.Fatal("the judging at two minutes is not due, or is due before")
}
held.Faulted = true
if dueAtTheMost(held, sent.Add(2*time.Minute)) {
t.Fatal("a send that was not healthy at a reading is judged at two minutes")
}
g, judge := threeReadings(t, "mesh-host", time.Now())
judge(100*time.Second, healthGood)
judge(2*time.Minute, healthGood)
if g.Verdict != inventory.GatePassed || g.Passes != 2 {
t.Fatalf("late readings, healthy throughout, at two minutes: %+v; want passed on two", g)
}
g, judge = threeReadings(t, "mesh-host", time.Now())
judge(2*time.Minute, healthGood)
if g.Verdict != inventory.GatePassed || g.Passes != 1 {
t.Fatalf("one healthy reading, at two minutes: %+v; want passed on it", g)
}
}
// A reading not yet healthy before the first healthy one is the build still starting: the two minutes still hold.
func TestANodeEngineStillStartingKeepsItsTwoMinutes(t *testing.T) {
withTheGatesTimes(t)
g, judge := threeReadings(t, "mesh-host", time.Now())
judge(10*time.Second, healthNotYet)
judge(100*time.Second, healthGood)
judge(2*time.Minute, healthGood)
if g.Verdict != inventory.GatePassed || g.Faulted {
t.Fatalf("%+v; want passed at two minutes", g)
}
}
// A reading that is not healthy after a healthy one takes the two minutes away: the module needs its three readings, and one
// never healthy fails at the bound and is put back; one found broken fails.
func TestANodeEngineThatFailsAReadingIsNotPassedAtTwoMinutes(t *testing.T) {
withTheGatesTimes(t)
g, judge := threeReadings(t, "mesh-host", time.Now())
judge(20*time.Second, healthNotYet)
judge(60*time.Second, healthGood)
judge(100*time.Second, healthGood)
judge(2*time.Minute+10*time.Second, healthGood)
if g.Verdict != inventory.GatePassed || g.Passes != 3 {
t.Fatalf("%+v; want passed only on its third healthy reading", g)
}
g, judge = threeReadings(t, "mesh-host", time.Now())
judge(60*time.Second, healthGood)
judge(100*time.Second, healthNotYet)
judge(2*time.Minute, healthGood)
if g.Verdict != "" || !g.Faulted {
t.Fatalf("passed at two minutes after a reading that was not healthy: %+v", g)
}
judge(10*time.Minute+time.Second, healthNotYet)
if g.Verdict != inventory.GateFailed || !strings.Contains(g.Why, "not healthy within") {
t.Fatalf("never healthy again: %+v; want failed at the bound", g)
}
g, judge = threeReadings(t, "mesh-host", time.Now())
judge(60*time.Second, healthGood)
judge(100*time.Second, healthBroken)
if g.Verdict != inventory.GateFailed {
t.Fatalf("a broken reading: %+v; want failed, to be put back", g)
}
}
// The tool runner passes on one reading after its witness window; mesh-delivery on one, at once.
func TestTheToolRunnerAndMeshDeliveryPassOnOneReading(t *testing.T) {
withTheGatesTimes(t)
sent := time.Now()
o := aGateOn("node-tools", "ace", sent)
o.g.Applied["ace"] = sent.Add(10 * time.Second)
o.read(sent.Add(30*time.Second), healthGood, "")
if o.g.Verdict != "" {
t.Fatalf("the tool runner passed inside its witness window: %+v", o.g)
}
o.read(sent.Add(70*time.Second), healthGood, "")
if o.g.Verdict != inventory.GatePassed || o.g.Passes != 1 || !strings.Contains(o.g.Why, "the tool runner's rule") {
t.Fatalf("the tool runner: %+v; want passed on one reading", o.g)
}
o = aGateOn(catalogue.DeliverySeat, "novox", sent)
o.read(sent.Add(5*time.Second), healthGood, "")
if o.g.Verdict != inventory.GatePassed || o.g.Passes != 1 || !strings.Contains(o.g.Why, "mesh-delivery's rule") {
t.Fatalf("mesh-delivery: %+v; want passed on one reading", o.g)
}
}
// A dip after a healthy reading taken before the controller's witness floor is a fault: the send is not passed at
// two minutes (novox/hq ADR 0352 decision 3).
func TestADipAfterAnUncountedHealthyReadingIsAFault(t *testing.T) {
withTheGatesTimes(t)
sent := time.Now()
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent}
modules := []string{"mesh-controller", "mesh-host"}
rules, floors := map[string]gateRule{}, map[string]time.Time{}
for _, m := range modules {
rules[m] = gateRuleOf(m, catalogue.Manifest{Module: m})
}
floors["mesh-host"] = sent
floors["mesh-controller"] = sent.Add(90 * time.Second)
judge := func(after time.Duration, h health) {
tallyGate(g, gateJudging{modules: modules, reading: map[string]health{"mesh-controller": h, "mesh-host": h},
worst: h, why: "not yet", rules: rules, floors: floors}, sent.Add(after))
}
judge(30*time.Second, healthGood)
if g.Passes != 0 {
t.Fatalf("a reading before the controller's floor was counted: %+v", g)
}
judge(70*time.Second, healthNotYet)
judge(2*time.Minute, healthGood)
if !g.Faulted || g.Verdict != "" {
t.Fatalf("%+v; want the dip a fault, and no pass at two minutes", g)
}
}
// A send carrying the controller and the node-engine is judged by the node-engine's three readings.
func TestASendIsJudgedByItsMostDemandingRule(t *testing.T) {
withTheGatesTimes(t)
sent := time.Now()
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent}
modules := []string{"mesh-controller", "mesh-host"}
rules, floors := map[string]gateRule{}, map[string]time.Time{}
for _, m := range modules {
rules[m] = gateRuleOf(m, catalogue.Manifest{Module: m})
floors[m] = gateFloor(m, rules[m], sent, []string{"novox"}, nil, nil)
}
for _, after := range []time.Duration{70 * time.Second, 110 * time.Second} {
tallyGate(g, gateJudging{modules: modules, reading: map[string]health{"mesh-controller": healthGood,
"mesh-host": healthGood}, worst: healthGood, rules: rules, floors: floors}, sent.Add(after))
}
if g.Verdict != "" || g.Wants != 3 || g.Passes != 2 || g.Healthy["mesh-controller"] != 2 {
t.Fatalf("%+v; want the send judged by the node-engine's three readings, the controller's own counted", g)
}
}
+40 -29
View File
@@ -36,6 +36,10 @@ func aGateMesh(t *testing.T) *gateMesh {
ctx := t.Context() ctx := t.Context()
inv := open.inventory inv := open.inventory
g := &gateMesh{open: open, health: map[string]health{}} g := &gateMesh{open: open, health: map[string]health{}}
// The gate's own tests judge a first machine, as walks a controller before novox/hq ADR 0360 started still are.
wasGated := gateFirstMachine
gateFirstMachine = true
t.Cleanup(func() { gateFirstMachine = wasGated })
g.keeper, _ = withConditionsInMemory(t) g.keeper, _ = withConditionsInMemory(t)
g.told = &conditions.Told{} g.told = &conditions.Told{}
was := doctorFrom was := doctorFrom
@@ -124,10 +128,14 @@ func aGateMesh(t *testing.T) *gateMesh {
} }
t.Cleanup(func() { gatherGateFacts = wasGather }) t.Cleanup(func() { gatherGateFacts = wasGather })
wasSettle, wasEvery, wasBound := gateSettle, gateEvery, gateBound wasWithin, wasEvery, wasBound := gateWithin, gateEvery, gateBound
// The witness window is the node-engine's, from a start no test makes (novox/hq ADR 0351).
wasWindow := gateWitnessWindow
gateWitnessWindow = 0
t.Cleanup(func() { gateWitnessWindow = wasWindow })
// One judging per advance until a test says otherwise: a pass waits gateEvery for the next. // One judging per advance until a test says otherwise: a pass waits gateEvery for the next.
gateSettle, gateEvery = 0, time.Hour gateWithin, gateEvery = time.Hour, time.Hour
t.Cleanup(func() { gateSettle, gateEvery, gateBound = wasSettle, wasEvery, wasBound }) t.Cleanup(func() { gateWithin, gateEvery, gateBound = wasWithin, wasEvery, wasBound })
built := time.Now().UTC() built := time.Now().UTC()
plan := inventory.Plan{ID: "plan-gate", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c2", plan := inventory.Plan{ID: "plan-gate", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c2",
@@ -267,12 +275,40 @@ func TestABuildThatPassesItsGateRollsEverywhereUnattended(t *testing.T) {
} }
} }
// A build rolls everywhere with no hand and no gate (novox/hq ADR 0360 §1): both machines in one send, no health
// judged, and the walk done once both reported on the send. Before, anchor was judged first, then laptop sent.
func TestABuildRollsToEveryMachineAtOnceWithNoGate(t *testing.T) {
g := aGateMesh(t)
gateFirstMachine = false
ctx := t.Context()
judged := false
wasGather := gatherGateFacts
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
judged = true
return wasGather(ctx, open, component)
}
t.Cleanup(func() { gatherGateFacts = wasGather })
for i := 0; i < 6; i++ {
advancePlans(ctx, g.open)
}
p := g.plan(t)
if !reflect.DeepEqual(g.sent, [][]string{{"anchor", "laptop"}}) {
t.Fatalf("sent %v, not both machines in one send", g.sent)
}
if s := p.Modules["app"]; p.State != inventory.PlanDone || s.Gate != nil || s.FirstAt != nil || judged {
t.Fatalf("the plan is %s (%s), app %+v, judged %v", p.State, p.Note, s, judged)
}
if obs, err := probeGates(ctx, doctorFrom); err != nil || len(obs) != 0 {
t.Fatalf("a rolled build left a condition: %+v %v", obs, err)
}
}
// A leaf module passes its gate on one reading, as soon as its first machine is healthy, without waiting out the // A leaf module passes its gate on one reading, as soon as its first machine is healthy, without waiting out the
// core's settle time (novox/hq ADR 0282 decision 4 as ADR 0304 decision 3 changed it, issue 524). // core's settle time (novox/hq ADR 0282 decision 4 as ADR 0304 decision 3 changed it, issue 524).
func TestALeafModulePassesOnOneHealthyReading(t *testing.T) { func TestALeafModulePassesOnOneHealthyReading(t *testing.T) {
g := aGateMesh(t) g := aGateMesh(t)
ctx := t.Context() ctx := t.Context()
gateSettle = time.Hour // a core module could not pass for an hour gateWithin = time.Hour // a core module could not pass on fewer readings for an hour
advancePlans(ctx, g.open) // the first machine is sent the new build advancePlans(ctx, g.open) // the first machine is sent the new build
advancePlans(ctx, g.open) // one healthy judging advancePlans(ctx, g.open) // one healthy judging
p := g.plan(t) p := g.plan(t)
@@ -285,31 +321,6 @@ func TestALeafModulePassesOnOneHealthyReading(t *testing.T) {
} }
} }
// Which modules are judged as the core: those on the controller's own path and the resolver's holder, three
// readings after the settle time; every other module one (novox/hq ADR 0282 decision 1, ADR 0304 decision 3).
func TestAModuleIsJudgedByItsClass(t *testing.T) {
for _, c := range []struct {
module string
claims []catalogue.Claim
passes int
settles bool
}{
{"app", nil, 1, false},
{"nzbget", nil, 1, false},
{"mesh-controller", nil, 3, true},
{"mesh-host", nil, 3, true},
{"node-tools", nil, 3, true},
{"nats", nil, 3, true},
{catalogue.DeliverySeat, nil, 3, true},
{"dnsmasq", []catalogue.Claim{{Name: resolverSeat}}, 3, true},
} {
p, s := gateWants(c.module, catalogue.Manifest{Module: c.module, Claims: c.claims})
if p != c.passes || s != c.settles {
t.Errorf("%s: %d passes, settles %v; want %d, %v", c.module, p, s, c.passes, c.settles)
}
}
}
// A first machine that never becomes healthy fails its gate at the bound, and is put back. // A first machine that never becomes healthy fails its gate at the bound, and is put back.
func TestABuildNeverHealthyFailsAtTheBound(t *testing.T) { func TestABuildNeverHealthyFailsAtTheBound(t *testing.T) {
judgeLeavesAsTheCore(t) judgeLeavesAsTheCore(t)
+20 -1
View File
@@ -2,6 +2,7 @@ package main
import ( import (
"context" "context"
"encoding/json"
"fmt" "fmt"
"io" "io"
"os" "os"
@@ -60,12 +61,30 @@ type sentRecord struct {
toldGeneration int64 toldGeneration int64
sender string sender string
modules []string modules []string
// plans is each module the send handed to the provisioner, its plan, as JSON (design 57 P6); nil for none.
plans []byte
} }
// sentRecordOf is a composed send's record: its sender is the caller this process acts for. // sentRecordOf is a composed send's record: its sender is the caller this process acts for.
func sentRecordOf(ctx context.Context, d sendable) sentRecord { func sentRecordOf(ctx context.Context, d sendable) sentRecord {
return sentRecord{sequence: d.Sequence, epoch: d.actingEpoch, generation: d.composedFrom, r := sentRecord{sequence: d.Sequence, epoch: d.actingEpoch, generation: d.composedFrom,
toldGeneration: d.Generation, sender: senderOf(callerOf(ctx)), modules: d.modules} toldGeneration: d.Generation, sender: senderOf(callerOf(ctx)), modules: d.modules}
if len(d.plans) > 0 {
kept := map[string]keptPlan{}
for module, p := range d.plans {
kept[module] = keptPlan{Plan: p, RequiredBy: d.requiredBy[module]}
}
// A map of plain strings and slices: it marshals.
r.plans, _ = json.Marshal(kept)
}
return r
}
// keptPlan is a handed module's plan as a send keeps it (migration 0100): the plan, and for a module only pulled in
// by another, the module that requires it (issue 765, inventory.SentProvisioningPlans).
type keptPlan struct {
catalogue.Plan
RequiredBy string `json:"required-by,omitempty"`
} }
// callerOf is who asked for what this process does: the seat call's caller when ctx belongs to one, the // callerOf is who asked for what this process does: the seat call's caller when ctx belongs to one, the
+6 -3
View File
@@ -8,7 +8,8 @@ import (
) )
// A rotation asked through the seat carries why to the command, which records it in the hand-act // A rotation asked through the seat carries why to the command, which records it in the hand-act
// log (novox/hq ADR 0228); why with a provision is refused as passed over, not dropped. // log (novox/hq ADR 0228); and with a provision too, since the vault asks why of every rotation of a
// credential it holds (issue 767).
func TestARotationThroughTheSeatCarriesWhy(t *testing.T) { func TestARotationThroughTheSeatCarriesWhy(t *testing.T) {
argv, err := argvFor("rotate", map[string]any{"node": "anchor", "module": "letta", argv, err := argvFor("rotate", map[string]any{"node": "anchor", "module": "letta",
"secret": "server-password", "why": "leaked into logs", "cause": "leaked"}) "secret": "server-password", "why": "leaked into logs", "cause": "leaked"})
@@ -19,8 +20,10 @@ func TestARotationThroughTheSeatCarriesWhy(t *testing.T) {
if err != nil || strings.Join(argv, " ") != "secret rotate anchor letta server-password" { if err != nil || strings.Join(argv, " ") != "secret rotate anchor letta server-password" {
t.Fatalf("without why: %v %v", argv, err) t.Fatalf("without why: %v %v", argv, err)
} }
if argv, err := argvFor("rotate", map[string]any{"provision": "postgres-database", "why": "leaked"}); err == nil { argv, err = argvFor("rotate", map[string]any{"provision": "postgres-database", "consumer": "ace", "why": "leaked",
t.Fatalf("why beside a provision was passed over: %v", argv) "cause": "leaked"})
if err != nil || strings.Join(argv, " ") != "rotate postgres-database --consumer ace --why leaked --cause leaked" {
t.Fatalf("why beside a provision: %v %v", argv, err)
} }
} }
+52 -20
View File
@@ -8,6 +8,8 @@ import (
"strings" "strings"
"time" "time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/catalogue" "github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory" "github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link" "github.com/novox/mesh-controller/internal/link"
@@ -34,32 +36,62 @@ type groupMember struct {
Repository, State string Repository, State string
} }
// groupsLine is one group as mesh-delivery's `groups` says it: its id (the head branch name), its members as text and,
// from a mesh-delivery that knows novox/hq ADR 0382, each member again as an object (`member_details`) naming its pull
// request's number, whole head, base branch and whether it is open.
type groupsLine struct {
ID string `json:"id"`
Members []string `json:"members"`
Details []memberLine `json:"member_details,omitempty"`
}
// memberLine is one member of a group, as `groups` says it beside its text.
type memberLine struct {
ID string `json:"id"`
Repository string `json:"repository"`
State string `json:"state"`
Number int `json:"number"`
Head string `json:"head"`
Base string `json:"base"`
Open bool `json:"open"`
}
// groupsAsked is mesh-delivery's `groups`: every group it keeps, or an error when it cannot be asked or did not
// answer within groupAskWithin. A seam a test replaces.
var groupsAsked = func(ctx context.Context, held bool) ([]groupsLine, error) {
if !held {
return nil, fmt.Errorf("no holder of %s is on record to say it", catalogue.DeliverySeat)
}
// On the bus as this process is: the serving controller's own connection, lent, or — in a verb that runs as a command
// of its own, such as a recheck's `delivery check` — one dialled for the moment (onTheBus).
var raw json.RawMessage
err := onTheBus(func(conn *nats.Conn) error {
asking, cancel := context.WithTimeout(ctx, groupAskWithin)
defer cancel()
var err error
raw, err = askDeliveryOwner(asking, conn, "groups", map[string]any{})
return err
})
if err != nil {
if errors.Is(err, link.ErrNothingServes) {
return nil, fmt.Errorf("nothing serves %s.groups", catalogue.DeliverySeat)
}
return nil, err
}
var lines []groupsLine
if err := json.Unmarshal(raw, &lines); err != nil {
return nil, fmt.Errorf("%s.groups answered something unreadable: %w", catalogue.DeliverySeat, err)
}
return lines, nil
}
// deliveryGroupOf is what mesh-delivery says of the group a head names: its members, and whether the head names a // deliveryGroupOf is what mesh-delivery says of the group a head names: its members, and whether the head names a
// group at all; an error when it cannot be asked or did not answer. A seam a test replaces. // group at all; an error when it cannot be asked or did not answer. A seam a test replaces.
var deliveryGroupOf = func(ctx context.Context, held bool, head string) ([]groupMember, bool, error) { var deliveryGroupOf = func(ctx context.Context, held bool, head string) ([]groupMember, bool, error) {
if !held { lines, err := groupsAsked(ctx, held)
return nil, false, fmt.Errorf("no holder of %s is on record to say it", catalogue.DeliverySeat)
}
serving := servingBus.Load()
if serving == nil {
return nil, false, errors.New("this controller is not on the bus")
}
asking, cancel := context.WithTimeout(ctx, groupAskWithin)
defer cancel()
raw, err := askDeliveryOwner(asking, serving.Conn(), "groups", map[string]any{})
if err != nil { if err != nil {
if errors.Is(err, link.ErrNothingServes) {
return nil, false, fmt.Errorf("nothing serves %s.groups", catalogue.DeliverySeat)
}
return nil, false, err return nil, false, err
} }
var lines []struct {
ID string `json:"id"`
Members []string `json:"members"`
}
if err := json.Unmarshal(raw, &lines); err != nil {
return nil, false, fmt.Errorf("%s.groups answered something unreadable: %w", catalogue.DeliverySeat, err)
}
for _, l := range lines { for _, l := range lines {
if l.ID != head { if l.ID != head {
continue continue
+9
View File
@@ -105,6 +105,12 @@ var handActVerbs = []handActVerb{
// (ADR 0257). // (ADR 0257).
{Verb: "mirrors", Decision: "which copies in the store no record names are the mesh's is a person's " + {Verb: "mirrors", Decision: "which copies in the store no record names are the mesh's is a person's " +
"word (ADR 0257)"}, "word (ADR 0257)"},
// A move of data a release runs once after its delivery (design 57): a person's step of the release, never a
// repair.
{Verb: "move-placement", Decision: "a release's move of data runs once after its delivery, on a person's " +
"word (design 57, issue 736)"},
{Verb: "move-config", Decision: "a release's move of data runs once after its delivery, on a person's " +
"word (design 57, issue 664)"},
{Verb: "bus upgrade", Decision: "the bus is never rolled by the mesh: replacing it is a planned step a " + {Verb: "bus upgrade", Decision: "the bus is never rolled by the mesh: replacing it is a planned step a " +
"person starts (ADR 0236)"}, "person starts (ADR 0236)"},
{Verb: "upgrade release-backlog", Decision: "after a release plan failed, the next opens only when a " + {Verb: "upgrade release-backlog", Decision: "after a release plan failed, the next opens only when a " +
@@ -118,6 +124,9 @@ var handActVerbs = []handActVerb{
{Verb: "secret accept", Decision: "a value an outside party issued is given by a person, at their desk"}, {Verb: "secret accept", Decision: "a value an outside party issued is given by a person, at their desk"},
{Verb: "secret rotate", Decision: "a value a person judged disclosed is replaced on their word", {Verb: "secret rotate", Decision: "a value a person judged disclosed is replaced on their word",
DecidedFor: []string{causeLeakedInLogs}}, DecidedFor: []string{causeLeakedInLogs}},
// A provider's credential the vault holds, rotated there on a person's word (issue 767): the same judgement.
{Verb: "rotate", Decision: "a value a person judged disclosed is replaced on their word",
DecidedFor: []string{causeLeakedInLogs}},
} }
// personsDecision is whether an act in the log is a person's decision by design, by the verb that // personsDecision is whether an act in the log is a person's decision by design, by the verb that
+1 -1
View File
@@ -55,7 +55,7 @@ func TestARepairByHandCarriesItsReason(t *testing.T) {
want string want string
}{ }{
{"push", map[string]any{"node": "anchor", "why": "stuck", "cause": "sent-not-reported"}, {"push", map[string]any{"node": "anchor", "why": "stuck", "cause": "sent-not-reported"},
"push anchor --wait 0 --why stuck --cause sent-not-reported"}, "push anchor --wait 0s --why stuck --cause sent-not-reported"},
{"plans", map[string]any{"close": "plan-1", "why": "the report will not come"}, {"plans", map[string]any{"close": "plan-1", "why": "the report will not come"},
"plans close plan-1 --why the report will not come"}, "plans close plan-1 --why the report will not come"},
{"plans", map[string]any{"retry": "plan-1"}, "plans retry plan-1"}, {"plans", map[string]any{"retry": "plan-1"}, "plans retry plan-1"},
+2 -1
View File
@@ -642,7 +642,8 @@ func planStale(ctx context.Context, inv *inventory.Inventory, p inventory.Plan)
if err != nil { if err != nil {
return "", "", err return "", "", err
} }
if u.RollOut { // A recorded module whose move goes out on a machine is still the walk's to send (novox/hq ADR 0377).
if u.RollOut || len(wentOut(s.Recorded)) > 0 {
return "", "", nil return "", "", nil
} }
} }
+66
View File
@@ -0,0 +1,66 @@
package main
import (
"context"
"fmt"
"strings"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
)
// A grant of a restricted stream that composes nothing (novox/hq ADR 0376 §2).
//
// A restricted stream is heard only by a module that runs as an account of its own (`runs-as`, ADR 0259): the
// tool runner's bus account is the one every agent session on its machine reaches the bus as, so the controller
// composes no consumer, grant or bind word for a module it carries (broker.ComposedHears). Where a grant still
// names such a module, the module hears nothing it was granted, and that is said — in `doctor` and in the
// controller's log on every raise and send — naming the module, the stream and the remedy. Never silent.
const (
// probeWithheldID is the self-check's probe of it, and kindGrantComposesNothing the condition it raises.
probeWithheldID = "D-heard"
kindGrantComposesNothing = "grant-composes-nothing"
)
// probeWithheldHearing says every granted hearing of a restricted stream composed for nothing.
func probeWithheldHearing(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
records, err := d.open.inventory.BusRecords(ctx)
if err != nil {
return nil, err
}
return withheldObservations(broker.WithheldOf(records)), nil
}
// withheldObservations is each withheld hearing as a condition of its module on its machine.
func withheldObservations(withheld []broker.Withheld) []conditions.Observation {
var out []conditions.Observation
for _, w := range withheld {
out = append(out, conditions.Observation{Scope: conditions.ScopeModule,
ID: w.Module + "." + strings.ToLower(w.Stream) + "." + w.Node,
Kind: kindGrantComposesNothing, Machine: w.Node, Severity: conditions.Warning,
Summary: w.Said(), Said: fmt.Sprintf("%s on %s granted %s; composed nothing", w.Module, w.Node, w.Stream)})
}
return sortedFound(out)
}
// sayWithheld puts every withheld hearing in the controller's log, once per raise or send that composes it.
func sayWithheld(records broker.Records, indent string) {
for _, w := range broker.WithheldOf(records) {
fmt.Printf("%sA GRANT COMPOSES NOTHING: %s\n", indent, w.Said())
}
}
// withheldWords are the plain words of a withheld hearing: the module, the machine and what to do.
func withheldWords(o conditions.Observation) words {
module, m := "a module", machineOr(o, "a machine")
if o.Scope == conditions.ScopeModule && idPart(o, 0) != "" {
module = idPart(o, 0)
}
return words{Headline: clip(conditions.Capital(fmt.Sprintf("%s on %s hears nothing it was granted", module, m)), 60),
Needs: fmt.Sprintf("give %s an account of its own in its manifest; the details say which stream.", module),
Explanation: fmt.Sprintf("You granted %s part of a stream that only modules running under an account of "+
"their own may hear. It runs inside the shared tool runner on %s, which every assistant session there "+
"also uses, so the mesh gives it none of that stream.", module, m),
Resolved: fmt.Sprintf("%s on %s hears what it was granted, or is no longer granted it", module, m)}
}
@@ -0,0 +1,43 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
)
// A grant naming a module the tool runner carries composes nothing, and doctor says so in plain words, naming the
// module, the stream and the remedy (novox/hq ADR 0376 §2).
func TestAGrantThatComposesNothingIsSaid(t *testing.T) {
records := broker.Records{Nodes: []string{"ace"}, Assigned: map[string][]broker.Declared{"ace": {
{Module: broker.RuntimeModule},
{Module: "reader", Hears: []broker.Heard{{Stream: "MAIL", Emitter: "mesh-mail", Restricted: true,
Filters: []string{"mesh.stream.mesh-mail.event.mail.mesh.>"}}}},
{Module: "mailer", RunsAs: "mailer", Hears: []broker.Heard{{Stream: "MAIL", Emitter: "mesh-mail", Restricted: true,
Filters: []string{"mesh.stream.mesh-mail.event.mail.mesh.>"}}}},
}}}
found := withheldObservations(broker.WithheldOf(records))
if len(found) != 1 {
t.Fatalf("said %+v", found)
}
o := found[0]
if o.Kind != kindGrantComposesNothing || o.Machine != "ace" || !strings.Contains(o.Summary, "reader on ace") ||
!strings.Contains(o.Summary, "MAIL") || !strings.Contains(o.Summary, "runs-as") {
t.Fatalf("%+v", o)
}
w := plainWordings[kindGrantComposesNothing](o)
if why, ok := conditions.PlainWords(w, "ace"); !ok {
t.Fatalf("not plain: %s (%+v)", why, w)
}
registered := false
for _, p := range probeRegistry {
if p.ID == probeWithheldID && p.Kind == kindGrantComposesNothing && p.run != nil {
registered = true
}
}
if !registered {
t.Fatal("doctor does not run the probe of a grant that composes nothing")
}
}
+18 -2
View File
@@ -146,12 +146,28 @@ func heldMachines(ctx context.Context, open *stores, names []string) (map[string
same[m] = current[m].Commit same[m] = current[m].Commit
} }
} }
why := heldBack(node, modules, same, known, current, plans) // A move under a module's own `record` that recreates nothing and moves no data goes as a module that rolls
// out goes (novox/hq ADR 0377): a walk's to send, not a person's.
here, copied := current, false
for m, was := range same {
if now, held := current[m]; held && now.OwnRecord() && !f.identical(m, was, now.Commit) && f.rolls(m, was) {
if !copied {
copied = true
here = make(map[string]inventory.CurrentBuild, len(current))
for k, v := range current {
here[k] = v
}
}
now.RollOut = true
here[m] = now
}
}
why := heldBack(node, modules, same, known, here, plans)
// And a build no gate has seen is not carried by a send that does not judge it (ADR 0236). // And a build no gate has seen is not carried by a send that does not judge it (ADR 0236).
for _, mv := range f.moves(node, modules, same, known, false) { for _, mv := range f.moves(node, modules, same, known, false) {
if planStillToSend(plans, mv.Module, node) == "" { if planStillToSend(plans, mv.Module, node) == "" {
why = append(why, fmt.Sprintf("%s would move from %s to %s, which has passed no gate yet — a release "+ why = append(why, fmt.Sprintf("%s would move from %s to %s, which has passed no gate yet — a release "+
"plan sends it, one machine at a time, judged", mv.Module, buildName(mv.From), buildName(mv.To))) "plan sends it, one machine first and then the rest at once, judged", mv.Module, buildName(mv.From), buildName(mv.To)))
} }
} }
if len(why) > 0 { if len(why) > 0 {
+8
View File
@@ -25,6 +25,7 @@ import (
// another send reaches anchor, unreported; the gate still passes. And a send that moves a judged module // another send reaches anchor, unreported; the gate still passes. And a send that moves a judged module
// to another build supersedes the judging: no verdict, nothing put back. // to another build supersedes the judging: no verdict, nothing put back.
func TestReplay352(t *testing.T) { func TestReplay352(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t) judgeLeavesAsTheCore(t)
t.Run("a newer send to the judged machine does not unreport the gate's", testANewerSendDoesNotUnreportTheGatesOwn) t.Run("a newer send to the judged machine does not unreport the gate's", testANewerSendDoesNotUnreportTheGatesOwn)
t.Run("a send that moves the module supersedes the judging", testASendThatMovesTheModuleSupersedesTheJudging) t.Run("a send that moves the module supersedes the judging", testASendThatMovesTheModuleSupersedesTheJudging)
@@ -67,6 +68,7 @@ func testANewerSendDoesNotUnreportTheGatesOwn(t *testing.T) {
// A plan's own first send waits while a release judges the same module on that machine with another build. // A plan's own first send waits while a release judges the same module on that machine with another build.
func TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere(t *testing.T) { func TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t) judgeLeavesAsTheCore(t)
b := aBacklog(t) b := aBacklog(t)
ctx := t.Context() ctx := t.Context()
@@ -82,6 +84,7 @@ func TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere(t *testing.T) {
// A merge plan's judging is superseded the same way: another send moved its module on the first machine. // A merge plan's judging is superseded the same way: another send moved its module on the first machine.
func TestAPlansJudgingIsSupersededByASendThatMovesItsModule(t *testing.T) { func TestAPlansJudgingIsSupersededByASendThatMovesItsModule(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t) judgeLeavesAsTheCore(t)
g := aGateMesh(t) g := aGateMesh(t)
ctx := t.Context() ctx := t.Context()
@@ -140,6 +143,7 @@ func testASendThatMovesTheModuleSupersedesTheJudging(t *testing.T) {
// A report is on the gate's own send: the declaration itself, or one sequenced after it; a gate kept // A report is on the gate's own send: the declaration itself, or one sequenced after it; a gate kept
// without its send reads the report against the send made last, as before. Pure. // without its send reads the report against the send made last, as before. Pure.
func TestAReportIsHeldAgainstTheGatesOwnSend(t *testing.T) { func TestAReportIsHeldAgainstTheGatesOwnSend(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
sent := inventory.SentDeclaration{Digest: "d-490", Sequence: 490} sent := inventory.SentDeclaration{Digest: "d-490", Sequence: 490}
for _, c := range []struct { for _, c := range []struct {
r inventory.Reported r inventory.Reported
@@ -185,6 +189,7 @@ func TestAReportIsHeldAgainstTheGatesOwnSend(t *testing.T) {
// A gate judges only the build the machine was last sent: last sent another build of the module, the // A gate judges only the build the machine was last sent: last sent another build of the module, the
// judging is superseded, whatever the machine reports. Pure. // judging is superseded, whatever the machine reports. Pure.
func TestAGateJudgesOnlyTheBuildTheMachineWasLastSent(t *testing.T) { func TestAGateJudgesOnlyTheBuildTheMachineWasLastSent(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
at := time.Now() at := time.Now()
f := gateFacts{now: at, reports: map[string]inventory.Reported{"anchor": {Node: "anchor", Outcome: inventory.OutcomeApplied, f := gateFacts{now: at, reports: map[string]inventory.Reported{"anchor": {Node: "anchor", Outcome: inventory.OutcomeApplied,
At: &at, Current: true}}, engines: map[string]string{}, served: map[string]served{}, rolledBack: map[string][]lease.Rollback{}, At: &at, Current: true}}, engines: map[string]string{}, served: map[string]served{}, rolledBack: map[string][]lease.Rollback{},
@@ -212,6 +217,7 @@ func TestAGateJudgesOnlyTheBuildTheMachineWasLastSent(t *testing.T) {
// A move of another build of a module to a machine where a release or a plan is judging that module // A move of another build of a module to a machine where a release or a plan is judging that module
// waits for that judging; the same build to that machine is already there. Pure. // waits for that judging; the same build to that machine is already there. Pure.
func TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine(t *testing.T) { func TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
at := time.Now() at := time.Now()
release := inventory.Plan{ID: "release-1", State: inventory.PlanRolling, Release: &inventory.PlanRelease{ release := inventory.Plan{ID: "release-1", State: inventory.PlanRolling, Release: &inventory.PlanRelease{
Gate: &inventory.PlanGate{Machines: []string{"novox"}, Carried: []inventory.CarriedMove{ Gate: &inventory.PlanGate{Machines: []string{"novox"}, Carried: []inventory.CarriedMove{
@@ -249,6 +255,7 @@ func TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine(t *testing.T) {
// The controller is never put back to a build that reaches less of the store's schema than the store // The controller is never put back to a build that reaches less of the store's schema than the store
// has, or to one that never said what it reaches: the current build is kept, and the condition says so. // has, or to one that never said what it reaches: the current build is kept, and the condition says so.
func TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema(t *testing.T) { func TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t) open := aMesh(t)
ctx := t.Context() ctx := t.Context()
inv := open.inventory inv := open.inventory
@@ -345,6 +352,7 @@ func TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema(t *testing.T)
// The lease's holder names the build the declaration told it it is, and the version stamp only without one. // The lease's holder names the build the declaration told it it is, and the version stamp only without one.
func TestTheHolderNamesTheBuildTheDeclarationToldIt(t *testing.T) { func TestTheHolderNamesTheBuildTheDeclarationToldIt(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
t.Setenv(RunningBuildVar, " ad62528c47c7 ") t.Setenv(RunningBuildVar, " ad62528c47c7 ")
if h := holderOf("x"); h.Build != "ad62528c47c7" { if h := holderOf("x"); h.Build != "ad62528c47c7" {
t.Fatalf("the holder's build is %q", h.Build) t.Fatalf("the holder's build is %q", h.Build)
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A tier whose module is no move — made from the source every machine running it runs, so nothing is
// sent — is not waited on for a report after its marking (novox/hq issue 648). Its machines answered the
// send that last carried that build before the walk got there, and nothing will make them report again:
// the walk asks the next tier in the same step, rather than "waiting for app on anchor, laptop" for ever.
func TestANoMoveTierWaitsForNoNewReport(t *testing.T) {
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
// app holds the build seat, so a module of the next tier is built by it; since novox/hq ADR 0360 §3 that gates
// nothing.
for _, m := range []catalogue.Manifest{
{Module: "app", Version: "c2", Claims: []catalogue.Claim{{Name: "node-build-agent", Scope: catalogue.ScopeNode}}},
{Module: "web", Version: "w1"},
} {
if err := inv.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/" + m.Module, BuiltFrom: m.Version, Head: m.Version, Asked: time.Now().Add(-time.Minute)}); err != nil {
t.Fatal(err)
}
}
// What anchor and laptop were last sent (c1) was made from the source the plan's build of c2 is.
for _, b := range []inventory.Build{
{ID: "build-1s", Module: "app", Commit: "c1", SourceFingerprint: "src1:app",
Asked: time.Now().Add(-30 * time.Second), At: time.Now().Add(-30 * time.Second)},
{ID: "build-2s", Module: "app", Commit: "c2", SourceFingerprint: "src1:app", Asked: time.Now(), At: time.Now()},
} {
b.Manifest, _ = json.Marshal(catalogue.Manifest{Module: "app", Version: "1"})
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
// Each machine answered that send, applied, before the walk's tier was built.
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.RecordDoing(ctx, nodeID(t, g.open, n), inventory.Doing{Node: n,
Outcome: inventory.OutcomeApplied, Declared: "d-" + n + "-c1", Applied: 1,
At: time.Now().Add(-10 * time.Second)}); err != nil {
t.Fatal(err)
}
}
built := time.Now().UTC()
p := g.plan(t)
p.Tiers = [][]string{{"app"}, {"web"}}
p.Modules["app"].Build, p.Modules["app"].BuiltAt = "build-2s", &built
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, g.open)
if len(g.sent) != 0 {
t.Fatalf("a no-move build was sent: %v", g.sent)
}
p = g.plan(t)
if s := p.Modules["app"]; s == nil || !strings.HasPrefix(s.Why, "no move") {
t.Fatalf("the build was not marked no move: %+v", s)
}
if p.Tier < 1 || strings.Contains(p.Note, "waiting for app") {
t.Fatalf("the walk waits on a no-move tier for a report nothing will cause: tier %d, %s %q", p.Tier, p.State, p.Note)
}
// A machine that has not answered its last send is not waited on either: a built-by edge gates no tier
// (novox/hq ADR 0360 §3). Before, the walk waited for app on laptop.
p.Tier, p.State, p.Note = 0, inventory.PlanRolling, ""
if err := inv.RecordSent(ctx, nodeID(t, g.open, "laptop"), "d-laptop-later", map[string]string{"app": "c1"}); err != nil {
t.Fatal(err)
}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, g.open)
p = g.plan(t)
if p.Tier < 1 || strings.Contains(p.Note, "waiting for app") {
t.Fatalf("the walk waits on a built-by edge: tier %d, %q", p.Tier, p.Note)
}
}
+60 -5
View File
@@ -43,6 +43,32 @@ const issuedAsk = 20 * time.Second
// issuedProvider is one assignment of a module whose offer says its software issues the provision's credential. // issuedProvider is one assignment of a module whose offer says its software issues the provision's credential.
type issuedProvider struct { type issuedProvider struct {
Node, Module, Tool string Node, Module, Tool string
// Holders are the provision's holders from this assignment's machine, named to the vault, whose tool may answer
// a value for each (issue 749).
Holders []issuedHolder
}
// issuedHolder is one holder of a provision from a provider: a consuming module on one machine (issue 749).
type issuedHolder struct {
Node string `json:"node"`
Module string `json:"module"`
}
func (h issuedHolder) String() string { return h.Module + " on " + h.Node }
// issuedHoldersFrom are the provision's holders from provider, each consuming module on each machine once.
func issuedHoldersFrom(holders []inventory.Holder, provider string) []issuedHolder {
seen := map[issuedHolder]bool{}
var out []issuedHolder
for _, h := range holders {
one := issuedHolder{Node: h.Consumer, Module: h.ConsumerModule}
if h.Provider != provider || seen[one] {
continue
}
seen[one] = true
out = append(out, one)
}
return out
} }
// issuedTaker is the line's work, with what it reaches handed in: the store, the bus and the sends. // issuedTaker is the line's work, with what it reaches handed in: the store, the bus and the sends.
@@ -53,6 +79,10 @@ type issuedTaker struct {
keep func(provision, provider, value string) (inventory.IssuedKept, error) keep func(provision, provider, value string) (inventory.IssuedKept, error)
record func(act link.HandAct) error record func(act link.HandAct) error
send func(machines []string) error send func(machines []string) error
// ofTheVault says whether a provider module's secrets are filled the new way; for one that is, the value goes
// into the vault (vault, vault_secrets.go) and nothing is kept, sealed or sent here. Nil is none.
ofTheVault vaultSwitch
vault vaultAsker
} }
// take asks every provider of the provision for its software's value and keeps each; it answers what it did, in // take asks every provider of the provision for its software's value and keeps each; it answers what it did, in
@@ -90,6 +120,15 @@ func (t issuedTaker) take(provision string) (string, error) {
// takeFrom is one provider: asked, opened, kept, recorded and sent when it changed. // takeFrom is one provider: asked, opened, kept, recorded and sent when it changed.
func (t issuedTaker) takeFrom(provision string, p issuedProvider) (string, error) { func (t issuedTaker) takeFrom(provision string, p issuedProvider) (string, error) {
if t.ofTheVault != nil {
ours, err := t.ofTheVault(p.Module)
if err != nil {
return "", err
}
if ours {
return t.takeIntoVault(provision, p)
}
}
public, private, err := secrets.Keypair() public, private, err := secrets.Keypair()
if err != nil { if err != nil {
return "", fmt.Errorf("no key could be made to take the value: %w", err) return "", fmt.Errorf("no key could be made to take the value: %w", err)
@@ -174,6 +213,16 @@ func issuedProvidersOf(ctx context.Context, inv *inventory.Inventory, provision
if err != nil { if err != nil {
return nil, true, err return nil, true, err
} }
out, err := issuedProvidersFrom(shelf, holders, func(node string) ([]string, error) {
return inv.Assigned(ctx, node)
}, provision)
return out, true, err
}
// issuedProvidersFrom are the assignments to ask, from the provision's holders: on every machine providing it, the
// assigned modules whose offer says their software issues it, each with that machine's holders (issue 749).
func issuedProvidersFrom(shelf map[string]catalogue.Manifest, holders []inventory.Holder,
assignedOn func(node string) ([]string, error), provision string) ([]issuedProvider, error) {
nodes := map[string]bool{} nodes := map[string]bool{}
for _, h := range holders { for _, h := range holders {
nodes[h.Provider] = true nodes[h.Provider] = true
@@ -185,13 +234,17 @@ func issuedProvidersOf(ctx context.Context, inv *inventory.Inventory, provision
sort.Strings(names) sort.Strings(names)
var out []issuedProvider var out []issuedProvider
for _, node := range names { for _, node := range names {
assigned, err := inv.Assigned(ctx, node) assigned, err := assignedOn(node)
if err != nil { if err != nil {
return nil, true, err return nil, err
} }
out = append(out, issuedOn(shelf, node, assigned, provision)...) on := issuedOn(shelf, node, assigned, provision)
for i := range on {
on[i].Holders = issuedHoldersFrom(holders, node)
}
out = append(out, on...)
} }
return out, true, nil return out, nil
} }
// offersIssued is whether any module offers the provision with a credential its software issues. // offersIssued is whether any module offers the provision with a credential its software issues.
@@ -262,7 +315,9 @@ func secretIssued(ctx context.Context, args []string) error {
return err return err
}) })
}, },
send: func(machines []string) error { return sendTo(ctx, open, machines) }, send: func(machines []string) error { return sendTo(ctx, open, machines) },
ofTheVault: filledTheNewWay,
vault: theVault(ctx),
} }
words, err := t.take(provision) words, err := t.take(provision)
if words != "" { if words != "" {
+82 -1
View File
@@ -193,7 +193,7 @@ func TestTheProvidersAskedAreTheOffersThatSaySo(t *testing.T) {
"jellyfin": {Module: "jellyfin", Provides: []catalogue.Offer{{Name: "plex-api"}}}, "jellyfin": {Module: "jellyfin", Provides: []catalogue.Offer{{Name: "plex-api"}}},
} }
got := issuedOn(shelf, "ace", []string{"jellyfin", "plex", "kometa"}, "plex-api") got := issuedOn(shelf, "ace", []string{"jellyfin", "plex", "kometa"}, "plex-api")
if len(got) != 1 || got[0] != (issuedProvider{Node: "ace", Module: "plex", Tool: "plex_issued_credential"}) { if len(got) != 1 || got[0].Node != "ace" || got[0].Module != "plex" || got[0].Tool != "plex_issued_credential" {
t.Fatalf("the providers asked: %+v", got) t.Fatalf("the providers asked: %+v", got)
} }
if err := issuedRefusesRotation(shelf, "plex-api"); err == nil || !strings.Contains(err.Error(), "secret issued plex-api") { if err := issuedRefusesRotation(shelf, "plex-api"); err == nil || !strings.Contains(err.Error(), "secret issued plex-api") {
@@ -216,3 +216,84 @@ func TestAnIssuedValueIsADecisionInTheHandActLog(t *testing.T) {
} }
t.Fatal("secret issued is not in the hand-act table") t.Fatal("secret issued is not in the hand-act table")
} }
// **A need's credential is the vault's for its holder alone** (issue 757) exactly when its providing module is marked
// moved, so `secret issued` goes to the vault naming each holder, and its software issues the provision's credential.
func TestANeedIsIssuedToItsHolderOnlyWhereTheProviderIsMovedAndItsSoftwareIssuesIt(t *testing.T) {
forge := func(moved bool, issued bool) catalogue.Manifest {
offer := catalogue.Offer{Name: "gitea-read"}
if issued {
offer.Credential = &catalogue.OfferCredential{IssuedBy: catalogue.IssuedBySoftware, Tool: "gitea_agent_token"}
}
return catalogue.Manifest{Module: "gitea", Moved: moved, Provides: []catalogue.Offer{offer}}
}
shared := func(moved bool) catalogue.Manifest {
return catalogue.Manifest{Module: "gitea", Moved: moved, OwnSecrets: catalogue.OwnSecrets{"key": {Path: "/k"}},
Provides: []catalogue.Offer{{Name: "gitea-read", Credential: &catalogue.OfferCredential{Own: "key"}}}}
}
given := func() catalogue.Manifest {
return catalogue.Manifest{Module: "gitea", Moved: true, Provides: []catalogue.Offer{{Name: "gitea-read",
Credential: &catalogue.OfferCredential{IssuedBy: catalogue.IssuedOutside}}}}
}
need := catalogue.Needed{Name: "gitea-read", From: "novox", Module: "gitea", For: "claude-code"}
for why, c := range map[string]struct {
shelf map[string]catalogue.Manifest
need catalogue.Needed
want bool
}{
"moved and issued by its software": {map[string]catalogue.Manifest{"gitea": forge(true, true)}, need, true},
"not moved": {map[string]catalogue.Manifest{"gitea": forge(false, true)}, need, false},
"a login for each consumer": {map[string]catalogue.Manifest{"gitea": forge(true, false)}, need, true},
"a login for each, not moved": {map[string]catalogue.Manifest{"gitea": forge(false, false)}, need, false},
"a credential every holder shares": {map[string]catalogue.Manifest{"gitea": shared(true)}, need, false},
"a credential a person gives": {map[string]catalogue.Manifest{"gitea": given()}, need, false},
"a provider not on the shelf": {map[string]catalogue.Manifest{}, need, false},
"answered by a record": {map[string]catalogue.Manifest{"gitea": forge(true, true)},
catalogue.Needed{Name: "gitea-read", ByRecord: true, Module: "gitea", For: "claude-code"}, false},
} {
if got := issuedToHolder(c.shelf, c.need); got != c.want {
t.Errorf("%s: issued to its holder is %v, want %v", why, got, c.want)
}
}
}
// **The vault is a moved provider for its consumers' credentials** (issue 793): its pair rows are the vault's since
// V3, so a need of `secret` is issued to its holder and answered by a moved provider, whether or not the module is
// marked moved; another module is not.
func TestTheModuleHoldingTheVaultSeatCountsAsMovedForItsConsumers(t *testing.T) {
vault := catalogue.Manifest{Module: "mesh-vault", Claims: []catalogue.Claim{{Name: "mesh-vault"}},
Provides: []catalogue.Offer{{Name: "secret"}}}
other := catalogue.Manifest{Module: "mesh-vault", Provides: []catalogue.Offer{{Name: "secret"}}}
need := catalogue.Needed{Name: "secret", Local: "admin", From: "novox", Module: "mesh-vault", For: "gitea"}
if !issuedToHolder(map[string]catalogue.Manifest{"mesh-vault": vault}, need) ||
!providerMoved(map[string]catalogue.Manifest{"mesh-vault": vault}, need) {
t.Error("the vault's consumer is not answered by a moved provider")
}
if issuedToHolder(map[string]catalogue.Manifest{"mesh-vault": other}, need) ||
providerMoved(map[string]catalogue.Manifest{"mesh-vault": other}, need) {
t.Error("a module not holding the vault seat counts as moved")
}
}
// **A provision answered by the helper makes its provider a moved one for that provision alone** (issue 792): the
// provides entry says `"answered-by": "helper"`, the module need not be marked moved, and no credential is the vault's
// for it. Another provision of the module, or a module without the entry, does not count.
func TestAnOfferAnsweredByTheHelperCountsAsMovedForThatProvisionOnly(t *testing.T) {
proxy := catalogue.Manifest{Module: "route-proxy", Provides: []catalogue.Offer{
{Name: "route", AnsweredBy: catalogue.AnsweredByHelper}, {Name: "other"}}}
plain := catalogue.Manifest{Module: "route-proxy", Provides: []catalogue.Offer{{Name: "route"}}}
shelf := func(m catalogue.Manifest) map[string]catalogue.Manifest {
return map[string]catalogue.Manifest{m.Module: m}
}
route := catalogue.Needed{Name: "route", From: "novox", Module: "route-proxy", For: "gitea"}
other := catalogue.Needed{Name: "other", From: "novox", Module: "route-proxy", For: "gitea"}
if !providerMoved(shelf(proxy), route) {
t.Error("a provision answered by the helper is not answered by a moved provider")
}
if providerMoved(shelf(proxy), other) || providerMoved(shelf(plain), route) {
t.Error("a provision not answered by the helper counts as moved")
}
if issuedToHolder(shelf(proxy), route) {
t.Error("the flag made a credential the vault's")
}
}
+67
View File
@@ -0,0 +1,67 @@
package main
import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A covered account is handed to the vault when it is minted (novox/hq ADR 0374, issue 787).
//
// **Covered** are the bus accounts `module issue` mints for a core module (catalogue.CoreCredentialModules) or for a
// module marked moved; no other account is. Until the vault makes bus accounts (ADR 0113 clause 6, issue 774) the
// controller mints such an account and writes its hash to the bus's user list, as before, and then hands the whole
// value at once to the vault's `accept` with origin `issued`: sealed here to the key the vault made for that call, so
// the vault alone opens it (ADR 0349 clause 8). The vault keeps it as the module's own secret on that machine and
// raises `secret-changed`; a re-issue is the same hand with a new value, which replaces the old in place. An account
// not covered is minted and kept as it was and handed to nobody.
// vaultAskerForIssue is the vault as `module issue` asks it. A test replaces it.
var vaultAskerForIssue = func(ctx context.Context) vaultAsker { return theVault(ctx) }
// accountIsCovered says whether the bus accounts minted for m are the vault's to hold (ADR 0374 clause 1).
func accountIsCovered(m catalogue.Manifest) bool {
return catalogue.CarriesVaultSealed(m.Module) || m.Moved
}
// vaultInstalled says whether the vault is assigned to any machine. Until it is, as in genesis (ADR 0113 §15 to §18,
// mesh-host's bootstrap, which issues the bus's and the provisioner's accounts before the vault step), there is no
// vault to hand to: the value is kept in the controller's store as before, and the vault takes it afterwards by the
// move (move-in takes a value of the controller's store with origin made, accepted or issued, as it is). Only a
// vault that is installed and does not answer fails the issue.
func vaultInstalled(ctx context.Context, inv *inventory.Inventory) (bool, error) {
entries, err := inv.Catalogued(ctx)
if err != nil {
return false, err
}
for _, e := range entries {
if e.Manifest.Module == "mesh-vault" && len(e.On) > 0 {
return true, nil
}
}
return false, nil
}
// handIssuedToVault hands the value just minted for module's own secret name on node to the vault, when the module's
// accounts are covered, and does nothing otherwise. Loud when the vault does not take it: the bus's user list
// already holds the new hash, so a vault that kept the old value would serve a password that no longer logs in.
func handIssuedToVault(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node, name, held string) error {
if !accountIsCovered(m) {
return nil
}
if installed, err := vaultInstalled(ctx, inv); err != nil {
return fmt.Errorf("whether the vault is installed cannot be read, so %s of %s was not handed to it: %w", name,
m.Module, err)
} else if !installed {
return nil
}
target := ownSecretArgs(node, m.Module, name)
target["origin"] = "issued"
if _, err := acceptIntoVault(vaultAskerForIssue(ctx), target, held, "module issue"); err != nil {
return fmt.Errorf("the credential is minted and kept here, and the bus's user list holds its hash, but the "+
"vault did not take %s of %s on %s: %w (`module issue` again hands a new one)", name, m.Module, node, err)
}
return nil
}
+157
View File
@@ -0,0 +1,157 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Every other test mints without a vault to hand to: a stand-in that takes anything, replaced by the tests below.
func init() {
vaultAskerForIssue = func(context.Context) vaultAsker { return newStandInVault(nil).ask }
}
// A covered account is handed to the vault when it is minted (novox/hq ADR 0374, issue 787): the whole value, sealed
// to the key the vault made for the call, with origin issued. A module that is neither core nor moved hands nothing.
func TestModuleIssueHandsACoveredAccountToTheVaultAndOnlyThat(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
theVaultIsInstalled(t, open)
inv := open.inventory
known := broker.Broker{Fingerprint: "ab"}
core := aProvisioner()
core.Module = "nats"
moved := aBot("moved-bot", true)
moved.OwnSecrets["broker"] = catalogue.OwnSecret{Path: "/var/lib/moved-bot/broker"}
plain := aBot("kept-bot", false)
plain.OwnSecrets["broker"] = catalogue.OwnSecret{Path: "/var/lib/kept-bot/broker"}
vault := newStandInVault(t)
vaultAskerForIssue = func(context.Context) vaultAsker { return vault.ask }
t.Cleanup(func() {
vaultAskerForIssue = func(context.Context) vaultAsker { return newStandInVault(nil).ask }
})
for _, m := range []catalogue.Manifest{core, moved, plain} {
register(t, open, m)
if _, err := assign(ctx, open, "laptop", m.Module); err != nil {
t.Fatal(err)
}
}
issue := func(m catalogue.Manifest, password string) {
t.Helper()
if err := issueWith(ctx, inv, m, "laptop", "", known, "10.0.0.1:4222", "module."+m.Module+".laptop", password); err != nil {
t.Fatal(err)
}
}
issue(core, "first-core-password")
issue(moved, "first-moved-password")
issue(plain, "kept-password")
if err := issueProvisionerClient(ctx, inv, core, "laptop", known, "10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
said := vault.said()
for _, want := range []string{`"module":"nats"`, `"module":"moved-bot"`, `"name":"broker"`,
`"name":"` + broker.ProvisionerSecret + `"`, `"node":"laptop"`} {
if !strings.Contains(said, want) {
t.Errorf("the vault was told %s, without %s", said, want)
}
}
if n := strings.Count(said, `"origin":"issued"`); n != 3 {
t.Errorf("%d accounts were handed with origin issued, want 3 (the core's two and the moved module's): %s", n, said)
}
if strings.Contains(said, "kept-bot") {
t.Errorf("an account that is neither core nor moved was handed to the vault: %s", said)
}
if strings.Contains(said, "first-core-password") || strings.Contains(said, "first-moved-password") {
t.Errorf("a value crossed the bus in the clear: %s", said)
}
// A re-issue hands the new value, which the vault opens: the whole {url, fingerprint, node, user, password}.
issue(moved, "second-moved-password")
if got := vault.taken["accept"]; !strings.Contains(got, "second-moved-password") ||
!strings.Contains(got, `"user":"module.moved-bot.laptop"`) || !strings.Contains(got, `"node":"laptop"`) ||
!strings.Contains(got, `"url":"nats://10.0.0.1:4222"`) || !strings.Contains(got, `"fingerprint":"ab"`) {
t.Errorf("the vault was handed %q after a re-issue, not the new account whole", got)
}
}
// A vault that does not take the value fails the issue loudly: the bus's user list holds the new hash already.
func TestModuleIssueFailsWhenTheVaultDoesNotTakeACoveredAccount(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
moved := aBot("moved-bot", true)
moved.OwnSecrets["broker"] = catalogue.OwnSecret{Path: "/var/lib/moved-bot/broker"}
theVaultIsInstalled(t, open)
register(t, open, moved)
if _, err := assign(ctx, open, "laptop", moved.Module); err != nil {
t.Fatal(err)
}
vault := newStandInVault(t)
vault.refuse = errors.New("the vault is not there")
vaultAskerForIssue = func(context.Context) vaultAsker { return vault.ask }
t.Cleanup(func() {
vaultAskerForIssue = func(context.Context) vaultAsker { return newStandInVault(nil).ask }
})
err := issueWith(ctx, open.inventory, moved, "laptop", "", broker.Broker{Fingerprint: "ab"}, "10.0.0.1:4222",
"module.moved-bot.laptop", "pw")
if err == nil || !strings.Contains(err.Error(), "vault did not take") {
t.Fatalf("a vault that took nothing was reported as %v", err)
}
}
func theVaultIsInstalled(t *testing.T, open *stores) {
t.Helper()
vault := catalogue.Manifest{Module: "mesh-vault", Version: "1"}
register(t, open, vault)
if _, err := assign(t.Context(), open, "laptop", vault.Module); err != nil {
t.Fatal(err)
}
}
// In genesis the bus's and the provisioner's accounts are issued before the vault is installed (ADR 0113 §15 to §18):
// there is no vault to hand to, so nothing is asked and nothing fails, and the value stays in the controller's store
// as before, for the move to carry into the vault. Once the vault is installed, one that does not answer fails.
func TestModuleIssueBeforeTheVaultIsInstalledKeepsTheValueAndHandsNothing(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
known := broker.Broker{Fingerprint: "ab"}
core := aProvisioner()
core.Module = "nats"
register(t, open, core)
if _, err := assign(ctx, open, "laptop", core.Module); err != nil {
t.Fatal(err)
}
vault := newStandInVault(t)
vault.refuse = errors.New("there is no vault yet")
vaultAskerForIssue = func(context.Context) vaultAsker { return vault.ask }
t.Cleanup(func() {
vaultAskerForIssue = func(context.Context) vaultAsker { return newStandInVault(nil).ask }
})
if err := issueWith(ctx, inv, core, "laptop", "", known, "10.0.0.1:4222", "module.nats.laptop", "pw"); err != nil {
t.Fatalf("genesis's issue of the bus's account failed with no vault installed: %v", err)
}
if err := issueProvisionerClient(ctx, inv, core, "laptop", known, "10.0.0.1:4222"); err != nil {
t.Fatalf("genesis's issue of the client's account failed with no vault installed: %v", err)
}
if said := vault.said(); said != "" {
t.Errorf("the vault was asked before it was installed: %s", said)
}
if _, held, err := inv.ModuleSecretIfIssued(ctx, "laptop", "nats", "broker"); err != nil || !held {
t.Errorf("the value is not kept in the controller's store (held %v, %v)", held, err)
}
theVaultIsInstalled(t, open)
if err := issueWith(ctx, inv, core, "laptop", "", known, "10.0.0.1:4222", "module.nats.laptop", "pw2"); err == nil {
t.Error("a vault that is installed and does not answer did not fail the issue")
}
}
+242
View File
@@ -0,0 +1,242 @@
package main
import (
"context"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq ADR 0360 §4, §5: a walk ends on its machines' reports — every machine sent has reported on its send,
// or is silent past ApplySilentAfter — and names a failed or refused apply; it never waits for a health reading.
func TestAWalkEndsOnItsMachinesReports(t *testing.T) {
now := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
sent := now.Add(-time.Minute)
long := now.Add(-inventory.ApplySilentAfter - time.Minute)
at := now.Add(-10 * time.Second)
p := inventory.Plan{Modules: map[string]*inventory.PlanModule{
"app": {Rest: map[string]inventory.SentDeclaration{
"ace": {Digest: "d-ace", At: sent},
"novox": {Digest: "d-novox", At: sent},
"shanks": {Digest: "d-shanks", At: sent},
"laptop": {Digest: "d-laptop", At: long},
}},
"proxy": {Rest: map[string]inventory.SentDeclaration{"ace": {Digest: "d-ace", At: sent}}},
}}
reports := []inventory.Reported{
{Node: "ace", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-ace"},
{Node: "novox", Outcome: inventory.OutcomeFailed, At: &at, Declared: "d-novox"},
{Node: "shanks", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-before"},
}
waiting, failed, silent := walkReports(p, reports, now, nil)
if !slices.Equal(waiting, []string{"shanks"}) || !slices.Equal(failed, []string{"novox"}) ||
!slices.Equal(silent, []string{"laptop"}) {
t.Fatalf("waiting %v, failed %v, silent %v", waiting, failed, silent)
}
reports[2].Declared = "d-shanks"
if waiting, _, _ := walkReports(p, reports, now, nil); len(waiting) != 0 {
t.Fatalf("the walk still waits once every machine reported: %v", waiting)
}
}
// withTheGate turns the first-machine gate on for a test of its own path: kept for the walks a controller before
// novox/hq ADR 0360 started.
func withTheGate(t *testing.T) {
t.Helper()
was := gateFirstMachine
gateFirstMachine = true
t.Cleanup(func() { gateFirstMachine = was })
}
// Every machine heard from is sent at once and passed on its applied report, with no judging (novox/hq ADR 0360 §7);
// before, the first machine was judged before the rest were sent (ADR 0353 §1).
func TestAReleaseSendsEveryMachineAtOnce(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
said := events(t)
for i := 0; i < 12; i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
if p.State != inventory.PlanDone || len(p.Release.Done) != 3 {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
for _, n := range []string{"desk", "laptop", "anchor"} {
if !slices.Contains(*said, "send "+n) {
t.Fatalf("%s was not sent: %v", n, *said)
}
}
if slices.Contains(*said, "judge") {
t.Fatalf("a machine was judged by its health: %v", *said)
}
if v, found, err := b.open.inventory.GateOf(ctx, "build-late-c2"); err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("late's pass on laptop was not kept: %+v %v %v", v, found, err)
}
}
// A machine that reports its apply failed is named and fails the release, with nothing put back, and every other
// machine goes on to its own report (novox/hq ADR 0360 §5, §7); before, it was put back (ADR 0353 §3).
func TestAMachineThatFailsItsApplyIsNamedAndNothingIsPutBack(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
inv := b.open.inventory
b.broken["desk"] = true // desk fails what it is sent
advancePlans(ctx, b.open)
for i := 0; i < 12 && b.release(t).Open(); i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
if p.State != inventory.PlanFailed || !slices.Equal(p.Release.Failed, []string{"desk"}) {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
if !slices.Contains(p.Release.Done, "laptop") || !slices.Contains(p.Release.Done, "anchor") {
t.Fatalf("the others have no verdict of their own: %+v", p.Release)
}
if v, found, err := inv.GateOf(ctx, "build-late-c2"); err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("late's pass on laptop was not kept: %+v %v %v", v, found, err)
}
if sent, _, _ := inv.SentBuilds(ctx, "desk"); sent["app"] != "c2" {
t.Fatalf("desk was put back: %v", sent)
}
}
// A release's send kept before it was made, and made, is judged by the controller that takes over without being
// made again (ADR 0353 §8), with every machine sent at once (ADR 0360 §7).
func TestAReleaseSendKeptAndMadeIsNotMadeAgainAtOnce(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
p, err := releaseBacklog(ctx, b.open, "")
if err != nil || p == nil {
t.Fatalf("no release opened: %v", err)
}
if _, err := advanceRelease(ctx, b.open, p); err != nil || p.Release.Gates["anchor"] == nil || !p.Release.Gates["anchor"].Pending {
t.Fatalf("the send was not kept before it was made: %+v %v", p.Release, err)
}
if err := inv.SavePlan(ctx, p); err != nil {
t.Fatal(err)
}
// The send is made, and the controller is replaced before it keeps it.
if _, err := sendRollout(ctx, b.open, []string{"anchor"}); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
if n := len(slices.DeleteFunc(slices.Clone(b.sent), func(s []string) bool { return !slices.Equal(s, []string{"anchor"}) })); n != 1 {
t.Fatalf("anchor was sent again: %v", b.sent)
}
g := b.release(t).Release.Gates["anchor"]
if g == nil || g.Pending || g.Since == nil || len(g.Carried) == 0 {
t.Fatalf("anchor is not judged on the send made: %+v", g)
}
}
// A machine of the rest that says nothing on its send is left silent after fifteen minutes: not failed, not put
// back; a rest that went silent whole ends failed, said as a condition (ADR 0353 §4); its step also sent anchor, so the release is done, not failed (ADR 0360 §7).
func TestAMachineSilentPastItsBoundIsLeftAndSaidAtOnce(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
reporting := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
if slices.Equal(names, []string{"laptop"}) { // laptop sleeps: sent, and says nothing
b.sent = append(b.sent, names)
current, _ := open.inventory.CurrentBuilds(ctx)
return names, open.inventory.RecordSent(ctx, nodeID(t, open, "laptop"), "d-laptop-asleep",
map[string]string{"app": current["app"].Commit, "late": current["late"].Commit, "same": current["same"].Commit})
}
return reporting(ctx, open, names)
}
t.Cleanup(func() { sendRollout = reporting })
for i := 0; i < 8; i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
g := p.Release.Gates["laptop"]
if !p.Open() || g == nil || g.Verdict != "" || g.Silent {
t.Fatalf("laptop is judged before fifteen minutes: %s %s %+v", p.State, p.Note, g)
}
since := time.Now().Add(-inventory.ApplySilentAfter - time.Minute)
g.Since = &since
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
p = b.release(t)
// anchor applied in the same step, so the release is done, laptop named silent (novox/hq ADR 0360 §7).
if p.State != inventory.PlanDone || !slices.Equal(p.Release.Silent, []string{"laptop"}) || len(p.Release.Failed) > 0 {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
if failed, _ := inv.GateFailed(ctx, "build-late-c2"); failed {
t.Fatal("a silent machine's build was marked failed")
}
if sent, _, _ := inv.SentBuilds(ctx, "laptop"); sent["late"] != "c2" {
t.Fatalf("laptop was put back: %v", sent)
}
obs, err := silentObservation(ctx, inv)
if err != nil {
t.Fatal(err)
}
if len(obs) != 1 || obs[0].Kind != "release-silent" || !slices.Equal(obs[0].Also, []string{"laptop"}) {
t.Fatalf("the silence is not said: %+v", obs)
}
}
// Issue 840 (novox/hq ADR 0362 §1-5): a walk's end counts a machine's send of a provisioned module applied only once
// the engine and the machine's provisioner client both said so.
func TestAWalkEndWaitsForTheProvisioner(t *testing.T) {
now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
at := now.Add(-10 * time.Second)
plan := func(sentAt time.Time) inventory.Plan {
return inventory.Plan{Modules: map[string]*inventory.PlanModule{"app": {Rest: map[string]inventory.SentDeclaration{
"ace": {Digest: "d-ace", At: sentAt, Provisioned: map[string]string{"app": "t1"}},
"g14": {Digest: "d-g14", At: sentAt, Provisioned: map[string]string{"app": "t1"}},
}}}}
}
reports := []inventory.Reported{
{Node: "ace", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-ace"},
{Node: "g14", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-g14"},
}
says := map[string]inventory.ProvReport{}
prov := func(node, module string) (inventory.ProvReport, bool) { r, ok := says[node+"/"+module]; return r, ok }
applied := inventory.ProvReport{Status: inventory.ProvApplied, Templates: "t1", At: at}
// (a) the engine applied, the client has not said: the end waits on both.
waiting, failed, silent := walkReports(plan(now.Add(-time.Minute)), reports, now, prov)
if !slices.Equal(waiting, []string{"ace", "g14"}) || len(failed)+len(silent) != 0 {
t.Fatalf("a: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
// ace's client applied, g14's failed on that build (b): ace done, g14 a failed apply naming the reason.
says["ace/app"] = applied
says["g14/app"] = inventory.ProvReport{Status: inventory.ProvFailed, Templates: "t1", Reason: "no value", At: at}
waiting, failed, silent = walkReports(plan(now.Add(-time.Minute)), reports, now, prov)
if len(waiting) != 0 || len(silent) != 0 || len(failed) != 1 || !strings.HasPrefix(failed[0], "g14 (") ||
!strings.Contains(failed[0], "no value") {
t.Fatalf("b: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
// (c) no client report past ApplySilentAfter: named silent, never applied.
delete(says, "g14/app")
waiting, failed, silent = walkReports(plan(now.Add(-inventory.ApplySilentAfter-time.Minute)), reports, now, prov)
if len(waiting) != 0 || len(failed) != 0 || !slices.Equal(silent, []string{"g14"}) {
t.Fatalf("c: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
// both said it: nothing waits.
says["g14/app"] = applied
if waiting, failed, silent = walkReports(plan(now.Add(-time.Minute)), reports, now, prov); len(waiting)+len(failed)+len(silent) != 0 {
t.Fatalf("both said: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
// (d) a send handing nothing to the provisioner is the engine's report alone, as before.
bare := plan(now.Add(-time.Minute))
bare.Modules["app"].Rest = map[string]inventory.SentDeclaration{"ace": {Digest: "d-ace", At: now.Add(-time.Minute)}}
if waiting, failed, silent = walkReports(bare, reports, now, func(string, string) (inventory.ProvReport, bool) {
return inventory.ProvReport{}, false
}); len(waiting)+len(failed)+len(silent) != 0 {
t.Fatalf("d: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
}
+67
View File
@@ -0,0 +1,67 @@
package main
import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// localNetworksOf is a machine's attached networks judged (novox/hq ADR 0340): which count as its local
// network, from what it last said of itself and the links it said face outside it. A machine that has
// said none judges to nothing, and a port reached from `local-network` is then admitted from the private
// network only, and said so.
func localNetworksOf(ctx context.Context, inv *inventory.Inventory, node string) ([]catalogue.LocalNetwork, error) {
account, had, err := inv.MachineOf(ctx, node)
if err != nil {
return nil, err
}
outward, err := inv.OutwardLinksOf(ctx, node)
if err != nil {
return nil, err
}
return judgedNetworks(account, had, outward, overlay.Interface), nil
}
// judgedNetworks is the judging alone, from what was read.
func judgedNetworks(account inventory.MachineAccount, had bool, outward []string, tunnel string) []catalogue.LocalNetwork {
if !had || account.Machine == nil {
return nil
}
attached := make([]catalogue.AttachedNetwork, 0, len(account.Machine.Networks))
for _, n := range account.Machine.Networks {
attached = append(attached, catalogue.AttachedNetwork{Link: n.Link, Network: n.Network})
}
return catalogue.JudgeLocalNetworks(attached, outward, tunnel)
}
// localNetworkLines is what `node` shows of a machine's attached networks (ADR 0340 §8): each one, with
// whether it counts as the local network and why, or why none is shown.
func localNetworkLines(account inventory.MachineAccount, had bool, judged []catalogue.LocalNetwork) []string {
out := []string{" its local network (what a port reached from local-network is admitted from):"}
switch {
case !had || account.Machine == nil:
return append(out, " not said — this machine has not reported what it is attached to")
case len(account.Machine.Networks) == 0:
why := account.Machine.Unknown["networks"]
if why == "" {
why = "its node-engine is older than the one that says it"
}
return append(out, " not said — "+why)
}
counted := 0
for _, j := range judged {
verdict := "not counted"
if j.Counts {
verdict = "counts"
counted++
}
out = append(out, fmt.Sprintf(" %-22s on %-10s %s: %s", j.Network, j.Link, verdict, j.Why))
}
if counted == 0 {
out = append(out, " none counts: a port reached from local-network is admitted from the private network only")
}
return out
}
+65
View File
@@ -0,0 +1,65 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// **node shows what the local network resolved to** (novox/hq ADR 0340 §8): every attached network,
// whether it counts, and why the others do not — the public subnet named as never counted.
func TestNodeShowsWhichAttachedNetworksCountAndWhy(t *testing.T) {
account := inventory.MachineAccount{Architecture: "amd64", Kernel: "linux", Machine: &inventory.MadeOf{
Networks: []inventory.AttachedNetwork{
{Link: "eth0", Network: "192.168.1.10/24"},
{Link: "eth1", Network: "203.0.113.7/24"},
{Link: "wg-mesh", Network: "10.10.0.4/24"},
}}}
judged := judgedNetworks(account, true, []string{"eth0", "eth1"}, "wg-mesh")
text := strings.Join(localNetworkLines(account, true, judged), "\n")
for _, want := range []string{
"192.168.1.0/24", "counts: in a private range (RFC 1918, 192.168.0.0/16)",
"203.0.113.0/24", "not counted: public, never counted",
"10.10.0.0/24", "the private network's own link",
} {
if !strings.Contains(text, want) {
t.Errorf("node does not say %q:\n%s", want, text)
}
}
}
// A machine that said none, or whose node-engine is older, is said as that — never as an empty list
// that reads like a machine on no network.
func TestNodeSaysWhenNoNetworkWasSaid(t *testing.T) {
older := inventory.MachineAccount{Machine: &inventory.MadeOf{}}
if text := strings.Join(localNetworkLines(older, true, nil), "\n"); !strings.Contains(text, "older than the one that says it") {
t.Errorf("an older node-engine is not said:\n%s", text)
}
public := inventory.MachineAccount{Machine: &inventory.MadeOf{
Networks: []inventory.AttachedNetwork{{Link: "eth0", Network: "198.51.100.9/24"}}}}
judged := judgedNetworks(public, true, []string{"eth0"}, "wg-mesh")
if text := strings.Join(localNetworkLines(public, true, judged), "\n"); !strings.Contains(text, "none counts") {
t.Errorf("a machine on a public network only is not said to have no local network:\n%s", text)
}
if text := strings.Join(localNetworkLines(inventory.MachineAccount{}, false, nil), "\n"); !strings.Contains(text, "not said") {
t.Errorf("a machine that never reported is not said:\n%s", text)
}
}
// The preview of a flip says what a port from the local network keeps: the local networks it stays
// open to, or that the machine reported none (ADR 0340 §6).
func TestTheFlipPreviewSaysWhatALocalNetworkPortKeeps(t *testing.T) {
rules := []catalogue.Rule{{Port: 1883, Protocol: "tcp", From: catalogue.FromLocalNetwork, Because: []string{"mosquitto"}}}
at := inventory.Reach{Protocol: "tcp", Address: "0.0.0.0", Port: 1883}
with := derivedFilter{rules: rules, mesh: []string{"10.10.0.1"}, outwardLinks: []string{"eth0"},
local: []string{"192.168.1.0/24"}}
if fate := with.fate(at); !strings.Contains(fate, "192.168.1.0/24") {
t.Errorf("the preview says %q, without the local network it stays open to", fate)
}
with.local = nil
if fate := with.fate(at); !strings.Contains(fate, "reported none") {
t.Errorf("the preview says %q, without saying no local network was reported", fate)
}
}
+27 -2
View File
@@ -104,6 +104,10 @@ func run() error {
return replayCommand(ctx, args[1:]) return replayCommand(ctx, args[1:])
case "kill": case "kill":
return killCommand(ctx, args[1:]) return killCommand(ctx, args[1:])
case "forget-checks":
return forgetChecksCommand(ctx, args[1:])
case "try":
return tryCommand(ctx, args[1:])
case "pause", "resume": case "pause", "resume":
return pauseCommand(ctx, args[0], args[1:]) return pauseCommand(ctx, args[0], args[1:])
case "collection": case "collection":
@@ -117,6 +121,10 @@ func run() error {
return imagesCommand(ctx, args[1:]) return imagesCommand(ctx, args[1:])
case "mirrors": case "mirrors":
return mirrorsCommand(ctx, args[1:]) return mirrorsCommand(ctx, args[1:])
case "move-placement":
return movePlacementCommand(ctx, args[1:])
case "move-config":
return moveConfigCommand(ctx, args[1:])
case "plans": case "plans":
return plansCommand(ctx, args[1:]) return plansCommand(ctx, args[1:])
case "delivery": case "delivery":
@@ -271,6 +279,10 @@ func usage() {
artifacts [--json] [--repository <r>] [--collected] every artifact the mesh made: kept and why, eligible, collected (ADR 0251) artifacts [--json] [--repository <r>] [--collected] every artifact the mesh made: kept and why, eligible, collected (ADR 0251)
collect [--json] [--most <n>] what the store's sweep would let go of, now; nothing is done collect [--json] [--most <n>] what the store's sweep would let go of, now; nothing is done
collect --confirm --why <text> [--most <n>] run the sweep now: hold every kept archive, let go of the eligible collect --confirm --why <text> [--most <n>] run the sweep now: hold every kept archive, let go of the eligible
move-placement [--json] which settings layers still hold placement keys (endpoints, expose, reach, ports); nothing is moved
move-placement --confirm --why <text> move them beside the assignment, once, after the delivery (issue 736)
move-config [--json] every config value the settings layers hold, and where in mesh-config it goes; nothing is set
move-config --confirm --why <text> set them in mesh-config, read them back, and read config from there (issue 664)
images <node> [--json] every container image the machine's declaration names, now and as last sent images <node> [--json] every container image the machine's declaration names, now and as last sent
seats [--json] every seat this mesh defines, what it delivers, and who holds it seats [--json] every seat this mesh defines, what it delivers, and who holds it
seat rename <from> <to> rename a seat; its former name still resolves (ADR 0122) seat rename <from> <to> rename a seat; its former name still resolves (ADR 0122)
@@ -327,7 +339,7 @@ func usage() {
licence add|list|use|key model access, under the name a person calls it licence add|list|use|key model access, under the name a person calls it
licence manager <name> <node> the node that holds a refreshable licence's refresh token licence manager <name> <node> the node that holds a refreshable licence's refresh token
licence refresh <name> mint a new access token and seal it to every holder licence refresh <name> mint a new access token and seal it to every holder
rotate <provision> [--consumer <n>] [--module <m>] a new credential for every holder, both ends at once rotate <provision> [--consumer <n>] [--module <m>] [--why <text>] a new credential for every holder, both ends at once (--why where the vault holds it)
ask <module> <tool> [json] call one of a module's tools over the broker, and print its answer ask <module> <tool> [json] call one of a module's tools over the broker, and print its answer
pin <node> <provision> <from-node> <module> pin <node> <provision> <from-node> <module>
which provider this one gets a provision from: the module, and its node which provider this one gets a provision from: the module, and its node
@@ -379,6 +391,13 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
// **A merge check builds nothing** (novox/hq to-be 45 §9): its verdict is said, and nothing of it is // **A merge check builds nothing** (novox/hq to-be 45 §9): its verdict is said, and nothing of it is
// recorded or registered. // recorded or registered.
if result.Check != nil || result.Checked != nil { if result.Check != nil || result.Checked != nil {
if isTry(result) {
// A branch tried before its pull request (novox/hq issue 686): its asker reads the outcome from the
// build's log. Never said as `checked`, so the forge's holder sets no status.
fmt.Printf("%s: a try of %s at %.8s, read with `try --id`: said to nobody, nothing taken in\n", result.ID,
result.Repository, result.Ref)
return nil
}
checked(ctx, result) checked(ctx, result)
return nil return nil
} }
@@ -429,8 +448,14 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
} }
fmt.Printf("%s: %s %s registered, built on %s from %s\n", fmt.Printf("%s: %s %s registered, built on %s from %s\n",
result.ID, manifest.Module, manifest.Version, result.On, short(result.Commit)) result.ID, manifest.Module, manifest.Version, result.On, short(result.Commit))
// What the walk's build did, kept on the walk whose ask it answers (novox/hq ADR 0350 §4); a build seat from
// before it says nothing, and that is what is kept.
made := result.MadeWords()
if result.Reused != "" || result.BuiltWhy != "" {
fmt.Printf("%s: %s %s\n", result.ID, manifest.Module, made)
}
saysWhenThePolicyActs(ctx, b.inv, manifest.Module) saysWhenThePolicyActs(ctx, b.inv, manifest.Module)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked, result.ID) planBuiltSaying(ctx, b.open, manifest.Module, result.Commit, "", asked, result.ID, made)
// A module registered may be one a machine is now behind: `status` is composed again. // A module registered may be one a machine is now behind: `status` is composed again.
statusFrom.nudge() statusFrom.nudge()
return nil return nil
+48 -5
View File
@@ -550,8 +550,17 @@ func shelfWithChange(base map[string]catalogue.Manifest, sources map[string]inve
out[k] = m out[k] = m
} }
var failures []string var failures []string
// The list of predefined keys at the tree's root (novox/hq ADR 0348 §7): what the build of this tree judges each
// manifest against and registration publishes, so what the gate judges each against too. A list that is
// wrong is refused by name, and every manifest is then judged against none.
predefined, err := catalogue.ReadPredefined(tree)
if err != nil {
failures = append(failures, fmt.Sprintf("%s: the controller judging this (%s) refuses it — %s",
catalogue.PredefinedFile, version, oneLine(err.Error())))
predefined = nil
}
found := map[string]string{} // module → its directory in the tree found := map[string]string{} // module → its directory in the tree
err := filepath.WalkDir(tree, func(path string, d fs.DirEntry, err error) error { err = filepath.WalkDir(tree, func(path string, d fs.DirEntry, err error) error {
if err != nil { if err != nil {
return err return err
} }
@@ -575,7 +584,19 @@ func shelfWithChange(base map[string]catalogue.Manifest, sources map[string]inve
if err != nil { if err != nil {
return err return err
} }
m, err := catalogue.ParseManifest(raw) m, err := catalogue.ParseManifestWith(raw, predefined)
// The manifest as written, still naming its templates: what the stand-in build resolves, as a real build
// does, so a moved module is handed its templates (issue 876).
written := m
var texts map[string]string
if err == nil {
// Each template's text from the tree beside the manifest, as the build reads it back from the archive it
// packs (novox/hq ADR 0349 §2): so a change to a template alone is a change to its module.
m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(path))
}
if err == nil && written.Templated() {
texts, err = catalogue.ReadTemplates(written, filepath.Dir(path))
}
if err != nil { if err != nil {
if inScope(dir) { if inScope(dir) {
failures = append(failures, fmt.Sprintf("%s: the controller judging this (%s) refuses it — %s", failures = append(failures, fmt.Sprintf("%s: the controller judging this (%s) refuses it — %s",
@@ -600,12 +621,12 @@ func shelfWithChange(base map[string]catalogue.Manifest, sources map[string]inve
"of one module name (issue 239)", orRoot(dir), m.Module, s.Repository)) "of one module name (issue 239)", orRoot(dir), m.Module, s.Repository))
return nil return nil
} }
resolved, err := standInBuild(m) resolved, err := standInBuild(written, texts)
if err == nil { if err == nil {
// As registration reads a built manifest: strictly, again (takeIn). // As registration reads a built manifest: strictly, again (takeIn).
var raw []byte var raw []byte
if raw, err = json.Marshal(resolved); err == nil { if raw, err = json.Marshal(resolved); err == nil {
resolved, err = catalogue.ParseManifest(raw) resolved, err = catalogue.ParseManifestWith(raw, predefined)
} }
} }
if err == nil { if err == nil {
@@ -723,15 +744,24 @@ func builtAlike(m catalogue.Manifest) string {
// standInBuild resolves a manifest as a build would, with artifacts nobody built: a digest made from the // standInBuild resolves a manifest as a build would, with artifacts nobody built: a digest made from the
// module's and the artifact's names, in the store's own vocabulary. Composition and validation read the // module's and the artifact's names, in the store's own vocabulary. Composition and validation read the
// shape of a reference, never its bytes. // shape of a reference, never its bytes.
func standInBuild(m catalogue.Manifest) (catalogue.Manifest, error) { func standInBuild(m catalogue.Manifest, texts map[string]string) (catalogue.Manifest, error) {
if m.Build == nil { if m.Build == nil {
return m.Resolve(nil) return m.Resolve(nil)
} }
// The templates archive carries each template's text by its path inside the archive, as a build reads it back.
archive, inside := catalogue.TemplatesArtifact(m), m.TemplatesDir()+"/"
read := map[string]string{}
for template, text := range texts {
read[strings.TrimPrefix(template, inside)] = text
}
var built []catalogue.Built var built []catalogue.Built
for _, a := range m.Build.Artifacts { for _, a := range m.Build.Artifacts {
sum := sha256.Sum256([]byte("gate\x00" + m.Module + "\x00" + a.Name)) sum := sha256.Sum256([]byte("gate\x00" + m.Module + "\x00" + a.Name))
digest := fmt.Sprintf("sha256:%x", sum) digest := fmt.Sprintf("sha256:%x", sum)
b := catalogue.Built{Name: a.Name, Kind: a.Kind, Digest: digest} b := catalogue.Built{Name: a.Name, Kind: a.Kind, Digest: digest}
if a.Name == archive && archive != "" {
b.Texts = read
}
switch a.Kind { switch a.Kind {
case catalogue.ArtifactImage, catalogue.ArtifactUpstream: case catalogue.ArtifactImage, catalogue.ArtifactUpstream:
b.Reference = catalogue.ArtifactStoreScheme + m.Module + "/" + a.Name + "@" + digest b.Reference = catalogue.ArtifactStoreScheme + m.Module + "/" + a.Name + "@" + digest
@@ -1083,6 +1113,19 @@ func raiseFromFacts(ctx context.Context, open *stores, f snapshot.Facts, shelf m
continue continue
} }
} }
// And the provisioner client's account, the same way (novox/hq ADR 0349 clause 11, design 57 P3).
if a.Provider == "" && a.Name == broker.ProvisionerSecret {
if pm, held := shelf[a.Module]; !held || !runsTheProvisionerClient(pm) {
continue
}
if _, err := inv.MintBusPassword(ctx, inventory.BusUser{Kind: inventory.BusProvisioner, Node: m.Name,
Module: a.Module, Username: broker.Principal{Kind: broker.KindProvisioner, Node: m.Name,
Module: a.Module}.Username()}); err != nil {
notes = append(notes, fmt.Sprintf("%s's provisioner client account for %s is not kept: %s",
m.Described(), a.Module, oneLine(err.Error())))
continue
}
}
var err error var err error
if a.Provider == "" { if a.Provider == "" {
err = inv.AcceptSecretForModule(ctx, m.Name, a.Module, a.Name, standIn) err = inv.AcceptSecretForModule(ctx, m.Name, a.Module, a.Name, standIn)
+15 -1
View File
@@ -33,7 +33,21 @@ import (
func aMesh(t *testing.T) *stores { func aMesh(t *testing.T) *stores {
t.Helper() t.Helper()
inventory.ForTest(t) // raises the store, migrates it, and points the environment at it inventory.ForTest(t) // raises the store, migrates it, and points the environment at it
licences.ForTest(t) // planning reaches this one too, by name and never by connection return aMeshOnTheStore(t)
}
// aMeshBefore is aMesh on a store migrated only below migration `number`, with the function that runs the
// rest: for a test of what a migration that moves data does to what the mesh composes (novox/hq issue 651).
func aMeshBefore(t *testing.T, number int) (*stores, func()) {
t.Helper()
_, migrate := inventory.ForTestBefore(t, number)
return aMeshOnTheStore(t), migrate
}
// aMeshOnTheStore is the mesh aMesh makes, on the store the environment points at.
func aMeshOnTheStore(t *testing.T) *stores {
t.Helper()
licences.ForTest(t) // planning reaches this one too, by name and never by connection
open, err := openStores(t.Context()) open, err := openStores(t.Context())
if err != nil { if err != nil {
+96 -3
View File
@@ -7,7 +7,10 @@ import (
"fmt" "fmt"
"os/exec" "os/exec"
"slices" "slices"
"sort"
"strings" "strings"
"sync"
"time"
"github.com/novox/mesh-controller/internal/inventory" "github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link" "github.com/novox/mesh-controller/internal/link"
@@ -123,8 +126,82 @@ func answerMeshCLI(inv *inventory.Inventory) link.CLIHandler {
// cliJournal says one line in the controller's journal (its standard output); a variable so a test can read it. // cliJournal says one line in the controller's journal (its standard output); a variable so a test can read it.
var cliJournal = func(line string) { fmt.Println(line) } var cliJournal = func(line string) { fmt.Println(line) }
// runForMeshCLI runs a judged line and answers what it said. Every line is said in the journal first: its call, // onlyReads says a line only reads, judged by the generic `command` verb's reading forms (novox/hq ADR 0266) — the
// who asked on which node, its command word only, and how it runs (review of ADR 0272). // same allow list that decides what an ordinary call may run, never a second list kept here — judged on the line's
// every word, so `plans go`, `conditions silence` and `settings set` are not reads while `plans`, `conditions` and
// `settings show` are. A line those forms do not name is not judged a read, and is said (novox/hq issue 762).
func onlyReads(asked link.CLIAsked, v cliVerdict) bool {
if v.refused != "" || len(asked.Line) == 0 || len(asked.Stdin) > 0 || cliServers[asked.Line[0]] {
return false
}
return commandReads(asked.Line) == nil && terminalOnly(asked.Line) == nil
}
// readCalls counts the read calls said in no line, per caller, until the next hourly line says them.
type readCalls struct {
mu sync.Mutex
counts map[string]int
}
// cliReads are this controller's read calls not yet said.
var cliReads = &readCalls{}
func (r *readCalls) add(caller string) {
r.mu.Lock()
defer r.mu.Unlock()
if r.counts == nil {
r.counts = map[string]int{}
}
r.counts[caller]++
}
// said is the line that says the read calls counted since the last, and starts counting again; empty when there
// were none, so an idle hour says nothing.
func (r *readCalls) said(over time.Duration) string {
r.mu.Lock()
counts := r.counts
r.counts = nil
r.mu.Unlock()
if len(counts) == 0 {
return ""
}
callers := make([]string, 0, len(counts))
for c := range counts {
callers = append(callers, c)
}
sort.Strings(callers)
parts := make([]string, len(callers))
for i, c := range callers {
parts[i] = fmt.Sprintf("%s %d", c, counts[c])
}
return fmt.Sprintf("mesh-cli read calls in the last %s, said in no line of their own: %s", over,
strings.Join(parts, ", "))
}
// readCallsEvery is how often the count of read calls is said.
const readCallsEvery = time.Hour
// sayReadCalls says, every readCallsEvery until ctx ends, how many read calls each caller made — when any did — so
// a runaway poller stays visible while its calls write no line each (novox/hq issue 762).
func sayReadCalls(ctx context.Context) {
tick := time.NewTicker(readCallsEvery)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
if line := cliReads.said(readCallsEvery); line != "" {
cliJournal(line)
}
}
}
}
// runForMeshCLI runs a judged line and answers what it said. Every line that changes something, is refused or fails
// is said in the journal: its call, who asked on which node, its command word only, and how it runs (review of ADR
// 0272). A line that only reads and succeeds is counted instead, and the count said hourly (novox/hq issue 762): a
// page polling `plans` every few seconds would otherwise write thousands of lines a day.
func runForMeshCLI(ctx context.Context, node string, asked link.CLIAsked, v cliVerdict) link.CLIAnswer { func runForMeshCLI(ctx context.Context, node string, asked link.CLIAsked, v cliVerdict) link.CLIAnswer {
how := "as an ordinary call" how := "as an ordinary call"
switch { switch {
@@ -137,7 +214,23 @@ func runForMeshCLI(ctx context.Context, node string, asked link.CLIAsked, v cliV
if call == "" { if call == "" {
call = "(no call)" call = "(no call)"
} }
cliJournal(fmt.Sprintf("mesh-cli %s: %s on %s asked %q, %s", call, asked.Account, node, asked.Line[0], how)) line := fmt.Sprintf("mesh-cli %s: %s on %s asked %q, %s", call, asked.Account, node, asked.Line[0], how)
if !onlyReads(asked, v) {
cliJournal(line)
return runJudgedLine(ctx, node, asked, v)
}
answer := runJudgedLine(ctx, node, asked, v)
if answer.Exit != 0 || answer.Refused != "" {
// A read that was refused or failed is said, as a change is.
cliJournal(fmt.Sprintf("%s, and it failed (exit %d)", line, answer.Exit))
} else {
cliReads.add(fmt.Sprintf("%s on %s", asked.Account, node))
}
return answer
}
// runJudgedLine runs a line runForMeshCLI has said or counted.
func runJudgedLine(ctx context.Context, node string, asked link.CLIAsked, v cliVerdict) link.CLIAnswer {
if v.refused != "" { if v.refused != "" {
return link.CLIRefusal(v.refused) return link.CLIRefusal(v.refused)
} }
+123
View File
@@ -0,0 +1,123 @@
package main
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// echoFails makes the echoing test binary end with a failure, as a read that failed would.
const echoFails = "MESH_TEST_ECHO_FAILS"
// journalled runs lines through runForMeshCLI and answers what the journal said, with a fresh count of read calls.
func journalled(t *testing.T) *[]string {
t.Helper()
t.Setenv(echoEnvironment, "1")
said := &[]string{}
was, wasReads := cliJournal, cliReads
cliJournal = func(line string) { *said = append(*said, line) }
cliReads = &readCalls{}
t.Cleanup(func() { cliJournal, cliReads = was, wasReads })
return said
}
// **A call that only reads writes no journal line; one that changes something does** (novox/hq issue 762).
func TestAReadCallWritesNoJournalLine(t *testing.T) {
said := journalled(t)
ctx := link.WithCallID(context.Background(), "call-1")
ordinary := cliVerdict{why: "not the terminal"}
for _, line := range [][]string{{"plans"}, {"status"}, {"conditions"}, {"node", "list"}, {"builds"}, {"queue"},
{"settings", "show", "zsh"}, {"plans", "--json"}} {
if a := runForMeshCLI(ctx, "g14", cliAsked("jochen", 1000, line...), ordinary); a.Exit != 0 || a.Refused != "" {
t.Fatalf("%q did not run: %+v", line, a)
}
}
runForMeshCLI(ctx, "control", cliAsked("jochen", 1000, "plans"), cliVerdict{terminal: true, why: "the terminal"})
if len(*said) != 0 {
t.Fatalf("read calls wrote journal lines: %q", *said)
}
runForMeshCLI(ctx, "control", cliAsked("jochen", 1000, "assign", "g14", "zsh"), cliVerdict{terminal: true, why: "the terminal"})
if len(*said) != 1 || !strings.Contains((*said)[0], `asked "assign"`) {
t.Fatalf("a call that changes something was not said: %q", *said)
}
}
// **A refused or failed read keeps its line** (novox/hq issue 762).
func TestAFailedOrRefusedReadIsSaid(t *testing.T) {
said := journalled(t)
runForMeshCLI(context.Background(), "g14", cliAsked("root", 0, "plans"), cliVerdict{refused: "root is not answered"})
t.Setenv(echoFails, "1")
a := runForMeshCLI(context.Background(), "g14", cliAsked("jochen", 1000, "plans"), cliVerdict{why: "not the terminal"})
if a.Exit == 0 {
t.Fatalf("the read did not fail: %+v", a)
}
if len(*said) != 2 || !strings.Contains((*said)[0], "refused") || !strings.Contains((*said)[1], "failed (exit 3)") {
t.Fatalf("the journal said %q", *said)
}
if line := cliReads.said(readCallsEvery); line != "" {
t.Fatalf("a failed read was counted as a read: %q", line)
}
}
// **A verb is classed by what the call does, not its name** (novox/hq issue 762): the same command word reads with
// some arguments and writes with others, and a line no reading form names is said.
func TestAVerbIsClassedByItsArguments(t *testing.T) {
ordinary := cliVerdict{why: "not the terminal"}
for _, c := range []struct {
line []string
reads bool
}{
{[]string{"settings", "show", "zsh"}, true},
{[]string{"settings", "preferences"}, true},
{[]string{"settings", "set", "zsh", `{"a":1}`}, false},
{[]string{"settings", "clear", "zsh"}, false},
{[]string{"plans"}, true},
{[]string{"plans", "p-1"}, true},
{[]string{"plans", "go", "p-1"}, false},
{[]string{"plans", "--json", "stop", "p-1"}, false},
{[]string{"plans", "close", "p-1"}, false},
{[]string{"plans", "retry", "p-1"}, false},
{[]string{"conditions"}, true},
{[]string{"conditions", "show", "x"}, true},
{[]string{"conditions", "silence", "x"}, false},
{[]string{"doctor", "run"}, false},
{[]string{"node", "account", "g14", "x"}, false},
{[]string{"assign", "g14", "zsh"}, false},
{[]string{"a-command-added-later"}, false},
{[]string{"serve"}, false},
} {
if got := onlyReads(cliAsked("jochen", 1000, c.line...), ordinary); got != c.reads {
t.Errorf("%q judged a read: %v, want %v", c.line, got, c.reads)
}
}
if onlyReads(cliAsked("jochen", 1000, "plans"), cliVerdict{refused: "no"}) {
t.Error("a refused line judged a read")
}
stdin := cliAsked("jochen", 1000, "status")
stdin.Stdin = []byte("x")
if onlyReads(stdin, cliVerdict{terminal: true}) {
t.Error("a line carrying standard input judged a read")
}
}
// **Once an hour the read calls are counted per caller, only if there were any** (novox/hq issue 762).
func TestTheReadCallsAreCountedPerCaller(t *testing.T) {
said := journalled(t)
ordinary := cliVerdict{why: "not the terminal"}
for range 3 {
runForMeshCLI(context.Background(), "g14", cliAsked("jochen", 1000, "plans"), ordinary)
}
runForMeshCLI(context.Background(), "ace", cliAsked("jochen", 1000, "status"), ordinary)
if len(*said) != 0 {
t.Fatalf("read calls wrote lines: %q", *said)
}
line := cliReads.said(readCallsEvery)
if !strings.Contains(line, "jochen on ace 1, jochen on g14 3") || !strings.Contains(line, "1h0m0s") {
t.Fatalf("the hourly line said %q", line)
}
if again := cliReads.said(readCallsEvery); again != "" {
t.Fatalf("an hour with no read calls said %q", again)
}
}
+1 -1
View File
@@ -86,7 +86,7 @@ func owedToTheRecord(ctx context.Context, announced merges, m link.SourceMoved,
// catchingUpOnMerges reads back the forge's announcements on a timer, until the context ends. // catchingUpOnMerges reads back the forge's announcements on a timer, until the context ends.
func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) { func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) {
f := following{open} f := following{open: open}
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) { catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
entries, err := open.inventory.Catalogued(ctx) entries, err := open.inventory.Catalogued(ctx)
if err != nil { if err != nil {
+16 -1
View File
@@ -75,7 +75,7 @@ func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Ke
State: r.State, Reason: r.Reason, Since: r.Since, Streak: r.Streak, Restarts: r.Restarts, State: r.State, Reason: r.Reason, Since: r.Since, Streak: r.Streak, Restarts: r.Restarts,
Check: r.Check, Needs: r.Needs, Account: r.Account, Root: r.Root} Check: r.Check, Needs: r.Needs, Account: r.Account, Root: r.Root}
for _, w := range r.Waits { for _, w := range r.Waits {
kept.Waits = append(kept.Waits, inventory.Wait{Part: w.Part, Secret: w.Secret, Setting: w.Setting, What: w.What}) kept.Waits = append(kept.Waits, inventory.Wait{Part: w.Part, Secret: w.Secret, Setting: w.Setting, Provision: w.Provision, What: w.What})
} }
resources = append(resources, kept) resources = append(resources, kept)
} }
@@ -240,6 +240,7 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
continue continue
} }
o := moduleUnhealthyObservation(m, node, unhealthy[m]) o := moduleUnhealthyObservation(m, node, unhealthy[m])
declaredSeverity(&o, hold, m)
if hold != nil { if hold != nil {
if by, held := hold.heldWith(node, m, unhealthy[m]); held { if by, held := hold.heldWith(node, m, unhealthy[m]); held {
// Held under the provider's condition: nothing of its own, and the provider's says it waits. The // Held under the provider's condition: nothing of its own, and the provider's says it waits. The
@@ -347,6 +348,7 @@ func sayWaiters(ctx context.Context, k *conditions.Keeper, hold *holding, p cata
} else { } else {
_, rest := besideAWait(p.Module, rs) _, rest := besideAWait(p.Module, rs)
o = moduleUnhealthyObservation(p.Module, p.Node, rest) o = moduleUnhealthyObservation(p.Module, p.Node, rest)
declaredSeverity(&o, hold, p.Module)
if said, waits := personWait(p.Module, p.Node, rest); waits { if said, waits := personWait(p.Module, p.Node, rest); waits {
o = reloginObservation(p.Module, p.Node, said, operatorOn(ctx, hold.inv, p.Node), rest) o = reloginObservation(p.Module, p.Node, said, operatorOn(ctx, hold.inv, p.Node), rest)
} }
@@ -481,6 +483,19 @@ func sayWaitingOn(o *conditions.Observation, waiters []string) {
o.Explanation += fmt.Sprintf(" %d module(s) that depend on it wait for it.", len(waiters)) o.Explanation += fmt.Sprintf(" %d module(s) that depend on it wait for it.", len(waiters))
} }
// declaredSeverity raises a module's unhealthy condition urgent when its manifest declares `unhealthy` urgent
// (novox/hq ADR 0355 §1). Only that condition changes (ADR 0355 §4): a wait for a new login, a module that needs
// the operator and a directory used as found keep their severities, and without the field, or without a manifest
// to read it from, it stays the warning it was.
func declaredSeverity(o *conditions.Observation, hold *holding, module string) {
if o.Kind != kindModuleUnhealthy || hold == nil {
return
}
if m, ok := hold.manifestOf(module); ok && m.UnhealthyIsUrgent() {
o.Severity = conditions.Urgent
}
}
// moduleUnhealthyObservation is a module unhealthy on a machine, in words: the summary names the module, // moduleUnhealthyObservation is a module unhealthy on a machine, in words: the summary names the module,
// the machine and what is wrong with each resource; the detail — targets, streaks, since — is evidence. // the machine and what is wrong with each resource; the detail — targets, streaks, since — is evidence.
func moduleUnhealthyObservation(module, node string, rs []inventory.ResourceHealth) conditions.Observation { func moduleUnhealthyObservation(module, node string, rs []inventory.ResourceHealth) conditions.Observation {
+86 -10
View File
@@ -8,6 +8,7 @@ import (
"fmt" "fmt"
"net" "net"
"os" "os"
"path/filepath"
"sort" "sort"
"strings" "strings"
"time" "time"
@@ -127,6 +128,10 @@ func moduleCommand(ctx context.Context, args []string) error {
if err != nil { if err != nil {
return err return err
} }
// A manifest handed over by hand takes each template's text from the tree beside it (novox/hq ADR 0349 §2).
if m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(positionals[0])); err != nil {
return err
}
from, err := whereItComesFrom(*repo, *ref, *commit, *path, *self) from, err := whereItComesFrom(*repo, *ref, *commit, *path, *self)
if err != nil { if err != nil {
return err return err
@@ -441,6 +446,10 @@ func settingsCommand(ctx context.Context, args []string) error {
if len(positionals) == 2 { if len(positionals) == 2 {
values = positionals[1] values = positionals[1]
} }
// Once its config values are mesh-config's, they are proposed there (issue 664).
if forwarded, err := forwardPropose(ctx, inv, positionals[0], *node, values, *clear); forwarded || err != nil {
return err
}
return proposeCommand(ctx, positionals[0], *node, values, *clear, *replace) return proposeCommand(ctx, positionals[0], *node, values, *clear, *replace)
case "proposals": case "proposals":
if len(positionals) > 1 || *node != "" || *clear || *replace || *history { if len(positionals) > 1 || *node != "" || *clear || *replace || *history {
@@ -461,6 +470,15 @@ func settingsCommand(ctx context.Context, args []string) error {
where = *node where = *node
} }
// Once the move of config has run, a module in the config form is read and set through mesh-config (issue 664).
if (args[0] == "set" && len(positionals) == 2) || ((args[0] == "show" && !*history) || args[0] == "clear") &&
len(positionals) == 1 {
if forwarded, err := forwardSettings(ctx, inv, args[0], positionals[0], *node, positionals, *replace); forwarded ||
err != nil {
return err
}
}
switch args[0] { switch args[0] {
case "set": case "set":
if len(positionals) != 2 { if len(positionals) != 2 {
@@ -537,6 +555,10 @@ func settingsCommand(ctx context.Context, args []string) error {
if p.SetBy != "" { if p.SetBy != "" {
by = "; " + p.SetBy by = "; " + p.SetBy
} }
if p.Why != "" {
// A layer whose placement moved out of it (migration 0097): still the layer, as it stands.
by += "; " + p.Why
}
fmt.Printf("%s on %s, until %s (%s%s):\n %s\n", positionals[0], where, fmt.Printf("%s on %s, until %s (%s%s):\n %s\n", positionals[0], where,
p.ReplacedAt.Local().Format("2006-01-02 15:04:05"), p.ReplacedBy, by, shown) p.ReplacedAt.Local().Format("2006-01-02 15:04:05"), p.ReplacedBy, by, shown)
} }
@@ -901,8 +923,10 @@ func brokerReachableAt(ctx context.Context, inv *inventory.Inventory, known brok
// module's own secret named broker; a module that declares none has nothing to read it with, and // module's own secret named broker; a module that declares none has nothing to read it with, and
// an account nothing reads is an orphan on the bus (novox/hq 04-ISSUES/078). Said before the // an account nothing reads is an orphan on the bus (novox/hq 04-ISSUES/078). Said before the
// account is made, not after. // account is made, not after.
// A module that runs its machine's provisioner client and declares no broker is issued the client's account
// alone (issue 917): the client speaks on the bus only as itself.
func mayIssue(m catalogue.Manifest) error { func mayIssue(m catalogue.Manifest) error {
if _, reads := m.OwnSecrets["broker"]; !reads { if _, reads := m.OwnSecrets["broker"]; !reads && !runsTheProvisionerClient(m) {
return fmt.Errorf("%s declares no own secret named broker, so there is nowhere to deliver "+ return fmt.Errorf("%s declares no own secret named broker, so there is nowhere to deliver "+
"an account; a module that speaks on the bus declares \"own-secrets\": {\"broker\": <path>}", m.Module) "an account; a module that speaks on the bus declares \"own-secrets\": {\"broker\": <path>}", m.Module)
} }
@@ -921,14 +945,6 @@ func mayIssue(m catalogue.Manifest) error {
func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest,
node, busAddress string) error { node, busAddress string) error {
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: m.Module}.Username()
password, err := inv.MintBusPassword(ctx, inventory.BusUser{
Username: user, Kind: busKindOf(m.Module), Node: node, Module: m.Module,
})
if err != nil {
return err
}
// Where the module is told to find the bus, and what certificate it must present. The same pair // Where the module is told to find the bus, and what certificate it must present. The same pair
// a node is told, for the same reason: a mesh's bus presents its own certificate, in no public // a node is told, for the same reason: a mesh's bus presents its own certificate, in no public
// trust store, so an address alone fails at TLS. // trust store, so an address alone fails at TLS.
@@ -940,7 +956,64 @@ func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue
if err != nil { if err != nil {
return err return err
} }
return issueWith(ctx, inv, m, node, busAddress, known, reachable, user, password) if _, reads := m.OwnSecrets["broker"]; reads {
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: m.Module}.Username()
password, err := inv.MintBusPassword(ctx, inventory.BusUser{
Username: user, Kind: busKindOf(m.Module), Node: node, Module: m.Module,
})
if err != nil {
return err
}
if err := issueWith(ctx, inv, m, node, busAddress, known, reachable, user, password); err != nil {
return err
}
}
if !runsTheProvisionerClient(m) {
return nil
}
return issueProvisionerClient(ctx, inv, m, node, known, reachable)
}
// runsTheProvisionerClient says a module runs its machine's provisioner client, which reaches the bus on an
// account of its own (novox/hq ADR 0349 clause 11, design 57 P3): it claims the seat and declares the own secret
// that account is delivered in. The same test the bus's user list makes (inventory's declaredFor).
func runsTheProvisionerClient(m catalogue.Manifest) bool {
_, reads := m.OwnSecrets[broker.ProvisionerSecret]
return reads && m.ClaimsSeat(broker.NodeProvisionerSeat)
}
// issueProvisionerClient gives a machine's provisioner client its account: minted under the client's own name,
// never the module's or the tool runner's, and sealed to the machine as the module's own secret
// broker.ProvisionerSecret, which the client's service reads (novox/hq ADR 0349 clause 11, design 57 P3). No
// claims and no consumer: the client holds no seat on this account and is delivered nothing but its answers.
func issueProvisionerClient(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node string,
known broker.Broker, reachable string) error {
user := broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: m.Module}.Username()
password, err := inv.MintBusPassword(ctx, inventory.BusUser{
Username: user, Kind: inventory.BusProvisioner, Node: node, Module: m.Module,
})
if err != nil {
return err
}
held, err := json.Marshal(struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Node string `json:"node"`
User string `json:"user"`
Password string `json:"password"`
}{URL: "nats://" + reachable, Fingerprint: known.Fingerprint, Node: node, User: user, Password: password})
if err != nil {
return err
}
if err := inv.AcceptSecretForModule(ctx, node, m.Module, broker.ProvisionerSecret, string(held)); err != nil {
return err
}
if err := handIssuedToVault(ctx, inv, m, node, broker.ProvisionerSecret, string(held)); err != nil {
return err
}
fmt.Printf("bus user %s minted for %s's provisioner client, scoped to %s's keys, its reports and the vault\n",
user, m.Module, node)
return nil
} }
// busKindOf is what a module's bus user is recorded as: the node's tool runtime where the module is // busKindOf is what a module's bus user is recorded as: the node's tool runtime where the module is
@@ -985,6 +1058,9 @@ func issueWith(ctx context.Context, inv *inventory.Inventory, m catalogue.Manife
if err := inv.AcceptSecretForModule(ctx, node, m.Module, "broker", string(held)); err != nil { if err := inv.AcceptSecretForModule(ctx, node, m.Module, "broker", string(held)); err != nil {
return err return err
} }
if err := handIssuedToVault(ctx, inv, m, node, "broker", string(held)); err != nil {
return err
}
// And how it hears what it consumes. Derived from its declaration, and only when it declared // And how it hears what it consumes. Derived from its declaration, and only when it declared
// something: a module that consumes nothing needs no consumer, and creating one would be a // something: a module that consumes nothing needs no consumer, and creating one would be a
+1 -1
View File
@@ -684,7 +684,7 @@ func artifactStoreOnNetwork(ctx context.Context, inv *inventory.Inventory,
if !offers { if !offers {
continue continue
} }
serves, err := servedOnNode(ctx, inv, machine, m, catalogue.ArtifactStoreProvision) serves, _, err := servedOnNode(ctx, inv, machine, m, catalogue.ArtifactStoreProvision)
if err != nil { if err != nil {
return "", "", false, fmt.Errorf("reading where %s puts the artifact store: %w", machine, err) return "", "", false, fmt.Errorf("reading where %s puts the artifact store: %w", machine, err)
} }
+1 -6
View File
@@ -2,7 +2,6 @@ package main
import ( import (
"context" "context"
"os"
"path/filepath" "path/filepath"
"reflect" "reflect"
"strings" "strings"
@@ -244,11 +243,7 @@ func TestATakeoverIsNotComposedForAHubPlacedOffItsTunnel(t *testing.T) {
// theResolver is the catalogue's dnsmasq module as it is (internal/beside). // theResolver is the catalogue's dnsmasq module as it is (internal/beside).
func theResolver(t *testing.T) catalogue.Manifest { func theResolver(t *testing.T) catalogue.Manifest {
t.Helper() t.Helper()
raw, err := os.ReadFile(filepath.Join(beside.Catalogue(t), "dnsmasq", "module.json")) m, err := catalogue.ReadTreeManifest(beside.Dir(t, "mesh-catalog"), filepath.Join(beside.Catalogue(t), "dnsmasq"))
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil { if err != nil {
t.Fatalf("dnsmasq does not parse:\n%v", err) t.Fatalf("dnsmasq does not parse:\n%v", err)
} }
+11
View File
@@ -19,6 +19,7 @@ import (
"github.com/novox/mesh-controller/internal/conditions" "github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory" "github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link" "github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/overlay"
"github.com/novox/mesh-controller/internal/token" "github.com/novox/mesh-controller/internal/token"
) )
@@ -884,6 +885,16 @@ func showNode(ctx context.Context, stored *stores, name string) error {
for _, line := range machineLines(account, had, time.Now()) { for _, line := range machineLines(account, had, time.Now()) {
fmt.Println(line) fmt.Println(line)
} }
// And which of the networks it is attached to count as its local network, and why the others
// do not (novox/hq ADR 0340 §8).
outward, err := inv.OutwardLinksOf(ctx, name)
if err != nil {
fmt.Printf(" its local network could NOT be judged: %v\n", err)
} else {
for _, line := range localNetworkLines(account, had, judgedNetworks(account, had, outward, overlay.Interface)) {
fmt.Println(line)
}
}
} }
held, err := inv.Profile(ctx, name) held, err := inv.Profile(ctx, name)
+139 -14
View File
@@ -3,6 +3,7 @@ package main
import ( import (
"context" "context"
"fmt" "fmt"
"slices"
"sort" "sort"
"strings" "strings"
"time" "time"
@@ -44,6 +45,28 @@ func needsOperatorKey(module, node string) string {
type operatorWaitFacts struct { type operatorWaitFacts struct {
manifests map[string]catalogue.Manifest manifests map[string]catalogue.Manifest
given map[string]map[string]time.Time given map[string]map[string]time.Time
// outside is, per provision some module offers with a credential issued outside the mesh, that provider module
// and the machines it is assigned to (novox/hq ADR 0337); pairsGiven, per "<module>@<machine>", the provisions
// it holds a value a person gave for, sealed to the keys both ends have now. Absent is not known: no wait for a
// provision of it is excused.
outside map[string]outsideProvider
pairsGiven map[string]map[string]bool
}
// outsideProvider is the module offering a provision whose credential only a person gives, and where it runs.
type outsideProvider struct {
Module string
Machines []string
}
// acceptLine is the one line that gives an outside provision's credential for every consumer of its provider: on
// the provider's machine when it runs on one, named as <machine> when on several or none.
func (p outsideProvider) acceptLine(provision string) string {
machine := "<its machine>"
if len(p.Machines) == 1 {
machine = p.Machines[0]
}
return inventory.AcceptLine(machine, p.Module, provision)
} }
// checkWait is nil when a wait is excused, and otherwise why not, in words. Pure. // checkWait is nil when a wait is excused, and otherwise why not, in words. Pure.
@@ -52,9 +75,18 @@ func checkWait(module, machine string, w inventory.Wait, f operatorWaitFacts) er
if !known { if !known {
return fmt.Errorf("says it waits for %s, and the mesh holds no manifest of %s to check it against", waitNames(w), module) return fmt.Errorf("says it waits for %s, and the mesh holds no manifest of %s to check it against", waitNames(w), module)
} }
named := 0
for _, n := range []string{w.Secret, w.Setting, w.Provision} {
if n != "" {
named++
}
}
switch { switch {
case w.Secret != "" && w.Setting != "", w.Secret == "" && w.Setting == "": case named != 1:
return fmt.Errorf("says it waits, naming %s, where a wait names one secret or one setting", waitNames(w)) return fmt.Errorf("says it waits, naming %s, where a wait names one secret, one setting or one provision",
waitNames(w))
case w.Provision != "":
return checkProvisionWait(module, machine, w, m, f)
case w.Setting != "": case w.Setting != "":
if _, declared := m.Settings[w.Setting]; !declared { if _, declared := m.Settings[w.Setting]; !declared {
return fmt.Errorf("says it waits for the setting %s, which %s does not declare", w.Setting, module) return fmt.Errorf("says it waits for the setting %s, which %s does not declare", w.Setting, module)
@@ -80,20 +112,46 @@ func checkWait(module, machine string, w inventory.Wait, f operatorWaitFacts) er
return nil return nil
} }
// checkProvisionWait excuses a wait for a provision's credential only when the store bears it out (novox/hq ADR 0337
// §6): the module requires the provision, a provider offers it with a credential issued outside the mesh, and the
// module on this machine holds no value a person gave for it under the keys both ends have now.
func checkProvisionWait(module, machine string, w inventory.Wait, m catalogue.Manifest, f operatorWaitFacts) error {
if !slices.Contains(m.Requires, w.Provision) {
return fmt.Errorf("says it waits for the provision %s, which %s does not require", w.Provision, module)
}
if _, outside := f.outside[w.Provision]; !outside {
return fmt.Errorf("says it waits for the provision %s, whose credential the mesh gives itself: only a "+
"credential issued outside the mesh waits for the operator", w.Provision)
}
given, readable := f.pairsGiven[module+"@"+machine]
if !readable {
return fmt.Errorf("says it waits for the provision %s, and what was given on %s could not be read", w.Provision, machine)
}
if given[w.Provision] {
return fmt.Errorf("says it waits for the provision %s, whose credential was given", w.Provision)
}
return nil
}
// waitRefusedPrefix opens every reason checkWait gives, so the words of a refused wait are told from a check's own. // waitRefusedPrefix opens every reason checkWait gives, so the words of a refused wait are told from a check's own.
const waitRefusedPrefix = "says it waits" const waitRefusedPrefix = "says it waits"
// waitNames is what a wait names, as "the secret x" or "the setting y". // waitNames is what a wait names, as "the secret x" or "the setting y".
func waitNames(w inventory.Wait) string { func waitNames(w inventory.Wait) string {
switch { var names []string
case w.Secret != "" && w.Setting != "": if w.Secret != "" {
return "the secret " + w.Secret + " and the setting " + w.Setting names = append(names, "the secret "+w.Secret)
case w.Secret != "":
return "the secret " + w.Secret
case w.Setting != "":
return "the setting " + w.Setting
} }
return "nothing" if w.Setting != "" {
names = append(names, "the setting "+w.Setting)
}
if w.Provision != "" {
names = append(names, "the provision "+w.Provision)
}
if len(names) == 0 {
return "nothing"
}
return strings.Join(names, " and ")
} }
// checkWaiting reads one machine's resources against the facts: every waiting resource whose waits all check out is // checkWaiting reads one machine's resources against the facts: every waiting resource whose waits all check out is
@@ -107,11 +165,17 @@ func checkWaiting(machine string, rs []inventory.ResourceHealth, f operatorWaitF
if len(r.Waits) == 0 { if len(r.Waits) == 0 {
why = fmt.Errorf("says it waits, and names nothing it waits for") why = fmt.Errorf("says it waits, and names nothing it waits for")
} }
waits := make([]inventory.Wait, 0, len(r.Waits))
for _, w := range r.Waits { for _, w := range r.Waits {
if why == nil { if why == nil {
why = checkWait(r.Module, machine, w, f) why = checkWait(r.Module, machine, w, f)
} }
if p, outside := f.outside[w.Provision]; outside && w.Provision != "" {
w.Act = p.acceptLine(w.Provision)
}
waits = append(waits, w)
} }
r.Waits = waits
if why != nil { if why != nil {
r.State, r.Reason = link.StateUnhealthy, why.Error() r.State, r.Reason = link.StateUnhealthy, why.Error()
} }
@@ -150,6 +214,10 @@ func operatorWaitSaid(module, machine string, waits []inventory.Wait) string {
if w.Secret != "" { if w.Secret != "" {
part += fmt.Sprintf(", given with `nox secret ask %s %s %s`", machine, module, w.Secret) part += fmt.Sprintf(", given with `nox secret ask %s %s %s`", machine, module, w.Secret)
} }
if w.Provision != "" && w.Act != "" {
// One line for the provider, which seals it to every consumer of it (ADR 0337 §3, §7).
part += fmt.Sprintf(", given for every consumer with `%s`", w.Act)
}
parts = append(parts, part+")") parts = append(parts, part+")")
} }
return fmt.Sprintf("%s on %s waits for the operator: %s", module, machine, strings.Join(parts, "; ")) return fmt.Sprintf("%s on %s waits for the operator: %s", module, machine, strings.Join(parts, "; "))
@@ -172,7 +240,7 @@ func needsOperatorObservation(module, node string, waits []inventory.Wait, rs []
func needsOperatorWords(module, node string, waits []inventory.Wait) words { func needsOperatorWords(module, node string, waits []inventory.Wait) words {
var acts []string var acts []string
seen := map[string]bool{} seen := map[string]bool{}
secret := false secret, provision := false, false
for _, w := range waits { for _, w := range waits {
var act string var act string
switch { switch {
@@ -180,6 +248,9 @@ func needsOperatorWords(module, node string, waits []inventory.Wait) words {
act, secret = fmt.Sprintf("type %s at %s's desk prompt", w.What, node), true act, secret = fmt.Sprintf("type %s at %s's desk prompt", w.What, node), true
case w.Setting != "": case w.Setting != "":
act = fmt.Sprintf("approve %s of %s on %s when it is proposed to you", w.Setting, module, node) act = fmt.Sprintf("approve %s of %s on %s when it is proposed to you", w.Setting, module, node)
case w.Provision != "":
// ADR 0337 §7: "give <what> of <provider> on <machine>"; the line is in the summary.
act, provision = "give "+w.What, true
} }
if act != "" && !seen[act] { if act != "" && !seen[act] {
seen[act] = true seen[act] = true
@@ -197,6 +268,10 @@ func needsOperatorWords(module, node string, waits []inventory.Wait) words {
explanation += " A hidden prompt opens at the desk when the secret is asked for, and what you type there " + explanation += " A hidden prompt opens at the desk when the secret is asked for, and what you type there " +
"is sealed to the machine." "is sealed to the machine."
} }
if provision {
explanation += " A key only you can give is given once for the app that issued it, and reaches every " +
"module that uses it."
}
explanation += " Its update is in place and nothing was undone; it carries on by itself once it is given." explanation += " Its update is in place and nothing was undone; it carries on by itself once it is given."
return words{ return words{
Headline: fmt.Sprintf("%s waits for you on %s", module, node), Headline: fmt.Sprintf("%s waits for you on %s", module, node),
@@ -217,6 +292,12 @@ func readWaitFacts(ctx context.Context, inv *inventory.Inventory, machine string
if f.given == nil { if f.given == nil {
f.given = map[string]map[string]time.Time{} f.given = map[string]map[string]time.Time{}
} }
if f.outside == nil {
f.outside = map[string]outsideProvider{}
}
if f.pairsGiven == nil {
f.pairsGiven = map[string]map[string]bool{}
}
if inv == nil { if inv == nil {
return return
} }
@@ -238,15 +319,59 @@ func readWaitFacts(ctx context.Context, inv *inventory.Inventory, machine string
} }
} }
} }
if _, read := f.given[module+"@"+machine]; read { if _, read := f.given[module+"@"+machine]; !read {
if given, err := inv.GivenOwnSecrets(ctx, machine, module); err == nil {
f.given[module+"@"+machine] = given
}
}
if _, read := f.pairsGiven[module+"@"+machine]; !read {
if given, err := inv.GivenPairsOf(ctx, machine, module); err == nil {
f.pairsGiven[module+"@"+machine] = given
}
}
// The provisions it requires whose credential only a person gives, and where their provider runs (ADR 0337).
m, known := f.manifests[module]
if !known {
continue continue
} }
if given, err := inv.GivenOwnSecrets(ctx, machine, module); err == nil { for _, provision := range m.Requires {
f.given[module+"@"+machine] = given if _, read := f.outside[provision]; read {
continue
}
if shelf == nil {
var err error
if shelf, err = inv.Catalogue(ctx); err != nil {
shelf = map[string]catalogue.Manifest{}
}
}
provider, outside := catalogue.OffersIssuedOutside(shelf, provision)
if !outside {
continue
}
f.outside[provision] = outsideProvider{Module: provider, Machines: machinesRunning(ctx, inv, provider)}
} }
} }
} }
// machinesRunning is every machine a module is assigned to, by name; a read that fails is left out.
func machinesRunning(ctx context.Context, inv *inventory.Inventory, module string) []string {
nodes, err := inv.Nodes(ctx)
if err != nil {
// Only the summary's accept line reads it, which then names the provider's machine "<its machine>".
// empty-on-error: no wait is excused or refused by it
return nil
}
var out []string
for _, n := range nodes {
assigned, err := inv.Assigned(ctx, n.Name)
if err == nil && slices.Contains(assigned, module) {
out = append(out, n.Name)
}
}
sort.Strings(out)
return out
}
// waitingModules is every module with a waiting resource in a statement. // waitingModules is every module with a waiting resource in a statement.
func waitingModules(rs []inventory.ResourceHealth) []string { func waitingModules(rs []inventory.ResourceHealth) []string {
seen := map[string]bool{} seen := map[string]bool{}
+2 -2
View File
@@ -55,8 +55,8 @@ func TestAWaitIsExcusedOnlyWhenItChecksOut(t *testing.T) {
{"a secret the mesh makes", inventory.Wait{Part: "p", Secret: "made", What: "w"}, factsGiven(nil), "mesh makes itself"}, {"a secret the mesh makes", inventory.Wait{Part: "p", Secret: "made", What: "w"}, factsGiven(nil), "mesh makes itself"},
{"the bus account", inventory.Wait{Part: "p", Secret: "broker", What: "w"}, factsGiven(nil), "mesh makes itself"}, {"the bus account", inventory.Wait{Part: "p", Secret: "broker", What: "w"}, factsGiven(nil), "mesh makes itself"},
{"a setting not declared", inventory.Wait{Part: "p", Setting: "logins", What: "w"}, factsGiven(nil), "does not declare"}, {"a setting not declared", inventory.Wait{Part: "p", Setting: "logins", What: "w"}, factsGiven(nil), "does not declare"},
{"both", inventory.Wait{Part: "p", Secret: "licence", Setting: "smb-users", What: "w"}, factsGiven(nil), "one secret or one setting"}, {"both", inventory.Wait{Part: "p", Secret: "licence", Setting: "smb-users", What: "w"}, factsGiven(nil), "one secret, one setting or one provision"},
{"neither", inventory.Wait{Part: "p", What: "w"}, factsGiven(nil), "one secret or one setting"}, {"neither", inventory.Wait{Part: "p", What: "w"}, factsGiven(nil), "one secret, one setting or one provision"},
{"no manifest known", passwordWait, operatorWaitFacts{}, "no manifest"}, {"no manifest known", passwordWait, operatorWaitFacts{}, "no manifest"},
{"what was given cannot be read", passwordWait, {"what was given cannot be read", passwordWait,
operatorWaitFacts{manifests: map[string]catalogue.Manifest{"mounts": mountsManifest}}, "could not be read"}, operatorWaitFacts{manifests: map[string]catalogue.Manifest{"mounts": mountsManifest}}, "could not be read"},
@@ -0,0 +1,83 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// One accept for a provider reaches every consumer that binds the provision from it, and each consumer's send
// places it; the per-pair form naming a consumer is refused for such a provision, naming the provider's form; a
// value the mesh made before the rule is not placed (novox/hq ADR 0337 §2, §3, §4, issue 164, issue 165). Needs the
// store (MESH_TEST_POSTGRES), and is named in testdata/store-tests.
func TestOneAcceptForAProviderIsPlacedForEveryConsumer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
// Before the rule: tv's offer said nothing of its credential, and the send made one per pair.
register(t, open, catalogue.Manifest{Module: "tv", Version: "1",
Provides: []catalogue.Offer{{Name: "tv-api", Scope: catalogue.ScopeMesh}}})
for _, m := range []string{"requests", "indexer"} {
register(t, open, catalogue.Manifest{Module: m, Version: "1", Requires: []string{"tv-api"},
Secrets: map[string]string{"tv-api": "/var/lib/mesh/" + m + "/tv-api.secret"}})
}
for _, a := range [][2]string{{"anchor", "tv"}, {"laptop", "requests"}, {"laptop", "indexer"}} {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatal(err)
}
}
placed := func(module string) bool {
t.Helper()
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatalf("laptop was refused: %v", err)
}
declared, err := declarationFor(ctx, open, "laptop", plan, settings)
if err != nil {
t.Fatalf("laptop did not compose: %v", err)
}
for _, r := range declared.Resources {
if r["path"] == "/var/lib/mesh/"+module+"/tv-api.secret" && r["sealed"] != nil {
return true
}
}
return false
}
if !placed("requests") || !placed("indexer") {
t.Fatal("before the rule, the send made and placed a credential per pair")
}
// The offer now says only a person gives it: the values the mesh made are not placed.
register(t, open, catalogue.Manifest{Module: "tv", Version: "2",
Provides: []catalogue.Offer{{Name: "tv-api", Scope: catalogue.ScopeMesh,
Credential: &catalogue.OfferCredential{IssuedBy: catalogue.IssuedOutside}}}})
if placed("requests") || placed("indexer") {
t.Fatal("a credential nobody gave was placed")
}
// The per-pair form is refused, naming the provider's form, and reads nothing.
file := filepath.Join(t.TempDir(), "key")
if err := os.WriteFile(file, []byte("the app's key\n"), 0o600); err != nil {
t.Fatal(err)
}
err := secretCommand(ctx, []string{"accept", "laptop", "requests", "tv-api", "--provider", "anchor", "--from", file})
if err == nil || !strings.Contains(err.Error(), "secret accept anchor tv tv-api") {
t.Fatalf("the per-pair form was not refused naming the provider's: %v", err)
}
consumers, unresolved, err := consumersFrom(ctx, open, "anchor", "tv", "tv-api")
if err != nil || len(unresolved) != 0 {
t.Fatalf("the consumers of tv on anchor could not be found: %v %v", err, unresolved)
}
if len(consumers) != 2 {
t.Fatalf("the consumers of tv on anchor are %+v; want requests and indexer on laptop", consumers)
}
if _, err := open.inventory.AcceptSecretForProvision(ctx, "tv-api", "anchor", "tv", "the app's key", consumers); err != nil {
t.Fatal(err)
}
if !placed("requests") || !placed("indexer") {
t.Fatal("the value given once for the provider was not placed for every consumer")
}
}
+113
View File
@@ -0,0 +1,113 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A consumer may wait for a provision's credential only a person gives (novox/hq ADR 0337 §5–§7, issue 164).
var requestsManifest = catalogue.Manifest{Module: "requests", Version: "1", Requires: []string{"tv-api", "db"},
Secrets: map[string]string{"tv-api": "/s/tv-api", "db": "/s/db"}}
var tvKeyWait = inventory.Wait{Part: "the tv library", Provision: "tv-api", What: "the API key of tv"}
func outsideFacts(given map[string]bool) operatorWaitFacts {
return operatorWaitFacts{manifests: map[string]catalogue.Manifest{"requests": requestsManifest},
outside: map[string]outsideProvider{"tv-api": {Module: "tv", Machines: []string{"media"}}},
pairsGiven: map[string]map[string]bool{"requests@laptop": given}}
}
// §6: excused only when the module requires the provision, its provider says it is issued outside, and no value a
// person gave is held for the pair under the keys both ends have now.
func TestAWaitForAProvisionIsExcusedOnlyWhenTheStoreBearsItOut(t *testing.T) {
for _, c := range []struct {
name string
w inventory.Wait
f operatorWaitFacts
says string
}{
{"required, issued outside, not given", tvKeyWait, outsideFacts(nil), ""},
{"given", tvKeyWait, outsideFacts(map[string]bool{"tv-api": true}), "was given"},
{"not required", inventory.Wait{Part: "p", Provision: "music-api", What: "w"}, outsideFacts(nil), "does not require"},
{"a credential the mesh gives itself", inventory.Wait{Part: "p", Provision: "db", What: "w"}, outsideFacts(nil),
"mesh gives itself"},
{"what was given cannot be read", tvKeyWait, operatorWaitFacts{
manifests: map[string]catalogue.Manifest{"requests": requestsManifest},
outside: map[string]outsideProvider{"tv-api": {Module: "tv"}}}, "could not be read"},
{"a provision and a secret", inventory.Wait{Part: "p", Provision: "tv-api", Secret: "x", What: "w"},
outsideFacts(nil), "one secret, one setting or one provision"},
} {
err := checkWait("requests", "laptop", c.w, c.f)
switch {
case c.says == "" && err != nil:
t.Errorf("%s: refused: %v", c.name, err)
case c.says != "" && (err == nil || !strings.Contains(err.Error(), c.says)):
t.Errorf("%s: %v; want a refusal saying %q", c.name, err, c.says)
}
}
}
// §7: an excused wait is the consumer's needs-operator condition, in plain words, its summary naming the one line
// that gives the credential for every consumer of the provider; the wait as kept is not changed.
func TestAWaitForAProvisionNamesTheOneAcceptForItsProvider(t *testing.T) {
r := inventory.ResourceHealth{Module: "requests", Resource: "requests.app", Kind: "container", Target: "requests",
State: link.StateWaiting, Check: "tool", Waits: []inventory.Wait{tvKeyWait}}
kept := []inventory.ResourceHealth{r}
got := checkWaiting("laptop", kept, outsideFacts(nil))
if got[0].State != link.StateWaiting {
t.Fatalf("an excused wait for a provision was judged: %+v", got[0])
}
if kept[0].Waits[0].Act != "" {
t.Fatal("checking the wait changed the statement as kept")
}
waits, waiting := operatorWait("requests", got)
if !waiting {
t.Fatal("the module does not read as waiting for the operator")
}
o := needsOperatorObservation("requests", "laptop", waits, got)
if !strings.Contains(o.Summary, "secret accept media tv tv-api") {
t.Fatalf("the summary does not name the accept for the provider: %q", o.Summary)
}
if !strings.HasPrefix(o.Needs, "give the API key of tv") {
t.Fatalf("the needs do not name the act: %q", o.Needs)
}
if why, ok := conditions.PlainWords(conditions.Words{Headline: o.Headline, Explanation: o.Explanation,
Resolved: o.Resolved, Needs: o.Needs, Actions: o.Actions}, "laptop"); !ok {
t.Fatalf("the condition is not in plain words: %s", why)
}
// A provider on several machines, or none: the line names its machine as a place to fill.
if line := (outsideProvider{Module: "tv", Machines: []string{"a", "b"}}).acceptLine("tv-api"); line !=
"secret accept <its machine> tv tv-api" {
t.Fatalf("a provider on two machines reads %q", line)
}
}
// §3: the consumers one accept reaches are those whose resolution answers the provision from that provider.
func TestTheConsumersOfAProviderAreThoseItAnswers(t *testing.T) {
plan := catalogue.Resolution{Needs: []catalogue.Needed{
{Name: "tv-api", From: "media", Module: "tv", For: "requests"},
{Name: "tv-api", From: "media", Module: "tv", For: "indexer", Local: "main"},
{Name: "tv-api", From: "other", Module: "tv", For: "elsewhere"},
{Name: "tv-api", From: "media", Module: "another-tv", For: "not-this-module"},
{Name: "db", From: "media", Module: "tv", For: "requests"},
{Name: "tv-api", ByRecord: true, For: "by-record"},
}}
got := bindsFrom(plan, "laptop", "media", "tv", "tv-api")
if len(got) != 2 || got[0].ConsumerModule != "indexer" || got[0].Local != "main" || got[1].ConsumerModule != "requests" {
t.Fatalf("the consumers from tv on media are %+v", got)
}
said := acceptedSaid("media", "tv", "tv-api", inventory.ProvisionAccepted{
Sealed: []inventory.PairEnd{{Consumer: "laptop", ConsumerModule: "requests"}},
Unsealable: []inventory.PairEnd{{Consumer: "new", ConsumerModule: "indexer"}}}, []string{"down: refused"})
for _, want := range []string{"sealed to requests on laptop", "no sealing key", "down: refused"} {
if !strings.Contains(said, want) {
t.Errorf("what the accept says lacks %q:\n%s", want, said)
}
}
}
+7
View File
@@ -30,6 +30,7 @@ func userUnit(module, account string) inventory.ResourceHealth {
// Only what depends on the new login alone is a wait; anything else unhealthy of the module is judged as // Only what depends on the new login alone is a wait; anything else unhealthy of the module is judged as
// before, so that a fault is never excused. // before, so that a fault is never excused.
func TestOnlyWhatDependsOnTheNewLoginIsAWait(t *testing.T) { func TestOnlyWhatDependsOnTheNewLoginIsAWait(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
notInGroup := relogin("lights", "operator") notInGroup := relogin("lights", "operator")
notInGroup.Reason = "not in the group lights: the apply has not put operator there; its outcome says why" notInGroup.Reason = "not in the group lights: the apply has not put operator there; its outcome says why"
systemUnit := userUnit("lights", "operator") systemUnit := userUnit("lights", "operator")
@@ -74,6 +75,7 @@ func TestOnlyWhatDependsOnTheNewLoginIsAWait(t *testing.T) {
// The gate reads a wait for a person as its own reading, never a fault, and passes with it; the same module // The gate reads a wait for a person as its own reading, never a fault, and passes with it; the same module
// with a container down beside it is not yet healthy, as before. // with a container down beside it is not yet healthy, as before.
func TestAWaitForAPersonIsAPassCarriedAlong(t *testing.T) { func TestAWaitForAPersonIsAPassCarriedAlong(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
now := time.Now() now := time.Now()
since := now.Add(-time.Minute) since := now.Add(-time.Minute)
account := relogin("lights", "operator") account := relogin("lights", "operator")
@@ -102,6 +104,7 @@ func TestAWaitForAPersonIsAPassCarriedAlong(t *testing.T) {
// The module's condition says the wait in one sentence for a person, and clears on the first statement that // The module's condition says the wait in one sentence for a person, and clears on the first statement that
// no longer says it — said as the module's own fault when its unit still fails after the new login. // no longer says it — said as the module's own fault when its unit still fails after the new login.
func TestTheReloginConditionSaysTheWaitAndClearsAfterTheLogin(t *testing.T) { func TestTheReloginConditionSaysTheWaitAndClearsAfterTheLogin(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t) open := aMesh(t)
ctx := t.Context() ctx := t.Context()
k, _ := withConditionsInMemory(t) k, _ := withConditionsInMemory(t)
@@ -158,6 +161,7 @@ func TestTheReloginConditionSaysTheWaitAndClearsAfterTheLogin(t *testing.T) {
// A module that is healthy on its own keeps its pass when another module of the same send fails at the // A module that is healthy on its own keeps its pass when another module of the same send fails at the
// bound: it is neither put back nor left without a verdict for every other walk to wait on. // bound: it is neither put back nor left without a verdict for every other walk to wait on.
func TestAModuleHealthyOnItsOwnKeepsItsPassWhenItsSendFails(t *testing.T) { func TestAModuleHealthyOnItsOwnKeepsItsPassWhenItsSendFails(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
b := aBacklog(t) b := aBacklog(t)
ctx := t.Context() ctx := t.Context()
inv := b.open.inventory inv := b.open.inventory
@@ -207,6 +211,7 @@ func TestAModuleHealthyOnItsOwnKeepsItsPassWhenItsSendFails(t *testing.T) {
// What the operator reads of the wait (ADR 0253, ADR 0254): it needs them, so it is never quiet, and it offers // What the operator reads of the wait (ADR 0253, ADR 0254): it needs them, so it is never quiet, and it offers
// no button, since nothing but their own new login can do it. // no button, since nothing but their own new login can do it.
func TestTheReloginConditionNeedsTheOperatorAndOffersNoButton(t *testing.T) { func TestTheReloginConditionNeedsTheOperatorAndOffersNoButton(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
o := reloginObservation("openrazer", "g14", "relogin needed on g14: …", "operator", []inventory.ResourceHealth{ o := reloginObservation("openrazer", "g14", "relogin needed on g14: …", "operator", []inventory.ResourceHealth{
relogin("openrazer", "operator"), userUnit("openrazer", "operator")}) relogin("openrazer", "operator"), userUnit("openrazer", "operator")})
plainExample(t, o, "openrazer waits for a new login on g14", plainExample(t, o, "openrazer waits for a new login on g14",
@@ -238,6 +243,7 @@ func TestTheReloginConditionNeedsTheOperatorAndOffersNoButton(t *testing.T) {
// When a module not working turns into one waiting for a new login, and back, the clearing line says what it // When a module not working turns into one waiting for a new login, and back, the clearing line says what it
// became, never that it works again (issue 318 review). // became, never that it works again (issue 318 review).
func TestAConditionThatBecameTheOtherKindSaysSoWhenItClears(t *testing.T) { func TestAConditionThatBecameTheOtherKindSaysSoWhenItClears(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t) open := aMesh(t)
ctx := t.Context() ctx := t.Context()
store := conditions.NewInMemory() store := conditions.NewInMemory()
@@ -286,6 +292,7 @@ func TestAConditionThatBecameTheOtherKindSaysSoWhenItClears(t *testing.T) {
// Which moves add an account group, read from the builds' manifests (issue 318 review). // Which moves add an account group, read from the builds' manifests (issue 318 review).
func TestOnlyAMoveThatAddsAnAccountGroupCanBringAWait(t *testing.T) { func TestOnlyAMoveThatAddsAnAccountGroupCanBringAWait(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
user := func(groups ...any) catalogue.Manifest { user := func(groups ...any) catalogue.Manifest {
return catalogue.Manifest{Module: "lights", Resources: []map[string]any{{"id": "operator", "type": "user", return catalogue.Manifest{Module: "lights", Resources: []map[string]any{{"id": "operator", "type": "user",
"name": "operator", "groups": groups}}} "name": "operator", "groups": groups}}}
+99
View File
@@ -0,0 +1,99 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
)
// movePlacementAnswer is what the move of placement says it did, or would do (issue 736).
type movePlacementAnswer struct {
// DryRun is true when nothing was moved: asked without --confirm.
DryRun bool `json:"dry-run"`
// Moved is each layer that held a placement key among its settings, and which keys.
Moved []inventory.MovedLayer `json:"moved"`
}
// movePlacementCommand is `move-placement`: the move of placement out of the settings layers (novox/hq ADR 0348
// clause 11, issue 736), one of the moves of data a release runs once after its delivery (design 57).
func movePlacementCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("move-placement", flag.ContinueOnError)
asJSON := set.Bool("json", false, "answer as JSON")
confirm := set.Bool("confirm", false, "move the keys, rather than only say which layers hold them")
f := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 0 {
return errors.New("move-placement [--json] [--confirm --why <text>]")
}
if !*confirm && f.given() {
return errors.New("move-placement takes --why only with --confirm: without it nothing is moved. Nothing was done")
}
if *confirm {
// A move a person asks for says why, before anything is asked of the store.
if err := f.require("move-placement"); err != nil {
return err
}
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
if *confirm {
f.record(ctx, "move-placement", nil)
}
answer, err := runMovePlacement(ctx, open.inventory, *confirm)
if err != nil {
return err
}
if *asJSON {
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
return encoder.Encode(answer)
}
fmt.Print(describeMovePlacement(answer))
return nil
}
// runMovePlacement is the move against a store: a dry run says what it would move.
func runMovePlacement(ctx context.Context, inv *inventory.Inventory, apply bool) (movePlacementAnswer, error) {
moved, err := inv.MovePlacement(ctx, apply)
if err != nil {
return movePlacementAnswer{}, err
}
return movePlacementAnswer{DryRun: !apply, Moved: moved}, nil
}
// describeMovePlacement says the answer as a person reads it.
func describeMovePlacement(a movePlacementAnswer) string {
if len(a.Moved) == 0 {
return "no settings layer holds a placement key: nothing to move\n"
}
var b strings.Builder
verb := "moved"
if a.DryRun {
verb = "would move"
}
for _, m := range a.Moved {
where := m.Node
if where == "" {
where = "the whole mesh"
}
fmt.Fprintf(&b, "%s on %s: %s %s\n", m.Module, where, verb, strings.Join(m.Keys, ", "))
}
if a.DryRun {
fmt.Fprintf(&b, "%d layer(s); nothing was moved — with --confirm --why <text> they are\n", len(a.Moved))
} else {
fmt.Fprintf(&b, "%d layer(s) moved beside their assignments, each kept whole in its history first\n", len(a.Moved))
}
return b.String()
}
+113
View File
@@ -0,0 +1,113 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// **Placement leaves the settings layers and no machine is told anything new** (novox/hq ADR 0348 clause
// 11, issue 651). A mesh is seeded as the store holds it before migration 0097 — `places`, `accesses` and a
// kept network among the settings, at the mesh's level and a machine's, one layer of placement only — then
// migrated, and every machine's declaration is composed. What each machine is told of the seeded modules is
// byte for byte what the controller before this change composed from the same seed: the golden files were
// written by the integration branch's controller at bfc79c6 running this same test (MESH_WRITE_GOLDEN=1),
// where there was no migration 0097 to run.
func TestADeclarationIsTheSameAfterPlacementMoves(t *testing.T) {
open, migrate := aMeshBefore(t, 97)
ctx := t.Context()
inv := open.inventory
image := "registry.invalid:5000/library@sha256:" + strings.Repeat("a", 64)
register(t, open, catalogue.Manifest{Module: "library", Version: "1",
Accesses: []catalogue.Access{{ID: "series"}, {ID: "downloads", Mode: catalogue.AccessRead}},
Resources: []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "server", "type": "container", "image": image,
"volumes": []any{"${dir:config}:/config", "${access:series}:/series", "${access:downloads}:/downloads:ro"}},
// Today's form: a mergeable file naming no keys takes every key of the layer (ADR 0326 §3).
{"id": "runtime-config", "type": "file", "path": "${dir:config}/runtime.json", "mode": "0640",
"content": `{"log":"info"}`, "merge": "json"},
}})
register(t, open, catalogue.Manifest{Module: "found", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "image": image}}})
if err := inv.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
for _, l := range []struct{ node, module, values string }{
{"", "library", `{"accesses": {"downloads": "/storage/downloads"}, "log": "warn"}`},
{"anchor", "library", `{"places": {"config": {"path": "/var/lib/library/config", "owner": "1001:2000"}},
"accesses": {"series": "/storage/media/series"}, "workers": 4}`},
{"laptop", "library", `{"accesses": {"series": "/home/media/series"}}`},
{"anchor", "found", `{"networks": {"server": ["predecessor_default"]}}`},
} {
if l.node == "" {
inv.ExecForTest(t, `insert into settings (node, module, values) values (null, $1, $2)`, l.module, l.values)
continue
}
inv.ExecForTest(t, `insert into settings (node, module, values) select id, $2, $3 from node where name = $1`,
l.node, l.module, l.values)
}
migrate()
// Assigned once the store is migrated: an assignment reads the layers, as this controller holds them.
for _, a := range [][]string{{"anchor", "library", "found"}, {"laptop", "library"}} {
if _, err := assign(ctx, open, a[0], a[1:]...); err != nil {
t.Fatalf("assign %v: %v", a, err)
}
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatal(err)
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Allocating)
if err != nil {
t.Fatal(err)
}
composed, err := plan.Compose(with)
if err != nil {
t.Fatal(err)
}
if len(composed.LeftOut) > 0 {
t.Fatalf("%s leaves out %v", node, composed.LeftOut)
}
var told []map[string]any
for _, r := range composed.Resources {
if owner := composed.Owner[r["id"].(string)]; owner == "library" || owner == "found" {
told = append(told, r)
}
}
sort.Slice(told, func(i, j int) bool { return told[i]["id"].(string) < told[j]["id"].(string) })
got, err := json.MarshalIndent(told, "", " ")
if err != nil {
t.Fatal(err)
}
golden := filepath.Join("testdata", "placement-moved", node+".json")
if os.Getenv("MESH_WRITE_GOLDEN") != "" {
if err := os.MkdirAll(filepath.Dir(golden), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(golden, append(got, '\n'), 0o644); err != nil {
t.Fatal(err)
}
continue
}
want, err := os.ReadFile(golden)
if err != nil {
t.Fatal(err)
}
if string(want) != string(got)+"\n" {
t.Errorf("%s is told something else after placement moved:\n%s\nwas:\n%s", node, got, want)
}
}
}
@@ -0,0 +1,243 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// wordedMesh is every live layer that says `endpoints`, `expose`, `reach` or `ports` (issue 736) — 36, read with
// the settings verb on 2026-10-14, ace's as anchor's and novox's as laptop's, one stand-in for an address — split as
// the store holds them once migration 0097 has run: the four words still among the settings, `places` and
// `accesses` already beside them. With each module shaped as its definition in the catalogue (mesh-catalog's, and
// the media catalogue's) is where the four words reach: its listens, what its containers publish and on which
// network, its route, its directories and accesses, and its mergeable files — every key of the layer, or the keys
// they name. The shapes are the modules'; the files' text and the images are the test's.
type wordedMesh struct {
Modules []json.RawMessage `json:"modules"`
Layers []struct {
Node string `json:"node"`
Module string `json:"module"`
Settings map[string]any `json:"settings"`
Placement map[string]any `json:"placement"`
} `json:"layers"`
}
func theWordedMesh(t *testing.T) wordedMesh {
t.Helper()
raw, err := os.ReadFile(filepath.Join("testdata", "placement-words-moved", "mesh.json"))
if err != nil {
t.Fatal(err)
}
var w wordedMesh
if err := json.Unmarshal(raw, &w); err != nil {
t.Fatal(err)
}
return w
}
// aMeshWithTheWordsInItsLayers is aMesh with every module of the worded mesh registered, a route provider on both
// machines, and its layers held as the store holds them before the four words move.
func aMeshWithTheWordsInItsLayers(t *testing.T) (*stores, wordedMesh) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
w := theWordedMesh(t)
image := "registry.invalid:5000/library@sha256:" + strings.Repeat("a", 64)
register(t, open, catalogue.Manifest{Module: "proxy", Version: "1", Provides: catalogue.FromAnywhere("route"),
Receives: map[string]string{"route": "/var/lib/proxy/routes.json"}})
for _, text := range w.Modules {
m, err := catalogue.ParseManifest([]byte(strings.ReplaceAll(string(text), "IMAGE", image)))
if err != nil {
t.Fatal(err)
}
register(t, open, m)
}
on := map[string][]string{"anchor": {"proxy"}, "laptop": {"proxy"}}
for _, l := range w.Layers {
settings, err := json.Marshal(l.Settings)
if err != nil {
t.Fatal(err)
}
inv.ExecForTest(t, `insert into settings (node, module, values) select id, $2, $3 from node where name = $1`,
l.Node, l.Module, string(settings))
if l.Placement != nil {
placement, err := json.Marshal(l.Placement)
if err != nil {
t.Fatal(err)
}
inv.ExecForTest(t, `insert into placement (node, module, values) select id, $2, $3 from node where name = $1`,
l.Node, l.Module, string(placement))
}
on[l.Node] = append(on[l.Node], l.Module)
}
for _, node := range []string{"anchor", "laptop"} {
if err := inv.SetPublicDomain(ctx, node, node+".example.test"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, node, on[node]...); err != nil {
t.Fatalf("assign %s %v: %v", node, on[node], err)
}
}
return open, w
}
// toldOfTheWords is what a machine is told of the modules whose layers say the four words, and of the route
// provider that serves their names: those modules' resources and the filter's rules, as JSON a person can diff.
func toldOfTheWords(t *testing.T, open *stores, node string) string {
t.Helper()
ctx := t.Context()
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatal(err)
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Allocating)
if err != nil {
t.Fatal(err)
}
composed, err := plan.Compose(with)
if err != nil {
t.Fatal(err)
}
if len(composed.LeftOut) > 0 {
t.Fatalf("%s leaves out %v", node, composed.LeftOut)
}
rules, err := plan.Rules(with)
if err != nil {
t.Fatal(err)
}
ours := map[string]bool{"proxy": true}
for _, l := range theWordedMesh(t).Layers {
ours[l.Module] = true
}
var told []map[string]any
for _, r := range composed.Resources {
if ours[composed.Owner[r["id"].(string)]] {
told = append(told, r)
}
}
sort.Slice(told, func(i, j int) bool { return told[i]["id"].(string) < told[j]["id"].(string) })
got, err := json.MarshalIndent(map[string]any{"resources": told, "rules": rules}, "", " ")
if err != nil {
t.Fatal(err)
}
return string(got) + "\n"
}
// sameAsTheGolden holds what a machine is told to the golden file the controller before issue 736 wrote from the
// same seed (MESH_WRITE_GOLDEN=1 writes it instead).
func sameAsTheGolden(t *testing.T, open *stores, when string) {
t.Helper()
for _, node := range []string{"anchor", "laptop"} {
got := toldOfTheWords(t, open, node)
golden := filepath.Join("testdata", "placement-words-moved", node+".json")
if os.Getenv("MESH_WRITE_GOLDEN") != "" {
if err := os.MkdirAll(filepath.Dir(golden), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(golden, []byte(got), 0o644); err != nil {
t.Fatal(err)
}
continue
}
want, err := os.ReadFile(golden)
if err != nil {
t.Fatal(err)
}
if string(want) != got {
t.Errorf("%s: %s is told something else:\n%s\nwas:\n%s", when, node, got, want)
}
}
}
// **The four words leave the settings layers and no machine is told anything new** (novox/hq ADR 0348 clause 11,
// issue 736). Every live layer that says `endpoints`, `expose`, `reach` or `ports` is seeded as the store holds it
// after delivery, on a module shaped as its definition is, and every machine's declaration is composed. What each
// is told is byte for byte what the controller before this change composed from the same seed.
func TestADeclarationIsTheSameWhileTheWordsAreStillInTheLayers(t *testing.T) {
open, _ := aMeshWithTheWordsInItsLayers(t)
sameAsTheGolden(t, open, "before the move")
}
// **And after the move, still nothing new** (issue 736). The move of placement, asked without confirming, says
// every layer that holds one of the four words and moves nothing; confirmed, it moves them all, no settings layer
// holds a placement key after, and each machine is told byte for byte what it was told before the move — and what
// the controller before this change told it. Asked again, it moves nothing.
func TestADeclarationIsTheSameAfterTheMoveOfPlacement(t *testing.T) {
open, w := aMeshWithTheWordsInItsLayers(t)
ctx := t.Context()
inv := open.inventory
asked, err := runMovePlacement(ctx, inv, false)
if err != nil || !asked.DryRun || len(asked.Moved) != len(w.Layers) {
t.Fatalf("the move, not confirmed, says %+v (%v); %d layers hold the words", asked, err, len(w.Layers))
}
if held := inv.QueryForTest(t, `select module from settings where values ?| $1`, catalogue.PlacementKeys()); len(held) != len(w.Layers) {
t.Fatalf("the move, not confirmed, moved something: %d layers still hold the words", len(held))
}
sameAsTheGolden(t, open, "after the move was asked without confirming")
moved, err := runMovePlacement(ctx, inv, true)
if err != nil || moved.DryRun || !reflect.DeepEqual(moved.Moved, asked.Moved) {
t.Fatalf("the move says %+v (%v), it said it would move %+v", moved, err, asked.Moved)
}
if held := inv.QueryForTest(t, `select module from settings where values ?| $1`, catalogue.PlacementKeys()); len(held) != 0 {
t.Fatalf("a settings layer still holds a placement key: %v", held)
}
sameAsTheGolden(t, open, "after the move")
if again, err := runMovePlacement(ctx, inv, true); err != nil || len(again.Moved) != 0 {
t.Fatalf("asked again, the move moved %+v (%v)", again, err)
}
if said := describeMovePlacement(movePlacementAnswer{}); !strings.Contains(said, "nothing to move") {
t.Fatalf("an empty move says %q", said)
}
}
// The verb's command line: a dry run unless confirmed, and a confirmed move says why, recorded as a person's step
// of the release, never a repair (issue 736).
func TestTheMoveOfPlacementIsADryRunUnlessConfirmed(t *testing.T) {
for _, c := range []struct {
args map[string]any
want string
}{
{map[string]any{}, "move-placement --json"},
{map[string]any{"confirm": "false"}, "move-placement --json"},
{map[string]any{"confirm": "true", "why": "release 57 is delivered"}, "move-placement --json --confirm --why release 57 is delivered"},
} {
argv, err := argvFor("move-placement", c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%v: %q %v, want %q", c.args, argv, err, c.want)
}
}
for _, args := range []map[string]any{
{"confirm": "true"}, // a move without a why
{"why": "tidy"}, // a why for a dry run, recorded nowhere
{"confirm": "yes", "why": "x"}, // a switch is true or false
{"node": "anchor"},
} {
if argv, err := argvFor("move-placement", args); err == nil {
t.Errorf("%v was composed as %q", args, argv)
}
}
if repairingCommand([]string{"move-placement", "--json", "--confirm", "--why", "x"}) != "move-placement" {
t.Error("a confirmed move through the generic verb would go unrecorded")
}
if repairingCommand([]string{"move-placement", "--json"}) != "" {
t.Error("a dry run was taken for an act by hand")
}
if !personsDecision(link.HandAct{Verb: "move-placement"}) {
t.Error("a move of the release would count toward a healer the mesh lacks")
}
}
+39
View File
@@ -117,6 +117,8 @@ var plainWordings = map[string]func(conditions.Observation) words{
"reaches it. It keeps running what it has.", m), "reaches it. It keeps running what it has.", m),
Resolved: m + " can get new instructions again"} Resolved: m + " can get new instructions again"}
}), }),
// A grant of a restricted stream that composes nothing (novox/hq ADR 0376 §2).
kindGrantComposesNothing: worded(withheldWords),
kindAgentCanBecomeRoot: worded(func(o conditions.Observation) words { kindAgentCanBecomeRoot: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine") m := machineOr(o, "a machine")
return words{Headline: "Sessions on " + m + " could become root", return words{Headline: "Sessions on " + m + " could become root",
@@ -184,6 +186,15 @@ var plainWordings = map[string]func(conditions.Observation) words{
}), }),
// The controller or the node-engine kept at the build a machine was last sent, over an older record (novox/hq // The controller or the node-engine kept at the build a machine was last sent, over an older record (novox/hq
// issue 201). // issue 201).
kindVaultSilent: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "The vault did not give " + m + " the core's credentials",
Needs: "check the vault answers (vault_status); the next send asks it again.",
Explanation: fmt.Sprintf("The bus's and the provisioner's credentials reach %s sealed by the vault, and "+
"the vault did not answer the last send for them. %s keeps the files it has; a core module not "+
"there yet is sealed by the controller until the vault holds it.", m, m),
Resolved: "Resolved: the vault gave " + m + " the core's credentials"}
}),
kindCoreDowngrade: worded(func(o conditions.Observation) words { kindCoreDowngrade: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine") m := machineOr(o, "a machine")
component := idPart(o, 0) component := idPart(o, 0)
@@ -456,6 +467,9 @@ var plainWordings = map[string]func(conditions.Observation) words{
"release-held": worded(func(o conditions.Observation) words { "release-held": worded(func(o conditions.Observation) words {
return releaseHeldWords(nil, o.Also) return releaseHeldWords(nil, o.Also)
}), }),
"release-silent": worded(func(o conditions.Observation) words {
return releaseSilentWords(o.Also)
}),
"facts-stale": worded(func(o conditions.Observation) words { "facts-stale": worded(func(o conditions.Observation) words {
return words{Headline: "Merge checks use outdated facts", return words{Headline: "Merge checks use outdated facts",
Explanation: "The facts a merge check judges a change against are out of date; they are gathered again " + Explanation: "The facts a merge check judges a change against are out of date; they are gathered again " +
@@ -617,6 +631,18 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "The bus reports a listener too slow to keep up, so messages to it are late.", Explanation: "The bus reports a listener too slow to keep up, so messages to it are late.",
Resolved: "The listener keeps up again"} Resolved: "The listener keeps up again"}
}), }),
kindBusFaultsUnread: worded(func(o conditions.Observation) words {
return words{Headline: "Slow clients of the bus cannot be checked",
Explanation: "The controller could not read what the bus said about its clients' connections, so a " +
"client the bus cut off would go unnoticed. It tries again at every check.",
Resolved: "Resolved: the bus's clients are checked again"}
}),
"connection-closed": worded(func(o conditions.Observation) words {
return words{Headline: "The bus cut a client off",
Explanation: "The bus closed a client's connection because it broke the bus's rules, so what it was " +
"sending was lost.",
Resolved: "Resolved: the client keeps to the bus's rules"}
}),
kindBusReconnects: worded(func(o conditions.Observation) words { kindBusReconnects: worded(func(o conditions.Observation) words {
return words{Headline: "A client keeps losing the bus", return words{Headline: "A client keeps losing the bus",
Explanation: "One of the mesh's clients lost its connection to the bus again and again in the last hour. " + Explanation: "One of the mesh's clients lost its connection to the bus again and again in the last hour. " +
@@ -841,6 +867,19 @@ func releaseHeldWords(modules, machines []string) words {
Resolved: "Resolved: the held updates are released"} Resolved: "Resolved: the held updates are released"}
} }
// releaseSilentWords are the plain words of machines that said nothing on an update they were sent (ADR 0353 §4).
func releaseSilentWords(machines []string) words {
where := "Some machines"
if len(machines) > 0 {
where = conditions.Capital(namesWords(machines, 4))
}
return words{Headline: "Machines did not answer an update",
Needs: "check that they are awake and on the mesh; the next walk sends them what waits.",
Explanation: where + " were sent module updates and said nothing about them within fifteen minutes, so " +
"they were left as they were. What they run is not known to be the update.",
Resolved: "Resolved: the next walk reached them"}
}
// reloginNeeds is what an account waiting for its groups needs (ADR 0252). // reloginNeeds is what an account waiting for its groups needs (ADR 0252).
func reloginNeeds(node string) string { func reloginNeeds(node string) string {
return fmt.Sprintf("log out of %s completely and log in again, or restart it.", node) return fmt.Sprintf("log out of %s completely and log in again, or restart it.", node)
+241 -13
View File
@@ -150,6 +150,10 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
// password a provider is told to create is the one its consumer was given — and sealed to // password a provider is told to create is the one its consumer was given — and sealed to
// this node before it was ever written down, so nothing between here and there can read it. // this node before it was ever written down, so nothing between here and there can read it.
for i, n := range resolved.Needs { for i, n := range resolved.Needs {
// **Issued for this holder alone and kept by the vault** (issue 757): where this consumer is handed to the
// provisioner, its machine's client asks the vault for it, and the declaration carries no value.
resolved.Needs[i].IssuedToHolder = issuedToHolder(shelf, n)
resolved.Needs[i].ProviderMoved = providerMoved(shelf, n)
if n.ByRecord { if n.ByRecord {
// Answered by something the mesh holds, so there is no pair-wise secret between two // Answered by something the mesh holds, so there is no pair-wise secret between two
// machines. Its key was supplied by a person and sealed to this node then; the mesh // machines. Its key was supplied by a person and sealed to this node then; the mesh
@@ -182,12 +186,21 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
} }
var secret inventory.Secret var secret inventory.Secret
var err error var err error
if n.SharedOwn != "" { if outsideModule, outside := issuedOutsideBy(shelf, n); outside {
// Only a person can give it (novox/hq ADR 0337, issue 164): read as given, never made.
secret, err = inv.GivenPairSecret(ctx, n.Name, nodeName, n.For, n.From, outsideModule, n.Local)
} else if n.SharedOwn != "" {
// The provider's one credential, sealed to this consumer too (novox/hq ADR 0158). // The provider's one credential, sealed to this consumer too (novox/hq ADR 0158).
secret, err = inv.SharedSecretFor(ctx, n.Name, nodeName, n.For, n.From, providerModuleOf(resolved, open, ctx, n), n.Local, n.SharedOwn) secret, err = inv.SharedSecretFor(ctx, n.Name, nodeName, n.For, n.From, providerModuleOf(resolved, open, ctx, n), n.Local, n.SharedOwn)
} else { } else {
secret, err = inv.SecretFor(ctx, n.Name, nodeName, n.For, n.From, n.Local) secret, err = inv.SecretFor(ctx, n.Name, nodeName, n.For, n.From, n.Local)
} }
if inventory.IsNotGiven(err) {
// **Not given, and the rest of the consumer runs** (ADR 0337 §4): no credential file, so the module can
// tell it was not given and says it waits for it; the machine is not refused for what only a person can
// give, and nothing the mesh made stands in for it.
continue
}
if err != nil { if err != nil {
// Said rather than skipped. A machine that resolves cleanly and receives no // Said rather than skipped. A machine that resolves cleanly and receives no
// credential is one that will fail to authenticate at some later, less obvious // credential is one that will fail to authenticate at some later, less obvious
@@ -280,9 +293,14 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
continue continue
} }
caps, _ := inv.ProfileOf(ctx, n.Name) caps, _ := inv.ProfileOf(ctx, n.Name)
// Its public domain too (novox/hq ADR 0346): a provider elsewhere serves its own composed names,
// and composes them as its own plan does, under its own domain.
// Read as its profile is: a machine whose domain cannot be read (removed since the list was read)
// composes no public name, which leaves the key out and says so, and fails no other machine's plan.
domain, _ := inv.PublicDomainOf(ctx, n.Name)
others = append(others, candidate{ others = append(others, candidate{
catalogue.Node{Name: n.Name, Site: siteOf[n.Name], Capabilities: caps, catalogue.Node{Name: n.Name, Site: siteOf[n.Name], Capabilities: caps,
At: onNetwork[n.Name]}, theirs}) At: onNetwork[n.Name], PublicDomain: domain}, theirs})
} }
offered := map[string][]catalogue.Provider{} offered := map[string][]catalogue.Provider{}
@@ -301,12 +319,18 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
// What that module says a consumer needs to know, with that node's settings on // What that module says a consumer needs to know, with that node's settings on
// it: a port somebody moved on the provider is a port its consumers must be told // it: a port somebody moved on the provider is a port its consumers must be told
// about, and the two coming from different places is how they come to disagree. // about, and the two coming from different places is how they come to disagree.
serves, err := servedOnNode(ctx, inv, o.node.Name, m, name) serves, layers, err := servedOnNode(ctx, inv, o.node.Name, m, name)
if err != nil {
return catalogue.World{}, err
}
// And its own composed names, as that node composes them (novox/hq ADR 0346): a name it
// does not compose leaves the key out, said under each consumer's need.
serves, leftOut, err := got.ServedOwnNames(m, name, serves, catalogue.WithDefaults(m, layers))
if err != nil { if err != nil {
return catalogue.World{}, err return catalogue.World{}, err
} }
offered[name] = append(offered[name], catalogue.Provider{ offered[name] = append(offered[name], catalogue.Provider{
Node: o.node.Name, At: o.node.At, Serves: serves, Module: m.Module}) Node: o.node.Name, At: o.node.At, Serves: serves, Module: m.Module, LeftOut: leftOut})
} }
} }
} }
@@ -442,6 +466,12 @@ func declarationWith(ctx context.Context, open *stores, node string,
if err != nil { if err != nil {
return sendable{}, err return sendable{}, err
} }
// Which ports the filter admits from the local network, and from which networks, for plan to say
// (novox/hq ADR 0340 §8). The same rules the composition rendered, never sent.
rules, err := plan.Rules(with)
if err != nil {
return sendable{}, err
}
// And what was taken on it, said in every declaration it is sent from this one place. // And what was taken on it, said in every declaration it is sent from this one place.
adoption, err := adoptionOf(ctx, open.inventory, record, plan, composed) adoption, err := adoptionOf(ctx, open.inventory, record, plan, composed)
if err != nil { if err != nil {
@@ -450,8 +480,23 @@ func declarationWith(ctx context.Context, open *stores, node string,
out := sendable{Resources: composed.Resources, Adoption: adoption, out := sendable{Resources: composed.Resources, Adoption: adoption,
Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers, Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut, withheld: with.Withheld, LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut, withheld: with.Withheld,
unbound: with.Unbound, foreseen: composed.Foreseen, unplaced: composed.Unplaced, unbound: with.Unbound, foreseen: composed.Foreseen, unplaced: composed.Unplaced, handedOnly: composed.HandedOnly,
modules: namedModules(plan, composed.LeftOut)} modules: namedModules(plan, composed.LeftOut),
Filled: composed.Filled, plans: composed.Plans,
localNetwork: catalogue.LocalNetworkPorts(rules, with.LocalNetworks)}
if len(composed.Filled) > 0 {
out.Provisioner = with.Provisioner
}
// A handed module only pulled in by another keeps what requires it, for its value of its own in the assignments
// mirror (issue 765).
for module := range composed.Plans {
if by := plan.RequiredBy(module); by != "" {
if out.requiredBy == nil {
out.requiredBy = map[string]string{}
}
out.requiredBy[module] = by
}
}
// And which build of each module it carries, for the send to record (novox/hq issue 259, ADR // And which build of each module it carries, for the send to record (novox/hq issue 259, ADR
// 0221). Read only on the send path: a question about what would be sent records nothing. // 0221). Read only on the send path: a question about what would be sent records nothing.
if choosing == Allocating { if choosing == Allocating {
@@ -564,6 +609,10 @@ func reportLeftOut(node string, declared sendable) {
for _, u := range declared.unplaced { for _, u := range declared.unplaced {
fmt.Printf("%s: %s\n", node, u) fmt.Printf("%s: %s\n", node, u)
} }
// And each file only the provisioner's client fills, left out because the module is not handed here.
for _, u := range declared.handedOnly {
fmt.Printf("%s: %s\n", node, u)
}
} }
// busCredentialIssued refuses an own secret called `broker` whose bus account nobody issued. // busCredentialIssued refuses an own secret called `broker` whose bus account nobody issued.
@@ -574,10 +623,17 @@ func reportLeftOut(node string, declared sendable) {
// cannot read and report the machine applied. Refused by name, with the verb — on the send, and on // cannot read and report the machine applied. Refused by name, with the verb — on the send, and on
// a question asked ahead of it (Foreseeing), so that one is never told the push will make it. // a question asked ahead of it (Foreseeing), so that one is never told the push will make it.
func busCredentialIssued(ctx context.Context, inv *inventory.Inventory, node, module, name string) error { func busCredentialIssued(ctx context.Context, inv *inventory.Inventory, node, module, name string) error {
if name != "broker" { var user string
switch name {
case "broker":
user = broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
case broker.ProvisionerSecret:
// And the machine's provisioner client's account, issued with its module (novox/hq ADR 0349 clause 11,
// design 57 P3): the same refusal, for the same reason.
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
default:
return nil return nil
} }
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, minted, err := inv.BusUserHash(ctx, user); err != nil { if _, minted, err := inv.BusUserHash(ctx, user); err != nil {
return err return err
} else if !minted { } else if !minted {
@@ -701,7 +757,18 @@ func renderingFor(ctx context.Context, open *stores, node string,
// per node, so a module running on three machines has three. // per node, so a module running on three machines has three.
needed := map[string]map[string]string{} needed := map[string]map[string]string{}
foreseen := map[string]map[string]bool{} foreseen := map[string]map[string]bool{}
// **A module handed to the provisioner here has its secrets from the vault** (novox/hq ADR 0349 §7, §11,
// design 57 V4): its machine's client asks for each, so the controller makes, reads and seals none of them for
// this machine. Every other module's are made and sealed as before.
hands, err := engineTakesHanded(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
handing := catalogue.Rendering{Hands: hands, Provisioner: provisionerAccount(plan)}
for _, m := range plan.Modules { for _, m := range plan.Modules {
if handing.Handing(m) {
continue
}
// **A secret family is never made** (novox/hq ADR 0283): each member a person gave on this machine is // **A secret family is never made** (novox/hq ADR 0283): each member a person gave on this machine is
// placed, and one not given is nothing — the module says it waits for it. // placed, and one not given is nothing — the module says it waits for it.
for _, family := range m.OwnSecrets.Families() { for _, family := range m.OwnSecrets.Families() {
@@ -795,6 +862,22 @@ func renderingFor(ctx context.Context, open *stores, node string,
needed[m.Module][name] = sealed needed[m.Module][name] = sealed
} }
} }
// **The core's own credentials as the vault sealed them** (novox/hq ADR 0349 §16, issue 755), for an engine that
// opens the vault's box (core_box.go): the declaration carries the box, never the seal made above. Below that
// contract nothing here is asked and the composition is as it was.
var core coreBoxes
opensBox, err := engineOpensVaultBox(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
if opensBox {
core, err = coreBoxesFor(ctx, inv, node, plan.Modules, func(m catalogue.Manifest) bool {
return handing.Handing(m)
}, choosing)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
}
// And a certificate for this machine's name inside the mesh, when anything on it asks. Issued // And a certificate for this machine's name inside the mesh, when anything on it asks. Issued
// rather than stored: the node's key does not change, so signing again produces an equally // rather than stored: the node's key does not change, so signing again produces an equally
// valid certificate and there is nothing to keep in step. // valid certificate and there is nothing to keep in step.
@@ -960,6 +1043,12 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil { if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err return catalogue.Rendering{}, inventory.Node{}, err
} }
// And which of the networks it is attached to count as its local network (novox/hq ADR 0340):
// private ranges only, on a link facing outside it. Reported by the machine, never set.
local, err := localNetworksOf(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// Where this machine reaches each mesh seat's holder, for ${seat:<seat>:reach} (novox/hq ADR // Where this machine reaches each mesh seat's holder, for ${seat:<seat>:reach} (novox/hq ADR
// 0222): the artifact store as this network reaches it, which the container runtime is told to // 0222): the artifact store as this network reaches it, which the container runtime is told to
// trust (ADR 0082). The same address composed into every reference the mesh built. // trust (ADR 0082). The same address composed into every reference the mesh built.
@@ -975,16 +1064,25 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil { if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err return catalogue.Rendering{}, inventory.Node{}, err
} }
readsBusy, err := engineReadsBusy(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
return catalogue.Rendering{ return catalogue.Rendering{
ReadsHealth: readsHealth, ReadsHealth: readsHealth,
JudgesRoot: judgesRoot, JudgesRoot: judgesRoot,
ReadsBusy: readsBusy,
Hands: hands,
Provisioner: provisionerAccount(plan),
BusMembership: memberships[node], BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Foreseen: foreseen, Ports: ports, Settings: settings, Generators: gens, Grants: grants, Needed: needed, Foreseen: foreseen, Ports: ports,
VaultSealed: core.sealed, NotPlaced: core.notPlaced,
Certificate: certificate, Authority: authority, Mesh: private, Names: names, Certificate: certificate, Authority: authority, Mesh: private, Names: names,
Machines: machines, Zones: zones, Holders: replicas, Machines: machines, Zones: zones, Holders: replicas,
Suffix: overlay.Suffix(), MeshRange: meshRange, TunnelInterface: overlay.Interface, Accounts: accounts, Foundation: foundation, Suffix: overlay.Suffix(), MeshRange: meshRange, TunnelInterface: overlay.Interface, Accounts: accounts, Foundation: foundation,
Kept: kept, Adopted: record.Adopted, OutwardLinks: outwardLinks, Kept: kept, Adopted: record.Adopted, OutwardLinks: outwardLinks,
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, SeatReach: reach, Built: built, LocalNetworks: catalogue.CountedLocalNetworks(local),
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, SeatReach: reach, Built: built,
BusUsers: busUsers, Withheld: withheld, Unbound: unbound, BusUsers: busUsers, Withheld: withheld, Unbound: unbound,
}, record, nil }, record, nil
} }
@@ -1010,6 +1108,64 @@ func engineJudgesRoot(ctx context.Context, inv *inventory.Inventory, node string
return had && stated.Contract >= link.RootContract, nil return had && stated.Contract >= link.RootContract, nil
} }
// engineReadsBusy says whether a machine's node-engine reads a container's `busy` (novox/hq ADR 0335), by its
// own newest statement, for the same reason as engineReadsHealth: an older engine parses strictly.
func engineReadsBusy(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.BusyContract, nil
}
// engineTakesHanded says whether a machine's node-engine writes a filled file on the provisioner client's word alone
// (link.HandedContract, design 57 P7), by its own newest statement: only there is a moved module handed to the
// provisioner (design 57 P6). Anywhere else a filled file would have two writers, and the module is sent as before.
func engineTakesHanded(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.HandedContract, nil
}
// provisionerAccount is the account the provisioner's client runs as on this machine (novox/hq ADR 0349 §4): the user
// the process of the module running it runs as, empty where no such module is on the machine.
func provisionerAccount(plan catalogue.Resolution) string {
for _, m := range plan.Modules {
if !runsTheProvisionerClient(m) {
continue
}
for _, r := range m.Resources {
if r["type"] == "process" {
if user, _ := r["user"].(string); user != "" && user != "root" {
return user
}
}
}
}
return ""
}
// busyNotSent is, for a node-engine that does not read a container's busy, each container whose module declares
// one: not sent it, so replaced at once while it may hold work (novox/hq ADR 0335). Nothing for one that reads it.
func busyNotSent(modules []catalogue.Manifest, reads bool) []string {
if reads {
return nil
}
var out []string
for _, m := range modules {
for _, r := range m.Resources {
if _, has := r[catalogue.BusyField]; has {
out = append(out, fmt.Sprintf("%s.%v declares busy, and this node-engine does not read it yet (its "+
"statement's contract is below %d): not sent, so the container is replaced at once, busy or not "+
"(novox/hq ADR 0335)", m.Module, r["id"], link.BusyContract))
}
}
}
return out
}
// zonesInTheMesh is every zone a module in the mesh declares, where the mesh placed it (novox/hq ADR // zonesInTheMesh is every zone a module in the mesh declares, where the mesh placed it (novox/hq ADR
// 0199): the zone settled from that node's settings, the node's private address, the port the // 0199): the zone settled from that node's settings, the node's private address, the port the
// answering listen is published on there. // answering listen is published on there.
@@ -1403,10 +1559,29 @@ func planCommand(ctx context.Context, args []string) error {
// in this output would have told anybody that. // in this output would have told anybody that.
for _, n := range plan.Needs { for _, n := range plan.Needs {
fmt.Printf(" needs %s from %s, for %s\n", n.Name, n.From, n.For) fmt.Printf(" needs %s from %s, for %s\n", n.Name, n.From, n.For)
// And a served key the provider left out, because its machine composes no name it asks for
// (novox/hq ADR 0346 §3).
for _, line := range n.LeftOut {
fmt.Printf(" %s\n", line)
}
}
// What the providers on this machine leave out of what they serve, and why: said where the remedy
// is (novox/hq ADR 0346 §3).
for _, line := range plan.ServedLeftOut(settings) {
fmt.Printf("%s: %s\n", args[0], line)
} }
for _, line := range unboundLines(plan) { for _, line := range unboundLines(plan) {
fmt.Println(line) fmt.Println(line)
} }
// And a container's busy that this machine's node-engine is too old to be sent (novox/hq ADR 0335): its
// container is replaced at once, as before. A statement that cannot be read is said as such.
if reads, err := engineReadsBusy(ctx, open.inventory, args[0]); err != nil {
fmt.Printf("%s: whether its node-engine reads a container's busy could not be read: %v\n", args[0], err)
} else {
for _, line := range busyNotSent(plan.Modules, reads) {
fmt.Printf("%s: %s\n", args[0], line)
}
}
declared, err := declarationFor(ctx, open, args[0], plan, settings) declared, err := declarationFor(ctx, open, args[0], plan, settings)
if err != nil { if err != nil {
return err return err
@@ -1417,6 +1592,11 @@ func planCommand(ctx context.Context, args []string) error {
fmt.Printf(" %-20s settings from %s\n", module, layer.From) fmt.Printf(" %-20s settings from %s\n", module, layer.From)
} }
} }
// What each port reached from the local network resolved to on this machine (novox/hq ADR 0340 §8),
// or that it resolved to none.
for _, p := range declared.localNetwork {
fmt.Println(" " + p.Said())
}
fmt.Printf("\n%d resource(s)\n", len(resources)) fmt.Printf("\n%d resource(s)\n", len(resources))
if *show { if *show {
@@ -1538,19 +1718,20 @@ func managerPublicKeyFor(
// artifact store's trust and the references composed through it (04-ISSUES/102). Unreadable // artifact store's trust and the references composed through it (04-ISSUES/102). Unreadable
// given ports are that node's refusal to report, not this reader's. // given ports are that node's refusal to report, not this reader's.
func servedOnNode(ctx context.Context, inv *inventory.Inventory, node string, func servedOnNode(ctx context.Context, inv *inventory.Inventory, node string,
m catalogue.Manifest, provision string) (map[string]any, error) { m catalogue.Manifest, provision string) (map[string]any, []catalogue.Layer, error) {
ports, layers, err := portsGivenOn(ctx, inv, node, m) ports, layers, err := portsGivenOn(ctx, inv, node, m)
if err != nil { if err != nil {
return nil, err return nil, nil, err
} }
serves := catalogue.ServedOn(m, provision, ports) serves := catalogue.ServedOn(m, provision, ports)
if len(serves) > 0 { if len(serves) > 0 {
serves, err = catalogue.Settle(serves, catalogue.WithDefaults(m, layers)) serves, err = catalogue.Settle(serves, catalogue.WithDefaults(m, layers))
if err != nil { if err != nil {
return nil, err return nil, nil, err
} }
} }
return serves, nil // The layers too, read once, for what is filled after (novox/hq ADR 0346).
return serves, layers, nil
} }
// portsGivenOn is where a node puts a module's ports — assigned, then given over them — and the // portsGivenOn is where a node puts a module's ports — assigned, then given over them — and the
@@ -1742,6 +1923,53 @@ func busUserList(ctx context.Context, inv *inventory.Inventory,
// providerModuleOf is which module answers a need on the providing node: the one in this node's // providerModuleOf is which module answers a need on the providing node: the one in this node's
// own set when the provider is here, else the one the catalogue says offers it. // own set when the provider is here, else the one the catalogue says offers it.
// issuedOutsideBy is whether a need's provision is offered by its provider with a credential issued outside the
// mesh (novox/hq ADR 0337), and that provider module: the one answering it when known, else any that offers it so.
func issuedOutsideBy(shelf map[string]catalogue.Manifest, n catalogue.Needed) (string, bool) {
if n.Module != "" {
if m, known := shelf[n.Module]; known {
return n.Module, m.IssuedOutsideOf(n.Name)
}
}
return catalogue.OffersIssuedOutside(shelf, n.Name)
}
// issuedToHolder is whether the vault keeps a need's credential for its holder alone (issues 749, 757 and 779): its
// providing module is marked moved, so the vault holds the credential in a row of its own for each holder
// (filledTheNewWay), and its offer says either that the provider's software issues the provision's credential, or
// that the provider gives each consumer a login of its own (form pair, ADR 0158 §11). A credential every holder
// shares, or one a person gives, is neither.
func issuedToHolder(shelf map[string]catalogue.Manifest, n catalogue.Needed) bool {
if n.ByRecord || n.Module == "" {
return false
}
provider, known := shelf[n.Module]
if !known || !movedProvider(provider) {
return false
}
if _, issued := provider.IssuedCredentialOf(n.Name); issued {
return true
}
return provider.CredentialFormOf(n.Name) == catalogue.CredentialFormPair
}
// providerMoved is whether a need is answered by a module marked moved (issue 788), or by one whose offer of the
// provision says the provider helper answers it (issue 792).
func providerMoved(shelf map[string]catalogue.Manifest, n catalogue.Needed) bool {
if n.ByRecord || n.Module == "" {
return false
}
provider, known := shelf[n.Module]
return known && (movedProvider(provider) || provider.AnsweredByHelperFor(n.Name))
}
// movedProvider is whether a provider's credentials are the vault's: the module is marked moved, or it holds the
// mesh-vault seat, whose pair rows have been in the vault since V3 (issue 793), so its consumers' credentials
// (gitea's internal token and admin password) are filled from those very rows with the values unchanged.
func movedProvider(m catalogue.Manifest) bool {
return m.Moved || slices.ContainsFunc(m.Claims, func(c catalogue.Claim) bool { return c.Name == "mesh-vault" })
}
func providerModuleOf(resolved catalogue.Resolution, open *stores, ctx context.Context, n catalogue.Needed) string { func providerModuleOf(resolved catalogue.Resolution, open *stores, ctx context.Context, n catalogue.Needed) string {
for _, m := range resolved.Modules { for _, m := range resolved.Modules {
if _, shared := m.SharedCredentialOf(n.Name); shared { if _, shared := m.SharedCredentialOf(n.Name); shared {
+7
View File
@@ -372,6 +372,13 @@ func joinAPlan(ctx context.Context, open *stores, module string) (bool, string,
if !found { if !found {
return false, "", nil return false, "", nil
} }
// **A walk that waits for its delivery's word asks nothing** (novox/hq ADR 0239, issue 754): a rebuild does not
// join it, and its outcome is not the walk's. The walk asks its own tier once it has its word.
if p.Waiting() {
return false, fmt.Sprintf("not joining %s (%s at %s): it waits for %s's word, so this build is not its "+
"outcome; once let go, the walk asks %s itself", p.ID, p.Repository, short(p.Commit), p.Delivery.Awaits,
module), nil
}
entries, err := inv.Catalogued(ctx) entries, err := inv.Catalogued(ctx)
if err != nil { if err != nil {
return false, "", err return false, "", err
+1 -1
View File
@@ -133,7 +133,7 @@ func TestASharedRepositoryIsPlannedFromTheRecordedBuildSources(t *testing.T) {
{"the route proxy's program: the route proxy alone", "mesh-controller", {"the route proxy's program: the route proxy alone", "mesh-controller",
[]string{"examples/route-proxy/main.go"}, "route-proxy", true}, []string{"examples/route-proxy/main.go"}, "route-proxy", true},
{"a package all three build from: all three", "mesh-controller", {"a package all three build from: all three", "mesh-controller",
[]string{"internal/broker/broker.go"}, "mesh-controller | build-agent | route-proxy", false}, []string{"internal/broker/broker.go"}, "mesh-controller,route-proxy | build-agent", false},
{"the route proxy's directory in the catalogue: it alone", "mesh-catalog", {"the route proxy's directory in the catalogue: it alone", "mesh-catalog",
[]string{"modules/route-proxy/module.json"}, "route-proxy", false}, []string{"modules/route-proxy/module.json"}, "route-proxy", false},
{"the build agent's directory: it alone, nothing it builds", "mesh-catalog", {"the build agent's directory: it alone, nothing it builds", "mesh-catalog",
+11 -9
View File
@@ -145,8 +145,9 @@ func TestAPlanIsWhatTheChangeTouchedAndWhatIsBuiltOnIt(t *testing.T) {
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a,x"}, repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a,x"},
{what: "built-by never widens", edges: []inventory.Edge{dep("x", builtBy, "a")}, {what: "built-by never widens", edges: []inventory.Edge{dep("x", builtBy, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a"}, repo: "one", paths: []string{"modules/a/x"}, want: "a"},
{what: "built-by orders", edges: []inventory.Edge{dep("x", builtBy, "a")}, // Whichever builder runs builds it (novox/hq ADR 0360 §3): built-by orders nothing, as before it ordered.
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a | x"}, {what: "built-by does not order", edges: []inventory.Edge{dep("x", builtBy, "a")},
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a,x"},
{what: "worker-of never widens", edges: []inventory.Edge{dep("x", workerOf, "a")}, {what: "worker-of never widens", edges: []inventory.Edge{dep("x", workerOf, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a"}, repo: "one", paths: []string{"modules/a/x"}, want: "a"},
{what: "worker-of orders", edges: []inventory.Edge{dep("x", workerOf, "a")}, {what: "worker-of orders", edges: []inventory.Edge{dep("x", workerOf, "a")},
@@ -296,18 +297,18 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
[]string{"internal/inventory/migrations/0088-a-build-says-its-build-source.sql"}, read, []string{"internal/inventory/migrations/0088-a-build-says-its-build-source.sql"}, read,
"mesh-controller | build-agent", ""}, "mesh-controller | build-agent", ""},
// A package all three build from: all three; the build seat's holder after the controller whose worker // A package all three build from: all three; the build seat's holder after the controller whose worker
// it binds (worker-of, issue 206), the proxy after the holder that builds it (built-by). // it binds (worker-of, issue 206), the proxy beside the controller: built-by orders nothing (ADR 0360 §3).
{"a package all three build from: all three", "mesh-controller", {"a package all three build from: all three", "mesh-controller",
[]string{"internal/broker/broker.go"}, read, "mesh-controller | build-agent | route-proxy", ""}, []string{"internal/broker/broker.go"}, read, "mesh-controller,route-proxy | build-agent", ""},
{"a vendored package all three build from: all three", "mesh-controller", {"a vendored package all three build from: all three", "mesh-controller",
[]string{"vendor/github.com/nats-io/nats.go/nats.go"}, read, "mesh-controller | build-agent | route-proxy", ""}, []string{"vendor/github.com/nats-io/nats.go/nats.go"}, read, "mesh-controller,route-proxy | build-agent", ""},
{"go.sum: all three", "mesh-controller", {"go.sum: all three", "mesh-controller",
[]string{"go.sum"}, read, "mesh-controller | build-agent | route-proxy", ""}, []string{"go.sum"}, read, "mesh-controller,route-proxy | build-agent", ""},
{"a file added to the proxy's package: the proxy", "mesh-controller", {"a file added to the proxy's package: the proxy", "mesh-controller",
[]string{"examples/route-proxy/new.go"}, read, "route-proxy", ""}, []string{"examples/route-proxy/new.go"}, read, "route-proxy", ""},
// Before any build source is said: as before. // Before any build source is said: as before.
{"no build source said: a README moves all three, as before", "mesh-controller", {"no build source said: a README moves all three, as before", "mesh-controller",
[]string{"README.md"}, unsaid, "mesh-controller | build-agent | route-proxy", ""}, []string{"README.md"}, unsaid, "mesh-controller,route-proxy | build-agent", ""},
// In the catalogue, where they live, each by its own build source. // In the catalogue, where they live, each by its own build source.
{"the route proxy's recipe: it alone", "mesh-catalog", {"the route proxy's recipe: it alone", "mesh-catalog",
[]string{"modules/route-proxy/Dockerfile"}, read, "route-proxy", ""}, []string{"modules/route-proxy/Dockerfile"}, read, "route-proxy", ""},
@@ -331,7 +332,7 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
// Files not all said: everything the repository builds, as before. // Files not all said: everything the repository builds, as before.
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main", Commit: "head", m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main", Commit: "head",
Paths: []string{"README.md"}, PathsTruncated: true} Paths: []string{"README.md"}, PathsTruncated: true}
if got := tiered(reachOfMerge(m, entries, read, edges).Plan.Tiers); got != "mesh-controller | build-agent | route-proxy" { if got := tiered(reachOfMerge(m, entries, read, edges).Plan.Tiers); got != "mesh-controller,route-proxy | build-agent" {
t.Errorf("a merge whose files were not all said: planned %q, wanted all three", got) t.Errorf("a merge whose files were not all said: planned %q, wanted all three", got)
} }
} }
@@ -452,8 +453,9 @@ func TestAPlanIsTheTouchedModulesAndWhatIsReachableAlongTheWideningEdges(t *test
kinds := []string{inventory.EdgeStandsOn, inventory.EdgeDeclared, inventory.EdgePackages, kinds := []string{inventory.EdgeStandsOn, inventory.EdgeDeclared, inventory.EdgePackages,
inventory.EdgeBuiltBy, inventory.EdgeWorkerOf} inventory.EdgeBuiltBy, inventory.EdgeWorkerOf}
widens := map[string]bool{inventory.EdgeStandsOn: true, inventory.EdgeDeclared: true} widens := map[string]bool{inventory.EdgeStandsOn: true, inventory.EdgeDeclared: true}
// Built-by orders nothing since novox/hq ADR 0360 §3.
orders := map[string]bool{inventory.EdgeStandsOn: true, inventory.EdgeDeclared: true, orders := map[string]bool{inventory.EdgeStandsOn: true, inventory.EdgeDeclared: true,
inventory.EdgeBuiltBy: true, inventory.EdgeWorkerOf: true} inventory.EdgeWorkerOf: true}
// Directory names drawn from one pool, so two repositories hold directories of the same name, and one // Directory names drawn from one pool, so two repositories hold directories of the same name, and one
// is a prefix of another. // is a prefix of another.
dirs := []string{"a", "ab", "b", "c", "lib/x", "lib/y", "modules/a", "modules/a/sub"} dirs := []string{"a", "ab", "b", "c", "lib/x", "lib/y", "modules/a", "modules/a/sub"}
+93
View File
@@ -0,0 +1,93 @@
package main
import (
"context"
"fmt"
"path"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The list of predefined keys the mesh publishes (novox/hq ADR 0348 §7, design 57 package K2, issue 661).
//
// **The newest trunk build of the catalogue wins.** A build of the catalogue reads predefined.json at the commit
// it builds and carries it with the commit's generation (link.PredefinedList). Registration keeps it in the
// controller's store when that generation is larger than the one kept (inventory.KeepPredefined), so an older
// build's list heard later never replaces a newer one, and every manifest the controller reads again is judged
// against the list kept (catalogue.UsePredefined). mesh-config publishes it on the bus from there (design 57
// package K3); the controller keeps no config value, only the keys.
// fromTheCatalogue is whether a build is of the catalogue's repository, whose list alone the mesh publishes: by
// its name, in the URL form or the seat form.
func fromTheCatalogue(result link.BuildResult) bool {
for _, repository := range []string{result.Repository, sourceRepository(result)} {
name := strings.TrimSuffix(path.Base(strings.TrimRight(repository, "/")), ".git")
if name == catalogue.CatalogueRepository {
return true
}
}
return false
}
func sourceRepository(result link.BuildResult) string {
if result.Source == nil {
return ""
}
return result.Source.Repository
}
// carriedList is the list a build of the catalogue carried, read strictly; nil for a build of another
// repository, or one that carried none.
func carriedList(result link.BuildResult) (catalogue.PredefinedList, error) {
if result.Predefined == nil || !fromTheCatalogue(result) {
return nil, nil
}
list, err := catalogue.ParsePredefined(result.Predefined.Keys)
if err != nil {
return nil, fmt.Errorf("%s carried a list of predefined keys that is refused: %w", result.ID, err)
}
return list, nil
}
// judgedWith is the list a build's manifest is first judged against at registration: the catalogue's own at the
// commit built, which its build judged it against, where it carried one; the list kept otherwise.
func judgedWith(result link.BuildResult) (catalogue.PredefinedList, error) {
list, err := carriedList(result)
if err != nil || list != nil {
return list, err
}
return catalogue.PublishedPredefined(), nil
}
// loadPredefined makes the list kept in the store the one this controller judges manifests against. A store
// keeping none leaves the list it has.
func loadPredefined(ctx context.Context, inv *inventory.Inventory) error {
kept, has, err := inv.PredefinedKept(ctx)
if err != nil || !has {
return err
}
catalogue.UsePredefined(kept.Keys)
return nil
}
// publishPredefined keeps a trunk build's list when it is newer than the one kept, says so, and loads what the
// store keeps then — this build's, or a newer one another controller process kept.
func publishPredefined(ctx context.Context, inv *inventory.Inventory, result link.BuildResult,
list catalogue.PredefinedList) error {
kept, err := inv.KeepPredefined(ctx, inventory.PredefinedRecord{Keys: list, Repository: result.Repository,
Commit: result.Commit, Generation: result.Predefined.Generation, Build: result.ID})
if err != nil {
return err
}
if kept {
fmt.Printf("%s: the list of predefined keys at %s (generation %d) is published: %s\n", result.ID,
short(result.Commit), result.Predefined.Generation, strings.Join(list.Keys(), ", "))
} else {
fmt.Printf("%s: the list of predefined keys at %s (generation %d) is not newer than the one kept, which "+
"stands\n", result.ID, short(result.Commit), result.Predefined.Generation)
}
return loadPredefined(ctx, inv)
}
+197
View File
@@ -0,0 +1,197 @@
package main
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// The list of predefined keys (novox/hq ADR 0348 §7, design 57 package K2, issue 661): `module check` and the
// merge gate judge each manifest against the list at the root of its tree; registration publishes the newest
// trunk build's list of the catalogue and judges every module against the list the mesh publishes.
const theFirstKeys = `{
"font-size": {"type": "number", "description": "the size text is shown in, in points", "minimum": 6, "maximum": 72},
"agent-managed-settings": {"type": "object", "description": "the managed settings of every agent session, a JSON object", "trusted": true}
}`
// notifierFallingBackTo is the notifier in the config form, its font size falling back to the key named.
func notifierFallingBackTo(key, kind string) string {
def := "10"
if kind == "string" {
def = `"10"`
}
return `{"module":"notifier","version":"1",
"config":{"font-size":{"type":"` + kind + `","description":"the notification's font size, in points",
"default":` + def + `,"predefined":"` + key + `"}},
"resources":[{"id":"conf","type":"file","path":"/etc/notifier/notifier.conf","mode":"0644",
"content":"font = ${config:font-size}\n"}]}`
}
// withoutAPublishedList judges the test against no list the mesh published, and puts the process's back after.
func withoutAPublishedList(t *testing.T) {
t.Helper()
was := catalogue.ReplacePublished(nil)
t.Cleanup(func() { catalogue.ReplacePublished(was) })
}
// aRepositoryTree is a repository's checkout holding one module at modules/notifier, and the list when given.
func aRepositoryTree(t *testing.T, manifest string, list *string) string {
t.Helper()
root := t.TempDir()
if err := os.Mkdir(filepath.Join(root, ".git"), 0o755); err != nil {
t.Fatal(err)
}
if list != nil {
if err := os.WriteFile(filepath.Join(root, catalogue.PredefinedFile), []byte(*list), 0o644); err != nil {
t.Fatal(err)
}
}
return aTemplatedModule(t, filepath.Join(root, "modules", "notifier"), manifest, nil)
}
func TestModuleCheckJudgesAManifestAgainstTheListAtItsRepositoryRoot(t *testing.T) {
withoutAPublishedList(t)
list := theFirstKeys
check := func(path string) (string, error) {
var out bytes.Buffer
err := moduleCheck([]string{path}, &out)
return out.String(), err
}
if out, err := check(aRepositoryTree(t, notifierFallingBackTo("font-size", "number"), &list)); err != nil ||
!strings.Contains(out, "notifier: ok") {
t.Fatalf("a module naming font-size was refused: %v\n%s", err, out)
}
for _, c := range []struct {
manifest string
list *string
says string
}{
{notifierFallingBackTo("font-scale", "number"), &list, `"font-scale", which is not on the list`},
{notifierFallingBackTo("font-size", "string"), &list, "which is a number, and this value is a string"},
{notifierFallingBackTo("font-size", "number"), nil, "is not present"},
{notifierFallingBackTo("font-size", "number"), ptr(`{"font-size": {"type": "number"}}`), "no description"},
} {
if out, err := check(aRepositoryTree(t, c.manifest, c.list)); err == nil || !strings.Contains(out, c.says) {
t.Errorf("wanted the check refused saying %q, got %v:\n%s", c.says, err, out)
}
}
}
func ptr(s string) *string { return &s }
// writeTheList puts the list at the root of a tree the gate judges.
func writeTheList(t *testing.T, tree, list string) string {
t.Helper()
if err := os.WriteFile(filepath.Join(tree, catalogue.PredefinedFile), []byte(list), 0o644); err != nil {
t.Fatal(err)
}
return tree
}
func TestTheMergeGateJudgesEachManifestAgainstTheListOfTheTree(t *testing.T) {
withoutAPublishedList(t)
f, manifests := templatedMesh(t)
v := gateJudged(t, f, writeTheList(t, aTree(t, withEdit(manifests, "notifier",
notifierFallingBackTo("font-size", "number"))), theFirstKeys), "modules/notifier/module.json")
if v.Verdict == "fail" || v.Modules["notifier"] != "changed" {
t.Fatalf("a module naming font-size on the tree's list failed the gate (%v):\n%s", v.Modules, v.Report())
}
for _, c := range []struct{ manifest, list, says string }{
{notifierFallingBackTo("font-scale", "number"), theFirstKeys, `"font-scale", which is not on the list`},
{notifierFallingBackTo("font-size", "string"), theFirstKeys, "which is a number"},
{inlineNotifier, `{"font-size": {"type": "number", "description": "points", "unit": "pt"}}`, `unknown field "unit"`},
} {
v := gateJudged(t, f, writeTheList(t, aTree(t, withEdit(manifests, "notifier", c.manifest)), c.list),
"modules/notifier/module.json", catalogue.PredefinedFile)
if v.Verdict != "fail" || !strings.Contains(v.Report(), c.says) {
t.Errorf("wanted the gate to fail saying %q:\n%s", c.says, v.Report())
}
}
}
// catalogueBuild is a trunk build of the catalogue carrying the list at a generation.
func catalogueBuild(id, manifest, list string, generation int) link.BuildResult {
r := link.BuildResult{ID: id, Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/notifier", Ref: "main", On: "anchor", Commit: strings.Repeat(id[len(id)-1:], 12),
Manifest: json.RawMessage(manifest), Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
if list != "" {
r.Predefined = &link.PredefinedList{Keys: json.RawMessage(list), Generation: generation}
}
return r
}
func TestRegistrationPublishesTheNewestTrunkListAndJudgesEveryModuleAgainstIt(t *testing.T) {
withoutAPublishedList(t)
open := aMesh(t)
ctx := t.Context()
// A manifest naming font-size registers, and its build's list is published.
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory,
catalogueBuild("b-1", notifierFallingBackTo("font-size", "number"), theFirstKeys, 10))); err != nil {
t.Fatalf("a module naming font-size was not registered: %v", err)
}
kept, has, err := open.inventory.PredefinedKept(ctx)
if err != nil || !has || kept.Generation != 10 || kept.Build != "b-1" {
t.Fatalf("the list was not published: %+v %v %v", kept, has, err)
}
if got := catalogue.PublishedPredefined(); len(got) != 2 {
t.Fatalf("the controller judges against %v, not the list published", got)
}
// One naming a key not on the list, or of another type, is refused and never enters the catalogue.
for i, c := range []struct{ manifest, says string }{
{notifierFallingBackTo("font-scale", "number"), "not on the list"},
{notifierFallingBackTo("font-size", "string"), "which is a number"},
} {
id := []string{"b-2", "b-3"}[i]
if _, _, err := takeIn(ctx, open.inventory, catalogueBuild(id, c.manifest, theFirstKeys, 10)); err == nil ||
!strings.Contains(err.Error(), c.says) {
t.Errorf("wanted the registration refused saying %q, got %v", c.says, err)
}
}
src, err := open.inventory.SourceOf(ctx, "notifier")
if err != nil || src.BuiltFrom != strings.Repeat("1", 12) {
t.Fatalf("a refused build replaced the registered one: %+v %v", src, err)
}
// A newer build's list, which drops agent-managed-settings, replaces the published one.
smaller := `{"font-size": {"type": "number", "description": "points", "minimum": 6, "maximum": 72}}`
if _, _, err := takeIn(ctx, open.inventory, catalogueBuild("b-4",
notifierFallingBackTo("font-size", "number"), smaller, 12)); err != nil {
t.Fatal(err)
}
// The older build's outcome heard later registers nothing new about the list: the newer stands.
if _, _, err := takeIn(ctx, open.inventory, catalogueBuild("b-5",
notifierFallingBackTo("font-size", "number"), theFirstKeys, 11)); err != nil {
t.Fatal(err)
}
kept, _, _ = open.inventory.PredefinedKept(ctx)
if kept.Build != "b-4" || len(kept.Keys) != 1 || len(catalogue.PublishedPredefined()) != 1 {
t.Fatalf("an older build's list heard later replaced the newer one: %+v", kept)
}
// A build of another repository publishes nothing, and its manifest is judged against the published list.
other := catalogueBuild("b-6", notifierFallingBackTo("font-size", "number"), theFirstKeys, 99)
other.Repository, other.Source.Repository = "http://forge.internal:20000/novox/notifier.git", "novox/notifier"
other.Manifest = json.RawMessage(strings.Replace(notifierFallingBackTo("font-size", "number"), `"notifier"`,
`"chimes"`, 1))
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, other)); err != nil {
t.Fatalf("a module of another repository naming a published key: %v", err)
}
if kept, _, _ = open.inventory.PredefinedKept(ctx); kept.Build != "b-4" {
t.Fatalf("a build of another repository published its list: %+v", kept)
}
other.ID, other.Manifest = "b-7", json.RawMessage(strings.Replace(notifierFallingBackTo("agent-managed-settings",
"number"), `"notifier"`, `"chimes"`, 1))
if _, _, err := takeIn(ctx, open.inventory, other); err == nil || !strings.Contains(err.Error(), "not on the list") {
t.Fatalf("a key only another repository's list holds was registered: %v", err)
}
}
+6
View File
@@ -929,6 +929,12 @@ func streamDiffers(want broker.Stream, have nats.StreamConfig) string {
differs = append(differs, fmt.Sprintf("keeps one of a message id for %s, defined %s", have.Duplicates, differs = append(differs, fmt.Sprintf("keeps one of a message id for %s, defined %s", have.Duplicates,
time.Duration(want.DuplicatesSeconds)*time.Second)) time.Duration(want.DuplicatesSeconds)*time.Second))
} }
if want.AllowMsgTTL && !have.AllowMsgTTL {
differs = append(differs, "takes no time to live of a message's own, defined to (novox/hq ADR 0376)")
}
if want.Restricted && have.Metadata[broker.RestrictedMetadata] != "true" {
differs = append(differs, "does not say it is restricted, so a message kept from it would not age out with it")
}
if want.DiscardNew && have.Discard != nats.DiscardNew { if want.DiscardNew && have.Discard != nats.DiscardNew {
differs = append(differs, "drops what it holds when full, defined to refuse what comes next") differs = append(differs, "drops what it holds when full, defined to refuse what comes next")
} }
+263 -65
View File
@@ -18,8 +18,9 @@ package main
// it beside the layer (`settings` says it back), and records the warrant in the hand-act log on the bus, where a // it beside the layer (`settings` says it back), and records the warrant in the hand-act log on the bus, where a
// person's decisions are read; the router edits the ask on every channel to its outcome. No seat event of its // person's decisions are read; the router edits the ask on every channel to its outcome. No seat event of its
// own: nothing consumes one, and the installer's first user list in the node-engine's repository names every // own: nothing consumes one, and the installer's first user list in the node-engine's repository names every
// controller event, so one would cost a node-engine change for a fact without a reader. The push afterwards is a // controller event, so one would cost a node-engine change for a fact without a reader. No push is left for
// separate act, as it is for a layer set at the terminal. // afterwards: the warrant that sets the layer opens the walk that sends it (novox/hq ADR 0344, settings_walk.go),
// so the operator's one Approve is the act that makes the change take effect.
// //
// **What the operator reads** is the module, the machine, each key with its exact new value and, for a changed // **What the operator reads** is the module, the machine, each key with its exact new value and, for a changed
// key, the value it replaces. A value that may not leave the mesh (an address, a path, a secret's shape: the // key, the value it replaces. A value that may not leave the mesh (an address, a path, a secret's shape: the
@@ -27,6 +28,13 @@ package main
// ask says so, and the whole is read with `settings proposals <id>` — whose fingerprint must be the one on the // ask says so, and the whole is read with `settings proposals <id>` — whose fingerprint must be the one on the
// phone. Fail closed: a proposal nothing can carry to the operator is refused at once, in words, and never left // phone. Fail closed: a proposal nothing can carry to the operator is refused at once, in words, and never left
// waiting for an answer that cannot come. // waiting for an answer that cannot come.
//
// **On a channel that proves who answers, the values are shown WHOLE** (novox/hq issue 383, the operator's
// decision of 2026-10-10): a mount point, a share name or a private address is the thing being approved and must
// be readable, so the ask carries them whole beside the explanation (asks.Ask.Whole), the router shows that only
// on a kind that verifies its sender, and only a value shaped like a secret is withheld there. The message is the
// headline, one line per key, who proposed it and when; the fingerprint and how to read the proposal whole are
// the Details answer's (asks.Ask.Details). The masking stays for conditions and for channels that prove nothing.
import ( import (
"context" "context"
@@ -46,7 +54,9 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks" "git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions" "github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link" "github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/outward" "github.com/novox/mesh-controller/internal/outward"
) )
@@ -154,47 +164,48 @@ func (p settingsProposal) ask(id string, machines []string) (asks.Ask, map[strin
if len([]rune(headline)) > asks.HeadlineLength { if len([]rune(headline)) > asks.HeadlineLength {
headline = verb + " settings?" headline = verb + " settings?"
} }
shown, whole := p.change(machines) // The message reads at a glance (issues 383 and 639): a summary of what changes where, then at most three
var b strings.Builder // points, a key each; every key with its whole value, who proposed it, the fingerprint and how to read it at the
if p.Clear { // terminal are its Details, which the router notes after the answers.
fmt.Fprintf(&b, "Clear the settings of %s on %s, back to what the module says?\n\n", p.Module, p.where()) shown, shownAll, shownWhole := p.change(machines, false)
} else { whole, wholeAll, _ := p.change(machines, true)
fmt.Fprintf(&b, "Set the settings of %s on %s to these values?\n\n", p.Module, p.where()) summary := p.summary()
compose := func(points []string) string {
return strings.Join(append([]string{summary}, points...), "\n")
} }
fmt.Fprintf(&b, "Proposed by %s, at %s. ", sayable(p.From, machines), p.At.Local().Format("15:04 on 2 Jan")) // The Details: every setting it sets, one line each, uncut — whole where the sender is proven (WholeDetails),
switch { // masked elsewhere — then who proposed it, the fingerprint and how to read it at the terminal.
case p.Clear && p.HadLayer: details := func(lines []string) string {
b.WriteString("The layer it removes:\n\n") var d strings.Builder
case p.Clear: d.WriteString(strings.Join(lines, "\n") + "\n")
b.WriteString("There is no layer to remove; approving changes nothing.") d.WriteString(p.onApprove() + "\n")
case !p.HadLayer: fmt.Fprintf(&d, "Proposed by %s at %s. Fingerprint %s.\n", sayable(p.From, machines), p.At.Local().Format("15:04 on 2 Jan"),
b.WriteString("There is no layer yet; this is the whole of it:\n\n") fingerprint(p.Digest))
default: if !shownWhole {
b.WriteString("The layer is replaced whole; what changes against it:\n\n") d.WriteString("Where who answers is not proven, its values are shown as ‹address›, ‹path› or ‹withheld›.\n")
}
fmt.Fprintf(&d, "Read it whole, with this fingerprint: settings proposals %s at the controller's terminal, or the "+
"controller's settings verb with proposal %s through the mesh MCP server.", id, id)
return d.String()
} }
b.WriteString(shown) q := asks.Ask{ID: id, Headline: headline, Explanation: compose(shown), Who: asks.Operator,
fmt.Fprintf(&b, "\n\nFingerprint %s.", fingerprint(p.Digest))
if !whole {
b.WriteString(" Parts shown as ‹address›, ‹path› or ‹withheld› may not leave the mesh: read it whole, with " +
"this fingerprint, through the mesh MCP server (mesh-controller.settings, proposal " + id + ") or with " +
"mesh-cli settings proposals " + id + ".")
}
if p.Clear {
b.WriteString(" Approved, the layer is removed at once and the machine takes it at its next push.")
} else {
b.WriteString(" Approved, the layer is set at once and the machine takes it at its next push.")
}
q := asks.Ask{ID: id, Headline: headline, Explanation: b.String(), Who: asks.Operator,
Expires: p.At.Add(askApproveFor), OnExpiry: "the proposal is discarded; nothing changes", Expires: p.At.Add(askApproveFor), OnExpiry: "the proposal is discarded; nothing changes",
About: p.about()} About: p.about(), Details: details(shownAll)}
// Only where a value was masked: an ask whose values all pass the content rule shows the same everywhere.
if w := compose(whole); w != q.Explanation {
q.Whole = w
}
if w := details(wholeAll); w != q.Details {
q.WholeDetails = w
}
options := map[string]int{} options := map[string]int{}
for i, act := range p.actions(id) { for i, act := range p.actions(id) {
binds, _ := asks.ActDigest(boundAct(act)) binds, _ := asks.ActDigest(boundAct(act))
oid := optionID(act.Label) oid := optionID(act.Label)
options[oid] = i options[oid] = i
does := "the settings are set; nothing is pushed yet" does := "the settings are set and sent to " + p.reaches() + " at once, each machine judged"
if p.Clear { if p.Clear {
does = "the layer is removed; nothing is pushed yet" does = "the layer is removed and that is sent to " + p.reaches() + " at once, each machine judged"
} }
if act.Arguments["answer"] == answerDecline { if act.Arguments["answer"] == answerDecline {
does = "nothing changes; the proposal is discarded" does = "nothing changes; the proposal is discarded"
@@ -205,55 +216,141 @@ func (p settingsProposal) ask(id string, machines []string) (asks.Ask, map[strin
return q, options return q, options
} }
// shownMost is the most of a change the phone is shown, in bytes; the rest is read whole with `settings proposals`. // summary is the proposal's first line, what changes where: how many settings, how many stay, and that Approve sends
const shownMost = 1400 // the layer at once by a walk, one machine at a time (novox/hq ADR 0344); the rest of what Approve does
// (onApprove) is the Details'.
func (p settingsProposal) summary() string {
if p.Clear {
if !p.HadLayer {
return "There is no layer to remove; approving changes nothing."
}
return fmt.Sprintf("Removes the %d setting(s) of %s set on %s; approved, the layer is removed and that is sent at once "+
"by a walk to %s, one machine at a time, each judged.", len(leaves(p.Before, "")), p.Module, p.where(), p.reaches())
}
added, changed, removed := settingsChange(p.Before, p.Values)
n := len(added) + len(changed) + len(removed)
unchanged := len(leaves(p.Values, "")) - len(added) - len(changed)
if n == 0 {
return fmt.Sprintf("Nothing changes: the %d setting(s) of %s on %s are as they stand.", unchanged, p.Module, p.where())
}
stay := ""
if unchanged > 0 {
stay = fmt.Sprintf(", and %d stay as they are", unchanged)
}
return fmt.Sprintf("Changes %d setting(s) of %s on %s%s; approved, the layer is set and sent at once by a walk to %s, "+
"one machine at a time, each judged.", n, p.Module, p.where(), stay, p.reaches())
}
// change is what changes, line by line, each value shown under the content rule, and whether every value was // change is what changes: as at most asks.MostPoints points for the glance, as every line whole for the note, and
// shown whole: "+ key: value" added, "~ key: value (was: old)" changed, "- key (was: old)" removed, and the // whether every value was shown exactly: "+ key: value"
// count of keys unchanged. // added, "~ key: value (was: old)" changed, "- key (was: old)" removed. Each value is shown under the content rule
func (p settingsProposal) change(machines []string) (string, bool) { // (sayableValue), or — exact, for a channel that proves who answers — as it is, withheld only when shaped like a
// secret (wholeValue). More changes than points: the first two, and the keys of the rest; a point longer than a
// phone shows at a glance is cut and says so. The note carries every line uncut (the review of #304: Approve sets
// exactly what the operator can read).
func (p settingsProposal) change(machines []string, exact bool) ([]string, []string, bool) {
whole := true whole := true
say := func(v any) string { say := func(v any) string {
s, w := sayableValue(v, machines) s, w := sayableValue(v, machines)
if exact {
s, w = wholeValue(v, machines)
}
whole = whole && w whole = whole && w
return s return s
} }
var lines []string type line struct{ key, text string }
var lines []line
if p.Clear { if p.Clear {
was := leaves(p.Before, "") was := leaves(p.Before, "")
for _, k := range layerKeys(was) { for _, k := range layerKeys(was) {
lines = append(lines, fmt.Sprintf("- %s: %s", k, say(was[k]))) lines = append(lines, line{k, fmt.Sprintf("- %s: %s", k, say(was[k]))})
} }
} else { } else {
added, changed, removed := settingsChange(p.Before, p.Values) added, changed, removed := settingsChange(p.Before, p.Values)
was, now := leaves(p.Before, ""), leaves(p.Values, "") was, now := leaves(p.Before, ""), leaves(p.Values, "")
for _, k := range added { for _, k := range added {
lines = append(lines, fmt.Sprintf("+ %s: %s", k, say(now[k]))) lines = append(lines, line{k, fmt.Sprintf("+ %s: %s", k, say(now[k]))})
} }
for _, k := range changed { for _, k := range changed {
lines = append(lines, fmt.Sprintf("~ %s: %s (was: %s)", k, say(now[k]), say(was[k]))) lines = append(lines, line{k, fmt.Sprintf("~ %s: %s (was: %s)", k, say(now[k]), say(was[k]))})
} }
for _, k := range removed { for _, k := range removed {
lines = append(lines, fmt.Sprintf("- %s (was: %s)", k, say(was[k]))) lines = append(lines, line{k, fmt.Sprintf("- %s (was: %s)", k, say(was[k]))})
}
unchanged := len(now) - len(added) - len(changed)
switch {
case len(lines) == 0 && unchanged > 0:
lines = append(lines, fmt.Sprintf("= nothing changes: the %d key(s) are as they stand", unchanged))
case unchanged > 0:
lines = append(lines, fmt.Sprintf("= %d key(s) unchanged", unchanged))
} }
} }
out := strings.Join(lines, "\n") // **Each line is judged as the router will judge it** (the review of #304, ADR 0347 §1 and §3): a line the
if len(out) > shownMost { // content rule (masked) or the secret rule (whole) would refuse — a value given to a secret's name, say
cut := shownMost // "+ password: hunter2" — is said with its key still named and its value withheld, never left for the
for cut > 0 && out[cut] != '\n' { // router to withhold whole and unnamed. A line refused even so is the propose's refusal (judgeLines).
cut-- for i, l := range lines {
if lineLeaves(l.text, exact, machines) {
continue
} }
out = out[:cut] + fmt.Sprintf("\n… NOT SHOWN IN FULL here: %d more bytes", len(strings.Join(lines, "\n"))-cut) lines[i].text = l.text[:2] + l.key + " (value withheld)"
whole = false whole = false
} }
return out, whole all := make([]string, len(lines))
for i, l := range lines {
all[i] = l.text
}
// What the note holds of a value: whole where the glance is the whole one (shown only where the sender is
// proven, with the whole details beside it), under the content rule elsewhere — so the marker says no more than
// the note it points to (the review of ADR 0347).
inNote := "in the note"
if exact {
inNote = "whole in the note"
}
var out []string
for i, l := range lines {
if len(lines) > asks.MostPoints && i == asks.MostPoints-1 {
var rest []string
for _, r := range lines[i:] {
rest = append(rest, r.key)
}
out = append(out, fmt.Sprintf("and %d more, each %s: %s", len(rest), inNote, strings.Join(rest, ", ")))
break
}
out = append(out, l.text)
}
// A point too long for a glance is cut, and says so: its value is the note's.
cut := "… (cut: " + inNote + ")"
for i, l := range out {
if r := []rune(l); len(r) > asks.PointLength {
out[i] = string(r[:asks.PointLength-len([]rune(cut))]) + cut
}
}
return out, all, whole
}
// lineLeaves says whether a line may leave the mesh as the router judges it: under the content rule for the masked
// form, the secret rule alone for the whole one.
func lineLeaves(line string, exact bool, machines []string) bool {
if exact {
_, ok := outward.Secret(line, machines...)
return ok
}
_, ok := outward.Check(line, machines...)
return ok
}
// judgeLines is the first line of an ask's details the router would refuse, or withhold, and why: masked under
// the content rule line by line, whole under the secret rule. Empty when every line may leave.
func judgeLines(details string, exact bool, machines []string) (string, string) {
for _, line := range strings.Split(details, "\n") {
if strings.TrimSpace(line) == "" {
continue
}
if exact {
if refusal, ok := outward.Secret(line, machines...); !ok {
return line, refusal.String()
}
continue
}
if refusal, ok := outward.Check(line, machines...); !ok {
return line, refusal.String()
}
}
return "", ""
} }
func layerKeys(m map[string]any) []string { func layerKeys(m map[string]any) []string {
@@ -300,6 +397,27 @@ func sayableValue(v any, machines []string) (string, bool) {
return out, out == text return out, out == text
} }
// wholeValue is a value as a channel that proves who answers is shown it (asks.Ask.Whole), and whether it was
// shown whole: as it is, unless it is shaped like a secret (outward.Secret), which is withheld whole — never in
// part, so no half of a key reaches the phone.
func wholeValue(v any, machines []string) (string, bool) {
text, ok := v.(string)
if !ok {
raw, err := json.Marshal(v)
if err != nil {
return markWithheld, false
}
text = string(raw)
}
if text == "" {
return `""`, true
}
if _, ok := outward.Secret(text, machines...); !ok {
return markWithheld, false
}
return text, true
}
// sayable is a text with what may not leave the mesh replaced in place by a marker; what the markers cannot make // sayable is a text with what may not leave the mesh replaced in place by a marker; what the markers cannot make
// pass is withheld whole. // pass is withheld whole.
func sayable(text string, machines []string) string { func sayable(text string, machines []string) string {
@@ -432,6 +550,23 @@ func (pr proposer) propose(ctx context.Context, in proposeInput) (string, error)
return refuse("the ask's words would carry %s, which may not leave the mesh, and the change could not be "+ return refuse("the ask's words would carry %s, which may not leave the mesh, and the change could not be "+
"shown without it; %s", refusal, atTheTerminalInstead(in)) "shown without it; %s", refusal, atTheTerminalInstead(in))
} }
// The whole words are shown only where the sender is proven, and never a secret's shape: wholeValue withholds
// one, and the router would refuse the ask if one were left, so it is refused here first, in words.
if refusal, ok := outward.Secret(q.Whole, machines...); !ok {
return refuse("the change shown whole would carry %s, which may not leave the mesh on any channel; %s",
refusal, atTheTerminalInstead(in))
}
// So are the details (ADR 0347 §1, §3): every line the router would withhold from the note, or refuse the ask
// for, is refused here, in the controller's own words, naming the line's key — Approve never sets a value the
// operator was shown nothing of.
if line, why := judgeLines(q.Details, false, machines); line != "" {
return refuse("the change's line %q would carry %s, which the operator's note may not show; %s",
keyOfLine(line), why, atTheTerminalInstead(in))
}
if line, why := judgeLines(q.WholeDetails, true, machines); line != "" {
return refuse("the change's line %q shown whole would carry %s, which may not leave the mesh on any channel; %s",
keyOfLine(line), why, atTheTerminalInstead(in))
}
// Fail closed, before anything is kept: no router, no grant, no channel means no ask. // Fail closed, before anything is kept: no router, no grant, no channel means no ask.
if pr.routerHere != nil { if pr.routerHere != nil {
here, err := pr.routerHere(ctx) here, err := pr.routerHere(ctx)
@@ -519,15 +654,29 @@ func (pr proposer) propose(ctx context.Context, in proposeInput) (string, error)
fingerprint(p.Digest)) fingerprint(p.Digest))
fmt.Fprintf(&b, " until %s nothing changes: the layer is set only on Approve, and discarded on Decline or at expiry\n", fmt.Fprintf(&b, " until %s nothing changes: the layer is set only on Approve, and discarded on Decline or at expiry\n",
q.Expires.Local().Format("2006-01-02 15:04")) q.Expires.Local().Format("2006-01-02 15:04"))
fmt.Fprintf(&b, " `settings proposals %s` shows it whole; once approved, `push %s` sends it", id, pushWord(p.Node)) fmt.Fprintf(&b, " `settings proposals %s` shows it whole; once approved, a walk sends it to %s at once, one machine at "+
"a time, each judged, and its end is told as every walk's is (novox/hq ADR 0344)", id, p.reaches())
return b.String(), nil return b.String(), nil
} }
func pushWord(node string) string { // reaches is the machines an approved layer is sent to, in words: the one it names, or every machine running the
if node == "" { // module.
return "--behind" func (p settingsProposal) reaches() string {
if p.Node == "" {
return "every machine running " + p.Module
} }
return node return p.Node
}
// onApprove is what the ask says Approve does (novox/hq ADR 0344 §4): the one Approve sets the layer and sends it.
func (p settingsProposal) onApprove() string {
what := "the layer is set"
if p.Clear {
what = "the layer is removed"
}
return fmt.Sprintf("Approved, %s and sent at once to %s, one machine at a time, each judged, every build there "+
"kept; its end is told as every walk's is. A module that reads its settings only when it starts takes them at "+
"its next start.", what, p.reaches())
} }
func newProposalID() string { func newProposalID() string {
@@ -758,6 +907,21 @@ func setLayerIn(open *stores) setOnWarrant {
return "", fmt.Errorf("the layer of %s on %s changed since the operator was shown the change: it is not set; "+ return "", fmt.Errorf("the layer of %s on %s changed since the operator was shown the change: it is not set; "+
"propose it again", p.Module, p.where()) "propose it again", p.Module, p.where())
} }
// A proposal that carries config values, raised before the module's config moved, is not applied after it: its
// values in the layer would turn ReadsConfig off and undo the move (issue 830, novox/hq ADR 0348). A proposal
// of placement only is set as before: forwardPropose sends it down this path on purpose.
carried := p.Values
if p.Clear {
carried = before
}
if config, _ := catalogue.SplitPlacement(carried); len(config) > 0 {
if reads, err := inv.ReadsConfig(ctx, p.Module); err != nil {
return "", err
} else if reads {
return "", fmt.Errorf("the config of %s moved to mesh-config since the operator was shown the change: it is "+
"not set; propose it again (it then goes to mesh-config)", p.Module)
}
}
var changed string var changed string
if p.Clear { if p.Clear {
if err := inv.ClearSettingsBy(ctx, p.Node, p.Module, setBy); err != nil { if err := inv.ClearSettingsBy(ctx, p.Node, p.Module, setBy); err != nil {
@@ -788,6 +952,40 @@ func setLayerIn(open *stores) setOnWarrant {
changed = "nothing changed" changed = "nothing changed"
} }
} }
return changed + " (ask " + askID + ")", nil // **And sends it** (novox/hq ADR 0344): the layer the operator approved is walked to the machines it reaches
// at once. A walk that cannot be opened is said as the act's failure — the layer is set and nothing sends it.
walked, err := openSettingsWalk(ctx, open, inventory.PlanSettings{Module: p.Module, Node: p.Node, Ask: askID,
Change: changed})
if err != nil {
return "", fmt.Errorf("%s (ask %s) is set, and the walk that sends it could not be opened: %v — `push %s` sends it",
changed, askID, err, pushTo(p.Node))
}
return changed + " (ask " + askID + "); " + walked, nil
} }
} }
// pushTo is the push that sends a layer by hand when its walk could not: the machine, or every one behind.
func pushTo(node string) string {
if node == "" {
return "--behind"
}
return node
}
// keyOfLine is what a refusal names of a line: the key of a change's line ("+ key: …"), or the line's first words.
func keyOfLine(line string) string {
if len(line) > 2 && strings.ContainsRune("+~-=", rune(line[0])) && line[1] == ' ' {
rest := line[2:]
if i := strings.IndexAny(rest, ": "); i > 0 {
rest = rest[:i]
}
if _, ok := outward.Secret(rest); !ok {
return "whose key is shaped like a secret"
}
return rest
}
if r := []rune(line); len(r) > 40 {
return string(r[:40]) + "…"
}
return line
}
+247 -14
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"io" "io"
"os" "os"
"slices" "slices"
@@ -116,24 +117,43 @@ func TestAProposalAsksAtTheLevelApproveWithTheExactChange(t *testing.T) {
if q.Options[0].Binds == q.Options[1].Binds { if q.Options[0].Binds == q.Options[1].Binds {
t.Error("Approve and Decline bind the same act") t.Error("Approve and Decline bind the same act")
} }
for _, line := range []string{ // The message reads at a glance (issues 383 and 639): a summary, then one point per key — and nothing else:
"Set the settings of mounts on shanks to these values?", // who proposed it, the fingerprint and the how-to are the Details, which the router notes after the answers.
"Proposed by g14/claude-code, through the mesh-controller seat, at " + r.now.Local().Format("15:04 on 2 Jan") + ".", summary := "Changes 3 setting(s) of mounts on shanks; approved, the layer is set and sent at once by a walk to shanks, one machine at a time, each judged."
"+ sources: recalbox=‹address›@‹path›:ro", if q.Explanation != summary+"\n+ sources: recalbox=‹address›@‹path›:ro\n~ shares: library=‹path› (was: none)\n- old (was: x)" {
"~ shares: library=‹path› (was: none)", t.Errorf("the explanation:\n%s", q.Explanation)
"- old (was: x)", }
"Fingerprint " + fingerprint(layerDigest(mountsSources)) + ".", if err := asks.Glance(q.Explanation); err != nil {
"read it whole, with this fingerprint", t.Error(err)
} {
if !strings.Contains(q.Explanation, line) {
t.Errorf("the explanation lacks %q:\n%s", line, q.Explanation)
}
} }
for _, leak := range []string{"nas.lan", "/mnt/recalbox", "/mnt/library", "smb://"} { for _, leak := range []string{"nas.lan", "/mnt/recalbox", "/mnt/library", "smb://"} {
if strings.Contains(q.Explanation, leak) { if strings.Contains(q.Explanation, leak) {
t.Errorf("the explanation carries %q, which may not leave the mesh", leak) t.Errorf("the explanation carries %q, which may not leave the mesh", leak)
} }
} }
// Where the sender is proven, the values whole: the mount point and the share are what is approved.
if q.Whole != summary+"\n+ sources: recalbox=smb://nas.lan/recalbox@/mnt/recalbox:ro\n~ shares: library=/mnt/library (was: none)\n- old (was: x)" ||
asks.Glance(q.Whole) != nil {
t.Errorf("the whole words:\n%s", q.Whole)
}
for _, line := range []string{
"Proposed by g14/claude-code, through the mesh-controller seat at " + r.now.Local().Format("15:04 on 2 Jan") +
". Fingerprint " + fingerprint(layerDigest(mountsSources)) + ".",
"Where who answers is not proven, its values are shown as ‹address›, ‹path› or ‹withheld›.",
"Read it whole, with this fingerprint: settings proposals " + kept(r).ID + " at the controller's terminal",
} {
if !strings.Contains(q.Details, line) {
t.Errorf("Details lack %q:\n%s", line, q.Details)
}
}
if refusal, ok := outward.Check(q.Details, "anchor", "laptop", "shanks"); !ok {
t.Errorf("the router would withhold a line of the details: %s", refusal)
}
for _, howTo := range []string{"ingerprint", "settings proposals", "settings verb", "not proven", "Proposed by"} {
if strings.Contains(q.Explanation, howTo) || strings.Contains(q.Whole, howTo) {
t.Errorf("the message carries the how-to %q", howTo)
}
}
all := []string{q.Headline, q.Explanation, q.OnExpiry} all := []string{q.Headline, q.Explanation, q.OnExpiry}
for _, o := range q.Options { for _, o := range q.Options {
all = append(all, o.Label, o.Does) all = append(all, o.Label, o.Does)
@@ -141,6 +161,9 @@ func TestAProposalAsksAtTheLevelApproveWithTheExactChange(t *testing.T) {
if refusal, ok := outward.Check(strings.Join(all, "\n"), "anchor", "laptop", "shanks"); !ok { if refusal, ok := outward.Check(strings.Join(all, "\n"), "anchor", "laptop", "shanks"); !ok {
t.Errorf("the router would refuse the ask: %s", refusal) t.Errorf("the router would refuse the ask: %s", refusal)
} }
if refusal, ok := outward.Secret(q.Whole, "anchor", "laptop", "shanks"); !ok {
t.Errorf("the router would refuse the whole words: %s", refusal)
}
// Kept as the controller's own ask, about no condition, with the proposal whole and its digests. // Kept as the controller's own ask, about no condition, with the proposal whole and its digests.
kept := r.theProposal(t) kept := r.theProposal(t)
p := kept.Proposal p := kept.Proposal
@@ -173,11 +196,98 @@ func TestAMeshWideLayerIsProposed(t *testing.T) {
} }
q := r.askSent(t) q := r.askSent(t)
if q.Headline != "Set notes on the whole mesh?" || q.About != "settings.notes.mesh" || if q.Headline != "Set notes on the whole mesh?" || q.About != "settings.notes.mesh" ||
!strings.Contains(q.Explanation, "Set the settings of notes on the whole mesh to these values?") { q.Explanation != "Changes 1 setting(s) of notes on the whole mesh; approved, the layer is set and sent at once by a walk to every machine running notes, one machine at a time, each judged.\n+ x: 1" ||
q.Whole != "" {
t.Errorf("%+v", q) t.Errorf("%+v", q)
} }
} }
// kept is the one proposal the rig keeps.
func kept(r *proposerRig) asked {
for _, a := range r.store {
if a.Proposal != nil {
return a
}
}
return asked{}
}
// The switch between the masked and the whole change (issue 383): the same change, under the content rule and
// exact, differs in the masked values alone; a value shaped like a secret is withheld in both, whole, with its
// key still named; and a change no value of which is masked carries no whole words of its own.
func TestAProposalShowsItsValuesWholeOnlyWhereTheSenderIsProvenAndNeverASecret(t *testing.T) {
machines := []string{"anchor", "laptop", "shanks"}
r := newProposerRig(t)
values := map[string]any{"shares": "media=/storage/media", "sources": "recalbox=smb://nas.lan/recalbox@/mnt/recalbox", "font": "Inter 13"}
if _, err := r.pr.propose(context.Background(), proposeInput{module: "mounts", node: "shanks", values: values}); err != nil {
t.Fatal(err)
}
q := r.askSent(t)
masked, _, _ := kept(r).Proposal.change(machines, false)
whole, _, _ := kept(r).Proposal.change(machines, true)
if !strings.HasSuffix(q.Explanation, "\n"+strings.Join(masked, "\n")) || !strings.HasSuffix(q.Whole, "\n"+strings.Join(whole, "\n")) {
t.Fatalf("the ask does not carry the change masked and whole:\n%s\n--\n%s", q.Explanation, q.Whole)
}
if want := []string{"+ font: Inter 13", "+ shares: media=‹path›", "+ sources: recalbox=‹address›@‹path›"}; strings.Join(masked, "\n") != strings.Join(want, "\n") {
t.Errorf("masked: %q", masked)
}
if want := []string{"+ font: Inter 13", "+ shares: media=/storage/media", "+ sources: recalbox=smb://nas.lan/recalbox@/mnt/recalbox"}; strings.Join(whole, "\n") != strings.Join(want, "\n") {
t.Errorf("whole: %q", whole)
}
if _, ok := outward.Secret(q.Whole, "shanks"); !ok {
t.Error("the router would refuse the whole words")
}
// Mutation: the masked and the whole change differ in exactly the lines whose value was masked.
differ := 0
for i := range masked {
if masked[i] != whole[i] {
differ++
if !strings.Contains(masked[i], "‹") || strings.Contains(whole[i], "‹") {
t.Errorf("the lines differ otherwise than by the mask:\n%s\n%s", masked[i], whole[i])
}
}
}
if differ != 2 {
t.Errorf("%d lines differ, and the mask covered 2", differ)
}
// A value shaped like a secret is withheld in both, whole, with its key still named.
r1 := newProposerRig(t)
secrets := map[string]any{"github": "ghp_abcdefghijklmnopqrstuvwxyz0123456789", "cert": "-----BEGIN CERTIFICATE-----"}
if _, err := r1.pr.propose(context.Background(), proposeInput{module: "mounts", node: "shanks", values: secrets}); err != nil {
t.Fatal(err)
}
q1 := r1.askSent(t)
for _, line := range []string{"+ cert: ‹withheld›", "+ github: ‹withheld›"} {
if !strings.Contains(q1.Explanation, line) {
t.Errorf("lacks %q:\n%s", line, q1.Explanation)
}
}
for _, secret := range []string{"ghp_", "BEGIN CERTIFICATE"} {
if strings.Contains(q1.Explanation, secret) || strings.Contains(q1.Whole, secret) || strings.Contains(q1.Details, secret) {
t.Errorf("the secret %q reaches the phone", secret)
}
}
// More keys than points (issue 639): the first two, then the keys of the rest, read whole at the terminal.
r3 := newProposerRig(t)
five := map[string]any{"a": "1", "b": "2", "c": "3", "d": "4", "e": "5"}
if _, err := r3.pr.propose(context.Background(), proposeInput{module: "notes", node: "laptop", values: five}); err != nil {
t.Fatal(err)
}
if q3 := r3.askSent(t); !strings.HasSuffix(q3.Explanation, "\n+ a: 1\n+ b: 2\nand 3 more, each in the note: c, d, e") || asks.Glance(q3.Explanation) != nil ||
!strings.Contains(q3.Details, "settings proposals") {
t.Errorf("five keys:\n%s", q3.Explanation)
}
// No value masked: the message is the same everywhere, and the ask says no whole words.
r2 := newProposerRig(t)
if _, err := r2.pr.propose(context.Background(), proposeInput{module: "dunst", node: "laptop", values: map[string]any{"font-size": 13, "width": 500}}); err != nil {
t.Fatal(err)
}
if q2 := r2.askSent(t); q2.Whole != "" || !strings.HasSuffix(q2.Explanation, "\n+ font-size: 13\n+ width: 500") ||
strings.Contains(q2.Details, "not proven") {
t.Errorf("%+v", q2)
}
}
// A proposal expired unanswered is kept so by the reconciling, and a warrant for it afterwards sets nothing. // A proposal expired unanswered is kept so by the reconciling, and a warrant for it afterwards sets nothing.
func TestAnExpiredProposalIsKeptExpired(t *testing.T) { func TestAnExpiredProposalIsKeptExpired(t *testing.T) {
r := newAskerRig(t) r := newAskerRig(t)
@@ -205,7 +315,8 @@ func TestAClearIsProposedAndShowsWhatItRemoves(t *testing.T) {
} }
q := r.askSent(t) q := r.askSent(t)
if q.Headline != "Clear notes on laptop?" || !strings.Contains(q.Explanation, "- places.data.owner: 1001:1001") || if q.Headline != "Clear notes on laptop?" || !strings.Contains(q.Explanation, "- places.data.owner: 1001:1001") ||
!strings.Contains(q.Explanation, "- places.data.path: ‹path›") { !strings.Contains(q.Explanation, "- places.data.path: ‹path›") || !strings.Contains(q.Whole, "- places.data.path: /srv/notes") ||
!strings.HasPrefix(q.Explanation, "Removes the 2 setting(s) of notes set on laptop; approved, the layer is removed and that is sent at once by a walk to laptop, one machine at a time") {
t.Errorf("%+v", q) t.Errorf("%+v", q)
} }
if p := r.theProposal(t).Proposal; !p.Clear || p.Digest != layerDigest(r.before) || len(r.judged) != 0 { if p := r.theProposal(t).Proposal; !p.Clear || p.Digest != layerDigest(r.before) || len(r.judged) != 0 {
@@ -744,3 +855,125 @@ func captureStdout(t *testing.T, f func()) string {
_ = w.Close() _ = w.Close()
return <-done return <-done
} }
// The review of #304: Approve sets exactly what the operator can read. A proposal of five keys, one value 300
// characters long, shows two keys at a glance, says what it cut, and carries every key with its whole value in its
// Details for a channel that proves who answers (WholeDetails), and every key masked for one that does not.
func TestEveryKeyAndItsWholeValueReachTheOperator(t *testing.T) {
var long strings.Builder
long.WriteString("library=/storage")
for i := 0; long.Len() < 300; i++ {
long.WriteString(fmt.Sprintf("/album-%02d", i))
}
value := long.String()[:300]
values := map[string]any{"a": "1", "b": "2", "c": "3", "d": "4", "shares": value}
r := newProposerRig(t)
if _, err := r.pr.propose(context.Background(), proposeInput{module: "mounts", node: "shanks", values: values}); err != nil {
t.Fatal(err)
}
q := r.askSent(t)
if err := q.Check(r.now); err != nil {
t.Fatal(err)
}
// What the operator reads on a proving channel: the whole words, then the note's whole details.
visible := q.Whole + "\n" + q.WholeDetails
if q.Whole == "" {
visible = q.Explanation + "\n" + q.WholeDetails
}
for k, v := range values {
if !strings.Contains(visible, "+ "+k+": "+v.(string)) {
t.Errorf("the operator never reads %s whole:\n%s", k, visible)
}
}
// The masked glance points to a note that holds the values masked, and says no more; the whole glance, shown only
// beside the whole details, says they are whole there.
if asks.Glance(q.Explanation) != nil || !strings.HasSuffix(q.Explanation, "\nand 3 more, each in the note: c, d, shares") ||
strings.Contains(q.Explanation, "whole in the note") || !strings.HasSuffix(q.Whole, "\nand 3 more, each whole in the note: c, d, shares") {
t.Errorf("the glance:\n%s", q.Explanation)
}
for k := range values {
if !strings.Contains(q.Details, "+ "+k+": ") {
t.Errorf("the masked details lack %s:\n%s", k, q.Details)
}
}
if strings.Contains(q.Details, "/storage") {
t.Errorf("the masked details show a path:\n%s", q.Details)
}
if refusal, ok := outward.Check(q.Details, "shanks"); !ok {
t.Errorf("the router would withhold the masked details: %s", refusal)
}
if _, ok := outward.Secret(q.WholeDetails, "shanks"); !ok {
t.Error("the router would refuse the whole details")
}
// A value too long for a glance is cut there, and says so.
r2 := newProposerRig(t)
if _, err := r2.pr.propose(context.Background(), proposeInput{module: "mounts", node: "shanks", values: map[string]any{"shares": value}}); err != nil {
t.Fatal(err)
}
q2 := r2.askSent(t)
// The cut point is 160 characters, its marker inside the bound.
if last := q2.Whole[strings.LastIndex(q2.Whole, "\n")+1:]; len([]rune(last)) != asks.PointLength {
t.Errorf("the cut point is %d characters, not %d: %q", len([]rune(last)), asks.PointLength, last)
}
if !strings.HasSuffix(q2.Whole, "… (cut: whole in the note)") || !strings.HasSuffix(q2.Explanation, "‹path›") && !strings.HasSuffix(q2.Explanation, "… (cut: in the note)") || !strings.Contains(q2.WholeDetails, "+ shares: "+value) || asks.Glance(q2.Whole) != nil {
t.Errorf("the cut is not said, or the note lacks the whole:\n%s\n--\n%s", q2.Whole, q2.WholeDetails)
}
}
// The review of #304: a value given to a secret's name, past the second key, is said with its key named and its
// value withheld, in the note on every channel; the router has nothing left to withhold unnamed or refuse. A line
// the router would refuse even so is refused at propose, in the controller's words.
func TestASecretNamedKeyIsNamedAndItsValueWithheld(t *testing.T) {
r := newProposerRig(t)
values := map[string]any{"a": "1", "b": "2", "c": "3", "password": "hunter2"}
if _, err := r.pr.propose(context.Background(), proposeInput{module: "notes", node: "laptop", values: values}); err != nil {
t.Fatal(err)
}
q := r.askSent(t)
for _, form := range []string{q.Details, q.WholeDetails, q.Explanation, q.Whole} {
if strings.Contains(form, "hunter2") {
t.Errorf("the value reaches the phone:\n%s", form)
}
}
if !strings.Contains(q.Details, "+ password (value withheld)") {
t.Errorf("the key is not named with its value withheld:\n%s", q.Details)
}
if q.WholeDetails != "" && !strings.Contains(q.WholeDetails, "+ password (value withheld)") {
t.Errorf("the whole details do not name the key:\n%s", q.WholeDetails)
}
if line, why := judgeLines(q.Details, false, nil); line != "" {
t.Errorf("the router would withhold %q: %s", line, why)
}
if line, why := judgeLines(q.WholeDetails, true, nil); line != "" {
t.Errorf("the router would refuse %q: %s", line, why)
}
if !strings.HasSuffix(q.Explanation, "\nand 2 more, each in the note: c, password") {
t.Errorf("the glance:\n%s", q.Explanation)
}
// A key shaped like a secret cannot be named on any channel: refused at propose, in the controller's words, and
// the key itself not echoed.
r2 := newProposerRig(t)
secretKey := "ghp_abcdefghijklmnopqrstuvwxyz0123456789"
_, err := r2.pr.propose(context.Background(), proposeInput{module: "notes", node: "laptop",
values: map[string]any{"a": "1", "b": "2", secretKey: "3"}})
if err == nil || !strings.Contains(err.Error(), "Nothing was proposed") || strings.Contains(err.Error(), secretKey) {
t.Errorf("a secret-shaped key: %v", err)
}
if len(r2.sent) != 0 {
t.Errorf("the refused proposal was asked")
}
}
// keyOfLine names a change's key, and never one shaped like a secret.
func TestARefusalNamesTheKeyAndNeverASecret(t *testing.T) {
for line, want := range map[string]string{
"+ password (value withheld)": "password",
"~ shares: ‹path› (was: none)": "shares",
"+ ghp_abcdefghijklmnopqrstuvwxyz0123456789: 1": "whose key is shaped like a secret",
"Proposed by g14 at 15:04.": "Proposed by g14 at 15:04.",
} {
if got := keyOfLine(line); got != want {
t.Errorf("%q: %q, want %q", line, got, want)
}
}
}
+100
View File
@@ -0,0 +1,100 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// What a machine's provisioner client said of a module's files (novox/hq ADR 0362, ADR 0349 §15, design 57 P6,
// issue 672), read where a walk's end and a gate count a machine's send applied: a module the send handed to the
// provisioner is applied there only once the client reported `prov.applied` on that build's templates as well.
// provAppliedOn is the client's newest report of a module on a machine, from the stream that keeps it
// (broker.ProvAppliedStream); false when it has said nothing of it. A variable so a test can say the client's words.
var provAppliedOn = func(ctx context.Context, node, module string) (inventory.ProvReport, bool, error) {
var raw []byte
var said bool
read := func(js nats.JetStreamContext) error {
var err error
raw, said, err = broker.LastProvApplied(js, node, module)
return err
}
var err error
if d := doctorFrom; d != nil && d.js != nil {
err = read(d.js.Context())
} else {
err = onTheBus(func(conn *nats.Conn) error {
js, err := conn.JetStream(nats.Context(ctx))
if err != nil {
return err
}
return read(js)
})
}
if err != nil || !said {
return inventory.ProvReport{}, false, err
}
var r inventory.ProvReport
if err := json.Unmarshal(raw, &r); err != nil {
return inventory.ProvReport{}, false, fmt.Errorf("what %s's provisioner client said of %s is not its report: %w",
node, module, err)
}
return r, true, nil
}
// provLookup answers the clients' reports for one reading; a report that cannot be read is not counted, which
// leaves the machine waiting, then silent, never applied — and is said.
func provLookup(ctx context.Context) inventory.ProvLookup {
return func(node, module string) (inventory.ProvReport, bool) {
r, said, err := provAppliedOn(ctx, node, module)
if err != nil {
fmt.Fprintf(os.Stderr, "what %s's provisioner client said of %s could not be read: %v\n", node, module, err)
return inventory.ProvReport{}, false
}
return r, said
}
}
// provisioningFailed is every module whose files a machine's provisioner client last said it failed to place, from
// the stream that keeps their reports.
func provisioningFailed(js nats.JetStreamContext) ([]inventory.ProvReport, error) {
all, err := broker.ProvAppliedReports(js)
if err != nil {
return nil, err
}
var out []inventory.ProvReport
for subject, raw := range all {
var r inventory.ProvReport
if err := json.Unmarshal(raw, &r); err != nil {
return nil, fmt.Errorf("what was said on %s is not a provisioner client's report: %w", subject, err)
}
if r.Status == inventory.ProvFailed {
out = append(out, r)
}
}
return out, nil
}
// provisionedOutcome is a machine's report of one send as a delivery's end counts it (novox/hq ADR 0362 §1-5, issue
// 840): the engine's own report, combined with its provisioner client's for a send that handed modules to it. The
// outcome of a client's failure begins inventory.OutcomeProvisioningFailed; a report not yet applied on both sides
// is none (false). A send handing nothing is the engine's report unchanged. onSend says the report is about the send.
func provisionedOutcome(node string, sent inventory.SentDeclaration, r inventory.Reported, onSend bool,
prov inventory.ProvLookup) (string, bool) {
if len(sent.Provisioned) == 0 {
return r.Outcome, onSend
}
engine := inventory.AppliedReport{Outcome: r.Outcome}
if r.At != nil {
engine.At = *r.At
}
got, ok := inventory.ProvisionedSend(node, sent, engine, onSend, prov)
return got.Outcome, ok
}
+114
View File
@@ -0,0 +1,114 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// The controller hands a moved module to the provisioner, and counts it applied only once its files are placed
// (design 57 P6, issue 672; novox/hq ADR 0349 §4, §15, ADR 0362).
// A declaration that hands files says which, and whose client fills them; one that hands none is byte for byte
// what it was.
func TestADeclarationNamesTheFilledFilesOnlyWhenItHandsAny(t *testing.T) {
plain := sendable{Resources: []map[string]any{{"id": "notify.configuration", "type": "file", "path": "/x",
"content": "a"}}, Sequence: 4}
before, err := plain.Body()
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(before), "filled_files") || strings.Contains(string(before), "provisioner_account") {
t.Fatalf("a declaration handing nothing names filled files: %s", before)
}
handing := plain
handing.Filled, handing.Provisioner = []string{"/x"}, "mesh-provisioner-client"
body, err := handing.Body()
if err != nil {
t.Fatal(err)
}
var envelope struct {
Filled []string `json:"filled_files"`
Provisioner string `json:"provisioner_account"`
}
if err := json.Unmarshal(body, &envelope); err != nil || len(envelope.Filled) != 1 || envelope.Filled[0] != "/x" ||
envelope.Provisioner != "mesh-provisioner-client" {
t.Fatalf("the handing declaration says %s", body)
}
}
func TestTheProvisionersAccountIsTheClientsProcessUser(t *testing.T) {
client := catalogue.Manifest{Module: "mesh-provisioner-client",
OwnSecrets: catalogue.OwnSecrets{"provisioner-broker": {Path: "/var/lib/x/broker"}},
Claims: []catalogue.Claim{{Name: "node-provisioner", Scope: "node"}},
Resources: []map[string]any{{"id": "account", "type": "user", "name": "mesh-provisioner-client"},
{"id": "client", "type": "process", "name": "mesh-provisioner-client", "user": "mesh-provisioner-client"}}}
if got := provisionerAccount(catalogue.Resolution{Modules: []catalogue.Manifest{client}}); got != "mesh-provisioner-client" {
t.Fatalf("the client's account is %q", got)
}
if got := provisionerAccount(catalogue.Resolution{}); got != "" {
t.Fatalf("a machine without a client names %q", got)
}
}
// The first-machine gate counts a handed module healthy only once its node's prov.applied for the new build has
// come (ADR 0349 §15, kept by ADR 0362 for a walk finishing at its gate); a failure there fails it.
func TestTheGateWaitsForTheProvisionersReport(t *testing.T) {
now := time.Date(2026, 10, 14, 23, 0, 0, 0, time.UTC)
since := now.Add(-time.Minute)
reported := now.Add(-50 * time.Second)
digest := "sha256:" + strings.Repeat("d", 64)
facts := func(said map[string]inventory.ProvReport) gateFacts {
return gateFacts{now: now,
reports: map[string]inventory.Reported{"g14": {Node: "g14", Outcome: inventory.OutcomeApplied,
At: &reported, Current: true}},
sent: map[string]inventory.SentDeclaration{"g14": {Digest: "sent", At: since,
Provisioned: map[string]string{"dunst": digest}}},
provisioned: func(node, module string) (inventory.ProvReport, bool) {
r, ok := said[node+"."+module]
return r, ok
}}
}
m := catalogue.Manifest{Module: "dunst"}
if h, why := judgeHealth("dunst", "", m, "g14", since, facts(nil)); h != healthNotYet ||
!strings.Contains(why, "has not reported dunst's files placed") {
t.Fatalf("judged without the provisioner's report: %v %q", h, why)
}
failed := map[string]inventory.ProvReport{"g14.dunst": {Status: inventory.ProvFailed, Templates: digest,
Reason: "a value is not there", At: now.Add(-10 * time.Second)}}
if h, why := judgeHealth("dunst", "", m, "g14", since, facts(failed)); h != healthBroken ||
!strings.Contains(why, "a value is not there") {
t.Fatalf("a failed placing is not the build's failure there: %v %q", h, why)
}
placed := map[string]inventory.ProvReport{"g14.dunst": {Status: inventory.ProvApplied, Templates: digest,
At: now.Add(-10 * time.Second)}}
if h, why := judgeHealth("dunst", "", m, "g14", since, facts(placed)); strings.Contains(why, "provisioner") {
t.Fatalf("placed, the module is still held on the provisioner: %v %q", h, why)
}
// A module the send did not hand is judged as before.
if h, why := judgeHealth("picom", "", catalogue.Manifest{Module: "picom"}, "g14", since, facts(nil)); strings.Contains(why, "provisioner") {
t.Fatalf("a module not handed waits on the provisioner: %v %q", h, why)
}
}
// A machine whose provisioner client failed to place a module's files has a failed apply, said as apply-failing
// once the failure outlasts an apply interval, with the client's reason (ADR 0362 §3).
func TestAFailedPlacingRaisesApplyFailing(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
f := calm(now)
f.provFailed = []inventory.ProvReport{{Node: "anchor", Module: "dunst", Status: inventory.ProvFailed,
Reason: "the templates cannot be fetched", At: now.Add(-time.Minute)}}
if got := watchApplyFailing(f); len(got) != 0 {
t.Fatalf("raised within an apply interval: %+v", got)
}
f.provFailed[0].At = now.Add(-10 * time.Minute)
got := watchApplyFailing(f)
if len(got) != 1 || got[0].Key() != "machine.anchor.apply-failing" ||
!strings.Contains(got[0].Summary, "dunst") || !strings.Contains(got[0].Said, "the templates cannot be fetched") {
t.Fatalf("a failed placing is not raised as anchor's apply-failing: %+v", got)
}
}
@@ -0,0 +1,92 @@
package main
import (
"slices"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
func aProvisioner() catalogue.Manifest {
return catalogue.Manifest{Module: "mesh-provisioner", Version: "1",
DefinesSeats: []catalogue.SeatDeclaration{{Name: broker.NodeProvisionerSeat, Scope: catalogue.ScopeNode,
Serves: []catalogue.Verb{{Name: "get"}}}},
Claims: []catalogue.Claim{{Name: broker.NodeProvisionerSeat, Scope: catalogue.ScopeNode}},
OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/run/mesh-provisioner/broker"},
broker.ProvisionerSecret: {Path: "/run/mesh-provisioner/client"}},
}
}
// A machine's provisioner client is issued an account of its own, under its own name, sealed to the machine as the
// module's own secret its service reads (novox/hq ADR 0349 clause 11, design 57 P3); until it is, a push refuses
// to invent one, as it does for a module's `broker`.
func TestTheProvisionerClientsAccountIsIssuedForItsServiceAndNeverInvented(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
m := aProvisioner()
register(t, open, m)
if _, err := assign(ctx, open, "laptop", m.Module); err != nil {
t.Fatal(err)
}
if err := busCredentialIssued(ctx, inv, "laptop", m.Module, broker.ProvisionerSecret); err == nil {
t.Fatal("a push would seal a placeholder for the client's account before it was issued")
}
if err := issueProvisionerClient(ctx, inv, m, "laptop", broker.Broker{Fingerprint: "ab"}, "10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
if _, minted, err := inv.BusUserHash(ctx, "provisioner.laptop"); err != nil || !minted {
t.Fatalf("the client's account is not minted: %v %v", minted, err)
}
states, err := inv.SecretsOf(ctx, "laptop", m.Module)
if err != nil {
t.Fatal(err)
}
if !slices.ContainsFunc(states, func(s inventory.SecretState) bool { return s.Name == broker.ProvisionerSecret }) {
t.Fatalf("the client's account is not sealed to the machine for its service: %+v", states)
}
if err := busCredentialIssued(ctx, inv, "laptop", m.Module, broker.ProvisionerSecret); err != nil {
t.Fatalf("an issued client account is refused: %v", err)
}
if !runsTheProvisionerClient(m) {
t.Fatal("the module is not read as running the client")
}
unread := m
unread.OwnSecrets = catalogue.OwnSecrets{"broker": {Path: "/run/broker"}}
if runsTheProvisionerClient(unread) {
t.Fatal("a module with nowhere to read the client's account is read as running it")
}
}
// A module that runs the client and declares no broker of its own, as the catalogue's mesh-provisioner-client does,
// may be issued: the client's account alone (issue 917). Before, `module issue` refused it for having no broker,
// while a push refused it for having no account, so no machine could run the client.
func TestAClientWithoutABrokerOfItsOwnMayBeIssued(t *testing.T) {
if err := mayIssue(aClientModule()); err != nil {
t.Fatalf("the client alone is refused an account: %v", err)
}
neither := aBot("bot", false)
if err := mayIssue(neither); err == nil {
t.Fatal("a module with nowhere to read any account may be issued one")
}
}
// Assigning the client tries to issue the client's account, as assigning any module issues its broker (issue 917):
// without a bus to issue on, the line says so and names the verb, never nothing.
func TestAssigningTheClientIssuesItsAccount(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
client := aClientModule()
register(t, open, client)
out, err := assign(ctx, open, "laptop", client.Module)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(out, "bus credential") {
t.Fatalf("assigning the client said nothing of its account:\n%s", out)
}
}
+110 -6
View File
@@ -17,6 +17,7 @@ import (
"strings" "strings"
"time" "time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker" "github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue" "github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory" "github.com/novox/mesh-controller/internal/inventory"
@@ -194,7 +195,7 @@ func serve(ctx context.Context) (err error) {
// And what the catalogue decided a build meant. The builder's own result is already handled // And what the catalogue decided a build meant. The builder's own result is already handled
// above; this is the other half — the control plane is the only one of the three that knows // above; this is the other half — the control plane is the only one of the three that knows
// which machines run the thing, so it is the one that acts (novox/hq ADR 0072). // which machines run the thing, so it is the one that acts (novox/hq ADR 0072).
if err := server.Follows(following{open}); err != nil { if err := server.Follows(following{open: open}); err != nil {
return err return err
} }
// And the merges the bus announced and never handed over, read back on a timer and acted on late // And the merges the bus announced and never handed over, read back on a timer and acted on late
@@ -202,7 +203,7 @@ func serve(ctx context.Context) (err error) {
go catchingUpOnMerges(ctx, open, server) go catchingUpOnMerges(ctx, open, server)
// And a catalogue that has just started, asking for what it missed. The same type answers // And a catalogue that has just started, asking for what it missed. The same type answers
// both: what a build meant and what the builds were are two questions about one record. // both: what a build meant and what the builds were are two questions about one record.
if err := server.Answers(following{open}); err != nil { if err := server.Answers(following{open: open}); err != nil {
return err return err
} }
// And what providers say about consumers they keep failing, kept for `status` (novox/hq ADR // And what providers say about consumers they keep failing, kept for `status` (novox/hq ADR
@@ -249,8 +250,14 @@ func serve(ctx context.Context) (err error) {
// (novox/hq to-be 45 §9). // (novox/hq to-be 45 §9).
checkEvents = bus checkEvents = bus
// And every walk this controller keeps in a new state, for the delivery it walks (novox/hq ADR 0239). // And every walk this controller keeps in a new state, for the delivery it walks (novox/hq ADR 0239).
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { sayPlanMoved(ctx, bus, open.inventory, p) } inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) {
if err := server.Checks(following{open}); err != nil { sayPlanMoved(ctx, bus, open.inventory, p)
// A walk ended has taken what it reuses: its merges' check builds go (novox/hq ADR 0350 §5).
if p.WalkSaid == inventory.WalkEnded {
go forgetChecksOfWalk(context.WithoutCancel(ctx), open.inventory, p)
}
}
if err := server.Checks(following{open: open}); err != nil {
return err return err
} }
// Composed now and kept current, before the verb that answers from it is served. // Composed now and kept current, before the verb that answers from it is served.
@@ -258,6 +265,12 @@ func serve(ctx context.Context) (err error) {
// The facts snapshot a merge check is fed (novox/hq to-be 45 §9): kept current while this // The facts snapshot a merge check is fed (novox/hq to-be 45 §9): kept current while this
// controller holds the lease, read by the build seat from the artifact store. // controller holds the lease, read by the build seat from the artifact store.
go exportingFacts(ctx, open, bus.Conn.ConnectedServerVersion) go exportingFacts(ctx, open, bus.Conn.ConnectedServerVersion)
// The record mirrors (novox/hq ADR 0289, design 57 F3): the assignments and each registered module's
// declared config on the bus, written from the store only when a record changed, by the lease holder.
// The provisioner's plan of each handed module, as its machine was last sent it (design 57 P6), in the module's
// assignment.
inv.SetProvisioningPlans(inv.SentProvisioningPlans)
go mirroringRecords(ctx, inv, bus.Conn, bus)
// Every call carries the lease's epoch, and its record is written only under the lease (novox/hq // Every call carries the lease's epoch, and its record is written only under the lease (novox/hq
// to-be 45 §6). // to-be 45 §6).
link.Calls.UnderLease(func() (uint64, error) { return theLease.epoch(ctx) }) link.Calls.UnderLease(func() (uint64, error) { return theLease.epoch(ctx) })
@@ -290,6 +303,8 @@ func serve(ctx context.Context) (err error) {
return err return err
} }
defer stopCLI() defer stopCLI()
// Its read calls write no line each; how many each caller made is said once an hour (novox/hq issue 762).
go sayReadCalls(ctx)
// And says so on the bus (novox/hq ADR 0197): what it serves, as the NATS services protocol asks. // And says so on the bus (novox/hq ADR 0197): what it serves, as the NATS services protocol asks.
stopAnnouncing, err := bus.Announce(seatAnnouncement(handlers), log.New(os.Stdout, "", log.LstdFlags)) stopAnnouncing, err := bus.Announce(seatAnnouncement(handlers), log.New(os.Stdout, "", log.LstdFlags))
if err != nil { if err != nil {
@@ -740,7 +755,14 @@ func waitForApplied(ctx context.Context, inv *inventory.Inventory, node, digest
fmt.Printf("%s applied it\n", node) fmt.Printf("%s applied it\n", node)
return nil return nil
case inventory.OutcomeFailed: case inventory.OutcomeFailed:
return fmt.Errorf("%s applied what it was sent but %d resource(s) failed", node, len(doing.Failed)) // Each failed resource by id with the machine's own words, so the answer says what to fix
// without a second look (novox/hq issue 629).
var each []string
for _, f := range doing.Failed {
each = append(each, f.ID+": "+f.Error)
}
return fmt.Errorf("%s applied what it was sent but %d resource(s) failed: %s", node, len(doing.Failed),
strings.Join(each, "; "))
case inventory.OutcomeRefused: case inventory.OutcomeRefused:
return fmt.Errorf("%s refused what it was sent: %s", node, doing.Refused) return fmt.Errorf("%s refused what it was sent: %s", node, doing.Refused)
} }
@@ -1373,6 +1395,13 @@ func issueMemberships(ctx context.Context, open *stores, server *link.Server, se
if issued > 0 { if issued > 0 {
fmt.Printf(" issued %d membership(s)\n", issued) fmt.Printf(" issued %d membership(s)\n", issued)
} }
// **And what every other machine hears of a module's stream** (novox/hq ADR 0363 §6): a grant the provider's
// setting changed reaches a consumer on a machine this send does not reach. Its membership is issued again
// as last issued, with only what it hears changed, so nothing else of it moves before its own send.
reheard := reissueHearing(ctx, bus, records, where, sent)
if reheard > 0 {
fmt.Printf(" issued %d membership(s) again for what they hear of a module's stream\n", reheard)
}
if len(refused) > 0 { if len(refused) > 0 {
// Returned, never passed over (novox/hq issue 249): the declarations of the machines they are // Returned, never passed over (novox/hq issue 249): the declarations of the machines they are
// for are not sent, and the rollout that asked is tried again rather than waiting for a push. // for are not sent, and the rollout that asked is tried again rather than waiting for a push.
@@ -1672,7 +1701,7 @@ func recordSent(ctx context.Context, inv *inventory.Inventory, node string, body
// send that is away look failed: it is said, loudly, and the machine's own record stands. // send that is away look failed: it is said, loudly, and the machine's own record stands.
err = inv.RecordSentWith(kept, record.ID, digest, builds, epoch, sent.toldGeneration, inventory.Send{ err = inv.RecordSentWith(kept, record.ID, digest, builds, epoch, sent.toldGeneration, inventory.Send{
Sequence: sent.sequence, Epoch: int64(sent.epoch), Sender: sent.sender, Generation: sent.generation, Sequence: sent.sequence, Epoch: int64(sent.epoch), Sender: sent.sender, Generation: sent.generation,
Digest: digest, Modules: sent.modules}) Digest: digest, Modules: sent.modules, Provisioning: sent.plans})
var unrecorded *inventory.SendNotRecordedError var unrecorded *inventory.SendNotRecordedError
switch { switch {
case errors.As(err, &unrecorded): case errors.As(err, &unrecorded):
@@ -1731,3 +1760,78 @@ func schemaReach() (int, error) {
} }
return reach, nil return reach, nil
} }
// reissueHearing issues again, on every machine not just sent, the membership of each assignment whose
// `consumes` names a module owning a stream, when what it hears changed: read as last issued from the bus,
// its `hears` replaced, and published. One that was never issued is left to its own send. Failures are said
// and passed over: the send that carries the grant is not the consumer's, and its own next send issues it.
func reissueHearing(ctx context.Context, bus link.OverNATS, records broker.Records, where broker.Placements,
sent []readyNode) int {
reached := map[string]bool{}
for _, s := range sent {
reached[s.node] = true
}
issued := 0
for node, declared := range records.Assigned {
if reached[node] {
continue
}
for _, d := range declared {
if !d.HearsAStream {
continue
}
subject := broker.MembershipSubject(node, d.Module)
last, err := bus.JS.GetLastMsg(broker.AssignmentsStream, subject)
if err != nil {
if !errors.Is(err, nats.ErrMsgNotFound) {
fmt.Printf(" %s on %s: its membership could not be read to issue what it hears again: %v\n",
d.Module, node, err)
}
continue
}
body, changed, err := withHearing(last.Data, broker.HeardIssuedFor(node, d))
if err != nil {
fmt.Printf(" %s on %s: its membership could not be read to issue what it hears again: %v\n",
d.Module, node, err)
continue
}
if !changed {
continue
}
if err := bus.PublishMembership(ctx, node, d.Module, body); err != nil {
fmt.Printf(" %s on %s: what it hears could not be issued again: %v\n", d.Module, node, err)
continue
}
issued++
}
}
return issued
}
// withHearing is a membership as issued with its `hears` replaced, and whether that changed it. Pure.
func withHearing(issued []byte, hears []broker.HeardIssued) ([]byte, bool, error) {
var fields map[string]json.RawMessage
if err := json.Unmarshal(issued, &fields); err != nil {
return nil, false, err
}
var had []broker.HeardIssued
if raw, ok := fields["hears"]; ok {
if err := json.Unmarshal(raw, &had); err != nil {
return nil, false, err
}
}
if slices.Equal(had, hears) {
return issued, false, nil
}
if len(hears) == 0 {
delete(fields, "hears")
} else {
raw, err := json.Marshal(hears)
if err != nil {
return nil, false, err
}
fields["hears"] = raw
}
body, err := json.Marshal(fields)
return body, true, err
}
+12 -2
View File
@@ -340,7 +340,10 @@ func cancelAsk(ctx context.Context, js *broker.JetStream, open *stores, seat str
return "", fmt.Errorf("%s is marked cancelled and could not be deleted from the queue (seq %d): %w — a "+ return "", fmt.Errorf("%s is marked cancelled and could not be deleted from the queue (seq %d): %w — a "+
"holder taking it ends it as cancelled", ask.ID, ask.Seq, err) "holder taking it ends it as cancelled", ask.ID, ask.Seq, err)
} }
recordCancelled(ctx, open, ask) // A check ended because its pull request has a newer head records nothing (novox/hq issue 726): open is nil.
if open != nil {
recordCancelled(ctx, open, ask)
}
return cancelSaid(ask, seat, removed), nil return cancelSaid(ask, seat, removed), nil
} }
@@ -518,7 +521,9 @@ func rebuildCommand(ctx context.Context, args []string) error {
} }
// **A module a plan holds unbuilt, or failed, joins that plan** rather than running beside it: the // **A module a plan holds unbuilt, or failed, joins that plan** rather than running beside it: the
// plan would ask it again, or stay failed on an outcome a rebuild has replaced. // plan would ask it again, or stay failed on an outcome a rebuild has replaced. A walk waiting for its word
// is not joined (novox/hq issue 754): the rebuild runs beside it, and says so.
var beside string
if module != "" { if module != "" {
joined, said, err := joinAPlan(ctx, open, module) joined, said, err := joinAPlan(ctx, open, module)
if err != nil { if err != nil {
@@ -528,6 +533,7 @@ func rebuildCommand(ctx context.Context, args []string) error {
fmt.Println(said) fmt.Println(said)
return nil return nil
} }
beside = said
} }
id, err := askABuild(ctx, source, path, ref) id, err := askABuild(ctx, source, path, ref)
if err != nil { if err != nil {
@@ -535,6 +541,10 @@ func rebuildCommand(ctx context.Context, args []string) error {
} }
recordAsked(ctx, inventory.BuildRequest{ID: id, Repository: source.Repository, Seat: source.Seat, Path: path, recordAsked(ctx, inventory.BuildRequest{ID: id, Repository: source.Repository, Seat: source.Seat, Path: path,
Ref: ref, For: "rebuild"}) Ref: ref, For: "rebuild"})
if beside != "" {
fmt.Printf("rebuild asked as %s, beside the walk: %s\n", id, beside)
return nil
}
fmt.Printf("rebuild asked as %s\n", id) fmt.Printf("rebuild asked as %s\n", id)
return nil return nil
} }
+4 -2
View File
@@ -616,6 +616,7 @@ func TestKillAndPauseAskTheHolderOnTheMachine(t *testing.T) {
// A plan that stopped at its first machine is retried: the module sent to that machine again, the // A plan that stopped at its first machine is retried: the module sent to that machine again, the
// send recorded as the first anew, and the plan goes on — unless a newer plan holds the module. // send recorded as the first anew, and the plan goes on — unless a newer plan holds the module.
func TestAPlanStoppedAtItsFirstMachineIsRetried(t *testing.T) { func TestAPlanStoppedAtItsFirstMachineIsRetried(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t) open := aMesh(t)
ctx := t.Context() ctx := t.Context()
asked := asksRecorded(t) asked := asksRecorded(t)
@@ -636,8 +637,9 @@ func TestAPlanStoppedAtItsFirstMachineIsRetried(t *testing.T) {
stopped := inventory.Plan{ID: "plan-rollout", Repository: "novox/a", Branch: "main", Commit: "c0ffee", stopped := inventory.Plan{ID: "plan-rollout", Repository: "novox/a", Branch: "main", Commit: "c0ffee",
Created: long, State: inventory.PlanFailed, Tiers: [][]string{{"a"}, {"b"}}, Created: long, State: inventory.PlanFailed, Tiers: [][]string{{"a"}, {"b"}},
Note: "a stopped at its first machine in tier 0: laptop refused what it was sent", Note: "a stopped at its first machine in tier 0: laptop refused what it was sent",
Modules: map[string]*inventory.PlanModule{"a": {State: "built", BuiltAt: &long, Commit: "c0ffee", // Asked by the plan, as every module a plan holds built is (novox/hq issue 754).
First: []string{"laptop"}, FirstAt: &long, Why: "laptop refused what it was sent"}}} Modules: map[string]*inventory.PlanModule{"a": {State: "built", AskedAt: &long, Build: "build-a", BuiltAt: &long,
Commit: "c0ffee", First: []string{"laptop"}, FirstAt: &long, Why: "laptop refused what it was sent"}}}
if err := open.inventory.SavePlan(ctx, &stopped); err != nil { if err := open.inventory.SavePlan(ctx, &stopped); err != nil {
t.Fatal(err) t.Fatal(err)
} }
+106
View File
@@ -0,0 +1,106 @@
package main
import (
"context"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/recordmirror"
)
// The controller's record mirrors on the bus (novox/hq ADR 0289, design 54, design 57 F3): the assignments,
// and each registered module's declared config, which mesh-config resolves a module's config values from.
//
// **Built from the store, on a tick of their own, by the lease holder alone.** A record changes through a
// verb that runs as a command of its own, a reconcile, a merge's registration or a pending assignment made,
// so no one call site sees every change; what every change has in common is that it is committed to the
// store. So the serving controller reads the store every mirrorEvery and writes only what differs from what
// the bucket holds (ADR 0289 §1): a quiet mesh writes nothing, and a change rolled back is never read. A
// mirror is therefore at most mirrorEvery behind its store.
// mirrorEvery is how often the mirrors are compared with the store: two small reads of the store.
const mirrorEvery = 10 * time.Second
// mirroringRecords keeps the mirrors current for as long as this controller serves: ctx ends when it stops.
func mirroringRecords(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn, announce recordmirror.Announcer) {
api, err := jetstream.New(conn)
if err != nil {
fmt.Printf("the record mirrors are not kept: %v\n", err)
return
}
holds := func() bool {
epoch, err := theLease.epoch(ctx)
return err == nil && epoch > 0
}
epoch := func() uint64 {
e, _ := theLease.epoch(ctx)
return e
}
sources := map[string]func(context.Context) (map[string][]byte, error){
recordmirror.Assignments.Name: inv.AssignmentsMirror,
recordmirror.DeclaredConfig.Name: inv.DeclaredConfigMirror,
recordmirror.Predefined.Name: inv.PredefinedMirror,
}
writers := map[string]*recordmirror.Writer{}
failing := map[string]string{}
tick := time.NewTicker(mirrorEvery)
defer tick.Stop()
for {
if holds() {
for _, m := range recordmirror.Mirrors() {
why := mirrorOnce(ctx, api, m, writers, sources[m.Name], announce, epoch, holds)
if why != failing[m.Name] {
if why != "" {
fmt.Printf("the %s mirror cannot be kept: %s\n", m.Name, why)
} else {
fmt.Printf("the %s mirror is kept again\n", m.Name)
}
failing[m.Name] = why
}
}
}
select {
case <-ctx.Done():
return
case <-tick.C:
}
}
}
// mirrorOnce brings one mirror to what the store holds, and answers why it could not; empty when it could.
func mirrorOnce(ctx context.Context, api jetstream.JetStream, m recordmirror.Mirror, writers map[string]*recordmirror.Writer,
source func(context.Context) (map[string][]byte, error), announce recordmirror.Announcer, epoch func() uint64,
holds func() bool) string {
reading, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
w := writers[m.Name]
if w == nil {
kv, err := api.KeyValue(reading, m.Bucket)
if err != nil {
return fmt.Sprintf("the bucket %s cannot be read: %v", m.Bucket, err)
}
w = &recordmirror.Writer{Mirror: m, KV: kv, Announce: announce, Epoch: epoch, Holds: holds}
writers[m.Name] = w
}
want, err := source(reading)
if err != nil {
return fmt.Sprintf("the store cannot be read: %v", err)
}
lost := w.LostEvents
r, err := w.Sync(reading, want)
if r.Written+r.Removed > 0 {
fmt.Printf("the %s mirror moved: %d written, %d removed\n", m.Name, r.Written, r.Removed)
}
if w.LostEvents > lost {
fmt.Printf("the %s mirror lost %d event(s): the bucket holds the change, its notice was refused twice\n",
m.Name, w.LostEvents-lost)
}
if err != nil {
return err.Error()
}
return ""
}
+36 -1
View File
@@ -2,6 +2,7 @@ package main
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"sort" "sort"
@@ -83,6 +84,27 @@ func everyBuildKept(ctx context.Context, node string) bool {
return on != "" && on == node return on != "" && on == node
} }
type keepForSettingsKey struct{}
// keepingBuildsForSettings is a context whose sends compose every module of one machine — recorded or rolling out —
// at the build that machine was last sent, as keepingEveryBuild does, for a settings walk's send (novox/hq ADR 0344):
// the layer the operator approved, and nothing of any module's new code. A build held for a person, or waiting
// for a gate, stays where it is; the walk that is its own carries it.
func keepingBuildsForSettings(ctx context.Context, node string) context.Context {
return context.WithValue(ctx, keepForSettingsKey{}, node)
}
// errSettingsCannotKeep is a settings walk's send refused because a build of its machine cannot be kept: what the
// machine was last sent is not known, or a build it runs is gone from the records. It does not clear by itself, so
// the walk fails on it rather than trying again (ADR 0344).
var errSettingsCannotKeep = errors.New("the settings cannot be sent with every build kept")
// settingsKeptOn is whether this context is a settings walk's send to this machine.
func settingsKeptOn(ctx context.Context, node string) bool {
on, _ := ctx.Value(keepForSettingsKey{}).(string)
return on != "" && on == node
}
// recordedKept is, for a send under keepingRecorded, every recorded module the machine was last sent a // recordedKept is, for a send under keepingRecorded, every recorded module the machine was last sent a
// build of that the mesh's build is not identical to: module → the commit it keeps. Nil when the context // build of that the mesh's build is not identical to: module → the commit it keeps. Nil when the context
// keeps nothing, or when what the machine was last sent is not known (it is then held whole elsewhere — // keeps nothing, or when what the machine was last sent is not known (it is then held whole elsewhere —
@@ -92,7 +114,7 @@ func everyBuildKept(ctx context.Context, node string) bool {
// 490), and a machine whose last send is not known refuses the send: there is nothing to keep it at, and // 490), and a machine whose last send is not known refuses the send: there is nothing to keep it at, and
// composing the mesh's builds would be the very move the grants step must not make. // composing the mesh's builds would be the very move the grants step must not make.
func recordedKept(ctx context.Context, open *stores, node string) (map[string]string, error) { func recordedKept(ctx context.Context, open *stores, node string) (map[string]string, error) {
every := everyBuildKept(ctx, node) every := everyBuildKept(ctx, node) || settingsKeptOn(ctx, node)
skip, on := keptExcept(ctx) skip, on := keptExcept(ctx)
if !on && !every { if !on && !every {
return nil, nil return nil, nil
@@ -106,6 +128,9 @@ func recordedKept(ctx context.Context, open *stores, node string) (map[string]st
return nil, err return nil, err
} }
if !known { if !known {
if settingsKeptOn(ctx, node) {
return nil, fmt.Errorf("%w: what %s was last sent is not known (novox/hq ADR 0344)", errSettingsCannotKeep, node)
}
if every { if every {
return nil, fmt.Errorf("what %s was last sent is not known, so the bus's user list cannot be sent "+ return nil, fmt.Errorf("what %s was last sent is not known, so the bus's user list cannot be sent "+
"there alone with every build kept (novox/hq issue 490)", node) "there alone with every build kept (novox/hq issue 490)", node)
@@ -133,6 +158,12 @@ func recordedKept(ctx context.Context, open *stores, node string) (map[string]st
if f.identical(m, was, now.Commit) { if f.identical(m, was, now.Commit) {
continue continue
} }
// **Under its own `record`, a move that recreates nothing and moves no data is not kept** (novox/hq ADR 0377
// §1): it goes as a module that rolls out goes. A person's `record`, the bus, and a move that cannot be
// judged are kept as before (§4, §5).
if !every && f.rolls(m, was) {
continue
}
out[m] = was out[m] = was
} }
// **And the controller and the node-engine at the build the machine was last sent, over an older record** // **And the controller and the node-engine at the build the machine was last sent, over an older record**
@@ -166,6 +197,10 @@ func keepRecorded(ctx context.Context, open *stores, node string, shelf map[stri
if err != nil { if err != nil {
return nil, err return nil, err
} }
if !found && settingsKeptOn(ctx, node) {
return nil, fmt.Errorf("%w: %s runs %s's build %s, which the build records no longer hold (novox/hq ADR 0344)",
errSettingsCannotKeep, node, m, short(kept[m]))
}
if !found && everyBuildKept(ctx, node) { if !found && everyBuildKept(ctx, node) {
return nil, fmt.Errorf("%s runs %s's build %s, which the build records no longer hold: the bus's user "+ return nil, fmt.Errorf("%s runs %s's build %s, which the build records no longer hold: the bus's user "+
"list cannot be sent there alone with it kept (novox/hq issue 490)", node, m, short(kept[m])) "list cannot be sent there alone with it kept (novox/hq issue 490)", node, m, short(kept[m]))

Some files were not shown because too many files have changed in this diff Show More