Compare commits

...
Author SHA1 Message Date
mesh-admin 660649eeed Test config end to end into a composed file, and hold the settings tests to no whole-mesh module layer (hq issue 630)
mesh/merge-gate head 660649eeed1c merged into trunk 6f1a05366c1d — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 660649eeed1c merged into trunk 6f1a05366c1d — pass: its merge-check.sh passed
mesh/delivery ready: it delivers once merged
2026-10-13 13:19:30 +00:00
mesh-admin c9b63f61bd Serve config as a seat verb, and say each value's source as one of the four (hq issue 630)
The listings named a node's own value "the node", which now could be node config; they say "<module> on <node>" for the module's own value instead.
2026-10-13 13:18:33 +00:00
mesh-admin 3f9b935c25 Keep the list from a catalogue build, and carry it and mesh and node config through the facts and the gate (hq issue 630)
The merge gate must compose with the same config the mesh holds, and judge a catalogue change against the list it brings.
2026-10-13 13:17:48 +00:00
mesh-admin 0c31ca0eaf Add the config command: mesh, node and module values in one call, with why and history (hq issue 630)
One place to set each of the three sources over a default, refusing what is not a predefined key and a module value without a node, recorded in the hand-act log like any act by hand.
2026-10-13 13:16:54 +00:00
mesh-admin ea4c318404 Test the four sources, the refusals and the move of whole-mesh module layers (hq issue 630) 2026-10-13 13:15:46 +00:00
mesh-admin df5254f308 Keep mesh and node config with their history, and move whole-mesh module layers onto each assigned node (hq issue 630)
An assignment is one module on one node, so a new whole-mesh layer is refused and the ones held are laid under each assigned node's layer, every replaced layer kept, so no machine composes differently.
2026-10-13 13:14:42 +00:00
mesh-admin fba6a7c454 Carry the catalogue's predefined.json with a build, and declare the config verb (hq issue 630)
The controller only learns manifests through builds, so the list of predefined keys travels the same way: the builder reads it at the catalogue's root, judges the manifest against it and hands it on.
2026-10-13 13:13:24 +00:00
mesh-admin 3d16515771 Declare preferences with a type and a predefined key, and lay mesh and node config between default and node (hq issue 630)
A preference may fall back to one reviewed mesh-wide or node-wide value instead of a module layer for the whole mesh; like a default, config fills ${setting:} only.
2026-10-13 13:08:37 +00:00
mesh-admin 6f1a05366c Merge pull request 'Admit a port from the machine's own local network, and never a public subnet (issue 154, ADR 0340)' (#299) from fix/154-local-network into main 2026-10-13 13:07:11 +00:00
mesh-admin 0a1dc5f93f Merge pull request 'seat: a push to a named machine waits for the machine's answer, so its call says how it went (issue 629)' (#301) from fix/629-a-push-says-how-it-went into main 2026-10-13 12:44:51 +00:00
mesh-admin ae0b384497 push-wait is untrusted, so it is set in one call like any preference (issue 629)
mesh/merge-gate head ae0b384497c2 merged into trunk f3875a6b6019 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head ae0b384497c2 merged into trunk f3875a6b6019 — pass: its merge-check.sh passed
mesh/delivery delivered
The gate warned that the push file asks for a setting without saying whether it is trusted, which makes push-wait terminal-only.
2026-10-13 12:38:31 +00:00
mesh-admin f3875a6b60 Merge pull request 'catalogue: say in the comments that a package may be declared by several modules (issue 628)' (#300) from fix/628-packages-are-shared into main 2026-10-13 12:38:30 +00:00
mesh-admin f09c5ebdec push: the wait is the call's or the controller's setting push-wait, never a length in the code (issue 629)
mesh/merge-gate head f09c5ebdec6c merged into trunk 584aa2d7d7a5 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head f09c5ebdec6c merged into trunk 584aa2d7d7a5 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The operator's rule: a wait is never hard-coded, at most a default is configured. The push verb takes `wait`; without it the controller's setting `push-wait` (module default 2m) applies; a setting that cannot be read waits for nothing, as a push did before, and says so.
2026-10-13 12:09:43 +00:00
mesh-admin 6d2e4c299a seat: a push to a named machine waits for the machine's answer, so its call says how it went (issue 629)
mesh/merge-gate head 6d2e4c299a34 merged into trunk 584aa2d7d7a5 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 6d2e4c299a34 merged into trunk 584aa2d7d7a5 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The seat ran `push <node> --wait 0`, so a push's call ended at "sent" and nobody learned whether the machine applied it. The caller is answered before a push starts (answersFirst, issue 265), so waiting holds nobody: the call now waits up to two minutes for the machine's own report and keeps it, applied, refused with the machine's reason, or each failed resource with its error.
2026-10-13 12:03:53 +00:00
mesh-admin b654b320a1 catalogue: say in the comments that a package may be declared by several modules (issue 628)
mesh/merge-gate head b654b320a18a merged into trunk 584aa2d7d7a5 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head b654b320a18a merged into trunk 584aa2d7d7a5 — pass: its merge-check.sh passed
mesh/delivery delivered
The resolver follows ADR 0303, but four comments still stated ADR 0210's retired rule that two modules declaring one package collide, and an agent took them as the rule. What stays exclusive is what a module owns: a path, a unit, a name, and a package declared present by one module and absent by another.
2026-10-13 11:52:17 +00:00
jochen 4cc8588afb Test that a global IPv6 subnet is public and never counts as the local network (issue 154, ADR 0340)
mesh/merge-gate head 4cc8588afb87 merged into trunk 0a1dc5f93fcb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 4cc8588afb87 merged into trunk 0a1dc5f93fcb — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/154-local-network delivered: every member is delivered
The review asked for the IPv6 side of the rule that local-network cannot open a port to the internet.
2026-10-13 13:20:50 +02:00
mesh-admin 584aa2d7d7 Merge pull request 'Never make a credential only a person can give, and give it once per provider (hq issue 164, ADR 0337)' (#298) from fix/164-an-accepted-credential-is-never-minted into main 2026-10-13 09:52:07 +00:00
jochen c9dd07f919 Never make a credential only a person can give, and give it once per provider (hq issue 164, ADR 0337)
mesh/merge-gate head c9dd07f919a2 merged into trunk 404b4409e70e — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head c9dd07f919a2 merged into trunk 404b4409e70e — pass: its merge-check.sh passed (run 2)
mesh/delivery delivered
A Servarr app's API key, an indexer's key and a downloader's password are values the provider's
software holds; the pair path made a random value for every consumer of them, sealed it to both ends
and reported success, and the provider refused it. An offer may now say `"credential": {"issued-by":
"outside"}`: SecretFor never makes a value for it, the plan reads such a pair as given or not given
and sends the consumer no credential file until a person gives it, without refusing its machine;
remakeShared and SharedSecretFor never make a provider's own secret issued outside either.
`secret accept <provider's node> <provider module> <provision>` seals one value to every consumer that
binds the provision from that provider, the provider and the operator, under one generation; the
per-pair form is refused for such a provision, and rotate names the accept. A consumer may wait for a
provision's credential: the controller excuses the wait only when the store bears it out, and its
needs-operator condition names the one accept line.

Adds the replay of issue 164 (TestReplay164), written with only what the controller had before.
2026-10-13 11:37:31 +02:00
mesh-admin 404b4409e7 Merge pull request 'A built module keeps its build edges: a new SDK plans the toolchain and its TypeScript bundles (hq issue 614)' (#296) from fix/614-a-built-module-keeps-its-build-edges into main 2026-10-13 09:35:26 +00:00
jochen 6e6cd5e26d Admit a port from the machine's own local network, and never a public subnet (issue 154, ADR 0340)
mesh/delivery-group group fix/154-local-network checking: 0 of 2 member(s) ready
mesh/merge-gate out of date: judged against trunk 1826b68a22b5, the trunk is now 404b4409e70e; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 1826b68a22b5, the trunk is now 404b4409e70e; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
A local service could only be declared public, which says the internet. local-network is a source and a reach: the private network and the private networks the machine reported on a link facing outside, judged here, shown by plan and node, and said in the rule set when there are none.
2026-10-13 11:20:22 +02:00
jochen e658a14296 Read a built module's build edges from what is kept of it, so a new SDK rebuilds its toolchain and bundles (hq issue 614)
mesh/merge-gate head e658a14296f9 merged into trunk 1826b68a22b5 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head e658a14296f9 merged into trunk 1826b68a22b5 — pass: its merge-check.sh passed
mesh/delivery delivered
A registered manifest carries no build, so mesh-tools' build.on naming the
SDK's package and every bundle's toolchain were edges only before a module was
first built: mesh-sdk 0.1.14 planned "sent nowhere". The relation now reads a
package base from the recorded name@version against the package the mesh holds,
and the toolchain from the bundles the resolved manifest keeps. Only records the
mesh already keeps are read, so it holds for every module built before it and a
rollback reads the same records. Replay R614.
2026-10-13 11:08:17 +02:00
mesh-admin 1826b68a22 Merge pull request 'Say which modules a walk left unchanged, so a merge that moved nothing is not read as stranded (hq issue 611)' (#295) from fix/611-an-unchanged-build-is-delivered into main 2026-10-13 08:25:45 +00:00
jochen 05576db018 Say which modules a walk left unchanged, so a merge that moved nothing is not read as stranded (hq issue 611)
mesh/merge-gate head 05576db018a4 merged into trunk a688dcff394b — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 05576db018a4 merged into trunk a688dcff394b — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/611-an-unchanged-build-is-delivered delivered: every member is delivered
A delivery plan made while another walk built a module reads it whole and names it; the walk, cut later,
finds none of the changed files in the build source that build recorded and moves nothing. Its record named
no module, so mesh-delivery read 'matched what runs' as 'never built', stopped the delivery and asked the
operator about a change already on every machine. The walk now says each such module in delivery.unchanged,
with why; its modules keep meaning what it builds.
2026-10-13 08:18:19 +02:00
mesh-admin a688dcff39 Merge pull request 'Add the replay of issue 239: a module's name is not taken over by another repository (hq issue 239)' (#294) from fix/239-replay into main 2026-10-13 05:37:08 +00:00
mesh-admin 8f1aec0832 Merge pull request 'Add the replay of issue 363: a README-only merge made a new digest of an unchanged Go program (hq issue 363)' (#293) from fix/363-replay into main 2026-10-13 05:08:13 +00:00
jochen 95556b8def Add the replay of issue 239: a module's name is not taken over by a build from another repository
mesh/merge-gate head 95556b8def3f merged into trunk 8f1aec0832b1 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 95556b8def3f merged into trunk 8f1aec0832b1 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/239-replay delivered: every member is delivered
The fix (#174, hq ADR 0266 §7) refuses an outcome built from a repository other than the module's
registered one; nothing replayed the photo app's outage that showed why. The replay uses only what
the controller had before the fix, so the prover can show it fails there and passes on the fix.
2026-10-13 07:02:05 +02:00
jochen 94c928b1f5 Add the replay of issue 363: a README-only merge made a new digest of an unchanged Go program (hq issue 363)
mesh/merge-gate head 94c928b1f5af merged into trunk cd6757f1e3c8 — pass: builds build-agent, route-proxy → ace, g14, novox, shanks; no bus step; every mach…
mesh/repo-check head 94c928b1f5af merged into trunk cd6757f1e3c8 — pass: its merge-check.sh passed
mesh/delivery stopped: a merged change never reached the machines: its walk plan-1791867648768608552 ended done before its word without building build-ag…
mesh/delivery-group group fix/363-replay stopped: a member was stopped
2026-10-13 07:00:44 +02:00
mesh-admin cd6757f1e3 Merge pull request 'Stamp no commit into a Go bundle, so one build source is one digest (hq issue 363, ADR 0267 rule 6)' (#292) from fix/363-a-go-bundle-does-not-vary-with-its-commit into main 2026-10-13 04:59:09 +00:00
jochen ef3e5ae9b0 Stamp no commit in this repository's own image recipes either (hq issue 363, review of #292)
mesh/merge-gate head ef3e5ae9b0dc merged into trunk 1d244d5aa24e — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head ef3e5ae9b0dc merged into trunk 1d244d5aa24e — pass: its merge-check.sh passed
mesh/delivery delivered
The genesis image copies the whole clone with .git, so its controller
carried the commit it was built at, as the bundles did.
2026-10-13 06:47:18 +02:00
mesh-admin 1d244d5aa2 Merge pull request 'Add the replay of issue 348, and refuse bind-dynamic now the catalogue binds once (hq issue 348)' (#291) from fix/348-replay into main 2026-10-13 04:39:40 +00:00
jochen bcdb1ad4f8 Stamp no commit into a Go bundle, so one build source is one digest (hq issue 363, ADR 0267 rule 6)
mesh/merge-gate out of date: judged against trunk e0d69ab7a05b, the trunk is now 1d244d5aa24e; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk e0d69ab7a05b, the trunk is now 1d244d5aa24e; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
A module built from its repository's root is compiled with the clone's .git
in view, and Go stamped the commit into the binary: a merge that changed only
a README made a new digest of an unchanged program.
2026-10-13 06:29:44 +02:00
mesh-admin e0d69ab7a0 Merge pull request 'mesh-builder: read the genesis document until the registry binds, and refuse a registry build with neither (hq issue 121, ADR 0336)' (#289) from fix/121-genesis-package-registry into main 2026-10-13 04:22:05 +00:00
jochen 6eea92c323 Add the replay of issue 348, and refuse bind-dynamic now the catalogue binds once (hq issue 348)
mesh/merge-gate head 6eea92c323f4 merged into trunk e0d69ab7a05b — pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes …
mesh/repo-check head 6eea92c323f4 merged into trunk e0d69ab7a05b — pass: its merge-check.sh passed (run 2)
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@ef3e5ae9b0dc (merged as cd6757f1 into main, walk plan-17918675…
mesh/delivery-group group fix/348-replay delivered: every member is delivered
The replay holds the operator's acceptance: a build sent after the resolver
stopped answering is not put back for that fault, and a restarted engine's
undecided statement does not forget it. Written only with what the controller
had before mesh-controller #179, so mesh-lab's prover lays it over the commit
before: it fails there and passes on the fix and on main.

The resolver's manifest test took either binding until the catalogue's main
bound once; mesh-catalog #161 (105cc7c0) did, so bind-dynamic is refused.
2026-10-13 06:09:09 +02:00
mesh-admin b12c4b82e8 Merge pull request 'Add the replay of issue 349: a merge acted on late superseded the merge after it (hq issue 349)' (#290) from fix/349-replay into main 2026-10-13 03:54:10 +00:00
jochen 5740874ce3 Drive the replay of issue 349 to where main decides (review of #290)
mesh/merge-gate head 5740874ce3b9 merged into trunk 1c9bc9cf0041 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 5740874ce3b9 merged into trunk 1c9bc9cf0041 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/349-replay delivered: every member is delivered
Since ADR 0276 the earlier merge waits in a queued batch with no modules, and the
replay skipped it, so its pass on main proved nothing. The replay now ends the
later merge's walk and has another repository's merge heard, which cuts the batch,
and fails on any plan of the two merges left standing with nothing planned.
2026-10-13 05:42:52 +02:00
jochen bbf70fbda4 Add the replay of issue 349: a merge acted on late superseded the merge after it (hq issue 349)
Two catalogue merges acted on in reverse order left the fixed module planned at
the commit without the fix. The replay uses only what the controller had before
Listed among the store tests, so a check without a database cannot pass its skip.
2026-10-13 05:42:52 +02:00
mesh-admin 1c9bc9cf00 Merge pull request 'Let a trusted file that names its keys hold only those keys at the terminal (hq issue 586, ADR 0329 §5)' (#286) from fix/586-a-named-key-is-trusted-alone into main 2026-10-13 03:38:13 +00:00
jochen 3275a16beb Let a trusted file that names its keys hold only those keys at the terminal (hq issue 586, ADR 0329 §5)
mesh/merge-gate head 3275a16bebbb merged into trunk 584ee991961c — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 3275a16bebbb merged into trunk 584ee991961c — pass: its merge-check.sh passed
mesh/delivery delivered
A module with a trusted mergeable file had its whole layer refused through the
settings verb, because such a file took any key. Once a module's files name the
keys they take (ADR 0326), each file takes only those, so only a trusted file's
named keys are the terminal's: betterbird's tags, in a file nothing trusts, is set
through the verb, while its accounts and identities stay the terminal's or the
operator's warrant. A module whose files name no keys keeps the whole-layer rule.
2026-10-13 05:22:35 +02:00
mesh-admin 584ee99196 Merge pull request 'Add the replay of issue 340: a verb set what every agent session obeys (hq issue 340)' (#288) from fix/340-replay into main 2026-10-13 01:04:46 +00:00
jochen e44c726a38 List TestReplay340 among the store tests, so a check without a database cannot count its skip as a pass (hq issue 340, review 144)
mesh/merge-gate head e44c726a3812 merged into trunk 49102e1645c9 — pass: the change touches no module of the mesh's graph
mesh/repo-check head e44c726a3812 merged into trunk 49102e1645c9 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/340-replay delivered: every member is delivered
2026-10-13 02:57:35 +02:00
jochen a641ed7d15 Let the build agent run on the genesis document until its registry binds (hq issue 121, ADR 0336)
mesh/merge-gate head a641ed7d15c7 merged into trunk b12c4b82e820 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head a641ed7d15c7 merged into trunk b12c4b82e820 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/121-genesis-package-registry delivered: every member is delivered
With npm-package-registry optional, the binding file the agent's environment
names may be absent, and mesh-builder failed every build on reading it. It now
reads the genesis document the installer seals at genesis's packages step when
no binding is present, prefers the binding when both are, and each build's log
says which it resolved from (§4). A build that needs the registry - a package,
an image naming .npmrc, TypeScript with an @novox/ dependency - on an agent with
neither is refused before anything is built or published, naming the machine,
the unbound requirement and gitea (§6); it never falls back to the public
registry. The captured catalogue moves to the mesh-catalog change it is judged
with.
2026-10-13 02:53:41 +02:00
jochen 2a13917d6e Add the replay of issue 340: a verb set what every agent session obeys (hq issue 340)
The settings verb took the agent module's managed settings and an empty
runtime configuration's url from any caller. The replay uses only what the
controller had before #172, so mesh-lab's prover can show it fails there.
2026-10-13 02:52:28 +02:00
mesh-admin 49102e1645 Merge pull request 'Add the replays of issues 279 and 412 (hq issues 279, 412)' (#285) from fix/279-412-replays into main 2026-10-13 00:45:26 +00:00
mesh-admin eb7ee921b5 Merge pull request 'Replay issue 166 with only what the catalogue had before its fix (for mesh-lab R166)' (#287) from test/166-replay into main 2026-10-13 00:32:48 +00:00
jochen b1baa8d8ba Replay issue 166 with only what the catalogue had before its fix (for mesh-lab R166)
mesh/merge-gate head b1baa8d8baf0 merged into trunk 4473bc7d7315 — pass: the change touches no module of the mesh's graph
mesh/repo-check head b1baa8d8baf0 merged into trunk 4473bc7d7315 — pass: its merge-check.sh passed
mesh/delivery delivered
A replay must fail on the commit before its fix and pass on it; the fix's own tests read what the fix
added, so this one uses only ParseManifest, Resolve and the resolve tests' helpers, and can be laid
over the commit before #281.
2026-10-13 02:27:07 +02:00
mesh-admin 4473bc7d73 Merge pull request 'Let a requirement be optional, binding only a provider that already runs (hq issue 166, ADR 0330)' (#281) from fix/166-optional-requirement into main 2026-10-13 00:05:33 +00:00
jochen 3bdab29a5a Add the replays of issues 279 and 412, written with only what the catalogue had before each fix
mesh/merge-gate head 3bdab29a5a0e merged into trunk eb7ee921b533 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 3bdab29a5a0e merged into trunk eb7ee921b533 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/279-412-replays delivered: every member is delivered
So the lab's prover can lay them over the commit before mesh-controller #260 and #267 and see them fail
there and pass on the fix (hq issues 279, 412).
2026-10-13 02:02:20 +02:00
jochen 79a3396340 Store optional requirements beside a plain requires list (hq issue 166, review of #281)
mesh/merge-gate head 79a3396340a7 merged into trunk 454a3db69a1e — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 79a3396340a7 merged into trunk 454a3db69a1e — pass: its merge-check.sh passed
mesh/delivery delivered
A controller older than optional requirements reads `requires` as names, and Inventory.Catalogue
stops at the first manifest it cannot read: the long form stored inside `requires` would have left
a rollback below this controller composing no machine. Stored, `requires` stays plain and the
optional names sit in `optional-requires`, a key an older controller sets aside, leaving out only
the module carrying it. A refusal of a hard requirement now names a module that wants it hard.
2026-10-13 01:53:18 +02:00
jochen 6ec4139871 Let a requirement be optional, binding only a provider that already runs (hq issue 166, ADR 0330)
Making every soft dependency a provision made it hard: a mesh without influxdb could not assign
grafana, and one with influxdb in its catalogue would install it for grafana. A requirement may now
be written {"name": ..., "optional": true}. It resolves as a hard one when a provider is assigned or
reachable, and is otherwise absent: never refused, nothing installed for it. Its binding, credential
and only-when-bound files are left out, its ${bound:...|default:...} placeholders take their
defaults, the module check refuses one that could render empty, and plan and node list it unbound.
2026-10-13 01:53:18 +02:00
mesh-admin 454a3db69a Merge pull request 'Refuse a declaring module's mergeable file that names no keys (hq issue 574, ADR 0326 §3)' (#282) from fix/574-a-declaring-module-names-its-files-keys into main 2026-10-12 23:28:21 +00:00
mesh-admin 7b2a9eae80 Merge pull request 'Read the one genesis bundle mesh-host carries now (issue 146)' (#284) from fix/146-controller-test-one-bundle into main 2026-10-12 23:21:14 +00:00
jochen 5795e03fad Read the one genesis bundle mesh-host carries now (hq issue 146, ADR 0331)
mesh/merge-gate head 5795e03fad7b merged into trunk cbac703e817d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 5795e03fad7b merged into trunk cbac703e817d — pass: its merge-check.sh passed
mesh/delivery delivered
mesh-host #96 replaced foundation-first-node-nats.lock with foundation-first-node.lock,
and the test that holds the installer's first user list to the controller's composition
still read the removed file, so repo-check failed on main and every pull request. The
captured copy moves to mesh-host 7a41e12, with the placement guard at the same commit.
2026-10-13 01:16:10 +02:00
mesh-admin cbac703e81 Merge pull request 'Say a build handed out again restarted: its log, the queue and the check's verdict (issue 594, part)' (#280) from fix/594-a-replaced-agent-finishes-its-build into main 2026-10-12 22:55:59 +00:00
jochen 18221c4079 Judge ADR 0326 §3 only where a definition is written (hq issue 574)
mesh/merge-gate head 18221c4079d1 merged into trunk 7b2a9eae8020 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 18221c4079d1 merged into trunk 7b2a9eae8020 — pass: its merge-check.sh passed
mesh/delivery delivered
ParseManifest also reads stored builds again: a build's manifest at a
commit, the gate, a release, a module put back. betterbird's builds from
before its accounts file named its keys would have become uncomposable
there, so a send to g14 or shanks could fail and a rollback be refused.
The refusal now runs in module check, module add, a build taken in, the
builder and the merge gate's registration, and a file is no longer
reported by both §2 and §3.
2026-10-13 00:54:26 +02:00
jochen db0cb1e15e Refuse a declaring module's mergeable file that names no keys (hq issue 574, ADR 0326 §3)
mesh/merge-gate head db0cb1e15e39 merged into trunk 56d6313262e8 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head db0cb1e15e39 merged into trunk 56d6313262e8 — fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheContro…
mesh/delivery superseded: a newer head of the same pull request
A module that declares a setting and owns a mergeable file naming no keys
still let every key reach that file. The refusal was held back in #272
because betterbird was one; mesh-catalog #252 named its keys, and every
module in mesh-catalog and mesh-media-catalog main now passes, so the
module check and registration refuse it again. A stored definition is
still composed as it is, never left out of a machine.
2026-10-13 00:40:24 +02:00
jschoubben ee1d7e5cae Mark a rerun check with a short (run N) after its verdict, and word a re-hand-out neutrally
mesh/merge-gate head ee1d7e5cae12 merged into trunk 56d6313262e8 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head ee1d7e5cae12 merged into trunk 56d6313262e8 — pass: its merge-check.sh passed
mesh/delivery delivered
The forge clips a status to 140 characters, so the long sentence pushed the
verdict's own words out; and a paused holder's hand-back or a bus delay also
count as a delivery, so the cause is not claimed (novox/hq issue 594).
2026-10-13 00:31:57 +02:00
jschoubben 43cab56452 Say a build handed out again restarted, in its log, the queue and the check's verdict
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 43cab564523d merged into trunk 56d6313262e8 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 43cab564523d merged into trunk 56d6313262e8 — pass: its merge-check.sh passed
A build agent replaced mid-build leaves its ask to be handed out again, and
the second run looked like a first: the queue and the pull request showed
nothing of the 5-8 minutes lost (novox/hq issue 594).
2026-10-13 00:26:08 +02:00
mesh-admin 56d6313262 Merge pull request 'The controller's check stops waiting on fixtures and fixed intervals (issue 591)' (#279) from fix/591-the-controller-check-runs-in-minutes into main 2026-10-12 22:22:24 +00:00
jschoubben 7f3ff32361 Give a fast-renewing test lease a serving controller's time to renew, and prove the minted hash's cost
mesh/merge-gate head 7f3ff3236149 merged into trunk 21ac8fd8586d — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 7f3ff3236149 merged into trunk 21ac8fd8586d — pass: its merge-check.sh passed
mesh/delivery delivered
A renewal timed out at the renew interval, so the lease test's 1 s renewals
could lose the lease on a slow build machine (novox/hq issue 591, review of #279).
2026-10-13 00:08:48 +02:00
mesh-admin 21ac8fd858 Merge pull request 'Let a mergeable file name the setting keys it takes (hq issue 168, ADR 0326)' (#272) from fix/168-a-file-names-the-settings-it-takes into main 2026-10-12 22:07:08 +00:00
jschoubben 2819915e8e Stop the controller's tests waiting out fixed intervals
mesh/merge-gate out of date: judged against trunk e63af0df17f5, the trunk is now 21ac8fd8586d; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk e63af0df17f5, the trunk is now 21ac8fd8586d; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Reading the build events waited two seconds on every queue read, the race
detector's exit sleep cost a second per command line a test runs, and the
lease and plan-wait tests waited out production intervals (novox/hq issue 591).
2026-10-12 23:53:06 +02:00
jschoubben 7892019d9e Copy each store test's database from a template migrated once, and hash test bus passwords at bcrypt's least cost
Running ~90 migrations into a fresh database per store test, and bcrypt at
the default cost under the race detector, were most of the controller's
check (novox/hq issue 591).
2026-10-12 23:53:06 +02:00
mesh-admin e63af0df17 Merge pull request 'End a walk's apply phase at each machine's own report of its send (hq issue 582)' (#278) from fix/582-apply-ends-at-the-machines-own-report into main 2026-10-12 21:36:33 +00:00
mesh-admin 961f2e82ea Merge pull request 'Hand a delivery build out before pull-request checks (issue 558)' (#276) from fix/558-a-delivery-build-goes-first into main 2026-10-12 21:20:19 +00:00
jschoubben e727589e48 End a walk's apply phase at each machine's own report of its send (hq issue 582)
mesh/merge-gate head e727589e483b merged into trunk 961f2e82ea1c — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head e727589e483b merged into trunk 961f2e82ea1c — pass: its merge-check.sh passed
mesh/delivery delivered
The walk records its send to the rest only after every machine was sent,
which takes 20-28 s, so the machine sent first fell outside the 2 s slack,
its report was missed and the phase ended at the next walk's send to it.
Keep each machine's own send time with what it was sent, match its apply
duration by that send and digest, and show each machine's report in the
walk's phases (ADR 0282 decision 6).
2026-10-12 23:18:28 +02:00
mesh-admin 2f157148c6 Merge pull request 'node shows what a machine said it is made of, with its age (issue 160)' (#269) from fix/160-node-shows-what-a-machine-is into main 2026-10-12 20:37:06 +00:00
mesh-admin 13be75eb65 Merge pull request 'Replay issue 201 with only what the controller had before its fix (for mesh-lab R201)' (#275) from test/201-replay into main 2026-10-12 20:22:37 +00:00
mesh-admin 13711fb5b9 Merge pull request 'Mark that the build agent takes work, so its health check can see it ready (hq issue 544)' (#277) from fix/544-the-build-agent-says-it-is-ready into main 2026-10-12 19:58:09 +00:00
jschoubben fc573f951a Remove only a move's proven second copy, and never delete a handed-out original (hq issue 558)
mesh/merge-gate head fc573f951a47 merged into trunk 2f157148c6c8 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head fc573f951a47 merged into trunk 2f157148c6c8 — pass: its merge-check.sh passed
mesh/delivery delivered
Second review of #276: a dead letter delivered again beside its original
shares its request id and was removed as a copy, losing the ask; and a
publish that outlived the pause could delete an original a holder had
taken. A copy now names the message it copies, removals are said, the
publish waits inside the pause's margin, and an original handed out
meanwhile keeps its place while the copy goes.
2026-10-12 21:46:49 +02:00
mesh-admin c0c379af2b Merge pull request 'Say the class of what a change plan moves, for a fix's budget (issue 403, hq ADR 0313)' (#265) from fix/403-fix-time into main 2026-10-12 19:41:26 +00:00
jschoubben 1ded7dff34 Make a queue move lossless and run its passes on the acting controller alone (hq issue 558)
mesh/merge-gate out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Review of #276: a move deleted before it published, so a failed publish
lost an ask; a verb's own process ran a pass beside the serving one, so
one could resume the queue while the other moved asks and a build ran
twice. The copy is now published first and second copies removed by id,
asks only signal the acting controller's loop, moves stop with 5 s of
the pause left, a cancel finds a moved ask by id, and an unreadable ask
keeps its age.
2026-10-12 21:38:11 +02:00
jschoubben 50c55a773e Mark that the build agent takes work, so its health check can see it ready (novox/hq issue 544)
mesh/merge-gate head 50c55a773eb8 merged into trunk c0c379af2b3c — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 50c55a773eb8 merged into trunk c0c379af2b3c — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/544-the-build-agent-says-it-is-ready delivered: every member is delivered
The agent listens on nothing, so the node-engine could judge it only alive and
held it starting for the whole grace on every machine a walk moved it to. It
now writes a mark once it is bound to its worker, after its seat verbs are
served, and 'mesh-builder ready' passes only while the mark is there.
2026-10-12 21:33:12 +02:00
jochen daf52b3aeb Merge remote-tracking branch 'origin/main' into fix/403-fix-time
mesh/merge-gate head daf52b3aeb79 merged into trunk 4dbd4066ca54 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head daf52b3aeb79 merged into trunk 4dbd4066ca54 — pass: its merge-check.sh passed
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
mesh/delivery-group group fix/403-fix-time failed: a member failed
# Conflicts:
#	internal/link/events.go
2026-10-12 21:28:52 +02:00
jschoubben c6e234455f Hand a delivery build out before pull-request checks (hq issue 558)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
A merged change's walk waited minutes on its build behind checks in a
first-come queue. The queue now hands deliveries out first, a check that
has waited ten minutes before deliveries asked after it, and says each
ask's kind; the order is kept in the work queue itself.
2026-10-12 21:25:46 +02:00
mesh-admin 4dbd4066ca Merge pull request 'A seat names the paths its holder owns, and a claim or a declaration without the seat is refused (issue 412, hq ADR 0287)' (#267) from fix/412-a-seat-names-its-paths into main 2026-10-12 19:22:11 +00:00
jochen 45a1783ba9 Replay issue 201 with only what the controller had before its fix
mesh/merge-gate head 45a1783ba9a4 merged into trunk 13711fb5b91b — pass: the change touches no module of the mesh's graph
mesh/repo-check head 45a1783ba9a4 merged into trunk 13711fb5b91b — pass: its merge-check.sh passed
mesh/delivery delivered
The fix's own TestReplay201 reads what #273 added (sendScope.puttingBack,
recordedKept's core keep, the condition), so it cannot be laid over the
commit before. TestReplay201ThePlansSend composes a plan's send to the
control node through keepingRecorded, planFor and declarationFor and holds
that it carries the controller the machine was last sent, not the older
record; it fails on 84deb53^1 and passes on 84deb53. For mesh-lab's R201
(hq issue 201).
2026-10-12 21:17:31 +02:00
mesh-admin 84deb530ee Merge pull request 'Keep the controller and node-engine at the build a machine was last sent over an older record (hq issue 201)' (#273) from fix/201-a-push-never-downgrades-the-core into main 2026-10-12 18:52:04 +00:00
jochen b3787e4d8e Merge origin/main into fix/201-a-push-never-downgrades-the-core
mesh/merge-gate head b3787e4d8eb2 merged into trunk a681227af373 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head b3787e4d8eb2 merged into trunk a681227af373 — pass: its merge-check.sh passed
mesh/delivery delivered
# Conflicts:
#	testdata/store-tests
2026-10-12 20:31:19 +02:00
mesh-admin a681227af3 Merge pull request 'Add issue 401's replay, written with what the controller had before its fix (hq issue 401)' (#274) from fix/401-replay into main 2026-10-12 18:22:09 +00:00
mesh-admin d8f36a8fb7 Merge pull request 'Read an xz release in Go, so the build seat needs no xz tool (hq issue 345)' (#264) from fix/345-xz-in-go into main 2026-10-12 17:35:56 +00:00
jochen f0fd7fe886 Leave a kept core move out of what a gate judges (issue 201, review 91)
mesh/merge-gate head f0fd7fe8868e merged into trunk a681227af373 — fail: builds mesh-controller → novox; no bus step; head f0fd7fe8868e does not merge clea…
mesh/repo-check head f0fd7fe8868e merged into trunk a681227af373 — fail: head f0fd7fe8868e does not merge cleanly into trunk a681227af373: in conflict test…
mesh/delivery superseded: a newer head of the same pull request
gatedSend took its judged moves from the record, so on a machine kept at
a newer controller the gate judged the controller moving back to the
record, read its own send's kept controller as a later build there, and
was superseded with nothing judged. The kept moves are now left out
there as in ungatedIn (withoutKept). The doc says the bus step keeps
too, and the three store tests are listed.
2026-10-12 19:16:26 +02:00
jochen 41a9ff15dd Merge origin/main into fix/201-a-push-never-downgrades-the-core 2026-10-12 19:07:20 +02:00
mesh-admin ba685b17f5 Merge pull request 'Let plans wait for a plan to move and answer it as data (issue 229)' (#266) from fix/229-plans-wait into main 2026-10-12 17:06:50 +00:00
jochen d170491256 Keep the core over an older record instead of refusing the send (issue 201, review 90)
mesh/merge-gate out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Refusing the whole send wedged a machine whose gate put back the
controller and whose put-back send was refused: every later automatic
send there was refused, retried each tick, and held up every other walk.

A send now composes the controller and the node-engine at the build the
machine was last sent when the mesh's record is older, and carries the
rest; nothing is refused or retried, and a plan whose own build is the
older one ends with its gate superseded. A sent build that failed its
gate is never kept, so the next send completes a put-back. The condition
names the two ways out and clears on any send that carries the record,
a person's push included.
2026-10-12 18:51:17 +02:00
jochen 6bb1c9b768 Merge remote-tracking branch 'origin/main' into fix/201-a-push-never-downgrades-the-core 2026-10-12 18:40:12 +02:00
mesh-admin f8daec01d3 Merge pull request 'Read a plan by its id as a parameter, never as SQL (issue 561)' (#268) from fix/561-plan-id-parameter into main 2026-10-12 16:37:07 +00:00
jochen f3c15e00af Add issue 401's replay, written with what the controller had before its fix, so the lab can prove it
mesh/merge-gate head f3c15e00af50 merged into trunk d8f36a8fb706 — pass: the change touches no module of the mesh's graph
mesh/repo-check head f3c15e00af50 merged into trunk d8f36a8fb706 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/401-replay delivered: every member is delivered
The fix's own test reads GivenOwnSecret, which the commit before does not have, so it cannot be laid over it. TestReplay401 asserts the same outcome with register, assign, planFor, declarationFor and the inventory's secret reads; it fails on 1e28342^1 and passes on main. Named in testdata/store-tests so a skip without the store fails the check. Replay for hq issue 401.
2026-10-12 18:36:22 +02:00
jochen 9038d4b0d3 Refuse a send that puts the controller or node-engine behind its machine's build (issue 201)
mesh/merge-gate out of date: judged against trunk f10aedac5880, the trunk is now f8daec01d3f0; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f10aedac5880, the trunk is now f8daec01d3f0; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
On 2026-10-02 a plan sent the control node a controller older than the one
running there, whose seat row the newer build had widened; the older binary
refused the row and the mesh went quiet for ten minutes. ADR 0185 took the
outage out of it, and the race stayed open.

Every send but a person's push now refuses a move of mesh-controller or
mesh-host to a build asked before the one its machine was last sent (the
builds' registration order, issue 219), whatever composed it — gated or not.
A gate's own put-back is the one exception, for the modules it puts back,
and says so in its send's scope. The refusal is the send's error, which the
plan keeps as its note, and a warning condition on the machine, cleared by
the next send there that carries no such move.
2026-10-12 18:21:17 +02:00
mesh-admin f10aedac58 Merge pull request 'A merge check judges the head merged into its trunk, and names the trunk commit (hq ADR 0323, issue 341)' (#271) from feat/341-a-merge-check-judges-the-merge into main 2026-10-12 16:11:27 +00:00
jochen b538397055 Name the issue that holds the refusal back (hq issue 574)
mesh/merge-gate head b5383970550b merged into trunk e63af0df17f5 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head b5383970550b merged into trunk e63af0df17f5 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 18:00:49 +02:00
jochen dd10889209 Hold back refusing a declaring module's silent mergeable file (hq issue 168)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
betterbird declares a preference and owns a mergeable file that names no
keys, and the catalogue can only name them once this controller runs: the
refusal made the catalogue check fail against catalogue main. It follows
once mesh-catalog names the keys.
2026-10-12 18:00:33 +02:00
jschoubben 875ebc093e Format PullUpdated, and keep the announced base head on a rechecked member
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/341-a-merge-check-judges-the-merge delivered: every member is delivered
A head checked again through mesh-delivery reached the direct verdict with no
base head, so a head no build judges was answered with no trunk commit and the
merge tool would refuse it (novox/hq ADR 0323).
2026-10-12 17:52:12 +02:00
mesh-admin 1e28342ede Merge pull request 'Make no value for an own secret issued outside the mesh that nobody gave (issue 401)' (#262) from fix/401-an-outside-secret-not-given-is-not-made into main 2026-10-12 15:48:12 +00:00
jochen 286257d802 Let a mergeable file name the setting keys it takes (hq issue 168, ADR 0326)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestModuleCheckPassesTheCatalogue (0.89s); and TestEveryCatalogueManifestParses, TestNoCatalogue…
mesh/delivery superseded: a newer head of the same pull request
Every key of a module's layers merged into every mergeable file it owns:
the mesh's own words, a key meant for a ${setting:} in another file, a
key for a second configuration file. A module could not own two
configurable files. A file now names its keys in takes-settings and
takes only those; a key no file names is stray where it is set; a
module that declares a setting names the keys of every mergeable file,
and one that declares none keeps today's form (ADR 0164 §15).
2026-10-12 17:47:05 +02:00
jschoubben bc623bc1f9 Name the announced trunk commit on a verdict no build judges
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: not gofmt'd by the toolchain's gofmt: internal/link/events.go
mesh/delivery-group group feat/341-a-merge-check-judges-the-merge rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
A change touching no module, in a repository with no merge-check.sh, was
answered by the controller with no trunk commit, so the merge tool refused it,
asked again and refused it for ever (novox/mesh-tools-go; novox/hq ADR 0323).
2026-10-12 17:44:18 +02:00
jochen 67d54d9d09 Say an account without its time as age unknown, not as two thousand years (issue 160)
mesh/merge-gate head 67d54d9d09b2 merged into trunk 13be75eb65c0 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 67d54d9d09b2 merged into trunk 13be75eb65c0 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 17:43:38 +02:00
jschoubben ce9dc8e157 Run no hook and no signature setting when a check makes its merge
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
A build machine's or a clone's git configuration must not run code or change
the merge a check judges (novox/hq ADR 0323).
2026-10-12 17:39:59 +02:00
mesh-admin c672b3df6d Merge pull request 'Stop composing ADR 0312's group grant, so deliveries to the media machine pass again (hq issue 567)' (#270) from fix/567-revert-the-group-grant into main 2026-10-12 15:36:12 +00:00
jschoubben 25deb499bf Say why a merge's changed files read as none on an error
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 17:31:36 +02:00
jschoubben 83a9cf6e3a Judge a pull request's head merged into its trunk, and name the trunk commit
A check judged the head alone, so two pull requests that each passed could
break the trunk together (node-engine #66 on 2026-10-09), and a recheck after
the trunk moved judged the same head again. The build seat now merges the head
into the trunk's head in its own clone, never pushed, fails a head that does not
merge cleanly naming the files, and the verdict names the head, the trunk
commit and the commits read beside it, for the forge and mesh-delivery to weigh
against the trunk's next move (novox/hq ADR 0323, issue 341).
2026-10-12 17:27:01 +02:00
jschoubben 7b050abdcb Stop composing ADR 0312's group grant, so deliveries to the media machine pass again
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Since #256 delivered, the node-engine judges every read-write access by the path's group alone and
states six media modules unhealthy on ace, though they write their folders as the owner; every
delivery there fails its gate (hq issue 567). ADR 0325 replaces ADR 0312; until it is built, the
controller sends no grant, as before #256.

This reverts commit 3cfc121670.
2026-10-12 17:25:46 +02:00
jochen 7953d2889d Show what a machine said it is made of in node, with its age (issue 160)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The architecture and kernel a machine reported were kept and read by nothing, and the facts the
node-engine now says (memory, filesystems, processor, distribution, virtual or physical, uptime,
timezone) would have joined them. node shows every one in one block with the age of the account,
and a fact the machine could not determine as unknown with its why, never as a zero.
2026-10-12 17:23:07 +02:00
jochen c3b4f187e1 Name issue 401's two store tests, so a skip without the store fails the check (issue 459's rule)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
2026-10-12 17:15:18 +02:00
jochen edd33f21d2 Make no value for an own secret issued outside the mesh that nobody gave (issue 401)
The send minted a random value for a plain own secret marked issued-by outside and placed it, so a
module could not tell "not given" from "given and wrong" (ADR 0228). It now places only a value a
person gave: not given is no file, an empty value where a module's file names it, and nothing to
restart on, as ADR 0283 decided for a family's member. A value the mesh made for one earlier is not
read as given.
2026-10-12 17:15:12 +02:00
jochen 24347ea5a0 Step back from a rendered resolver file only, and compare a declared path cleaned (issue 412, review 80)
mesh/merge-gate head 24347ea5a02b merged into trunk 84deb530ee5a — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 24347ea5a02b merged into trunk 84deb530ee5a — pass: its merge-check.sh passed
mesh/delivery delivered
A resource on the resolver file was left out by the check and still composed by the declaration, so an
uplink declaring it as a resource beside the resolver passed with two writers; it is refused again, as
before. And /etc//hosts or /etc/../etc/hosts is /etc/hosts to the machine, so it is compared cleaned.
A seat names no root or top-level directory.
2026-10-12 17:08:55 +02:00
jochen 5ad36127ee Hold the plans schema's longest bound to the value, and say what a slow store gives (review 75)
mesh/merge-gate head 5ad36127ee8b merged into trunk f8daec01d3f0 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 5ad36127ee8b merged into trunk f8daec01d3f0 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 16:50:59 +02:00
jochen a1a6f6493d Refuse a plans argument that is not a plan's id (issue 561)
mesh/merge-gate head a1a6f6493d50 merged into trunk f10aedac5880 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head a1a6f6493d50 merged into trunk f10aedac5880 — pass: its merge-check.sh passed
mesh/delivery delivered
Every argument of plans that names a plan (id, wait, stop, close, retry,
go) is checked once where the call becomes a command line, against the
ids the mesh gives plans (plan-<digits>, release-<digits>), so a quote
from a caller never reaches a command or the store (review 75 on #266).
2026-10-12 16:46:29 +02:00
jochen e9f060e0be Read a plan by its id as a parameter, never as SQL (issue 561)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
PlanByID pasted its id into the query, and the id comes from a caller of
the plans verb: an id holding a quote changed the query (x' or '1'='1
answered another plan). The id is now $1, and RecentPlans's limit too, so
no plan query is built from a value.
2026-10-12 16:33:23 +02:00
mesh-admin 3cfc121670 Merge pull request 'Compose each read-write access's grant for the node-engine (hq ADR 0312, issue 537)' (#256) from feat/537-a-read-write-access-grants-the-paths-group into main 2026-10-12 14:20:27 +00:00
jochen a3bf5da9b1 Name the paths a seat's holder owns, and refuse a claim or a declaration without the seat (issue 412)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Which module owns /etc/hosts or /etc/resolv.conf on a machine was answered only by reading every
manifest, and a module not holding node-hostname could declare /etc/hosts where nobody held it.
hq ADR 0287 makes the paths part of the seat's definition: node-hostname names /etc/hosts and
/etc/hostname, node-uplink names /etc/resolv.conf stepping back for node-resolver, node-resolver
names /etc/resolv.conf.

CanHold refuses a holder that does not declare every path (registration and handover), registration
refuses a module declaring a seat's path while claiming none of the seats naming it, a module's own
seat names paths under the same rules (node only, literal, not named by another seat), composition
refuses a generated resource on a seat's path, the uplink's step-back is read from the seat instead
of a special case, and `seats` shows each seat's paths and who owns each on each machine.
2026-10-12 16:19:49 +02:00
jochen d20b52cb19 Let plans wait for a plan to move and answer it as data (issue 229)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
An agent following a rollout had plans answer once, in prose, and polled the
console with a curl loop of its own to be told a plan moved. plans now takes
wait (a plan's id), until (moved, tier or done) and bound, and answers when the
plan moves or the bound passes, saying which, with the plan as data and its
line as the output.

A call's one answer has to reach the console inside its thirty seconds, so
link.AnswersWithin lets a handler that waits on purpose hold its answer up to
LongestAnswer (25s) instead of AnswerWithin; the bound is at most 20s, and a
caller following a plan further asks again.
2026-10-12 16:18:54 +02:00
jochen 7ca5a45ecc Class a new module by its own name, so adding a core module is not called leaf (issue 403, review 69)
mesh/delivery-group group fix/403-fix-time checking: composed check out of date: novox/mesh-controller@7ca5a45ecc5a was judged against trunk ?, the trunk is no…
mesh/merge-gate out of date: judged against trunk 84deb530ee5a, the trunk is now 4dbd4066ca54; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 84deb530ee5a, the trunk is now 4dbd4066ca54; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 16:16:02 +02:00
mesh-admin db7e4869ae Merge pull request 'Raise a machine that refuses or only partly applies its declarations past one apply interval (issue 233)' (#263) from fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 into main 2026-10-12 14:11:10 +00:00
jochen 539a1c16cd Refuse an xz file of more than 1,024 blocks before decoding it (review 68, hq issue 345)
mesh/merge-gate head 539a1c16cd4e merged into trunk ba685b17f5e1 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 539a1c16cd4e merged into trunk ba685b17f5e1 — pass: its merge-check.sh passed
mesh/delivery delivered
The xz reader keeps one delta or BCJ decoder for every block of a stream and frees them only when the
stream ends, so a file of many tiny blocks grows the heap with its length: about 67 MiB live for a 6 MB
`xz --block-size=1 --delta` file, about 5.6 GiB at the 512 MiB download cap.

The file's framing is now walked first, without decompressing: stream header, each block header, the
LZMA2 chunk headers (each carries its compressed size, so a chunk is skipped unread), padding, check,
index and footer. A file of more than 1,024 blocks or 16 streams is refused, as is an index whose block
count is not the count walked. The index alone is not trusted because the reader compares it with the
blocks only after decoding them all, so a lying index would still be decoded. Refusing delta and BCJ
outright was not chosen: it needs the same walk to see each block header, and would refuse sound
releases filtered with BCJ.

1,024 blocks keep at most about 0.35 MiB of decoders. Betterbird's archive is 1 stream with 1 block;
multi-threaded xz cuts blocks of 3 times the dictionary, so 512 MiB in blocks of 512 KiB still fits.
The library stays unforked.
2026-10-12 16:08:33 +02:00
mesh-admin 9eddf36503 Merge pull request 'A directory's owner may follow the account its module runs as (issue 279)' (#260) from fix/279-owner-follows-a-setting into main 2026-10-12 14:00:23 +00:00
jochen 05eccbf22b Say the class of what a change plan moves, so a fix's budget follows its modules (issue 403)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/403-fix-time ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
The issue tracker holds a fix to the budget of the slowest-budgeted module it
moves, and gives none to a fix that moves no module (hq ADR 0313 §2). The
walk's class covers every merge a walk carries; the plan's class is the
change's own: core, leaf, or none.
2026-10-12 15:50:11 +02:00
jochen 01fc9e1d73 Read xz with a real dictionary maximum (review 65, hq issue 345)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
ulikunitz/xz's ReaderConfig.DictCap is a minimum: a stream asking for a
1 GiB or 4 GiB dictionary was given it. github.com/therootcompany/xz, a port
of xz-embedded, takes a maximum and refuses a block whose LZMA2 dictionary is
larger before allocating it. 64 MiB, xz's largest preset.
2026-10-12 15:39:34 +02:00
jochen d4de4543cc Judge S21 by time since the first failing report while the machine is heard, not by how many reports said it (issue 233 review)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 delivered: every member is delivered
2026-10-12 15:39:08 +02:00
jochen a371de37fe Keep a directory owner's settings at the controller's terminal (issue 279)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Root chowns a directory to the number its owner's setting names, so a key
an owner asks for must not change through the settings verb: any caller
could have set puid to 0 or to its own uid. TerminalKeys now takes every
${setting:…} in a directory's owner, whatever else asks for it, and the
owner is written as the parsed number so "0001" never reaches the
node-engine as text (review 60).
2026-10-12 15:37:29 +02:00
jochen 77acb40092 Let a directory's owner follow the account its module runs as (issue 279)
A media manager runs as the puid/pgid its assignment sets per node, but a
declared directory's owner was literal, so a folder the app must write was
refused on a machine whose identity was not the image's default. A
directory's owner now takes ${setting:<key>} (user, or user:group), filled
at composition and refused unless the value is a whole number, and the
module check refuses any other shape or a key the module asks for nowhere
else.
2026-10-12 15:37:23 +02:00
mesh-admin 5facd8b08e Merge pull request 'An act's answer is said from one resolution after its write (hq issue 237)' (#253) from fix/237-one-resolution-for-assign into main 2026-10-12 13:31:03 +00:00
mesh-admin 72edb458cc Merge pull request 'The node-reads seat: its verbs, its holder's own account, and the node's secret places (hq ADR 0310, issue 418)' (#257) from fix/418-node-reads into main 2026-10-12 13:20:23 +00:00
jochen 10f76cafc4 Read an xz release in Go, so the build seat needs no xz tool (hq issue 345)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Betterbird's pinned .tar.xz failed on the build seat with "this build seat has
no xz to read it": the seat's image is the catalogue's build-agent recipe,
which installs git and docker-cli alone, and xz had been added only to
cmd/mesh-builder/Dockerfile, which the seat does not build from. The stream
is now read with github.com/ulikunitz/xz (vendored), its dictionary capped
at xz's own largest preset, and what it unpacks to bounded as before; the
xz line in the unused Dockerfile is taken back out.
2026-10-12 15:10:52 +02:00
mesh-admin 2843834813 Merge pull request 'The build queue answers as data too (issue 455)' (#259) from fix/455-the-queue-answers-as-data into main 2026-10-12 13:04:26 +00:00
jochen 3e10ad1535 Raise a machine that refuses or only partly applies its declarations past one apply interval, naming the package manager and its words, so the mesh says it rather than only status (issue 233)
mesh/delivery-group group fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 ready: every member ready, and composed together they pass
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 14:49:33 +02:00
mesh-admin 27b13e4586 Merge pull request 'Ask a node's engine which of its directories the walk would refuse, read-only (issue 358)' (#258) from fix/358-judge-directories-read-only into main 2026-10-12 12:48:56 +00:00
mesh-admin 142a5ad71d Merge pull request 'An archive may be an upstream release pinned by URL and sha256, mirrored by the build seat (hq issue 345, ADR 0316)' (#255) from fix/345-builder-archive-pins into main 2026-10-12 12:33:58 +00:00
jochen 63a99d4323 Name the vault's placed directories among a node's secret places, fail loudly on one that cannot be resolved, and keep the reads account out of every group (review 54, issue 418)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/418-node-reads delivered: every member is delivered
The real mesh-vault manifest declares its directories without a path, so the
list left its state and ledger out, and a place still holding a placeholder
vanished from it without a word. The provider's directories are now resolved
where the node places them, an unresolved place fails the composition naming
the module, and a node-reads holder whose account is put in a group is refused.
2026-10-12 14:30:00 +02:00
jschoubben 0cd8bc0bfb Answer the build queue as data too, so a check can say how far back it waits
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The queue verb answered only a table, so nothing outside the controller
could say an ask's place in the queue. It now asks its command for JSON:
each ask's state, kind, repository, commit, machine, since and position
among the waiting, with the table carried beside it and answered as the
verb's output, both written from one read. A repository's password, if
one were ever written into it, is left out of what is shown.

novox/hq issue 455
2026-10-12 14:22:22 +02:00
jschoubben 7385d9c092 Ask a node's engine which of its directories the walk would refuse, read-only (issue 358)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/358-judge-directories-read-only delivered: every member is delivered
Before the node-engine gives a mesh directory its mode and owner only through
a walk that follows no link another account could make, the operator is to
see which directories that walk would refuse on every node, the control-node
included, where no tool reads a path. `node judge-directories <node>` signs
the ask as a hand-over is, under its own context, and prints the engine's
one line per directory; it changes nothing, so it is a seat verb an agent may
call. The node is granted the subject, the controller alone its publish.
2026-10-12 14:17:01 +02:00
mesh-admin db9e9c5b87 Merge pull request 'Advise a recheck for an unanswered check on a ready or rejected delivery (issue 451)' (#254) from fix/451-a-recheck-is-the-step into main 2026-10-12 12:14:43 +00:00
jochen 4ead5a3df7 Define the node-reads seat, refuse a holder without an account of its own, and tell it where its node keeps a secret (hq ADR 0310, issue 418)
mesh/delivery-group group fix/418-node-reads rejected: a member's own check failed
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Reading a file, searching one, listing a directory, the owner and mode of each
component of a path and looking up an account had no mesh tool, so they went
through the login shell's execute as the operator's account. node-reads is a
node seat with six verbs, each saying the commands it replaces. Its holder must
run as an account of its own (ADR 0310 §3), and its own runtime is given
MESH_SECRET_PLACES: the directory of every module's own secret, every credential
file and grant directory, the vault's directories and the sealed-secret keep,
so it refuses them by name before any read (§5).
2026-10-12 13:57:24 +02:00
jschoubben a23b190747 Merge remote-tracking branch 'origin/main' into fix/451-a-recheck-is-the-step
mesh/merge-gate pass: builds mesh-controller, route-proxy → ace, novox; no bus step; every machine composes with the change as it did without (4 of 4 compo…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/451-a-recheck-is-the-step delivered: every member is delivered
2026-10-12 13:56:52 +02:00
jochen 3be6e86f77 Say an act's answer from one resolution after its write (issue 237)
mesh/merge-gate pass: builds mesh-controller, route-proxy → ace, novox; no bus step; every machine composes with the change as it did without (4 of 4 compo…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
assign read the whole mesh before its write, to record the seats held by
derivation, and that read printed on standard error that the node did not
resolve for lack of the seat the act was about to give it. The seat verb
answers with both streams, so the stale line came last and named the
assignment just made as its remedy.

The reads inside an act no longer print 'left out' lines; the node's
unmet seat dependencies and whether it resolves come from one planFor
taken after the write, and every other machine from blockedElsewhere.
unassign now says when its node does not resolve, from the same
resolution.
2026-10-12 13:53:19 +02:00
jochen b0c4b849a7 Judge an upstream pin's host like any name, and harden the upstream unpack (review 35, hq issue 345)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The url was skipped by the installation check, so an installation's host or
an address passed; it is judged whole now, the vendor's host declared in the
artifact's names-on-purpose, and an address or a reserved name as the host
is refused. Only the version's dotted-quad reading is skipped. The unpack
caps its entries, drains xz before waiting for it, reads a pax global
header and a nested within's parents, and a redirect off https is not
followed.
2026-10-12 13:51:35 +02:00
jochen b849af53dd Let an archive be an upstream release pinned by URL and sha256, mirrored by the build seat (hq issue 345, ADR 0316)
A mail client's release is about 80 MB per version, too large to keep in a
repository at every upgrade, and an add-on whose licence forbids copying it
to third parties may not be in a repository at all. An archive artifact may
now name url, sha256 and version (and file or within) instead of from: the
build seat fetches it once, refuses it when its digest is not the pin's and
names both, keeps it in its workspace under the digest (a vendor that serves
only its newest version cannot break a later build of the same pin), and
packs it like any archive, so a node gets the mesh's own archive by digest
and never asks the vendor. The build seat's image carries xz, which Go's
standard library does not read. The installation check takes the vendor's
address and version of such a pin for the world's, as a registry is an
image's.
2026-10-12 13:51:35 +02:00
jschoubben 74bd90df7e Pin the node-engine at its merge of the access grant, and capture it beside (hq ADR 0312, issue 537)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
2026-10-12 13:47:28 +02:00
jschoubben 30599c1858 Pin the node-engine at its reviewed grant, and hold the mount match to the path itself (hq ADR 0312, issue 537 review)
A path beginning with the same letters is another path, and a read-only mount
of a read-write access is still granted: the grant follows what mounts it.
2026-10-12 13:47:17 +02:00
jschoubben 10d4712f82 Name who a read-write access grants its path's group to, so modules write shared data through the group the operator gave it (hq ADR 0312, issue 537)
Each read-write access is composed with a grant: the module's containers that
mount its path and the account its tools run as (the operator's, or the module's
own under ADR 0259 §8); that runtime restarts on the access, so it holds a group
the engine put its account in. A read access grants nothing. Sent only to an
engine whose statement says GrantContract (4), since an older one refuses the
field. Pins the node-engine at the branch that reads it.
2026-10-12 13:47:17 +02:00
jschoubben e737e13bbb Advise a recheck for an unanswered check on a ready or rejected delivery (issue 451)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.66s)
mesh/delivery-group group fix/451-a-recheck-is-the-step rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
That check answered once, so the build queue is the wrong place to look.
mesh-delivery names the delivery's state as data (held); the controller
reads it and never the words. A line without it keeps the advice of 438.
2026-10-12 13:47:11 +02:00
mesh-admin 4b1dee6b79 Merge pull request 'A merge that shares nothing with the assembling batch, and whose group waits for no one, is cut at once (hq issue 525, ADR 0304 §1, ADR 0306 §2-3)' (#250) from feat/525-a-lone-merge-skips-the-window into main 2026-10-12 11:46:59 +00:00
mesh-admin 4ab441ce16 Merge pull request 'Ask only once the bus enforces the controller's own grant, judged for real and failing closed (review of hq issue 353)' (#188) from fix/353-the-asker-judges-the-controllers-own-grant into main 2026-10-12 11:34:14 +00:00
jochen f996cf7b6b Take an older bus module's allowed as held, and name only remedies that work (review 31 of PR 188, hq issue 353)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
After mesh-catalog #166 merges and before the bus upgrade, the bus's
machine cannot be pushed, so any change to the composed list made the
list check read behind for good: every ask was withheld and the condition
named a push that is refused. An older module's allowed now counts as
held, the unconfirmed reload said once; the worst case is one ask refused
in the seconds before a reload, which is asked again (issue 369).

The list check stays for a module that says not allowed or cannot answer,
and its remedy is the plan's own reason when the list cannot be composed,
the bus upgrade while sends there wait on it, and a push otherwise. An
in_force false is a wait only when the module gives both times; otherwise
what is not known is named.
2026-10-12 13:21:32 +02:00
jochen bcfcfaaaf3 Keep asking on an older bus module and name the real remedy for each grant not held (review of hq issue 353)
A bus module older than its in_force answer read as not held, so a merge
before mesh-catalog #166 is live would silence every ask until a person's
bus upgrade. Such an answer now falls back to the check main makes (is the
list sent to the bus's machine the one composed now), says the reload is
not confirmed, and each case names its own remedy: push <node> for a list
behind or not allowing it, a ~5 s wait for a reload, the bus upgrade when
an older module's list cannot be judged.
2026-10-12 13:21:32 +02:00
jschoubben a3181e356b Ask only once the bus enforces the controller's own grant, judged for real and failing closed (review of hq issue 353)
change held asking back, a list sent was taken for a list the bus enforces, a plan that could not be
worked out read as not behind, and no test held the real judgement. Now the asker asks only when the
composition grants the controller itself to publish its ask, and the bus's own module says that grant
is in force on the running server (nats_user_can's in_force, mesh-catalog); each fact not known is not
held, said with why. brokerBehind takes a list nobody could compose as behind.
2026-10-12 13:21:32 +02:00
mesh-admin dd2fd28c92 Merge pull request 'closedNames waits for the verb's own connection by name, not the first closed one (issue 543)' (#252) from fix/543-wait-for-the-verbs-own-connection-by-name into main 2026-10-12 11:21:18 +00:00
mesh-admin 9d165c7500 Merge pull request 'plan-moved says what a walk touches, its bound and its outcome when it starts and ends (ADR 0308, issue 532)' (#247) from adr-0308-walk-touches into main 2026-10-12 11:08:53 +00:00
jochen 7630072e63 Wait for the verb's own connection by name, not the first closed one (issue 543)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
closedNames returned as soon as the bus's closed-connections list held
anything, which could be the test's own setup connection closing before
the verb subprocess's connection did. That raced TestAVerbsOwnProcessNamesItsConnection
red about once in 80 runs. It now waits until the connection it is named
for appears, or the deadline passes.
2026-10-12 13:05:15 +02:00
jochen 1e9dcf0667 Say every walk a verb keeps, and clear a stop from a walk that walks again
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
cancel, clear and rebuild kept a walk's end or restart without saying it, as plans stop, retry and
release-backlog did before them. The controller's run now says every save a verb makes on the bus, so the
next verb cannot lose a mark the same way. A stop is cleared when a walk is resumed, so one that fails again
ends failed and the search for its merge runs (review 21, ADR 0308, issue 532).
2026-10-12 12:54:20 +02:00
jochen cd6610cdf7 Say a walk's start or end when a person stops, closes, retries or releases it, and a hand stop as stopped
plans stop|close, plans retry and upgrade release-backlog kept a walk's start or end mark without saying it,
so the router would hold what the walk touched until its bound; each now runs on the bus as plans go does.
A hand stop keeps who stopped it, so its outcome is stopped, not failed (review 15, ADR 0308, issue 532).
2026-10-12 12:54:20 +02:00
jochen facc551ff2 Say what a walk touches, its bound and its outcome when it starts and ends (ADR 0308, issue 532)
The router holds what an open walk touches and tells its result once; it can do so
only if plan-moved names the walk's start and end, the modules and machines it moves,
what a moved tool runner or node-engine carries there, and how long to hold.
2026-10-12 12:54:20 +02:00
jschoubben 71b3aeaf2c Keep a group member of the other kind's batch from being overtaken (hq issue 525)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A merge's group member counted as merged when its merge waited in any batch,
but kinds never share a batch: the later member got a batch of its own, cut
at once, and walked before the earlier member's batch was cut, breaking the
group's order (ADR 0276 §5). Only a member kept in a batch of the merge's own
kind counts as merged now; one kept in the other kind's keeps the merge
waiting, and its batch names that member, so the two are cut in the order
their windows close (review of #250).
2026-10-12 12:54:11 +02:00
mesh-admin fafa3fd9ef Merge pull request 'Require the service manager's reset-failed, wanted-by and unlink-dangling verbs (issue 332)' (#248) from fix/332-require-service-manager-verbs into main 2026-10-12 10:53:20 +00:00
jschoubben 6e5a1df575 Judge the installer's user list against mesh-host with the groups grant (hq issue 525)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.78s)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
The controller now asks mesh-delivery.groups, and mesh-host a76baa4b carries
that grant in the installer's first user list (mesh-host #85). The captured
copy is moved to that commit so the genesis test judges the list that ships.
2026-10-12 12:31:50 +02:00
mesh-admin d881f4764d Merge pull request 'Keep a condition's looks in the store, so a look that changes nothing writes no bucket (issue 420)' (#249) from fix/420-conditions-bucket-on-transition into main 2026-10-12 10:27:26 +00:00
jochen 61d00a234a Give the store tests' service manager holders the three verbs it now must serve (issue 332)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
They named only the eight verbs from before issue 332 and were refused once the seat required
reset-failed, wanted-by and unlink-dangling.
2026-10-12 12:25:05 +02:00
jschoubben 4d203b5a5d Ask mesh-delivery whether a merge's group still assembles (hq issue 525)
A merge from any head waited for the window because the controller could not
see a group's members, so the window skip changed nothing in production and
unrelated merges walked one after another. The controller now asks
mesh-delivery's groups, within 3 s and before the plans are held: a merge
waits only when its head's group has another member whose pull request is
open and whose merge it does not keep yet (ADR 0306 §2). No answer means it
waits, and its batch says why. A headed merge in no waiting group and related
to no batch is cut at once, as one from no head is (ADR 0304 §1, ADR 0306 §3).

The controller's grant names mesh-delivery.groups. The installer's first user
list in mesh-host must carry it too before the genesis test can pass.
2026-10-12 12:14:46 +02:00
jschoubben 6196a4815e Let a merge that shares nothing skip the merge window (hq issue 525)
A merge waited the whole window even when nothing assembling shared anything
with it. It now joins only a batch it is related to (ADR 0304 §1, read by ADR
0306 §3: repository, a module the cut would move, a graph edge, a group head),
folding the newer into the oldest when it relates to two. A merge related to
none starts a batch whose window is closed at once (PlanBatch.CutAtOnce),
unless its delivery group may still have a member to merge (ADR 0306 §2).

The controller cannot see a group's unmerged members (that is mesh-delivery's
record), so a merge from any head still waits, in a batch of its own: the safe
error is to wait. Walks stay one at a time; a batch cut at once queues behind
an open walk. A stop's merges walked again join a related batch, or their own.
2026-10-12 12:14:46 +02:00
jochen dcdafd37f0 Keep a condition's looks in the store, so a look that changes nothing writes no bucket (issue 420)
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
Every look rewrote the condition's value in the conditions bucket, because the
value carried the count of looks, the last look and the newest evidence: 131 647
writes in four days for twelve open conditions, against ADR 0289's rule that a
bucket is written only when what it describes changed. The looks now live in the
controller's store (migration 0095, condition_looks), the bucket holds what a
condition says and is written only when that changes, and the conditions verb and
the condition events put the looks back.
2026-10-12 12:11:13 +02:00
mesh-admin f549647d18 Merge pull request 'Give the unheld log test its own writer and memory, so it passes on a repeat (issue 486)' (#246) from fix/486-the-unheld-log-test-stands-alone into main 2026-10-12 10:10:39 +00:00
jochen 0a0f5b4a5a Require the service manager's reset-failed, wanted-by and unlink-dangling verbs now every holder serves them (issue 332)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestAVerbsOwnProcessNamesItsConnection (4.62s); and TestASeatsVerbGainsTheArgumentsTheBinaryNames
mesh/delivery superseded: a newer head of the same pull request
ADR 0246 step 3: the systemd module, the seat's only holder, serves all three on every node,
so a holder that stops serving one is refused rather than leaving a node without them.
2026-10-12 12:03:37 +02:00
mesh-admin 22a2f73f7b Merge pull request 'plans names the hold a waiting walk waits on (issue 529)' (#245) from fix/529-a-waiting-walk-is-answered-or-named into main 2026-10-12 09:55:17 +00:00
mesh-admin d854598d8c Merge pull request 'Wait out a forge not served for a moment when registering a build (issue 486)' (#244) from fix/486-the-forge-is-waited-out into main 2026-10-12 09:43:50 +00:00
jschoubben ac5d34f613 Give the unheld log test its own writer and memory, so it passes on a repeat (novox/hq issue 486)
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
logUnheld kept what it had logged in package state and the test swapped os.Stderr, so a second run in one process logged nothing and failed.
2026-10-12 11:41:41 +02:00
jschoubben b03ec3b64e Offer no hand start for a walk that is only waiting its turn (issue 529)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/529-a-waiting-walk-is-answered-or-named delivered: every member is delivered
A hand start there would skip its group's order; point at it only for a
walk nothing holds, and say why nothing holds it.
2026-10-12 11:34:54 +02:00
jschoubben 4aacd98f8d Cap the forge wait as a whole, and wait out only a forge not served (novox/hq issue 486)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The registration holds the controller's one receive loop. Checking the bound
only between asks let an ask in flight hold it near two minutes, the
deaf-controller limit. The whole wait now runs under a 60 s deadline, still
says it waited when that deadline ends it, and a tool that did not answer in
time is not asked again: it has already held the loop its whole wait.
2026-10-12 11:32:14 +02:00
jschoubben ed037784e2 Name the hold a waiting walk waits on in plans (issue 529)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group fix/529-a-waiting-walk-is-answered-or-named checking: 0 of 2 member(s) ready
A walk whose delivery was held for a person said only that it waits for
mesh-delivery's word and that plans go starts it by hand, pointing the
reader at a hand start instead of the person's decision. Ask mesh-delivery
what holds each waiting walk and say the held delivery, its reason and
mesh-delivery.release, or the delivery not at its turn.
2026-10-12 11:27:05 +02:00
jschoubben 47205403d3 Wait out a forge not served for a moment when registering a build (novox/hq issue 486)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A registration asked the forge while node-tools re-served after a tool-runner
rollout, heard "nothing serves gitea.gitea_api", and failed a good build's
plan. The bus's not-served answer is now link.ErrNothingServes (and a timeout
link.ErrNoAnswer), and the registration asks again for up to 90 s on those
alone, never on a refusal; past the bound it fails saying how long it waited.
2026-10-12 11:23:03 +02:00
mesh-admin da1473e9db Merge pull request 'Say a unit's restart count as what it counts, not since its state began (issue 458)' (#243) from fix/458-restarts-say-what-they-count into main 2026-10-12 08:56:34 +00:00
jschoubben cde0dc287b Say a restart count runs from when the node-engine began watching it
mesh/delivery-group group fix/458-restarts-say-what-they-count delivered: every member is delivered
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The engine reports its own count, kept across recreates and starts by hand, not
systemd's NRestarts or the runtime's RestartCount (novox/hq issue 458).
2026-10-12 10:46:38 +02:00
jschoubben 63079e2c67 Say a restart count is automatic restarts since the last start or creation
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
NRestarts and RestartCount reset on a start by hand or a recreation, so
"whole life" was as wrong as "since" (novox/hq issue 458).
2026-10-12 10:45:40 +02:00
jschoubben 0d08d5c730 Say a unit's restart count is its whole life, not since its state began
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The count comes from systemd's NRestarts or the runtime's restart count; beside
"since" it read as thousands of restarts tonight (novox/hq issue 458).
2026-10-12 10:42:19 +02:00
mesh-admin 3bd984c41e Merge pull request 'Let a module keep a log the controller creates and never removes, and warn before a bucket or log is full (ADR 0297)' (#231) from adr-0297-module-logs into main 2026-10-12 00:42:55 +00:00
mesh-admin ed1b106c19 Merge pull request 'Say what relates two changes, before anything acts on it (hq issue 525)' (#242) from feat/525-what-relates-two-changes into main 2026-10-12 00:35:18 +00:00
jochen 6b77dd96f4 Merge remote-tracking branch 'origin/main' into adr-0297-module-logs
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 02:29:00 +02:00
jschoubben 9460689fff Relate a batch whose moves are not known to every change
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The review found that a merge kept without its moves, judged again at
the cut, gave its batch no modules, so it could read as unrelated to a
walk it will build beside. An unknown change is never unrelated.
2026-10-12 02:06:54 +02:00
jschoubben 96005a8bfd Say what relates two changes, so unrelated ones can walk beside each other
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
The helper decides, by repository, module (widened to dependents), delivery
group head or module graph edge, whether two changes are related, as novox/hq
ADR 0304 decision 1 reads in ADR 0306 decision 3. Nothing calls it yet; it is
the first step of issue 525's parallel walks.
2026-10-12 02:04:38 +02:00
mesh-admin b12da96906 Merge pull request 'Hold a leaf delivery to three minutes (hq ADR 0304 decision 4, issue 525)' (#241) from feat/525-leaf-budget-three-minutes into main 2026-10-12 00:02:24 +00:00
mesh-admin eba0a301cc Merge pull request 'Judge a leaf module on one healthy reading, a core module on three (hq issue 524)' (#240) from fix/524-judgement-by-class into main 2026-10-11 23:52:00 +00:00
jschoubben e6bd8f1dee Say a leaf delivery's budget as three minutes in the times verb's description
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/525-leaf-budget-three-minutes delivered: every member is delivered
2026-10-12 01:46:26 +02:00
jschoubben 5b9c81bf26 Hold a leaf delivery to three minutes (hq ADR 0304 decision 4, issue 525)
mesh/delivery-group group feat/525-leaf-budget-three-minutes checking: 0 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
ADR 0304 lowered a leaf module's delivery budget from five minutes to
three, the time its measured phases support once a leaf is judged on
one reading and waits only for what it shares.
2026-10-12 01:44:18 +02:00
jschoubben d8ec8baa22 Judge a send on no machine as the core, and test a send carrying the core with the real classes
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
The review found that a send whose first machine runs none of its
modules would pass at once with no reading, and that every mixed send
in the tests judged its leaf as the core.
2026-10-12 01:38:09 +02:00
mesh-admin 9c750a80de Merge pull request 'A provider's software-issued credential is taken from the provider, sealed (issue 526, ADR 0305)' (#239) from feat/526-a-provider-delivers-what-its-software-issued into main 2026-10-11 23:29:10 +00:00
jschoubben 4ee131e990 Let the controller ask an issued credential's tool in its own name, and seal it again over a remade pair (hq issue 526, ADR 0305)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/526-a-provider-delivers-what-its-software-issued delivered: every member is delivered
The bus left the controller out of the caller-named grants, so issued could never reach plex's tool; the
controller is now granted exactly the tools of offers whose software issues the credential, composed from
the records, so the installer's first user list is unchanged. A pair remade on a key change or accepted by
hand kept the issued stamp and was never sealed again; those writes now clear it.
2026-10-12 01:21:20 +02:00
jschoubben faf8b76b84 Cite ADR 0305's clauses by their accepted-for-review numbers (issue 526)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 01:09:26 +02:00
jschoubben ad3b4f185b Judge a leaf module on one healthy reading, a core module on three (hq issue 524)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery superseded: a newer head of the same pull request
ADR 0282 decided readings by class and ADR 0304 took a leaf module to
one, but the gate never learned it: every build, leaf or core, needed
three readings over at least two minutes, so a container module that
was healthy at once still waited about two and a half minutes on its
first machine. The number of readings and the settle time now follow
the module's class: three after two minutes for the controller's own
path and the resolver, one for every other module. Tests that judge a
gate's other rules keep leaf modules on three readings.
2026-10-12 01:08:14 +02:00
jschoubben db86dd52c8 Take a provider's software-issued credential from the provider, sealed (hq issue 526, ADR 0305)
mesh/delivery-group group feat/526-a-provider-delivers-what-its-software-issued rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
A token a media server got from its vendor reached its consumers only as a value the mesh minted, which
the server refused, or as a copy typed by hand. An offer may now say its software issues the credential;
the new issued verb carries no value and asks the provider's tool, in the controller's own name, for the
value sealed to one call, then seals it to every pair that does not hold it yet.
2026-10-12 01:07:16 +02:00
mesh-admin b26499bb9f Merge pull request 'The store-tests step fails on a malformed line, a failed go test and a shortened list (issue 515)' (#238) from fix/515-store-tests-step into main 2026-10-11 22:35:40 +00:00
jschoubben e25fdf776f Make the store-tests step fail on a malformed line, a failed go test and a shortened list
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
The step let tests escape it: a one-field line ran nothing, go test's exit
status was ignored, a pull request could drop entries main lists, and a
result written mid-line by go test -v read as NOT RUN, hiding a FAIL
(novox/hq issue 515).
2026-10-12 00:28:32 +02:00
mesh-admin e1953ad3f5 Merge pull request 'Let several modules declare one package, and refuse only present beside absent (hq ADR 0303)' (#237) from feat/303-shared-packages into main 2026-10-11 20:39:23 +00:00
jschoubben 6d2138217f Let several modules declare one package, and refuse only present beside absent (hq ADR 0303)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Assigning the media-importer beside youtube was refused because both
declare ffmpeg. The node-engine installs a missing package and never
removes one a declaration stopped naming, so a second declaration
changes nothing on the machine; what two modules can contradict is one
declaring a package present and another absent, which stays refused.
2026-10-11 21:55:20 +02:00
mesh-admin 0e86664191 Merge pull request 'delivery-stop leaves merges out and walks the others again (hq issue 459, ADR 0299)' (#232) from feat/459-a-stop-re-walks-the-others into main 2026-10-11 19:31:57 +00:00
mesh-admin 92aa3fb179 Merge pull request 'A build-log bus test waits for the stream to store both lines (hq issue 507)' (#236) from fix/507-a-build-log-test-waits into main 2026-10-11 19:12:47 +00:00
mesh-admin c2abb3759a Merge pull request 'Stop tracking built binaries at the repository root (issue 470)' (#235) from fix/470-no-built-binary-is-tracked into main 2026-10-11 19:07:25 +00:00
jschoubben 0fb7de7ef8 Merge remote-tracking branch 'origin/main' into feat/459-a-stop-re-walks-the-others
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/459-a-stop-re-walks-the-others delivered: every member is delivered
# Conflicts:
#	testdata/store-tests
2026-10-11 21:06:47 +02:00
jschoubben bc4662d656 End the line a walk's ask leaves open as each ADR 0299 store test ends, so its pass is read (issue 459)
A walk's ask prints "  tier N: " and the build ask the rest; the tests' fake
ask prints nothing, so go test -v put three tests' --- PASS after it and the
store-tests step read them as not run.
2026-10-11 21:06:44 +02:00
mesh-admin ece3759647 Merge pull request 'Say each machine's last send in status and node show (hq issue 485)' (#234) from fix/485-the-last-send-is-said into main 2026-10-11 19:06:00 +00:00
jschoubben 8d73f705af Wait for the build's log lines to be stored before counting them (hq issue 507)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A line is said with a plain publish, so a live subscriber can hold both
lines while the stream has stored one or none yet; the test read the
stream once and failed under load. It now waits up to 5 s for both.
2026-10-11 21:05:04 +02:00
jschoubben 0d4d94ff24 Stop tracking built binaries at the repository root, and ignore them (hq issue 470)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Built artifacts belong in build/; three binaries (51 MB) were committed by a local build and rode in every clone and build context.
2026-10-11 20:52:45 +02:00
jschoubben e2801773c3 Take an ordered engine's report as a send's answer only by its epoch and sequence (novox/hq issue 485)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A periodic report about an identical earlier send names the same digest and
can arrive after a later send the machine never received, so the digest and
the time read it as applied. The digest-and-time rule stays for engines that
report no order. Tests now hold the digest condition and the sequence one.
2026-10-11 20:44:54 +02:00
jschoubben fae6bd0a87 Say each machine's last send in status and node show, and as fields in status --json (novox/hq issue 485)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
The send record of issue 234 is now readable only through the controller's
verbs (issue 506), and only plan said it. status says per machine when it was
last sent a declaration, by whom, from which generation and what it answered;
node show says it in full with the refusal's words. A machine's answer is read
from its report only when that report is about the send and came after it.
2026-10-11 20:40:48 +02:00
mesh-admin afc7dd8f68 Merge pull request 'Fail the repo check when a listed store test did not run against the store (hq issue 459)' (#233) from feat/459-store-tests-are-proven into main 2026-10-11 18:31:45 +00:00
jschoubben 201bd28278 Keep a walk's withheld modules on its record, name every machine a stopped send reached, and list the stop's store tests (review of #232)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: 3 store test(s) listed in testdata/store-tests did not run and pass against the store
mesh/delivery-group group feat/459-a-stop-re-walks-the-others rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-11 20:25:20 +02:00
jschoubben 5446be1abb Merge remote-tracking branch 'origin/feat/459-store-tests-are-proven' into feat/459-a-stop-re-walks-the-others 2026-10-11 20:23:14 +02:00
jschoubben 932203df20 Leave the store tests' proof to the check of its own (#233) 2026-10-11 20:23:14 +02:00
jschoubben 949990f5a8 Merge remote-tracking branch 'origin/main' into feat/459-a-stop-re-walks-the-others 2026-10-11 20:23:12 +02:00
jschoubben 731143f5b9 Fail the repo check when a listed store test did not run against the store (hq issue 459)
mesh/merge-gate pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery covered after its stop: a later merge that contains it was delivered: novox/mesh-controller@e2801773c300 (merged as ece37596 into main, wal…
A test that needs the store skips without one, and a skip passes unseen, so a
change's store tests could pass the check without ever running. Tests named in
testdata/store-tests must each run and pass, said by name.
2026-10-11 20:22:56 +02:00
mesh-admin b555e995b0 Merge pull request 'A walk sends the bus's user list alone before its gate, so a new grant is on the bus when the first machine is judged (issue 490)' (#230) from fix/490-grants-reach-the-bus-before-the-gate into main 2026-10-11 18:19:33 +00:00
jschoubben 309475bf6f Walk a stopped walk's merges again in the open batch, in one act, and say what stays on the machines (review of #232, hq issue 459)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery-group group feat/459-a-stop-re-walks-the-others ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
A new batch beside the open one walked an older commit after a newer one, and a
stop that failed partway could leave a merge on two plans. A module shared with
the stopped merge, built at its branch, would deliver it; it is left out or
withheld. A batch not yet cut can drop a merge, so mesh-delivery can stop a
merge walked again. The stopped walk names what it left on which machines.
2026-10-11 20:12:27 +02:00
jschoubben fbfdded74b Let delivery-stop leave merges out and walk the others again, so a group stop ends only its own change (hq issue 459, ADR 0299)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group feat/459-a-stop-re-walks-the-others ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
One walk carries every merge of its batch, so ending it for one group member
ended every other delivery on it. delivery-stop now takes without: the walk
ends, every other merge it answered is batched again, and each later merge of
a left-out merge's repository is left out too, since it contains it.
2026-10-11 19:54:12 +02:00
mesh-admin 315cbd1f54 Merge pull request 'Issue 496: a definition's resolved directory and file paths are judged by the node-engine's rules where the definition is judged' (#228) from fix/496-a-resolved-path-is-judged into main 2026-10-11 17:49:40 +00:00
jschoubben eb8233ac7c Give the grants step's restic fixture a check beside its tool, as the module rules require (repo-check of #230)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A tool check alone is refused, so the store test's c2 never registered;
the fixtures are now parsed by a test that needs no store.
2026-10-11 19:43:50 +02:00
mesh-admin 3b017b228c Merge pull request 'Say a module assigned and left out of its machine's declaration as a condition (issue 380)' (#223) from fix/380-a-left-out-module-is-said into main 2026-10-11 17:36:58 +00:00
jochen bf82d30865 Keep a fill warning for an unreadable bucket or log only when its condition was read open
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Second review of #231: when neither the stream nor its condition can be
read, nothing is known of its fill, so none is said; the unread warning
covers it. Only a condition read back as open is kept, with its own words.
2026-10-11 19:34:42 +02:00
jochen c432b9b5f6 Name the fill condition for buckets and logs, and judge each one on its own
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
Review of #231: "store" is the glossary's word for the postgres server, so
the condition is bucket-or-log-filling. And one stream that cannot be read
no longer fails the whole check: it is said as unread with its reason, an
open fill condition for it is kept, and every other bucket and log is
still judged (ADR 0297 §6).
2026-10-11 19:30:59 +02:00
jschoubben 325575b292 Keep the builds of the grants step's machine alone, not of every machine its composition resolves (repo-check of #230)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheGrantsStepSendsTheUserListAndNothingElse (2.14s)
mesh/delivery superseded: a newer head of the same pull request
Composing the bus's machine resolves the others on the same context, and
a machine never recorded as sent refused the whole composition.
2026-10-11 19:30:10 +02:00
jschoubben c385ed2a17 Merge remote-tracking branch 'origin/main' into fix/380-a-left-out-module-is-said
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-11 19:28:39 +02:00
jochen 7bd61332cb Let a module declare a log the controller creates and never removes (ADR 0297)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The issue tracker's history lasts seven days on the events stream and its
bucket fills toward a cap with no warning (issues 501, 502). A module may
now declare `logs`; the controller creates each as a stream on the bus
(create-or-update, never deleted or recreated, reported when undeclared),
issues it to the owner's assignments in their membership, grants the
runtime exactly the append and direct-read subjects, and raises a
condition when any module's bucket or log reaches 75% of its cap, cleared
below 70%.
2026-10-11 19:22:43 +02:00
jschoubben 0f853e93fa Exempt only the user list's content from the grants step's guard, and test the guard in the send itself (review of #230)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheGrantsStepSendsTheUserListAndNothingElse (1.69s)
mesh/delivery superseded: a newer head of the same pull request
The guard let any field of the list's resource move; its path or mode
changing is a change the step must not carry. The store test now calls
sendToEach, so removing the guard there fails it.
2026-10-11 19:17:28 +02:00
jschoubben 78205d7405 Judge the grants step on the declaration it sends, so nothing composed between the check and the send slips through (review of #230)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheGrantsStepSendsTheUserListAndNothingElse (2.55s)
The guard composed its own declaration ahead of the send; now sendToEach
judges the one it composed and stamped, and refuses it unsent.
2026-10-11 19:12:33 +02:00
jschoubben 538c4d5115 Merge main (#227) into the grants step's branch 2026-10-11 19:10:09 +02:00
jschoubben fbeffb608d Refuse the grants step when its send would change more than the bus's user list (review of #230)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheGrantsStepSendsTheUserListAndNothingElse (1.68s)
Kept builds are not the whole declaration: a new assignment, settings or
a provision's logins went out as the mesh holds them now. The step now
compares what it would send with the last send's summary and fails, said,
when anything but the user list differs; ungatedIn reads the kept builds
instead of passing the step unread.
2026-10-11 19:09:23 +02:00
mesh-admin fad655532d Merge pull request 'A declaration says the assignment generation it came from, and every send is recorded (issue 234)' (#227) from fix/234-a-declaration-says-the-generation-it-came-from into main 2026-10-11 17:08:47 +00:00
jschoubben e1367d185a Send the bus's user list alone before a walk's gate, so a new grant is on the bus when the first machine is judged (issue 490)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A walk left the machine holding the bus out of a gated send whenever a
build waited there for that same gate, so a module's new health tool was
refused by the bus on its first machine and the gate could not pass. The
grants step sends that machine its declaration with every build kept,
before the first machine's send; plans and the change plan say it.
2026-10-11 19:01:41 +02:00
jschoubben 0c675bcdb5 Merge remote-tracking branch 'origin/main' into fix/234-a-declaration-says-the-generation-it-came-from
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-11 19:00:24 +02:00
jschoubben 8115f1ac42 Judge a definition's resolved paths where the definition is judged, by the node-engine's own rules, so a refusal never freezes a machine (review of #228, issue 496)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-11 18:59:07 +02:00
jschoubben 93289dc88b Merge remote-tracking branch 'origin/main' into fix/496-a-resolved-path-is-judged 2026-10-11 18:56:02 +02:00
mesh-admin 9edc5c758d Merge pull request 'A controller test judges the runtime's daemon.json by the docker module's own keys, not live-restore alone (issue 498)' (#229) from fix/498-docker-gives-log-rotation-too into main 2026-10-11 16:52:51 +00:00
jschoubben 483c387b72 Judge the runtime's daemon.json against the docker module's own keys, not a frozen copy
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The test pinned daemon.json to live-restore alone, so every controller check failed once the
docker module gained log rotation as a preference (mesh-catalog #205, #217). It still refuses
dns (ADR 0196) and any key the module does not declare, and checks the log defaults where the
module declares them (novox/hq issue 498).
2026-10-11 18:46:31 +02:00
jschoubben 174c8b5d53 Judge every path the controller resolves for a directory or file where the plan is composed, so the merge gate refuses a directory in Docker's data (issue 496)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheResolverAndWhatAsksItComposeOnOneMachine (0.02s)
mesh/delivery superseded: a newer head of the same pull request
2026-10-11 18:19:22 +02:00
mesh-admin fe2e5e91b5 Merge pull request 'A check never sees the forge credential, and what it publishes is redacted (issue 462)' (#226) from fix/462-a-check-never-sees-the-forge-credential into main 2026-10-11 10:19:01 +00:00
jschoubben 5dd0e3c056 Raise S20 for every generation refusal, ask for a push when the counter was behind, and write a send's generation with its digest (hq issue 234 review)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheResolverAndWhatAsksItComposeOnOneMachine (0.03s)
mesh/delivery superseded: a newer head of the same pull request
A refusal of a send the mesh never recorded went only to stderr; it is now kept
with its sender said to be unknown, and raised. A counter behind the machine (a
store put back) is raised and the condition names push, since the receive loop
must not push. The node's digest and generation are one transaction, and a send
record that fails is said loudly without failing the send. The trigger ignores
an update that changes nothing, and the put-back mark is dropped: a put-back is
composed afresh with the current generation.
2026-10-11 11:55:27 +02:00
mesh-admin 51db0b5273 Merge pull request 'broker: each credential may call tools only in its own name on the caller-named subjects (hq issue 365)' (#224) from fix/365-a-tool-call-names-its-caller into main 2026-10-11 09:31:36 +00:00
jschoubben 313efa826c Say in every declaration the assignment generation it came from, and record every send (hq issue 234)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A declaration newer in sequence than every other named four fewer modules
than the assignments held, a machine applied it, and nothing the mesh kept
said who sent it or from what view. The store now raises an assignment
generation in the same transaction as every assignment change (a trigger,
so a cascade counts too); a send reads it before composing, carries it to
engines that said they read it, marks a gate's put-back, and is recorded
with its sequence, sender, generation and modules. The would-send is
stamped with what was last sent, so nothing reads behind for it; a refusal
is kept on the send it refused and raised as S20 naming the sender; plan
says what the machine was last told.
2026-10-11 11:30:16 +02:00
jschoubben 72fde0ee1a Capture the SDK's conformance fixtures at the commit the node-engine's pin moved it to (hq issue 449's rule) 2026-10-11 11:30:16 +02:00
jschoubben 5c4fa43f8b Leave no package-registry credential or clone userinfo in the workspace a check mounts (issue 462)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A build wrote .npmrc into its source tree and never removed it, and its clone
recorded the URL's userinfo; a later check's container mounts the workspace as
HOME. The .npmrc and the tree now go when the build ends, clones record their
URL without userinfo, and a check first removes any .npmrc an older builder left.
2026-10-11 11:24:58 +02:00
jschoubben 7bd04342b6 Pin the node-engine at its pull request's generation refusal, so the validator reads a declaration's generation (hq issue 234) 2026-10-11 11:23:12 +02:00
jschoubben c494ed03a7 Keep the forge credential out of what a check's container sees, and redact what a check publishes (issue 462)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
The toolchain container mounts the workspace as HOME, so the credential kept
there, and a clone's recorded userinfo, were readable by any pull request; its
output is kept on the bus for days. The credential now lives in a private
directory outside the workspace only while cloning, clones record their URL
without userinfo, and every line said to the build's log and the verdict
passes a redactor copied from the journal tool (sharing it: issue 471).
2026-10-11 11:20:10 +02:00
mesh-admin 02c61b983c Merge pull request 'A build waits out a registry held still, and a retry asks every failed build of the tier (issue 457)' (#225) from fix/457-a-build-waits-out-a-registry-pause into main 2026-10-11 09:17:09 +00:00
jschoubben c14fe2776c Restore the tracked controller binary a local build overwrote, and hold that D1 never clears a wait (review of #223)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The binary in 419d662 came from a build into the worktree, not from the
change. node show says send, as the glossary does. A test now reconciles D1
beside a wait's needs-operator, which a wait raised under D1's source would fail.
2026-10-11 11:14:03 +02:00
jschoubben 419d662ad8 Use the glossary's words, say which modules raise a condition, and judge left-out modules in D1 (review of #223)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
The operator reads manifest, declaration and send, not definition,
composition and update. D1 already resolves every machine, so the
left-out judgement rides on it instead of resolving each machine again.
A test holds that a wait's own needs-operator still clears beside it.
2026-10-11 11:10:22 +02:00
mesh-admin 3b6b54a501 Merge pull request 'The SDK conformance test reads the SDK the controller pins, never a desktop's checkout (issue 449)' (#220) from fix/449-the-conformance-test-reads-what-it-carries into main 2026-10-11 09:08:57 +00:00
jschoubben 83ce19b9c0 Say a registry came back only when the call worked, and retry from toRetry's set (issue 457 review)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The streaming blob PUT is left unwaited, with why, since its body cannot be
read twice and the POST before it already waited.
2026-10-11 11:05:30 +02:00
jschoubben 7758301444 Ask every failed build of the tier on a retry, not only the first (issue 457)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
An outcome arriving after its plan failed is kept only in the build records,
so a retry read from the plan alone asked one failed build and the records
then failed the plan again on the next. Settle the tier from the records first.
2026-10-11 10:51:41 +02:00
jschoubben 094d3d5bc6 Wait out a registry held still, for up to five minutes, instead of failing the build (issue 457)
The store's nightly collection stops the registry for about a minute and a
half; builds in that window failed on connection refused and failed their
whole plans. A refusal is now waited for with a growing pause, said in the
build's log, and still fails the build past the bound.
2026-10-11 10:51:41 +02:00
mesh-admin e7bcc107c8 Merge pull request 'A failed repository check names what failed, from its whole output (issue 460)' (#222) from fix/460-a-failed-check-names-what-failed into main 2026-10-11 08:47:56 +00:00
jschoubben ebca7816bf inventory: a person's one tool is granted naming that person as the caller too (hq issue 365)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/365-a-tool-call-names-its-caller delivered: every member is delivered
2026-10-11 05:47:40 +02:00
jschoubben 9bed7d6398 broker: grant each credential the tool calls that name it as the caller, and no other (hq issue 365)
Every grant to call a tool is granted again under the call kind, with the
credential's own bus user as the last token, and every grant to answer one is
granted for calls naming any caller: the caller of a tool call becomes a fact
the bus enforces, as ADR 0259 section 3 made the asker of an ask. A kind of its
own because every existing tool grant is a wildcard that would match any caller
appended. The old tool grants stay for one release so nothing loses its way to a
tool (hq issue 464); the controller's own grants move with that issue, since
they are also the installer's first user list.
2026-10-11 05:25:29 +02:00
jschoubben a330c564ba Say a module assigned and left out of its machine's composition as a condition (issue 380)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A push leaves out a module whose settings do not compose and sends the rest,
which is right, but only plan said so: nfs-server ran nowhere for days while
the operator believed it ran. The self-check now raises needs-operator for a
setting nobody gave, naming the setting and the command, and left-out for any
other cause; both warnings, cleared once the module composes or is unassigned.
status and node show list the same modules as assigned, not applied.
2026-10-11 04:43:58 +02:00
mesh-admin 0a2e58c070 Merge pull request 'Deliver again an ask the controller made, removing the original from its queue (issue 334, part)' (#219) from fix/334-a-given-up-ask-can-be-delivered-again into main 2026-10-11 02:36:26 +00:00
jschoubben 1747e33923 Name what failed in a repository check from its whole output, not its tail (issue 460)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A long run's logs pushed the --- FAIL lines out of the last 200 lines the
summary was named from, so a failed check could not say which test failed.
Pick the lines that name a failure as the output streams, keep them whole at
the end of the verdict's report, and say the whole output in the build's log.
2026-10-11 04:24:34 +02:00
jschoubben 9b6acf0ef5 Read the captured SDK, saying so, when the seat placed no clone beside the check (issue 449)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The seat runs the running controller's besideRefs, which clones no mesh-sdk until this
change has rolled out, so a missing clone failing the test kept this change from ever
passing its own check. A follow-up makes it a failure again after the rollout.
2026-10-11 04:20:54 +02:00
mesh-admin 45b4ae92bc Merge pull request 'A provider whose wait fails its check lists who waits on it (issue 450)' (#221) from fix/450-a-provider-whose-wait-fails-lists-its-waiters into main 2026-10-11 02:19:02 +00:00
jschoubben 4f5fab81fe Read the SDK fixtures at the pin of the tree under check, not the running controller's (issue 449 review)
A pull request moving the SDK passed its check against the old SDK and then failed
everywhere once it rolled out. In a merge check the test now reads the clone's history
at the tree's own go.mod pin, and holds the captured copy to the clone byte for byte.
2026-10-11 04:19:01 +02:00
jschoubben 4632b6a508 Judge the SDK conformance fixtures against the SDK the controller pins, never a desktop's checkout (issue 449)
A check clones mesh-sdk beside the controller at the commit go.mod pins; elsewhere
the test reads a copy captured at that commit, held to go.mod. A missing clone fails
instead of skipping.
2026-10-11 04:19:01 +02:00
mesh-admin 9d6a12eb53 Merge pull request 'Say an unanswered merge check's time in the operator's zone (issue 443)' (#218) from fix/443-a-stalled-lines-times-are-local into main 2026-10-11 01:51:53 +00:00
jschoubben 984d85865d Give the words' zone through a seam, so no test writes time.Local under the race detector (novox/hq issue 443)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/443-a-stalled-lines-times-are-local delivered: every member is delivered
2026-10-11 03:41:14 +02:00
jschoubben ed45cc6415 Check a provider's waits before saying who waits on it (issue 450)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A provider whose wait fails the check raises unhealthy, but the holding read
its stored statement with the wait unchecked: sayWaiters built the
needs-operator key, found it not open and listed no held consumer. The
holding now reads each statement as judged (ADR 0283 decision 3), in
sayWaiters and in the provider's state its consumers are held by.
2026-10-11 03:29:10 +02:00
jschoubben ef551fdfb6 Remove an ask's original only when its sequence still holds it, and only with a worker (issue 334)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A seat's queue made again numbers from one, so an old dead letter's sequence
can name a live ask; deleting by number alone would drop it silently. An ask
with no worker would wait unseen while counted as delivered.
2026-10-11 03:24:52 +02:00
mesh-admin 7493bd8f18 Merge pull request 'A provider waiting for the operator holds its consumers, and a wait beside another condition is said (issue 405)' (#216) from fix/405-a-waiting-provider-holds-its-consumers into main 2026-10-11 01:23:35 +00:00
mesh-admin 8305e4e3d1 Merge pull request 'A test that reads another repository judges what the check clones, never the desktop's checkout (issue 432)' (#217) from fix/432-a-test-reads-what-it-carries into main 2026-10-11 01:21:14 +00:00
mesh-admin fbc7bc976b Merge pull request 'make check runs merge-check.sh, so it and the gate agree (issue 431)' (#215) from fix/431-make-check-runs-what-the-gate-runs into main 2026-10-11 01:20:19 +00:00
jschoubben f510b46319 Deliver again an ask the controller made, removing the original from its queue (issue 334)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A build ask its seat's worker gave up on could only be dropped, though the
controller already holds the publish on that seat's accepts and the stream
API to remove the original. Asks to other seats stay refused: delivering them
needs a grant ADR 0264 withholds, in the asker's name ADR 0259 protects.
2026-10-11 03:18:34 +02:00
jschoubben 926fde2fda Say an unanswered merge check's time in the operator's zone, from the checks given as data (novox/hq issue 443)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: FAIL github.com/novox/mesh-controller/cmd/mesh-controller 268.072s
mesh/delivery-group group fix/443-a-stalled-lines-times-are-local rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-11 03:10:30 +02:00
jschoubben d5cf3b42fe Say a waiting provider in the operator's words, and which state it is when a consumer is held (issue 405 review)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
2026-10-11 03:05:34 +02:00
mesh-admin 47c7877e8d Merge pull request 'A consumer's max-deliveries condition has one watcher and counts what was given up (issue 440)' (#214) from fix/440-one-key-one-watcher into main 2026-10-11 01:04:28 +00:00
jschoubben 68fbd99e89 Say how a check's clones are chosen and what the captured copy carries (issue 432 review)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-11 03:00:28 +02:00
jschoubben 9a37c143e4 Keep a waiting provider's hold within ADR 0283 (issue 405 review)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
A consumer held under a provider that only waits for the operator now
passes its gate as a wait for a person, carrying the wait, so no walk
waits on the operator's secret. Only consumers of the waiting part are
held; one that cannot be matched is raised on its own and says its
provider waits. needs-operator stays a warning while consumers wait.
2026-10-11 03:00:27 +02:00
jschoubben eb72075104 Judge tests that read another repository against what the check clones, never the desktop's checkout (issue 432)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Tests that read ../../../mesh-catalog or ../../../mesh-host gave a verdict
that depended on what sat beside the checkout: a stale or dirty sibling
failed them on a desktop, and a missing one skipped them unseen. They now
read the clone the build seat puts in MESH_CHECK_BESIDE, failing when it is
absent there, and elsewhere a copy captured at a named commit.

The skip had hidden that the builder test read a module retired by ADR 0190.
The systemd reading test no longer counts the machine's own environment.d.
2026-10-11 02:55:04 +02:00
jschoubben d6b867a87a Restart what reads a secret family member when the member is given again (issue 405)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheSecondControllerWaitsAndTakesAHigherEpochOnHandover (4.67s)
mesh/delivery superseded: a newer head of the same pull request
secretsReadBy looked only at secrets declared by name, so a container
mounting a member kept the value it started with.
2026-10-11 02:51:11 +02:00
jschoubben 1dc9961c43 Hold consumers under a provider waiting for the operator, and say a wait beside another condition (issue 405)
A provider whose only part not healthy waits for the operator's secret or
setting let its consumers raise their own unhealthy conditions, and a
wait beside a relogin, a directory used as found or a fault was not said
until the other cleared.
2026-10-11 02:51:11 +02:00
mesh-admin 11ffab887b Merge pull request 'Say an unanswered merge check in its own words, with no action it cannot take (issue 438)' (#213) from fix/438-a-check-that-never-answered-is-said into main 2026-10-11 00:47:10 +00:00
jschoubben cfbbad8209 Count a dead letter by when it was kept, so one kept late still counts (issue 440)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The give-up time is not newer for every letter kept: a notice that could
not be kept is offered again a minute later, so a later give-up can be kept
first and the one after it read as not fresh. The stored time rises with the
stream's sequence. A consumer whose letters vanish between the two reads is
left out of the look instead of counted as a look, and the skip of a
token-less max-deliveries advisory now has a test of its own.
2026-10-11 02:36:58 +02:00
jschoubben d2e9dc6159 Inline the check's teardown so make -n check stays a dry run
mesh/merge-gate pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@ef551fdfb6b2 (merged as 0a2e58c0 into main, walk plan-17916861…
GNU make runs a recipe line holding $(MAKE) even under -n, so a dry run of
check ran the whole suite. novox/hq issue 431.
2026-10-11 02:35:37 +02:00
jschoubben 5557a01f06 Make check run merge-check.sh, so a developer's check and the gate cannot drift
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
make check listed its own steps, and its gofmt walked vendor/, failing on
third-party files no change could fix; the steps after it never ran. It now
raises the throwaway database and runs the script repo-check runs.
novox/hq issue 431.
2026-10-11 02:33:30 +02:00
mesh-admin 5bddb16514 Merge pull request 'A misspelled placeholder is refused, never written out as text (issue 231)' (#211) from fix/231-a-misspelled-placeholder-is-refused into main 2026-10-11 00:31:46 +00:00
jschoubben 671e350f56 Name the build queue and the merge check as the glossary does (issue 438)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/438-a-check-that-never-answered-is-said delivered: every member is delivered
Review of #213: the queue is the controller's build queue, not the build seat's,
and the merge check is the pair, so the headline says the pull request's merge
check has not answered.
2026-10-11 02:30:19 +02:00
jschoubben f83fcdc15f Give a consumer's max-deliveries key one watcher, counting what was given up (issue 440)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The dead-letter row and a max-deliveries advisory without a token both said
bus.<stream>.<consumer>.max-deliveries: each look added an observation and a
line of evidence through the row, and the two overwrote each other's words.
The row owns the key since issue 330, so the advisory watcher leaves it, and
the row now says when the newest held message was given up, so a letter held
for a day no longer reads as observed every 30 seconds. Times without Happened
is refused, since the raise ignored it and an update counted it.
2026-10-11 02:30:09 +02:00
jschoubben 585caa4371 Say an unanswered merge check in its own words, with no action it cannot take (issue 438)
mesh/delivery-group group fix/438-a-check-that-never-answered-is-said checking: 0 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh-delivery now says a head whose merge check started and never answered as a
stalled line in state unanswered. Through the generic delivery words it would read
as a delivery and offer a Stop that mesh-delivery refuses, since no delivery of
the head exists; it now names the checks waited on and what the operator can do.
2026-10-11 02:25:11 +02:00
mesh-admin f8d08b0637 Merge pull request 'A bus refusal is counted by what the bus said, not by the controller's looks (issue 402)' (#212) from fix/402-a-refusal-is-counted-once into main 2026-10-11 00:21:00 +00:00
jschoubben dc62fd0075 Let a shell parameter operator after a known word pass, so ${PORT:-8080} is not refused (issue 231)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The case-insensitive rule for the mesh's namespace words took ${PORT:-8080},
${SHELL:-/bin/sh}, ${SECRET:?unset} and ${dir:-/tmp} for misspellings, though they
are among the commonest lines of a script or env file. A :-, :=, :+ or :? after the
colon is the shell's, whatever the word's case.
2026-10-11 02:20:43 +02:00
jschoubben 525b2c1a11 Sweep the definition, not the filled values, and judge each field alike at check and composition (issue 231)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
The first sweep ran at composition over the resource after settings, bindings and
machine facts were filled, so an operator's value such as ${labels:instance} was
refused as a misspelling its author never wrote, and the whole machine got nothing.
Both points now sweep the resource as the manifest wrote it, against one table of
which fields each pass fills, so the check and composition refuse the same things.
Any ${<known namespace>: its pattern does not take is refused whatever its case or key.

Issue 231's undeclared-setting half is not met here: refusing a key the module does
not declare (ADR 0164) is not built and is handed to issue 398.
2026-10-11 02:15:37 +02:00
jschoubben c11222026a Count a bus refusal by what the bus said, not by how often the controller looked (issue 402)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
S9 reads the one remembered advisory again on every look for an hour, and
each look was kept as an observation and a line of evidence, so a single
refusal read as one repeated every 30 seconds. An observation may now say
when what it reads happened and how often; the keeper counts that, and a
look with nothing newer adds nothing. Sources that look at the present
keep counting their looks.
2026-10-11 02:10:48 +02:00
jschoubben 360c318e85 Refuse a placeholder no pass consumes, so a misspelling never reaches a machine as text (issue 231)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-11 02:06:14 +02:00
mesh-admin f008fab9d8 Merge pull request 'A kept call holds a command's first word, never the line (issue 397)' (#210) from fix/397-a-kept-call-holds-no-command-line into main 2026-10-10 23:23:49 +00:00
jschoubben 15a9919df5 A kept command's first word is parted by any whitespace, and capped (review of issue 397)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The review of #210 found that the fix left the hole open and widened
another.

A command's words were cut on a space alone, while the verb's own
splitter parts them on a space, a tab or a newline. The same line
written with tabs found no space, so all of it was kept — the very hole
this closes. Its words are now parted as the splitter parts them.

And because the command arm sits above the arm that caps a string at
120 bytes, a command kept whole was no longer capped: for a 300-byte
single token the change kept more than the code it replaced. The first
word now carries the same cap.

A one-word command is still kept whole, but the comment no longer
claims it carries nothing to withhold: the record is written before the
verb judges the line, so one word may be a token or compact JSON. The
cap is what bounds that.

The test now says the whole of what is kept for each line, which is
also what proves the rest is gone, and looks for forbidden words as
whole words rather than as substrings — so "set" is back in the list,
and "show" cannot hide in a word such as "shown". All eight cases fail
against the unfixed code.
2026-10-11 01:04:40 +02:00
jschoubben 1390836b73 A kept call holds a command's first word, never the line (issue 397)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
kept() withheld the arguments line, values, secret and stdin, and kept
the command argument of the generic command verb verbatim. A line such
as `settings set <module> '<value>' --node <node>` therefore put the
value into the calls record, which answers anyone who may call the seat.

It is now kept as a mesh-cli line is: its first word, with the rest said
to have been given. A command of one word is kept whole, since there is
nothing after it to withhold, and one that is not a string is kept as
"(given, not kept)".

The record as it stands holds no such line: its whole answer, read at
2026-10-10 22:52 UTC, carries no call of the command verb. That says
nothing of calls older than its window.
2026-10-11 00:53:37 +02:00
mesh-admin dac7e5f7f6 Merge pull request 'Say a part waiting for the operator as needs-operator and pass its gate; add secret families (issue 386, ADR 0283)' (#209) from feat/386-a-wait-for-the-operator into main 2026-10-10 19:49:29 +00:00
mesh-admin c06a2cd972 Merge pull request 'Say a walk's phases out of order unknown, never a negative time (hq issue 382)' (#208) from fix/382-phases-out-of-order into main 2026-10-10 19:36:52 +00:00
jochen d5f6b67b94 Say a part waiting for the operator as needs-operator and pass its gate; add secret families (issue 386, hq ADR 0283)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A module waiting for the operator's secret failed its first-node gate, held
every later walk and was said as 'nothing for you to do'. The node-engine's
new waiting state is checked against the manifest and the secrets given,
read by the gate as a wait for a person, and raised as needs-operator naming
the act. A secret family gives each part its own one-line secret, which the
mesh never makes, so the desk prompt can take each password.
2026-10-10 21:19:58 +02:00
jochen 12d4025d30 Say a walk's phases out of order unknown, never a negative time (hq issue 382)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
#206's own walk kept its first send 783 ms before its build, and delivery
walks said send-first measured at -783 ms. A moment recorded before the
one before it now makes both phases unknown: the earlier one's time joins
the unknown time, the later one has none, and the walk goes on from the
later moment. Measured phases and unknown time still add up to the total.
2026-10-10 21:15:48 +02:00
mesh-admin 1ca4d8ce60 Merge pull request 'Test that times is optional on the delivery seat (hq issue 382)' (#207) from test/382-times-optional into main 2026-10-10 19:13:56 +00:00
jochen a4f93b52a8 Test that a delivery seat holder without times still holds the seat, and one serving it is not refused (hq issue 382, review of #206)
mesh/merge-gate pass: builds build-agent, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 o…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 20:55:43 +02:00
mesh-admin a2a671adb7 Merge pull request 'Keep each walk's phases and say a delivery over its budget (hq ADR 0282 slice 1, issue 382)' (#206) from feat/382-walk-phases into main 2026-10-10 18:49:18 +00:00
jochen 412f11b079 Mark times optional on the delivery seat until mesh-delivery serves it (hq ADR 0282, design 33 §7)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 20:35:24 +02:00
jochen 5d4eb974ab Promise times among the mesh-delivery seat's verbs, so its holder may serve it (hq ADR 0282)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.75s)
mesh/delivery superseded: a newer head of the same pull request
2026-10-10 20:22:34 +02:00
jochen 6805e2bcb3 Walk phases: a report past the silent bound never moves a walk's end; keep phases outside the hold on the plans; first-node gate in words (review of #206)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.71s)
2026-10-10 20:20:07 +02:00
jochen c640341c50 Keep each walk's phases and say a delivery over its budget (hq ADR 0282 slice 1, issue 382)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.62s)
The operator's budget (a leaf module running everywhere within five minutes
of its merge, a core module within ten) cannot be held to without knowing
where a walk's time goes. A walk now keeps when its batch's window closed,
when it was cut and its class (migration 0093), each gate reading, and what
each machine of the rest was sent; 'delivery walks' and plan-moved say its
phases from the merge to every machine of the rest reporting the build
applied, and ended walks keep them as walk-phase durations per class. Probe
D16 reads mesh-delivery's 'times' and raises delivery.<class>.over-budget.
A phase that cannot be measured is said unknown, never zero. Measurement
only: no walk is held, sent or judged differently.
2026-10-10 20:15:01 +02:00
mesh-admin 690b75f659 Merge pull request 'Say where a command line's quote is left open and how a quote is written (hq issue 294)' (#205) from fix/294-quote-in-a-quoted-value into main 2026-10-10 17:21:57 +00:00
jochen d52de218c8 Quote none of the line in the unclosed-quote refusal: a refusal is kept on the bus as the call's answer (hq issue 294)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 18:39:20 +02:00
jochen 6188e6b1da Say where a command line's quote is left open and how a quote is written (hq issue 294)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
An apostrophe inside a single-quoted value ends that quote, and the line then failed as a bare
'unclosed quote' with no position and no way out named. The splitter's rules stay the shell's; the
refusal now names the character, shows the line from there and gives the two ways to write a quote.
2026-10-10 18:36:47 +02:00
mesh-admin 798738cc12 Merge pull request 'A shrink is read against the item's own path: a moved directory starts its size history again (hq issue 368)' (#204) from fix/368-data-shrank-path-change into main 2026-10-10 15:56:12 +00:00
jochen babfef4f3a Keep one reading when two paths are measured at one moment, rather than failing the machine's record (hq issue 368 review)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 17:48:49 +02:00
jochen 96b0966ad8 Read a shrink against the item's own path, so a moved directory starts its size history again (hq issue 368)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A reading now keeps the path it was measured at, and the peak is read only from
readings at the item's path now. Before, an agent's home that moved to its own
account was compared with the operator's home it left, and data-shrank was
raised for data that was never lost. Existing readings take their item's path
now, so a shrink that is real stays raised.
2026-10-10 17:42:46 +02:00
520 changed files with 54513 additions and 1151 deletions
+5
View File
@@ -3,3 +3,8 @@
# A built binary. The lab writes one here when pointed at the repository root by mistake;
# built artifacts belong in build/, which is already ignored.
/mesh-builder
/mesh-controller
/postgres-provisioner
/redis-provisioner
# And where `go build` in cmd/mesh-controller leaves it (novox/hq issue 233).
/cmd/mesh-controller/mesh-controller
+3 -1
View File
@@ -27,9 +27,11 @@ WORKDIR /src
COPY go.mod go.sum ./
COPY vendor/ vendor/
# `COPY . .` brings `.git` along, so Go is told not to stamp the commit: the version is said by -X, and a
# rebuild of an unchanged program is the same bytes (novox/hq ADR 0267 rule 6, issue 363).
COPY . .
ARG VERSION=development
RUN CGO_ENABLED=0 GOFLAGS=-mod=vendor GOPROXY=off go build -trimpath \
RUN CGO_ENABLED=0 GOFLAGS=-mod=vendor GOPROXY=off go build -trimpath -buildvcs=false \
-ldflags "-s -w -X main.version=${VERSION}" \
-o /mesh-controller ./cmd/mesh-controller
+18 -10
View File
@@ -99,16 +99,22 @@ proxy-image:
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The whole gate. Raises a database, runs everything against it, and takes it down again --
# including when the tests fail, which is why the teardown is not conditional.
# The whole check. Raises a database, runs the repository's own check against it -- merge-check.sh,
# the very script `mesh/repo-check` runs -- and takes the database down again, including when the
# check fails, which is why the teardown is not conditional and the script's exit status is the
# target's.
#
# **Packages in parallel, under the race detector, each test on a bus of its own** (internal/testbus).
# It was one package at a time against one shared bus, because the live tests assert, read and remove
# the mesh's own objects by their fixed names, and two packages at once deleted what the other read; the
# suite was red run as Go runs it and read as noise. A bus per test, of the release the mesh runs, made
# it the same in any order. The timeout bounds a hang to a failure with a stack, never a stalled gate.
check: fmt vet postgres
@go test -race -timeout 15m ./... ; status=$$? ; $(MAKE) postgres-stop ; exit $$status
# **It calls the script rather than restating it** (novox/hq issue 431): this target used to list its
# own steps, its formatting step walked vendor/ where the script's does not, and it failed on
# third-party code no change could fix -- so a developer's check and the gate disagreed, and the steps
# after formatting never ran through it. The script is the one list of steps; this target only gives it
# a database (MESH_TEST_POSTGRES, exported above).
#
# **Packages in parallel, under the race detector, each test on a bus of its own** (internal/testbus),
# as the script runs them: a bus per test, of the release the mesh runs, makes the suite the same in any
# order, and the timeout bounds a hang to a failure with a stack, never a stalled gate.
check: postgres
@sh merge-check.sh ; status=$$? ; docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
# Without a database the store's tests skip rather than fail, so this is the honest subset and not
# the gate.
@@ -118,8 +124,10 @@ test:
vet:
go vet ./...
# Quick steps for a developer, not part of `check`. The directories gofmt reads must be the ones
# merge-check.sh lists, never `.`, which walks vendor/ (novox/hq issue 431).
fmt:
@unformatted=$$(gofmt -l . 2>/dev/null) ; \
@unformatted=$$(gofmt -l cmd internal examples) ; \
if [ -n "$$unformatted" ] ; then echo "not gofmt'd:" ; echo "$$unformatted" ; exit 1 ; fi
postgres:
+1 -1
View File
@@ -11,7 +11,7 @@ WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags '-s -w' -o /mesh-builder ./cmd/mesh-builder
RUN CGO_ENABLED=0 go build -trimpath -buildvcs=false -ldflags '-s -w' -o /mesh-builder ./cmd/mesh-builder
FROM ${ALPINE_BASE}
# git to clone what it is asked to build, and the docker client to build and push it. The daemon
+174 -56
View File
@@ -18,11 +18,13 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"net/url"
"os"
"os/signal"
"sort"
"strings"
"syscall"
"time"
@@ -54,6 +56,8 @@ is dialled except the broker.
MESH_BINDING a file the mesh wrote saying where the artifact store is
MESH_PACKAGE_BINDING a file the mesh wrote saying where the package registry is
MESH_NPM_TOKEN_FILE a file the mesh sealed holding the token for it
MESH_GENESIS_PACKAGE_REGISTRY the document the installer sealed at genesis, read while no
binding is present (novox/hq ADR 0336)
MESH_NPM_REGISTRY a package registry URL, when the mesh has not said (a person, the bootstrap)
MESH_NPM_TOKEN the token for it, likewise
MESH_NPM_SCOPE the scope it answers for (default: @novox)
@@ -66,6 +70,11 @@ broker to take work from or a registry to publish into:
Without --registry the artifacts stay in this machine's container runtime, named by the
digest of their own configuration. The result is printed as JSON.
mesh-builder ready
exits 0 once the agent running in this container is bound to its build work and pulls it, and
1 before: its container's health check.
`
func run() error {
@@ -76,12 +85,18 @@ func run() error {
return nil
case "build":
return buildOnce(context.Background(), os.Args[2:])
case "ready":
// The container's health check (novox/hq issue 544): exits non-zero until the agent takes work.
return ready()
default:
fmt.Print(usage)
return nil
}
}
if err := notReady(); err != nil {
return err
}
credential, err := brokerFrom()
if err != nil {
return err
@@ -134,7 +149,8 @@ func run() error {
// **What an earlier holder here left goes before anything is taken** (novox/hq issue 306): a holder
// recreated mid-check left the check's store and bus running, and the ask went to another machine.
h.removeLeftBehind()
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused})
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused,
Taking: func() { taking(seat) }})
defer machine.Close()
// The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired
@@ -242,7 +258,10 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
}
say("build", what)
npmrc, err := packagesFrom()
npmrc, err := packagesFrom(on)
if err == nil {
say("packages", saysRegistry(npmrc))
}
var built builder.Result
if request.Check != nil {
// **A pull request's merge check, not a build** (novox/hq to-be 45 §9): nothing is built,
@@ -256,7 +275,13 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
v, err = builder.Check(building, builder.Command, checkSpecOf(request), workspace, registry, forgeFrom(), say)
if err == nil {
result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report,
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo)}
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo),
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside}
// What each of a group's heads was judged as (novox/hq ADR 0323 §6).
for _, m := range v.Members {
result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo,
Commit: m.Head, Trunk: m.Trunk})
}
}
} else if err == nil {
// The package-registry credential is a build input, so it is resolved before the clone: a
@@ -302,6 +327,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
}
result.Against = built.Against
result.SourceFingerprint = built.Source
result.Predefined = built.Predefined
result.Trunk, result.OnTrunk, result.Branches = built.Trunk, built.OnTrunk, built.Branches
for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
@@ -354,7 +380,7 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
}
for _, m := range c.Members {
spec.Group = append(spec.Group, builder.GroupHead{Owner: m.Owner, Repo: m.Repo, Repository: m.Repository,
Ref: m.Ref, Paths: m.Paths})
Ref: m.Ref, Paths: m.Paths, Base: m.Base})
}
return spec
}
@@ -368,79 +394,171 @@ func layerOf(l *builder.Layer) *link.CheckLayer {
}
// packagesFrom is where a build resolves the mesh's own published packages — the SDK above all
// (novox/hq ADR 0076, issue 053).
// (novox/hq ADR 0076, issue 053) — on this machine.
//
// Preferably from the mesh: a package-registry binding names the endpoint the way the artifact
// store's binding does, and a sealed token file the credential the way the broker's does. The
// environment variables remain for a builder run by a person, and for the bootstrap, where there is
// no registry yet — there the result is disabled and a build that needs no mesh-published dependency
// builds anyway.
func packagesFrom() (builder.Npmrc, error) {
// **The binding wins, then the genesis document** (novox/hq ADR 0336 §3, §4). The build agent's
// npm-package-registry requirement is optional: on genesis's node, before the gitea module is assigned,
// nothing provides it, and the installer sealed the forge it raised to this agent as one document
// instead. Once the requirement binds, the binding names the endpoint the way the artifact store's
// does and a sealed token file the credential, and the genesis document is left unread. A binding
// file named and absent is a requirement not bound here, never a fault: the environment names the
// path whether or not the mesh wrote it.
//
// The environment variables remain for a builder run by a person. With none of the three the result
// is disabled and says this machine, so a build that needs the registry is refused naming it (§6),
// and a build that needs none builds anyway.
func packagesFrom(machine string) (builder.Npmrc, error) {
scope := strings.TrimSpace(os.Getenv("MESH_NPM_SCOPE"))
if scope == "" {
scope = "@novox"
}
none := builder.Npmrc{Machine: machine}
registry := strings.TrimSpace(os.Getenv("MESH_NPM_REGISTRY"))
var username string
if path := strings.TrimSpace(os.Getenv("MESH_PACKAGE_BINDING")); path != "" {
raw, err := os.ReadFile(path)
if err != nil {
return builder.Npmrc{}, fmt.Errorf("cannot read what the mesh said about the package registry: %w", err)
switch {
case errors.Is(err, os.ErrNotExist):
// Not bound on this machine: the genesis document, or nothing.
case err != nil:
return none, fmt.Errorf("cannot read what the mesh said about the package registry: %w", err)
default:
return fromBinding(path, raw, scope, machine)
}
var told struct {
From string `json:"from"`
At string `json:"at"`
As string `json:"as"`
Serves map[string]any `json:"serves"`
}
if path := strings.TrimSpace(os.Getenv("MESH_GENESIS_PACKAGE_REGISTRY")); path != "" {
raw, err := os.ReadFile(path)
switch {
case errors.Is(err, os.ErrNotExist):
// Not this mesh's genesis node, or the installer has not sealed it yet.
case err != nil:
return none, fmt.Errorf("cannot read the genesis document for the package registry: %w", err)
default:
return fromGenesis(path, raw, scope, machine)
}
if err := json.Unmarshal(raw, &told); err != nil {
return builder.Npmrc{}, fmt.Errorf("%s is not a binding: %w", path, err)
}
if told.At == "" {
return builder.Npmrc{}, fmt.Errorf(
"%s says the package registry is on %q and gives no address for it", path, told.From)
}
// Composed from what the provider serves, so nothing here knows gitea's URL shape from
// another registry's: it states its port, the path its registry answers on, and the scheme.
scheme := "https"
if s, ok := told.Serves["scheme"]; ok {
scheme = fmt.Sprintf("%v", s)
}
port, ok := told.Serves["port"]
if !ok {
return builder.Npmrc{}, fmt.Errorf("%s says nothing about which port the package registry answers on", path)
}
npmPath, ok := told.Serves["npm-path"]
if !ok {
return builder.Npmrc{}, fmt.Errorf("%s says nothing about the path the package registry answers on", path)
}
registry = fmt.Sprintf("%s://%s:%v%v", scheme, told.At, port, npmPath)
username = told.As
}
// The credential the mesh sealed to this machine. The mesh authenticates the ordinary way — a
// generated password the provider only applies (novox/hq ADR 0048) — so with a username this is
// a password (basic auth); without one it is a bearer token a provider minted.
// A builder run by a person.
registry := strings.TrimSpace(os.Getenv("MESH_NPM_REGISTRY"))
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
if path := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN_FILE")); path != "" {
raw, err := os.ReadFile(path)
username := strings.TrimSpace(os.Getenv("MESH_NPM_USER"))
if registry == "" && secret == "" {
return none, nil
}
n := builder.Npmrc{Scope: scope, Registry: registry, From: builder.FromEnvironment, Machine: machine}
if username != "" {
n.Username, n.Password = username, secret
} else {
n.Token = secret
}
return n, nil
}
// fromBinding composes the registry from the npm-package-registry binding the mesh wrote and the
// credential it sealed beside it. The mesh authenticates the ordinary way — a generated password the
// provider only applies (novox/hq ADR 0048) — so with a username the secret is a password (basic
// auth); without one it is a bearer token a provider minted.
func fromBinding(path string, raw []byte, scope, machine string) (builder.Npmrc, error) {
none := builder.Npmrc{Machine: machine}
var told struct {
From string `json:"from"`
At string `json:"at"`
As string `json:"as"`
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal(raw, &told); err != nil {
return none, fmt.Errorf("%s is not a binding: %w", path, err)
}
if told.At == "" {
return none, fmt.Errorf(
"%s says the package registry is on %q and gives no address for it", path, told.From)
}
// Composed from what the provider serves, so nothing here knows gitea's URL shape from
// another registry's: it states its port, the path its registry answers on, and the scheme.
scheme := "https"
if s, ok := told.Serves["scheme"]; ok {
scheme = fmt.Sprintf("%v", s)
}
port, ok := told.Serves["port"]
if !ok {
return none, fmt.Errorf("%s says nothing about which port the package registry answers on", path)
}
npmPath, ok := told.Serves["npm-path"]
if !ok {
return none, fmt.Errorf("%s says nothing about the path the package registry answers on", path)
}
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
if file := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN_FILE")); file != "" {
raw, err := os.ReadFile(file)
if err != nil {
return builder.Npmrc{}, fmt.Errorf("cannot read this builder's package-registry credential: %w", err)
return none, fmt.Errorf("cannot read this builder's package-registry credential: %w", err)
}
secret = strings.TrimSpace(string(raw))
}
if u := strings.TrimSpace(os.Getenv("MESH_NPM_USER")); u != "" {
username = u
n := builder.Npmrc{Scope: scope, Registry: fmt.Sprintf("%s://%s:%v%v", scheme, told.At, port, npmPath),
From: builder.FromBinding, Machine: machine}
if told.As != "" {
n.Username, n.Password = told.As, secret
} else {
n.Token = secret
}
return n, nil
}
if registry == "" && secret == "" {
return builder.Npmrc{}, nil
// GenesisRegistry is the genesis document (novox/hq ADR 0336 §3): what the installer sealed to the
// build agent on its node at genesis's packages step, as the `genesis-package-registry` own-secret —
// the forge it raised, the port this node gave the package registry, its npm path, and the account it
// made with that account's password. mesh-host writes it; the field names are the two repositories'
// agreement.
type GenesisRegistry struct {
Scheme string `json:"scheme"`
At string `json:"at"`
Port int `json:"port"`
NpmPath string `json:"npm-path"`
As string `json:"as"`
Password string `json:"password"`
}
// fromGenesis composes the registry from the genesis document.
func fromGenesis(path string, raw []byte, scope, machine string) (builder.Npmrc, error) {
none := builder.Npmrc{Machine: machine}
var g GenesisRegistry
if err := json.Unmarshal(raw, &g); err != nil {
return none, fmt.Errorf("%s is not the genesis document for the package registry: %w", path, err)
}
if username != "" {
return builder.Npmrc{Scope: scope, Registry: registry, Username: username, Password: secret}, nil
var missing []string
for name, value := range map[string]string{"at": g.At, "npm-path": g.NpmPath, "as": g.As, "password": g.Password} {
if strings.TrimSpace(value) == "" {
missing = append(missing, name)
}
}
return builder.Npmrc{Scope: scope, Registry: registry, Token: secret}, nil
if g.Port <= 0 {
missing = append(missing, "port")
}
if len(missing) > 0 {
sort.Strings(missing)
return none, fmt.Errorf("%s, the genesis document for the package registry, says nothing of %s",
path, strings.Join(missing, ", "))
}
scheme := g.Scheme
if scheme == "" {
scheme = "http"
}
return builder.Npmrc{Scope: scope, Registry: fmt.Sprintf("%s://%s:%d%s", scheme, g.At, g.Port, g.NpmPath),
Username: g.As, Password: g.Password, From: builder.FromGenesis, Machine: machine}, nil
}
// saysRegistry is the line each build's log carries about the package registry (novox/hq ADR 0336
// §4): which of the binding and the genesis document it resolved from, or that it has neither here.
func saysRegistry(n builder.Npmrc) string {
if n.Enabled() {
return "the package registry is resolved from " + n.From
}
machine := n.Machine
if machine == "" {
machine = "this machine"
}
return fmt.Sprintf("no package registry: %s is unbound on %s and no genesis document is here; "+
"a build needing it is refused", builder.RegistryRequirement, machine)
}
// forgeFrom is the git credential this builder may offer a clone, composed from the same binding
+6 -1
View File
@@ -84,10 +84,15 @@ func buildOnce(ctx context.Context, args []string) error {
}
fmt.Fprintln(os.Stderr)
npmrc, err := packagesFrom()
machine := os.Getenv("MESH_NODE")
if machine == "" {
machine, _ = os.Hostname()
}
npmrc, err := packagesFrom(machine)
if err != nil {
return err
}
fmt.Fprintf(os.Stderr, " [packages] %s\n", saysRegistry(npmrc))
built, buildErr := builder.Build(ctx, builder.Command, publisher, repository, *path, *ref, where, bases, npmrc,
forgeFrom(),
func(step, message string) { fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message) })
+102
View File
@@ -0,0 +1,102 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/builder"
)
// The build agent's package registry (novox/hq ADR 0336, issue 121): its npm-package-registry
// requirement is optional, so the binding may be absent, and on genesis's node the installer seals
// the forge it raised as one genesis document instead.
// registryFiles names the three paths the build agent's environment always names, writing only those
// given.
func registryFiles(t *testing.T, binding, token, genesis string) {
t.Helper()
dir := t.TempDir()
for env, content := range map[string]string{
"MESH_PACKAGE_BINDING": binding,
"MESH_NPM_TOKEN_FILE": token,
"MESH_GENESIS_PACKAGE_REGISTRY": genesis,
} {
path := filepath.Join(dir, env)
t.Setenv(env, path)
if content != "" {
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
}
}
for _, env := range []string{"MESH_NPM_REGISTRY", "MESH_NPM_TOKEN", "MESH_NPM_USER", "MESH_NPM_SCOPE"} {
t.Setenv(env, "")
}
}
const (
aBinding = `{"from":"gitea","at":"anchor.mesh","as":"anchor-agent",
"serves":{"scheme":"http","port":3000,"npm-path":"/api/packages/novox/npm/"}}`
aGenesisDocument = `{"scheme":"http","at":"127.0.0.1","port":3100,"npm-path":"/api/packages/novox/npm/",
"as":"mesh-builder","password":"genesis-secret"}`
)
func TestUntilTheRequirementBindsTheGenesisDocumentIsRead(t *testing.T) {
registryFiles(t, "", "", aGenesisDocument)
n, err := packagesFrom("anchor")
if err != nil {
t.Fatalf("an absent binding beside a genesis document was refused: %v", err)
}
if n.From != builder.FromGenesis || n.Registry != "http://127.0.0.1:3100/api/packages/novox/npm/" ||
n.Username != "mesh-builder" || n.Password != "genesis-secret" || n.Scope != "@novox" {
t.Fatalf("the genesis document was not what the registry came from: %+v", n)
}
file, err := n.File()
if err != nil || !strings.Contains(file, "//127.0.0.1:3100/api/packages/novox/npm/:username=mesh-builder") {
t.Fatalf("the .npmrc is not written from the genesis document: %v\n%s", err, file)
}
if got := saysRegistry(n); !strings.Contains(got, "genesis document") {
t.Errorf("the log does not say the genesis document: %s", got)
}
}
func TestARealBindingWinsOverTheGenesisDocument(t *testing.T) {
registryFiles(t, aBinding, "bound-secret\n", aGenesisDocument)
n, err := packagesFrom("anchor")
if err != nil {
t.Fatal(err)
}
if n.From != builder.FromBinding || n.Registry != "http://anchor.mesh:3000/api/packages/novox/npm/" ||
n.Username != "anchor-agent" || n.Password != "bound-secret" {
t.Fatalf("the binding did not win over the genesis document: %+v", n)
}
if got := saysRegistry(n); !strings.Contains(got, "binding") {
t.Errorf("the log does not say the binding: %s", got)
}
}
func TestWithNeitherTheAgentHasNoRegistryAndSaysWhere(t *testing.T) {
registryFiles(t, "", "", "")
n, err := packagesFrom("workstation")
if err != nil {
t.Fatalf("an unbound optional requirement was a fault: %v", err)
}
if n.Enabled() || n.Machine != "workstation" {
t.Fatalf("a registry came from nowhere: %+v", n)
}
got := saysRegistry(n)
for _, want := range []string{"npm-package-registry", "unbound", "workstation"} {
if !strings.Contains(got, want) {
t.Errorf("the log does not say %q: %s", want, got)
}
}
}
func TestAGenesisDocumentMissingAFieldIsRefusedByName(t *testing.T) {
registryFiles(t, "", "", `{"at":"127.0.0.1","port":3000,"npm-path":"/api/packages/novox/npm/","as":"mesh-builder"}`)
if _, err := packagesFrom("anchor"); err == nil || !strings.Contains(err.Error(), "password") {
t.Fatalf("a genesis document with no password was taken: %v", err)
}
}
+44
View File
@@ -0,0 +1,44 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
)
// **The build agent says it is ready** (novox/hq issue 544). It listens on nothing, so no http or tcp
// check can see it work; it writes a mark once it is bound to its worker and pulls build work, and its
// container's health check runs `mesh-builder ready`, which passes only while the mark is there. Without
// it the node-engine could judge the agent only alive, and every walk that moved it waited out the whole
// grace on each machine.
//
// The mark lives in the container's own temporary directory, never in the workspace the machine keeps:
// a new container starts without it, and a restarted one removes it before it binds again.
func readyMark() string { return filepath.Join(os.TempDir(), "mesh-builder.ready") }
// notReady removes the mark: a holder starting is not taking work until it says so again.
func notReady() error {
if err := os.Remove(readyMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove the mark that this agent takes build work (%s): %w", readyMark(), err)
}
return nil
}
// taking writes the mark. A mark that cannot be written is said, and the agent goes on building: its
// check then says it is not ready, which is the truth the gate should hear.
func taking(seat string) {
if err := os.WriteFile(readyMark(), []byte(seat+"\n"), 0o644); err != nil {
fmt.Fprintf(os.Stderr, "cannot mark that this agent takes build work (%s): %v\n", readyMark(), err)
return
}
fmt.Fprintf(os.Stderr, "taking build work from %s\n", seat)
}
// ready is the health check: nil when this container's agent is bound and pulling build work.
func ready() error {
if _, err := os.Stat(readyMark()); err != nil {
return fmt.Errorf("not taking build work yet: %s is not there", readyMark())
}
return nil
}
+32
View File
@@ -0,0 +1,32 @@
package main
import (
"os"
"testing"
)
// The health check passes only between the agent saying it takes work and its next start (novox/hq
// issue 544).
func TestTheAgentIsReadyOnlyOnceItTakesWork(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := ready(); err == nil {
t.Fatal("an agent that never took work is ready")
}
taking("mesh-build-machine")
if err := ready(); err != nil {
t.Fatalf("an agent taking work is not ready: %v", err)
}
// Started again, it is not ready until it binds again.
if err := notReady(); err != nil {
t.Fatal(err)
}
if err := ready(); err == nil {
t.Fatal("an agent starting again is ready before it takes work")
}
if err := notReady(); err != nil {
t.Fatalf("removing a mark that is not there failed: %v", err)
}
if _, err := os.Stat(readyMark()); err == nil {
t.Fatal("the mark is still there")
}
}
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A provision whose credential only a person outside the mesh can give (novox/hq ADR 0337, issue 164, issue 165):
// given once for its provider, at the controller's terminal, and sealed in one act to every consumer that binds the
// provision from that provider now. The value is never read back, printed or kept.
// outsideOffer is the provider module's own name when `secret accept <node> <module> <name>` names a provision that
// module offers with a credential issued outside the mesh, and "" when it names anything else (an own secret, or a
// pair with --provider).
func outsideOffer(ctx context.Context, module, name, provider string) (string, error) {
if provider != "" {
return "", nil
}
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
if m, known := shelf[module]; known && m.IssuedOutsideOf(name) {
return module, nil
}
return "", nil
}
// offeredOutsideBy is a module that offers the provision with a credential issued outside the mesh, or "".
func offeredOutsideBy(ctx context.Context, provision string) (string, error) {
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
module, _ := catalogue.OffersIssuedOutside(shelf, provision)
return module, nil
}
// consumersFrom is every consumer that binds a provision from one provider now, found by resolving every machine as
// its next send would, and the machines that could not be resolved, with why.
func consumersFrom(ctx context.Context, open *stores, provider, providerModule, provision string) (
[]inventory.PairEnd, []string, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, nil, err
}
var out []inventory.PairEnd
var unresolved []string
for _, n := range nodes {
plan, _, err := planFor(ctx, open, n.Name)
if err != nil {
unresolved = append(unresolved, fmt.Sprintf("%s: %s", n.Name, oneLine(err.Error())))
continue
}
out = append(out, bindsFrom(plan, n.Name, provider, providerModule, provision)...)
}
return out, unresolved, nil
}
// bindsFrom is every consumer on one machine whose resolution answers the provision from that provider. Pure.
func bindsFrom(plan catalogue.Resolution, machine, provider, providerModule, provision string) []inventory.PairEnd {
var out []inventory.PairEnd
for _, need := range plan.Needs {
if need.ByRecord || need.Name != provision || need.From != provider {
continue
}
if need.Module != "" && need.Module != providerModule {
continue
}
out = append(out, inventory.PairEnd{Consumer: machine, ConsumerModule: need.For, Local: need.Local})
}
sort.Slice(out, func(i, j int) bool {
return out[i].ConsumerModule+"\x00"+out[i].Local < out[j].ConsumerModule+"\x00"+out[j].Local
})
return out
}
// acceptForProvision gives a provision's credential issued outside the mesh, once for its provider (ADR 0337 §3).
func acceptForProvision(ctx context.Context, provider, providerModule, provision, value string) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
consumers, unresolved, err := consumersFrom(ctx, open, provider, providerModule, provision)
if err != nil {
return err
}
accepted, err := open.inventory.AcceptSecretForProvision(ctx, provision, provider, providerModule, value, consumers)
value = ""
if err != nil {
return err
}
fmt.Print(acceptedSaid(provider, providerModule, provision, accepted, unresolved))
if len(accepted.Sealed) == 0 {
return nil
}
if err := sendTo(ctx, open, accepted.Machines); err != nil {
return fmt.Errorf("%s is sealed and could not be sent: %w; `push %s` sends it", provision, err,
strings.Join(accepted.Machines, "`, `push "))
}
fmt.Printf(" sent %s; each consumer starts again on it\n", strings.Join(accepted.Machines, ", "))
return nil
}
// acceptedSaid is what one accept for a provider did, in words that never carry the value. Pure.
func acceptedSaid(provider, providerModule, provision string, a inventory.ProvisionAccepted, unresolved []string) string {
var b strings.Builder
if len(a.Sealed) == 0 {
fmt.Fprintf(&b, "%s from %s on %s: no consumer binds it now, so nothing was sealed; a consumer that binds "+
"later waits for it, and this line given again seals it to every consumer then\n", provision,
providerModule, provider)
} else {
var names []string
for _, c := range a.Sealed {
names = append(names, fmt.Sprintf("%s on %s%s", c.ConsumerModule, c.Consumer, asLocal(c.Local)))
}
fmt.Fprintf(&b, "%s from %s on %s is given, sealed to %s and to %s\n", provision, providerModule, provider,
strings.Join(names, ", "), provider)
fmt.Fprintf(&b, " the mesh cannot read it back, never makes one of its own for it, and will not rotate it\n")
}
for _, c := range a.Unsealable {
fmt.Fprintf(&b, " not sealed to %s on %s: no sealing key on that machine; give it again once it has one\n",
c.ConsumerModule, c.Consumer)
}
for _, u := range unresolved {
fmt.Fprintf(&b, " not known whether a consumer on %s binds it, which did not resolve; give it again once it does\n", u)
}
return b.String()
}
+6 -1
View File
@@ -64,6 +64,11 @@ type actor struct {
// the highest epoch issued, and what was said of it; zero when it was not (S12).
reset time.Time
resetSaid string
// renewEvery and poll, when set, are how often the serving controller renews the lease it holds and
// looks again at one another holds, in place of the lease's own: a test's, so two controllers hand the
// lease over in a second rather than in a renewal of five (novox/hq issue 591). A renewal is given
// the lease's own RenewEvery to answer however often it is made, so a slow bus does not lose it.
renewEvery, poll time.Duration
}
// theLease is this process's standing to the lease.
@@ -223,7 +228,7 @@ func (a *actor) serveUnderTheLease(ctx context.Context, inv *inventory.Inventory
}
say := func(format string, args ...any) { fmt.Printf(format+"\n", args...) }
l, err := lease.Open(ctx, api, broker.LeaseBucket, lease.Options{Holder: holderOf(instance),
Floor: inv.HighestEpoch, Say: say, Health: controllerHealth, Moved: func(was, floor uint64) {
RenewEvery: a.renewEvery, RenewWithin: lease.RenewEvery, Poll: a.poll, Floor: inv.HighestEpoch, Say: say, Health: controllerHealth, Moved: func(was, floor uint64) {
a.mu.Lock()
defer a.mu.Unlock()
a.reset = time.Now()
+70 -17
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/broker"
"io"
"slices"
"sort"
"strings"
@@ -54,6 +55,9 @@ func assignWith(ctx context.Context, open *stores, node string, opts assignOptio
if err := refusedMovingTheController(modules); err != nil {
return "", err
}
// Every line of the answer comes from the one resolution after the write (novox/hq issue 237):
// what the reads before it would say about a machine left out is not the answer's.
ctx = leftOutSaidBy(ctx, io.Discard)
// Held while it is recorded, so it cannot land between a converge's preview and its flip and
// be taken without ever having been previewed (novox/hq ADR 0100).
ctx, release, err := holdNodes(ctx, open, []string{node})
@@ -125,22 +129,28 @@ func assignWith(ctx context.Context, open *stores, node string, opts assignOptio
for _, line := range settled {
answer += "\n " + line
}
// What this act changed about this node's unmet seat dependencies, and nothing else (novox/hq
// ADR 0207): a dependency of a module just assigned, or one this assignment met. The rest of the
// node's list, and every other node's, is `status`'s.
for _, line := range unheldChange(shelf, node, before, append(append([]string(nil), before...), added...)) {
answer += "\n " + line
}
// Its bus credential, in the same act (novox/hq issue 203): an assignment pushed before its
// credential exists delivers a process that cannot authenticate and crash-loops until somebody
// runs a second verb and a second push. Issued here when the module speaks on the bus and has
// no credential yet; kept when it has one, so re-assigning rotates nothing.
var issued []string
for _, module := range added {
if line := issueOnAssign(ctx, open, node, module); line != "" {
answer += "\n " + line
issued = append(issued, line)
}
}
plan, _, err := planFor(ctx, open, node)
// **The one resolution the rest of the answer is said from** (novox/hq issue 237), taken after
// the write: what this act changed about this node's unmet seat dependencies, and nothing else
// (novox/hq ADR 0207) — a dependency of a module just assigned, or one this assignment met; the
// rest of the node's list, and every other node's, is `status`'s — and whether the node resolves.
after := resolvedAfter(ctx, open, shelf, node)
for _, line := range unheldChangeTo(shelf, node, before, after.modules, after.unheld) {
answer += "\n " + line
}
for _, line := range issued {
answer += "\n " + line
}
plan, err := after.plan, after.err
if err != nil {
// Kept, and still refused. Both halves are the answer, and the rest of the mesh is still
// worth reporting: this machine's refusal is rarely the only consequence.
@@ -193,8 +203,9 @@ func seatDependenciesOnAssign(ctx context.Context, open *stores, node string, mo
if _, err := catalogue.AssignRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err
}
// Two modules declaring one package, path or unit is refused before anything is recorded
// (novox/hq ADR 0210, 04-ISSUES/235): kept, the node would not resolve until one came off again.
// Two modules declaring one path or unit, or one package present and absent, is refused before anything
// is recorded (novox/hq ADR 0210, ADR 0303, 04-ISSUES/235): kept, the node would not resolve until one came
// off again. Two modules declaring one package present is not a collision (ADR 0303, issue 628).
if err := catalogue.CollisionRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err
}
@@ -218,6 +229,8 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
if len(modules) == 0 {
return "", fmt.Errorf("unassign %s names no module", node)
}
// One resolution after the write, as assign's answer (novox/hq issue 237).
ctx = leftOutSaidBy(ctx, io.Discard)
ctx, release, err := holdNodes(ctx, open, []string{node})
if err != nil {
return "", err
@@ -262,15 +275,22 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
}
answer := fmt.Sprintf("%s no longer runs %s — run `push %s` to make it so",
node, strings.Join(modules, ", "), node)
var left []string
for _, a := range assigned {
if !slices.Contains(modules, a) {
left = append(left, a)
}
}
for _, line := range unheldChange(shelf, node, assigned, left) {
after := resolvedAfter(ctx, open, shelf, node)
for _, line := range unheldChangeTo(shelf, node, assigned, after.modules, after.unheld) {
answer += "\n " + line
}
// Whether the node itself still resolves, from the same resolution: before, only the rest of the
// mesh's reads said it, on standard error and once per other machine.
if after.err != nil {
if unresolvable(after.err) {
answer += fmt.Sprintf("\n but %s does not resolve now, so nothing will be sent to it:", node)
} else {
answer += fmt.Sprintf("\n but whether %s still resolves could not be read:", node)
}
for _, line := range strings.Split(strings.TrimRight(after.err.Error(), "\n"), "\n") {
answer += "\n " + strings.TrimSpace(line)
}
}
// What it leaves behind that is irreplaceable is kept and retired, never removed (novox/hq ADR 0233).
for _, line := range keptOnUnassign(ctx, open.inventory, node, modules) {
answer += "\n " + line
@@ -278,6 +298,39 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
return answer + blockedElsewhere(ctx, open, node), nil
}
// afterAct is the one resolution of a node an act's answer is said from, taken after its write
// (novox/hq issue 237): the plan or why there is none, the modules it has on the node, and its unmet
// seat dependencies.
type afterAct struct {
plan catalogue.Resolution
err error
modules []string
unheld []catalogue.Unheld
}
// resolvedAfter resolves the node as `plan` does, once, after the act's write. When the node resolves,
// its modules and unmet seat dependencies are the resolution's — the closure, so a holder pulled in by
// a requirement counts as `plan` counts it. When it does not, they are judged over the assignments read
// after the write, which can only find more unmet than the closure would: a seat said held there is
// held in the closure too.
func resolvedAfter(ctx context.Context, open *stores, shelf map[string]catalogue.Manifest, node string) afterAct {
plan, _, err := planFor(ctx, open, node)
if err == nil {
out := afterAct{plan: plan, unheld: plan.Unheld}
for _, m := range plan.Modules {
out.modules = append(out.modules, m.Module)
}
return out
}
out := afterAct{err: err}
assigned, readErr := open.inventory.Assigned(ctx, node)
if readErr != nil {
return out
}
out.modules, out.unheld = assigned, unheldOver(shelf, node, assigned)
return out
}
// splitModules is a surface's one `module` field as the modules it names: several, comma-separated,
// are one act (novox/hq ADR 0207), so the holders that depend on each other go on together from the
// command API and the controller seat's verbs as they do from the command line.
+18 -1
View File
@@ -408,6 +408,9 @@ func comparisonOf(module string, c comparison, opts takeOptions) (preview string
switch {
case sec.Origin == inventory.OriginAccepted:
fmt.Fprintf(&b, " %s %s: accepted from a person, carried in as it is\n", what, name)
case sec.Origin == inventory.OriginIssued:
// Taken from the provider's software (novox/hq ADR 0305): the value the running service uses.
fmt.Fprintf(&b, " %s %s: issued by the provider's software, carried in as it is\n", what, name)
case opts.Mint[sec.Name]:
fmt.Fprintf(&b, " %s %s: minted by the mesh; the service takes the new value, as --mint said\n", what, name)
case foundData:
@@ -719,7 +722,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
return "", err
}
derived := derivedFilter{rules: rules, foundation: with.Foundation, mesh: with.Mesh,
outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks}
outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks, local: with.LocalNetworks}
filtering, err := inv.FilteringOf(ctx, node)
if err != nil {
return "", err
@@ -927,6 +930,9 @@ type derivedFilter struct {
// The filter constrains what arrives on them; everything arriving elsewhere is this machine's
// own guest and is not filtered.
outwardLinks []string
// local is the machine's local networks, which a port from `local-network` stays open to (novox/hq
// ADR 0340).
local []string
}
// closesOutside is what a narrowing from everywhere to the private network is called: it closes.
@@ -974,6 +980,17 @@ func (d derivedFilter) fate(r inventory.Reach) string {
if !onMesh {
return fmt.Sprintf("%s — declared by %s from the mesh only", closesOutside, by)
}
case catalogue.FromLocalNetwork:
// The private network and this machine's local networks (novox/hq ADR 0340): it closes to
// everything else outside, and says which local networks stay.
if !onMesh {
if len(d.local) == 0 {
return fmt.Sprintf("%s — declared by %s from the local network, and this machine "+
"reported none", closesOutside, by)
}
return fmt.Sprintf("WILL CLOSE to everything outside the private network and %s — declared "+
"by %s from the local network", strings.Join(d.local, ", "), by)
}
}
return fmt.Sprintf("declared by %s (from %s)", by, rule.From)
}
@@ -0,0 +1,180 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **A refusal the bus said once is counted once** (novox/hq issue 402). The bus refused the controller
// one publish at 18:52:59 UTC on 2026-10-10 and never again, yet S9 looked at the one remembered
// advisory every half minute for the hour it is kept, and the condition said "observed 15 time(s), last
// 13s ago" with a refusal in its evidence every 30 seconds: two of us read it as a refusal going on and
// went looking for a missing grant. The condition counts what the bus said, with when it last said it,
// never how often the controller looked.
func TestARefusalSaidOnceIsCountedOnceHoweverOftenItIsLookedAt(t *testing.T) {
refused := time.Date(2026, 10, 10, 18, 52, 59, 0, time.UTC)
now := refused.Add(10 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
// The words the live condition bus.controller.refused carried on 2026-10-10.
said := "the bus refused the controller: nats: permissions violation: Permissions Violation for " +
`Publish to "mesh.seat.mesh-delivery.tool.times"`
advisory := link.Advisory{Kind: link.AdvisoryRefused, ID: "controller", Said: said,
First: refused, Last: refused, Count: 1}
look := func() conditions.Condition {
t.Helper()
f := &signalFacts{now: now, host: "novox", advisories: []link.Advisory{advisory}}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
c, found, err := conditions.ReadOne(t.Context(), store, "bus.controller.refused")
if err != nil || !found {
t.Fatalf("bus.controller.refused: found %v, %v", found, err)
}
return c
}
var c conditions.Condition
for range 15 {
c = look()
now = now.Add(30 * time.Second)
}
if c.Observations != 1 || len(c.Evidence) != 1 {
t.Fatalf("one refusal, looked at 15 times, reads as observed %d time(s) with %d evidence lines: %+v",
c.Observations, len(c.Evidence), c.Evidence)
}
if !c.LastObserved.Equal(refused) || !c.Evidence[0].At.Equal(refused) {
t.Fatalf("last observed %s, evidence at %s: the refusal was at %s", c.LastObserved, c.Evidence[0].At, refused)
}
// The bus refuses it three times more between two looks: the condition counts four refusals, says
// the newest, and adds one line of evidence for the look that saw them.
again := now.Add(-5 * time.Second)
advisory.Last, advisory.Count = again, 4
c = look()
if c.Observations != 4 || len(c.Evidence) != 2 || !c.LastObserved.Equal(again) || !c.Evidence[0].At.Equal(again) {
t.Fatalf("four refusals read as observed %d time(s), last %s, evidence %+v", c.Observations, c.LastObserved, c.Evidence)
}
if !strings.Contains(c.Summary, "(4 times since 18:52 UTC)") {
t.Fatalf("summary %q", c.Summary)
}
now = now.Add(30 * time.Second)
if c = look(); c.Observations != 4 || len(c.Evidence) != 2 {
t.Fatalf("a look with nothing new counted: observed %d time(s), evidence %+v", c.Observations, c.Evidence)
}
}
// **A condition that looks at the present still counts its looks** (novox/hq issue 402 changes only
// what reads a record): a machine silent for three looks was observed three times, and says so.
func TestAConditionOfThePresentCountsItsLooks(t *testing.T) {
now := time.Date(2026, 10, 10, 19, 0, 0, 0, time.UTC)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
var c conditions.Condition
for range 3 {
var err error
if c, err = k.Observe(t.Context(), conditions.Observation{Scope: conditions.ScopeMachine, ID: "ace",
Kind: "silent", Machine: "ace", Severity: conditions.Warning, Summary: "ace has not been heard from",
Source: "S1"}); err != nil {
t.Fatal(err)
}
now = now.Add(30 * time.Second)
}
if c.Observations != 3 || len(c.Evidence) != 3 || !c.LastObserved.Equal(now.Add(-30*time.Second)) {
t.Fatalf("observed %d time(s), %d evidence lines, last %s", c.Observations, len(c.Evidence), c.LastObserved)
}
}
// **One key, one watcher** (novox/hq issue 440). A consumer's max-deliveries condition is said from
// DEAD_LETTERS while it holds a message the consumer gave up on (issue 330). A max-deliveries advisory
// without a token names the same key; read beside it, each look added an observation and a line of
// evidence through the dead-letter half, and the two summaries overwrote each other on every look. The
// dead-letter row alone says that key, and counts the messages given up on, never the looks.
func TestAHeldDeadLetterIsCountedByWhatWasGivenUpNotByTheLooks(t *testing.T) {
gaveUp := time.Date(2026, 10, 10, 21, 4, 0, 0, time.UTC)
now := gaveUp.Add(20 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
const key = "bus.EVENTS.media_sonarr.max-deliveries"
held := map[string]int{"EVENTS.media_sonarr": 1}
newest := map[string]time.Time{"EVENTS.media_sonarr": gaveUp}
advisory := link.Advisory{Kind: link.AdvisoryMaxDeliveries, ID: "EVENTS.media_sonarr", Stream: "EVENTS",
Consumer: "media_sonarr", Said: "sonarr on media handed message 7 over 5 times and gave up on it",
First: gaveUp, Last: gaveUp, Count: 1}
look := func() conditions.Condition {
t.Helper()
f := &signalFacts{now: now, host: "novox", advisories: []link.Advisory{advisory},
deadLetters: held, deadLettersNewest: newest}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
c, found, err := conditions.ReadOne(t.Context(), store, key)
if err != nil || !found {
t.Fatalf("%s: found %v, %v", key, found, err)
}
if !strings.Contains(c.Summary, "dead-letters verb") {
t.Fatalf("the summary is not the dead-letter row's: %q", c.Summary)
}
return c
}
var c conditions.Condition
for range 5 {
c = look()
now = now.Add(30 * time.Second)
}
if c.Observations != 1 || len(c.Evidence) != 1 || !c.LastObserved.Equal(gaveUp) {
t.Fatalf("one message given up on, looked at five times, reads as observed %d time(s), last %s, "+
"with %d evidence lines: %+v", c.Observations, c.LastObserved, len(c.Evidence), c.Evidence)
}
// The consumer gives up on a second message between two looks: two given up on, one more line.
again := now.Add(-10 * time.Second)
held["EVENTS.media_sonarr"], newest["EVENTS.media_sonarr"] = 2, again
c = look()
if c.Observations != 2 || len(c.Evidence) != 2 || !c.LastObserved.Equal(again) {
t.Fatalf("two given up on read as observed %d time(s), last %s, evidence %+v", c.Observations,
c.LastObserved, c.Evidence)
}
now = now.Add(30 * time.Second)
if c = look(); c.Observations != 2 || len(c.Evidence) != 2 {
t.Fatalf("a look with nothing new counted: observed %d time(s), evidence %+v", c.Observations, c.Evidence)
}
}
// The advisory still says the max-deliveries it alone knows: a message given up on that could not be
// kept, under a key of its own (issue 330), which issue 440's change leaves as it was.
func TestAMessageThatCouldNotBeKeptIsStillSaidFromTheAdvisory(t *testing.T) {
at := time.Date(2026, 10, 10, 21, 4, 0, 0, time.UTC)
f := &signalFacts{now: at, host: "novox", advisories: []link.Advisory{{Kind: link.AdvisoryMaxDeliveries,
ID: "EVENTS.media_sonarr", Stream: "EVENTS", Consumer: "media_sonarr", Token: link.AdvisoryNotKept,
Said: "sonarr on media gave up on message 7, and it could not be kept", First: at, Last: at, Count: 1}}}
said := watchAdvisories(f)
if len(said) != 1 || said[0].Key() != "bus.EVENTS.media_sonarr.not-kept" {
t.Fatalf("%+v", said)
}
}
// A max-deliveries advisory without a token names a consumer's dead-letter key, which only DEAD_LETTERS
// says (novox/hq issues 330 and 440): with nothing held, the advisory alone raises nothing.
func TestAMaxDeliveriesAdvisoryAloneRaisesNothing(t *testing.T) {
at := time.Date(2026, 10, 10, 21, 4, 0, 0, time.UTC)
f := &signalFacts{now: at, host: "novox", advisories: []link.Advisory{{Kind: link.AdvisoryMaxDeliveries,
ID: "EVENTS.media_sonarr", Stream: "EVENTS", Consumer: "media_sonarr",
Said: "sonarr on media handed message 7 over 5 times and gave up on it", First: at, Last: at, Count: 1}}}
if said := watchAdvisories(f); len(said) != 0 {
t.Fatalf("said %+v", said)
}
}
+118
View File
@@ -0,0 +1,118 @@
package main
import (
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// kindApplyFailing is S21's kind: a machine whose declarations have been refused, or applied only in part,
// for longer than one apply interval (novox/hq issue 233).
const kindApplyFailing = "apply-failing"
// applyEvery is how often a node-engine applies what it was last told when nothing new arrives: its
// ReconcileEvery. A failure still there after one more of them is not one the next apply clears.
const applyEvery = 5 * time.Minute
// packageManagerKey is how a node-engine names a package manager that does not answer among what failed
// (mesh-host's probe.go): "the package manager (pacman)".
const packageManagerKey = "the package manager ("
// watchApplyFailing is S21: **a machine that does not apply what it was sent is said by the mesh, not only
// to someone who asks for `status`** (novox/hq issue 233, issue 187's family). A workstation refused every
// declaration for an hour and a half because its package manager did not answer, and the mesh's only word
// on it was a line in `status`. Raised once the machine's first report of this failure is older than one apply
// interval **while the machine is heard** — judged by time, not by how many reports said it: a node-engine
// says a re-apply only when how it went changed, so the same failure is one report however long it lasts.
// Names the refusal's or the failure's words; cleared by the clean apply the node-engine then says. Not while
// the machine is silent, asleep or not known: S1 says silent, and asleep is not lost.
func watchApplyFailing(f *signalFacts) []conditions.Observation {
heard := map[string]bool{}
for _, m := range f.machines {
if !m.asleep() && !m.lastHeard.IsZero() && f.now.Sub(m.lastHeard) <= heartbeatBound(m.every) {
heard[m.name] = true
}
}
var out []conditions.Observation
for _, d := range f.doing {
if !d.Wrong() || d.Since == nil || f.now.Sub(*d.Since) <= applyEvery || !heard[d.Node] {
continue
}
out = append(out, applyFailing(d, f.now))
}
return out
}
// applyFailing is one machine's S21 condition.
func applyFailing(d inventory.Doing, now time.Time) conditions.Observation {
m := d.Node
since := d.Since.UTC().Format("2006-01-02 15:04 MST")
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: m, Kind: kindApplyFailing, Machine: m,
Severity: conditions.Warning, Said: doingWords(d),
Resolved: m + " applies what it is sent again"}
how := "refused its declaration whole"
if d.Outcome != inventory.OutcomeRefused {
how = "applied its declaration only in part"
}
o.Summary = fmt.Sprintf("%s has %s since %s (bound %s): %s", m, how, since, ago(applyEvery), failedNames(d))
o.Headline = m + " is not applying its update"
o.Explanation = fmt.Sprintf("%s has %s for %s, and trying again has not helped. The details say what fails.",
m, plainHow(d), humanDuration(now.Sub(*d.Since)))
if tool := unansweringTool(d); tool != "" {
o.Summary = fmt.Sprintf("%s: %s does not answer, so no package reaches it; everything else it was sent "+
"is applied, since %s (bound %s)", m, tool, since, ago(applyEvery))
o.Headline = fmt.Sprintf("%s cannot install packages", m)
o.Explanation = fmt.Sprintf("The package manager on %s (%s) has not answered for %s, so no package reaches "+
"it. Everything else is still applied, so an update that repairs it can land; if none does, it needs a "+
"look on the machine.", m, tool, humanDuration(now.Sub(*d.Since)))
}
return o
}
// plainHow is how a machine failed, as the operator reads it.
func plainHow(d inventory.Doing) string {
if d.Outcome == inventory.OutcomeRefused {
return "refused every update"
}
return "applied its updates only in part"
}
// unansweringTool is the package manager a machine said does not answer, or "".
func unansweringTool(d inventory.Doing) string {
for _, r := range d.Failed {
if rest, ok := strings.CutPrefix(r.ID, packageManagerKey); ok {
return strings.TrimSuffix(rest, ")")
}
}
return ""
}
// failedNames is what failed, by name: the refusal, or the failed resources' ids.
func failedNames(d inventory.Doing) string {
if d.Outcome == inventory.OutcomeRefused {
return "refused"
}
ids := make([]string, 0, len(d.Failed))
for _, r := range d.Failed {
ids = append(ids, r.ID)
}
sort.Strings(ids)
return "failed " + strings.Join(ids, ", ")
}
// doingWords is the machine's own words on what failed: the evidence, kept inside the mesh.
func doingWords(d inventory.Doing) string {
if d.Outcome == inventory.OutcomeRefused {
return "refused: " + oneLine(d.Refused)
}
var parts []string
for _, r := range d.Failed {
parts = append(parts, r.ID+": "+oneLine(r.Error))
}
sort.Strings(parts)
return strings.Join(parts, "; ")
}
+123
View File
@@ -0,0 +1,123 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// unansweringPackageManager is anchor's last report as the workstation's read on 2026-10-04, once its
// node-engine applies the rest (novox/hq issue 233): the package failed and the package manager did not answer,
// the same failure on times reports since for before now.
func unansweringPackageManager(now time.Time, forHowLong time.Duration, times int) inventory.Doing {
since := now.Add(-forHowLong)
return inventory.Doing{Node: "anchor", Outcome: inventory.OutcomeFailed, At: now.Add(-time.Second),
Since: &since, Times: times, Applied: 41, Failed: []inventory.FailedResource{
{ID: "apply", Error: `applying "pacman.base": nothing can be said about "base": this is the arch host and ` +
"pacman does not answer here. Either this machine is not Arch, or its package database is broken: " +
"pacman exited 1: error: config file /etc/pacman.conf could not be read: No such file or directory"},
{ID: "the package manager (pacman)", Error: "pacman does not answer here, so every package in this " +
"declaration fails and the rest of it is applied (novox/hq issue 233): pacman exited 1: error: config " +
"file /etc/pacman.conf could not be read: No such file or directory"}}}
}
// **A machine whose package manager does not answer is raised by the mesh itself, naming the tool and the
// probe's words** (novox/hq issue 233): the words in the evidence, the path kept out of the summary.
func TestAPackageManagerThatDoesNotAnswerIsRaisedNamingItsWords(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
f := calm(now)
f.doing = []inventory.Doing{unansweringPackageManager(now, 90*time.Minute, 18)}
got := watchApplyFailing(f)
if len(got) != 1 {
t.Fatalf("raised %d conditions, want one: %+v", len(got), got)
}
o := got[0]
if o.Key() != "machine.anchor.apply-failing" || o.Machine != "anchor" {
t.Errorf("the condition is not the machine's: %s", o.Key())
}
if !strings.Contains(o.Summary, "pacman does not answer") || !strings.Contains(o.Summary, "since 2026-10-04 12:30") {
t.Errorf("the summary does not name the tool and since when: %s", o.Summary)
}
if strings.Contains(o.Summary, "/etc/") {
t.Errorf("the summary carries a path, which belongs in the evidence: %s", o.Summary)
}
if !strings.Contains(o.Said, "could not be read") {
t.Errorf("the evidence does not carry the probe's words: %s", o.Said)
}
if !strings.Contains(o.Headline, "cannot install packages") || o.Explanation == "" || o.Resolved == "" {
t.Errorf("the plain words do not say it: %+v", o)
}
}
// **A whole refusal is raised too**, with the node-engine's words: what the workstation's report said before the
// node-engine applied the rest.
func TestARefusalThatGoesOnIsRaised(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
since := now.Add(-time.Hour)
f := calm(now)
f.doing = []inventory.Doing{{Node: "anchor", Outcome: inventory.OutcomeRefused, Since: &since, Times: 12,
Refused: "this is the arch host and pacman does not answer here", At: now.Add(-time.Minute)}}
got := watchApplyFailing(f)
if len(got) != 1 || !strings.Contains(got[0].Summary, "refused its declaration whole") ||
!strings.Contains(got[0].Said, "pacman does not answer") {
t.Fatalf("a refusal that went on for an hour was not raised with its words: %+v", got)
}
}
// **Judged by time while the machine is heard, not by how many reports said it** (review of novox/hq issue 233):
// a node-engine says the same failure once, so one report an apply interval old is a fault that goes on; one
// younger is not yet; and a silent or sleeping machine is S1's.
func TestOneReportOlderThanAnApplyIsRaisedAndAQuietMachineIsNot(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
f := calm(now)
f.doing = []inventory.Doing{unansweringPackageManager(now, 6*time.Minute, 1)}
if got := watchApplyFailing(f); len(got) != 1 {
t.Errorf("one report of a failure six minutes old was not raised: %+v", got)
}
f.doing = []inventory.Doing{unansweringPackageManager(now, 4*time.Minute, 1)}
if got := watchApplyFailing(f); len(got) != 0 {
t.Errorf("a failure four minutes old was raised before one apply could clear it: %+v", got)
}
f.doing = []inventory.Doing{unansweringPackageManager(now, time.Hour, 1)}
f.machines[0].lastHeard = now.Add(-time.Hour)
if got := watchApplyFailing(f); len(got) != 0 {
t.Errorf("a silent machine was raised for its last report: %+v", got)
}
f = calm(now)
f.doing = []inventory.Doing{unansweringPackageManager(now, time.Hour, 1)}
f.doing[0].Node = "a-machine-the-mesh-does-not-know"
if got := watchApplyFailing(f); len(got) != 0 {
t.Errorf("a machine not heard of was raised: %+v", got)
}
}
// **A failed re-apply followed by a clean one clears S21** (review of novox/hq issue 233): the node-engine says
// the clean re-apply, the store keeps it as applied with no failure since, and the condition the failure raised
// is cleared through the keeper on the next tick.
func TestACleanReapplyAfterAFailedOneClearsTheCondition(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
defer k.Close(context.Background())
w := &watchdogs{keeper: k, started: now.Add(-time.Hour)}
failing := calm(now)
failing.doing = []inventory.Doing{unansweringPackageManager(now, 20*time.Minute, 1)}
w.see(t.Context(), failing)
open, err := k.Open(t.Context())
if err != nil || len(open) != 1 || open[0].Kind != kindApplyFailing {
t.Fatalf("the failed re-apply raised %+v (%v), want apply-failing", open, err)
}
clean := calm(now.Add(5 * time.Minute))
clean.doing = nil // the clean re-apply recorded: NotDoingWhatTheyWereTold no longer lists the machine
w.see(t.Context(), clean)
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("the condition stayed open after the clean re-apply: %+v", open)
}
}
+9
View File
@@ -213,6 +213,8 @@ type asker struct {
saidNoRouter bool
saidNoGrant bool
// saidGrantCaveat is whether a grant held with a caveat was said, so it is said once.
saidGrantCaveat bool
mu sync.Mutex
nudged chan struct{}
@@ -317,6 +319,13 @@ func (a *asker) reconcile(ctx context.Context) error {
}
return a.sayUnasked(ctx, unasked, "the bus does not hold the controller's grant to ask yet: "+why)
}
if why != "" && !a.saidGrantCaveat {
// Held, with a caveat worth one line (an older bus module cannot confirm the reload).
a.logf("the controller asks: %s", why)
a.saidGrantCaveat = true
} else if why == "" {
a.saidGrantCaveat = false
}
a.saidNoGrant = false
}
channels := ""
+205
View File
@@ -12,6 +12,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
@@ -808,3 +809,207 @@ func TestARefusalIsAskedAgainAndTheUndeliveredConditionClearsOnceTaken(t *testin
t.Errorf("the explanation says its verdict twice: %q", e)
}
}
// hq issue 353, the review: the bus holds the controller's grant to ask only when both facts hold — the mesh
// composes the controller's own grant, and the bus's module says it is in force on the running server — and
// each fact not known is not held, said with why.
func TestTheGrantIsHeldOnlyWhenComposedAndInForceOnTheBus(t *testing.T) {
yes := func(context.Context) (bool, string, error) { return true, "", nil }
no := func(why string) grantJudge {
return func(context.Context) (bool, string, error) { return false, why, nil }
}
broken := func(context.Context) (bool, string, error) { return false, "", errors.New("the store is away") }
for _, c := range []struct {
name string
composed, inForce grantJudge
held bool
says string
}{
{"composed and in force", yes, yes, true, ""},
{"not composed", no("no router is assigned"), yes, false, "no router is assigned"},
{"composed, written, not reloaded", yes, no("the server has not reloaded"), false, "has not reloaded"},
{"the composition unknown", broken, yes, false, "could not be worked out: the store is away"},
{"the bus unknown", yes, broken, false, "could not be worked out: the store is away"},
{"nothing judges the bus", yes, nil, false, "not judged"},
} {
held, why, err := grantHeldBy(c.composed, c.inForce, time.Now)(context.Background())
if err != nil || held != c.held || !strings.Contains(why, c.says) {
t.Errorf("%s: held %v (%q, %v)", c.name, held, why, err)
}
}
// Remembered for grantLookEvery, then judged again.
now := time.Date(2026, 10, 9, 18, 0, 0, 0, time.UTC)
inForce := false
judge := grantHeldBy(yes, func(context.Context) (bool, string, error) { return inForce, "not yet", nil },
func() time.Time { return now })
if held, _, _ := judge(context.Background()); held {
t.Fatal("held before the bus enforces it")
}
inForce = true
if held, _, _ := judge(context.Background()); held {
t.Error("judged again inside grantLookEvery")
}
now = now.Add(grantLookEvery)
if held, _, _ := judge(context.Background()); !held {
t.Error("not judged again after grantLookEvery")
}
}
// The controller's own grant, as composed: present while a router that takes asks is assigned, absent
// otherwise — whatever else in the user list changes.
func TestTheControllersOwnGrantToAskIsReadFromTheComposition(t *testing.T) {
router := broker.Declared{Module: "messenger", Holds: []broker.Seat{{Name: broker.AsksSeat, Scope: "mesh",
Accepts: []string{"ask", "cancel"}, Emits: []string{"decided"}, ByCaller: []string{"ask", "cancel", "decided"}}}}
other := broker.Declared{Module: "dunst"}
with := broker.Records{Nodes: []string{"anchor"}, Assigned: map[string][]broker.Declared{"anchor": {router, other}}}
without := broker.Records{Nodes: []string{"anchor"}, Assigned: map[string][]broker.Declared{"anchor": {other}}}
if may, err := controllerMayAsk(with); err != nil || !may {
t.Errorf("with a router assigned: %v %v", may, err)
}
if may, err := controllerMayAsk(without); err != nil || may {
t.Errorf("with no router assigned: %v %v", may, err)
}
}
// The bus module's answer, with the record's word on the list sent as the check only where the answer does
// not settle it (reviews 22 and 31 of PR 188); each way it is not held names its own remedy.
func TestTheBusModulesAnswerNamesTheRemedyForEachCase(t *testing.T) {
const (
reloaded = `{"allowed":true,"in_force":true,"file_written":"2026-10-09T15:55:56Z","server_loaded":"2026-10-09T15:56:05Z"}`
notLoaded = `{"allowed":true,"in_force":false,"why":"the user list on the machine allows it, and the server has not reloaded since it was written","file_written":"2026-10-09T15:55:56Z","server_loaded":"2026-10-09T15:50:00Z"}`
noVarz = `{"allowed":true,"in_force":false,"why":"the server's last configuration load could not be read: refused"}`
noStat = `{"allowed":true,"in_force":false,"why":"when the bus's user list was written could not be read inside mesh-broker-nats: exit 1"}`
notAllowed = `{"allowed":false,"in_force":false,"why":"the user list on the machine does not allow it"}`
)
current := busListState{Machine: "anchor"}
behind := busListState{Machine: "anchor", Behind: true}
stepped := busListState{Machine: "anchor", Behind: true, BusStep: "sending anchor would replace the bus; `bus upgrade --why …`"}
uncomposed := busListState{Machine: "anchor", Behind: true, Uncomposed: errors.New("module foo requires a provision nobody provides")}
for _, c := range []struct {
name, answer string
askErr error
list busListState
listErr error
held bool
says, not string
}{
{"in force", reloaded, nil, behind, nil, true, "", "push"},
{"written, not reloaded", notLoaded, nil, current, nil, false, "within about 5 s", "push"},
{"in force not known: varz", noVarz, nil, current, nil, false, "is not known: the server's last configuration load", "wait"},
{"in force not known: stat", noStat, nil, current, nil, false, "is not known: when the bus's user list was written", "wait"},
{"not allowed, list behind", notAllowed, nil, behind, nil, false, "`push anchor` carries it", "5 s"},
{"not allowed, bus step pending", notAllowed, nil, stepped, nil, false, "`bus upgrade", "`push anchor`"},
{"not allowed, list uncomposed", notAllowed, nil, uncomposed, nil, false, "`plan anchor`", "`push anchor`"},
{"older module, allowed, list current", `{"allowed":true}`, nil, current, nil, true, "not confirmed", "push"},
{"older module, allowed, list behind", `{"allowed":true}`, nil, behind, nil, true, "not confirmed", "push"},
{"older module, allowed, list unknown", `{"allowed":true}`, nil, current, errors.New("the store is away"), true, "`bus upgrade`", "push"},
{"older module, not allowed, list behind", `{"allowed":false}`, nil, behind, nil, false, "`push anchor` carries it", "5 s"},
{"module cannot answer, list current", "", errors.New("no responders"), current, nil, true, "could not answer", "push"},
{"module cannot answer, list behind", "", errors.New("no responders"), behind, nil, false, "`push anchor` carries it", ""},
{"module cannot answer, list unknown", "", errors.New("no responders"), current, errors.New("the store is away"), false, "could not be worked out", ""},
{"answer unreadable, list behind", `not json`, nil, behind, nil, false, "could not be read", ""},
} {
judge := busGrantBy(
func(context.Context) (string, json.RawMessage, error) {
return "anchor", json.RawMessage(c.answer), c.askErr
},
func(context.Context) (busListState, error) { return c.list, c.listErr })
held, why, _ := judgeGrantOnce(judge)
if held != c.held || !strings.Contains(why, c.says) || (c.not != "" && strings.Contains(why, c.not)) {
t.Errorf("%s: held %v (%q)", c.name, held, why)
}
}
none := busGrantBy(func(context.Context) (string, json.RawMessage, error) { return "", nil, nil }, nil)
if held, why, _ := judgeGrantOnce(none); held || !strings.Contains(why, "no module holds the bus") {
t.Errorf("no bus: held %v (%q)", held, why)
}
}
// judgeGrantOnce is the whole judgement with the composed grant held, as the asker makes it.
func judgeGrantOnce(bus grantJudge) (bool, string, error) {
yes := func(context.Context) (bool, string, error) { return true, "", nil }
return grantHeldBy(yes, bus, time.Now)(context.Background())
}
// The orders reviews 22 and 31 of PR 188 found: this controller live before the bus module that answers
// in_force, with the bus's machine behind on its list — and, after mesh-catalog #166 merged, unpushable until a
// person's bus upgrade. An older module that allows the ask asks (once, with the caveat said); a new module that
// has not reloaded names the wait, not a push; one that cannot say whether it reloaded names what is not known.
func TestAnOlderBusModuleNeitherSilencesTheAsksNorNamesTheWrongRemedy(t *testing.T) {
for _, c := range []struct {
name, answer string
list busListState
asks bool
says, not string
}{
{"older module, list current", `{"allowed":true}`, busListState{Machine: "anchor"}, true, "not confirmed", ""},
{"older module, list behind, bus step pending", `{"allowed":true}`,
busListState{Machine: "anchor", Behind: true, BusStep: "a planned step"}, true, "not confirmed", ""},
{"new module, not reloaded",
`{"allowed":true,"in_force":false,"why":"not reloaded","file_written":"2026-10-09T15:55:56Z","server_loaded":"2026-10-09T15:50:00Z"}`,
busListState{Machine: "anchor"}, false, "within about 5 s", "push"},
{"new module, reload not known", `{"allowed":true,"in_force":false,"why":"the server's last configuration load could not be read: refused"}`,
busListState{Machine: "anchor"}, false, "is not known", "wait"},
} {
t.Run(c.name, func(t *testing.T) {
r := newAskerRig(t)
var said []string
r.a.logf = func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }
var raised [][]conditions.Observation
r.a.raise = func(_ context.Context, obs []conditions.Observation) error {
raised = append(raised, obs)
return nil
}
yes := func(context.Context) (bool, string, error) { return true, "", nil }
r.a.grantHeld = grantHeldBy(yes, busGrantBy(
func(context.Context) (string, json.RawMessage, error) {
return "anchor", json.RawMessage(c.answer), nil
},
func(context.Context) (busListState, error) { return c.list, nil }),
func() time.Time { return r.now })
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
_ = r.a.reconcile(context.Background())
sent := r.asksSent(t)
if c.asks {
if len(sent) != 1 || sent[0].About != heldCondition().Key {
t.Fatalf("not asked: %+v", sent)
}
n := 0
for _, s := range said {
if strings.Contains(s, c.says) {
n++
}
}
if n != 1 {
t.Errorf("the caveat said %d times: %q", n, said)
}
return
}
if len(sent) != 0 {
t.Fatalf("asked while the bus does not hold the grant: %+v", sent)
}
if len(raised) == 0 || len(raised[len(raised)-1]) != 1 ||
raised[len(raised)-1][0].Kind != "asks-undelivered" {
t.Fatalf("not said as undelivered: %+v", raised)
}
summary := raised[len(raised)-1][0].Summary
if !strings.Contains(summary, c.says) || (c.not != "" && strings.Contains(summary, c.not)) {
t.Errorf("the condition names the wrong remedy: %q", summary)
}
})
}
}
// The serving asker is wired to the real judgement: with records and a bus it cannot read, the grant is not
// held, and so nothing is asked (a stub saying yes, or no judgement at all, would ask).
func TestTheServingAskerJudgesTheGrantForReal(t *testing.T) {
a := servingAsker(&stores{}, nil, nil, nil)
if a.grantHeld == nil {
t.Fatal("the serving asker does not judge the bus's grant")
}
held, why, err := a.grantHeld(context.Background())
if err != nil || held || !strings.Contains(why, "could not be worked out") {
t.Errorf("held %v (%q, %v) with nothing to read", held, why, err)
}
}
+271 -21
View File
@@ -235,12 +235,27 @@ func routerHereIn(inv *inventory.Inventory) func(ctx context.Context) (bool, err
}
}
// grantHeldIn says whether the bus holds the controller's grant to ask (novox/hq issue 353): the user list the
// machine holding the bus was last sent is the one the mesh composes now (brokerBehind, the same judgement a
// push makes to send that machine first). While it is behind, the controller's ask is refused by the bus,
// whatever the record says of the router, so nothing is asked and the operator is told to push that machine.
// Judged at most every grantLookEvery: composing the list resolves the bus's machine whole.
func grantHeldIn(open *stores) func(ctx context.Context) (bool, string, error) {
// grantHeldIn says whether the bus holds the controller's grant to ask (novox/hq issue 353), judged on the two
// facts that make it so, and failing closed on each:
//
// 1. **composed**: the user list the mesh composes now grants the controller itself to publish its ask
// (controllerMayAsk) — the controller's own grant, not whether any user of the list changed;
// 2. **in force**: the bus's own module, on the machine holding the bus, says the user list there allows it
// and the server reloaded after that list was written (the nats module's `nats_user_can`, `in_force`) — a
// list sent is not a list the bus enforces until it reloads.
//
// Anything that cannot be worked out is not held, with why, so the conditions that need the operator are said
// as undelivered rather than asked into a refusal.
func grantHeldIn(open *stores, conn *nats.Conn) func(ctx context.Context) (bool, string, error) {
return grantHeldBy(composedGrantIn(open), busGrantIn(open, conn), time.Now)
}
// grantJudge is one of the two facts: whether it holds, and why not.
type grantJudge func(ctx context.Context) (bool, string, error)
// grantHeldBy is the judgement over the two facts, remembered for grantLookEvery: composing the list resolves
// the bus's machine whole, and asking the bus's module is a call.
func grantHeldBy(composed, inForce grantJudge, now func() time.Time) func(ctx context.Context) (bool, string, error) {
var mu sync.Mutex
var at time.Time
var held bool
@@ -248,22 +263,251 @@ func grantHeldIn(open *stores) func(ctx context.Context) (bool, string, error) {
return func(ctx context.Context) (bool, string, error) {
mu.Lock()
defer mu.Unlock()
if !at.IsZero() && time.Since(at) < grantLookEvery {
if !at.IsZero() && now().Sub(at) < grantLookEvery {
return held, why, nil
}
machine, behind, err := brokerBehind(ctx, open, nil)
held, why = judgeGrant(ctx, composed, inForce)
at = now()
return held, why, nil
}
}
// judgeGrant is held only when both facts hold; an error is a fact not known, never a yes. A fact that holds
// with a caveat (a reload not confirmed) gives that caveat as the why of a yes.
func judgeGrant(ctx context.Context, composed, inForce grantJudge) (bool, string) {
var caveat []string
for _, fact := range []struct {
judge grantJudge
what string
}{{composed, "whether the mesh composes the controller a grant to ask"}, {inForce, "whether the bus enforces it"}} {
if fact.judge == nil {
return false, fact.what + " is not judged here"
}
ok, why, err := fact.judge(ctx)
if err != nil {
return false, fact.what + " could not be worked out: " + err.Error()
}
if !ok {
return false, why
}
if why != "" {
caveat = append(caveat, why)
}
}
return true, strings.Join(caveat, "; ")
}
// controllerMayAsk says whether the controller's own grant, as composed from these records, lets it publish its
// ask (ADR 0259 §3: composed while a router that takes asks under its asker's name is assigned).
func controllerMayAsk(records broker.Records) (bool, error) {
users, err := broker.Users(records)
if err != nil {
return false, err
}
for _, u := range users {
if u.Kind != broker.KindController {
continue
}
perms, err := broker.PermissionsFor(u)
if err != nil {
return false, err
}
return broker.MayPublish(perms, asks.AskSubject(askerName)), nil
}
return false, errors.New("the composition holds no controller")
}
// composedGrantIn is the first fact, from the mesh's records.
func composedGrantIn(open *stores) grantJudge {
return func(ctx context.Context) (bool, string, error) {
if open == nil || open.inventory == nil {
return false, "", errors.New("the mesh's records are not open")
}
records, err := open.inventory.BusRecords(ctx)
if err != nil {
return false, "", err
}
at, held, why = time.Now(), !behind, ""
if behind {
why = fmt.Sprintf("the bus's user list on %s is behind what the mesh composes, so the bus has not been "+
"given the controller's grant to ask; `push %s` carries it", machine, machine)
may, err := controllerMayAsk(records)
if err != nil || may {
return may, "", err
}
return held, why, nil
return false, "the mesh composes the controller no grant to ask: no router that takes asks under its " +
"asker's name is assigned", nil
}
}
// busGrantIn is the second fact, asked of the bus's own module on the machine holding the bus, with the check
// main made before it (novox/hq issue 353, PR 186: is the user list sent to the bus's machine the one composed
// now) kept for the cases the module's word does not settle.
func busGrantIn(open *stores, conn *nats.Conn) grantJudge {
ask := func(ctx context.Context) (string, json.RawMessage, error) {
if open == nil || open.inventory == nil || conn == nil {
return "", nil, errors.New("the mesh's records or the bus are not open")
}
holders, err := seatHolders(ctx, open.inventory)
if err != nil {
return "", nil, err
}
h, held := holders[theBrokerSeat]
if !held || h.Node == "" || h.Module == "" {
return "", nil, nil
}
a, err := link.AskModuleToolOn(ctx, conn, h.Module, busUserCanTool, h.Node, map[string]any{
"user": broker.ControllerName, "action": "publish", "subject": asks.AskSubject(askerName)}, 15*time.Second)
if err != nil {
return h.Node, nil, err
}
if a.Error != "" {
return h.Node, nil, errors.New(a.Error)
}
return h.Node, a.Result, nil
}
list := func(ctx context.Context) (busListState, error) {
if open == nil || open.inventory == nil {
return busListState{}, errors.New("the mesh's records are not open")
}
s, err := brokerListState(ctx, open, nil)
if err != nil || s.Machine == "" || !s.Behind {
return s, err
}
step, err := busHeld(ctx, open.inventory, []string{s.Machine})
if err != nil {
return busListState{}, err
}
s.BusStep = step[s.Machine]
return s, nil
}
return busGrantBy(ask, list)
}
// busListState is the user list on the bus's machine as the controller's record knows it: whether the list
// sent there is behind the one composed now, why it could not be composed, and the bus step a send there waits
// on (that machine cannot be pushed until a person's `bus upgrade`).
type busListState struct {
Machine string
Behind bool
Uncomposed error
BusStep string
}
// remedy is the one thing that sends the bus's machine the list composed now: the plan's own reason when it
// cannot be composed, the bus upgrade while sends there wait on it, and otherwise a push of that machine.
func (s busListState) remedy(node string) string {
if s.Machine != "" {
node = s.Machine
}
switch {
case s.Uncomposed != nil:
return fmt.Sprintf("the user list for %s cannot be composed (%v); `plan %s` says what to fix first",
node, s.Uncomposed, node)
case s.BusStep != "":
return fmt.Sprintf("sends to %s wait on the bus's planned step, so a push will not carry it: %s", node, s.BusStep)
}
return fmt.Sprintf("`push %s` carries it", node)
}
// busGrantBy judges the bus's word, and names for each way it is not held the one thing that makes it held:
//
// - **in force**: held;
// - **allowed, from a module older than `in_force`**: held, with the caveat that the reload is not confirmed.
// The file allows the publish, and the bus reloads within about 5 s of a change to it, so the worst case is
// one ask refused in those seconds, which is asked again (issue 369). The list check is not consulted:
// while a bus step is pending the bus's machine cannot be pushed, so any change to the composed list would
// read as behind for good and silence every ask until a person's bus upgrade (the review of PR 188);
// - **allowed, written, not reloaded** (the module gives both times): wait about 5 s;
// - **allowed, in force not known** (the module could not read a time): what is not known, said;
// - **not allowed, or the module cannot answer**: the list check decides — the remedy that sends the list
// when it is behind (busListState.remedy), and, for a module that cannot answer, held while the list sent
// is current, as main judged it.
//
// ask answers the machine holding the bus and its module's answer ("" when nothing holds the bus); list is the
// record's word on the list sent to that machine.
func busGrantBy(ask func(ctx context.Context) (string, json.RawMessage, error),
list func(ctx context.Context) (busListState, error)) grantJudge {
return func(ctx context.Context) (bool, string, error) {
node, raw, askErr := ask(ctx)
if askErr == nil && node == "" {
return false, "no module holds the bus, so nothing says what it enforces", nil
}
var answer busAnswer
if askErr == nil {
if answer, askErr = readBusAnswer(raw); askErr != nil {
askErr = fmt.Errorf("its answer could not be read: %w", askErr)
}
}
if askErr != nil {
if node == "" {
node = "the bus's machine"
}
s, err := list(ctx)
if err != nil {
return false, "", fmt.Errorf("the bus module on %s could not answer (%v), and whether the list sent "+
"there is the one composed now could not be worked out: %w", node, askErr, err)
}
if s.Behind {
return false, fmt.Sprintf("the bus module on %s could not answer (%v), and the user list sent there is "+
"behind what the mesh composes; %s", node, askErr, s.remedy(node)), nil
}
return true, fmt.Sprintf("the bus module on %s could not answer (%v); judged by the user list sent there, "+
"which is the one composed now", node, askErr), nil
}
notAllowed := func(older string) (bool, string, error) {
s, err := list(ctx)
if err != nil {
return false, "", fmt.Errorf("the user list on %s does not allow the controller to ask%s, and whether "+
"the list sent there is the one composed now could not be worked out: %w", node, older, err)
}
if s.Behind {
return false, fmt.Sprintf("the user list on %s does not allow the controller to ask%s: it is behind "+
"what the mesh composes; %s", node, older, s.remedy(node)), nil
}
return false, fmt.Sprintf("the user list on %s does not allow the controller to ask%s, though the list "+
"sent there is the one composed now; %s again", node, older, s.remedy(node)), nil
}
switch {
case answer.InForce == nil && answer.Allowed:
return true, fmt.Sprintf("the user list on %s allows the controller to ask, but its reload is not "+
"confirmed: the bus module there is older than the answer whether a grant is in force, and the bus "+
"upgrade (`bus upgrade`) carries one that confirms it", node), nil
case answer.InForce == nil:
return notAllowed(" (the bus module there is older than the answer whether a grant is in force)")
case !answer.Allowed:
return notAllowed("")
case *answer.InForce:
return true, "", nil
case answer.FileWritten != "" && answer.ServerLoaded != "":
return false, fmt.Sprintf("the bus on %s has the controller's grant to ask written (%s) but has not "+
"reloaded since (last load %s); it reloads in place within about 5 s, so nothing is to be done but wait",
node, answer.FileWritten, answer.ServerLoaded), nil
}
why := answer.Why
if why == "" {
why = "it gives no reason"
}
return false, fmt.Sprintf("the user list on %s allows the controller to ask, but whether the bus enforces it "+
"is not known: %s", node, why), nil
}
}
// busUserCanTool is the bus module's answer to whether a user may do something, and whether it is in force.
const busUserCanTool = "nats_user_can"
// busAnswer is the bus module's answer to busUserCanTool; InForce is nil from a module older than that field,
// and FileWritten and ServerLoaded are given only when it could read both times (mesh-catalog's nats InForce).
type busAnswer struct {
Allowed bool `json:"allowed"`
InForce *bool `json:"in_force"`
Why string `json:"why"`
FileWritten string `json:"file_written"`
ServerLoaded string `json:"server_loaded"`
}
func readBusAnswer(raw json.RawMessage) (busAnswer, error) {
var answer busAnswer
err := json.Unmarshal(raw, &answer)
return answer, err
}
// grantLookEvery is how often the bus's user list is judged against the one its machine was last sent.
const grantLookEvery = 30 * time.Second
@@ -302,7 +546,19 @@ func startAsking(ctx context.Context, open *stores, server *link.Server, conn *n
fmt.Printf("the operator cannot be asked: %v\n", err)
return
}
a := &asker{
a := servingAsker(open, conn, js, keeper)
if err := server.Decides(a); err != nil {
fmt.Printf("the operator's answers cannot be heard, so nothing is asked: %v\n", err)
return
}
askerFrom = a
go a.keep(ctx)
}
// servingAsker is the serving controller's asker, wired to the mesh: what it reads, publishes, performs and
// records, and the judgements it asks by — the bus's grant to ask among them (novox/hq issue 353).
func servingAsker(open *stores, conn *nats.Conn, js jetstream.JetStream, keeper *conditions.Keeper) *asker {
return &asker{
open: keeper.Open,
silence: func(ctx context.Context, key string, d time.Duration, by, why string) error {
_, err := keeper.Silence(ctx, key, d, by, why)
@@ -322,7 +578,7 @@ func startAsking(ctx context.Context, open *stores, server *link.Server, conn *n
},
routerRecord: routerRecordOf(conn, open.inventory),
routerHere: routerHereIn(open.inventory),
grantHeld: grantHeldIn(open),
grantHeld: grantHeldIn(open, conn),
channels: channelsIn(open.inventory),
raise: func(ctx context.Context, obs []conditions.Observation) error {
return keeper.Reconcile(ctx, sourceAsker, obs)
@@ -330,10 +586,4 @@ func startAsking(ctx context.Context, open *stores, server *link.Server, conn *n
now: time.Now,
logf: func(format string, args ...any) { fmt.Printf(format+"\n", args...) },
}
if err := server.Decides(a); err != nil {
fmt.Printf("the operator's answers cannot be heard, so nothing is asked: %v\n", err)
return
}
askerFrom = a
go a.keep(ctx)
}
+374 -17
View File
@@ -37,6 +37,11 @@ import (
// own commit, the newest first, one walk each, until one is delivered; the older ones are then answered
// by that one.
//
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, ADR 0306 decisions
// 2 and 3, issue 525): it joins a batch it is related to, and one that is related to none and whose delivery
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Walks stay one at
// a time: such a batch is cut when no walk is open, and queued behind the one that is otherwise.
//
// A merge on the controller's own path (a module whose walk waits for nobody's word) never shares a batch with
// one that waits for mesh-delivery's: each kind has a batch of its own, an own-path batch is cut first and folds
// no waiting catalogue walk, and a catalogue walk let go while another walk runs starts once it ended — so no
@@ -173,6 +178,12 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
fmt.Printf("%s merged into %s (%.8s); nothing the mesh holds reads it\n", repository, m.Base, m.Commit)
return nil
}
// **Its delivery group's word, before the plans are held** (ADR 0306 decision 2): a slow answer holds up no walk.
own := ownPath(touches)
waits, err := groupWait(ctx, inv, m, deliverySeatHeld(entries), own)
if err != nil {
return notNow(err)
}
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return notNow(err)
@@ -198,7 +209,6 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
// **A merge on the controller's own path never shares a batch with one that waits for the delivery's
// word** (decided during the build, 2026-10-10): batched together, the catalogue's deliveries would start
// with the controller's and skip their turn. Each kind has a batch of its own.
own := ownPath(touches)
if p, ok, err := answeredByALaterMerge(ctx, inv, heard, touches, own, deliverySeatHeld(entries)); err != nil {
return notNow(err)
} else if ok {
@@ -218,7 +228,7 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
return cutBatchesHeld(ctx, f.open, now)
}
batch, err := theOpenBatch(ctx, inv, heard, now, own)
batch, why, err := batchToJoin(ctx, inv, heard, entries, read, now, own, waits)
if err != nil {
return notNow(err)
}
@@ -229,7 +239,7 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
if err := keepBatch(ctx, inv, &batch, now, ""); err != nil {
return notNow(err)
}
fmt.Printf("%s merged into %s (%.8s); it joins %s — %s\n", repository, m.Base, m.Commit, batch.ID,
fmt.Printf("%s merged into %s (%.8s); %s, it joins %s — %s\n", repository, m.Base, m.Commit, why, batch.ID,
batchWords(batch, now))
return cutBatchesHeld(ctx, f.open, now)
}
@@ -320,22 +330,211 @@ func ownPath(touches []inventory.Entry) bool {
return false
}
// theOpenBatch is the batch a merge heard now joins: the one of its kind not yet cut, or a new one.
func theOpenBatch(ctx context.Context, inv *inventory.Inventory, first inventory.BatchedMerge, now time.Time, own bool) (inventory.Plan, error) {
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, as ADR 0306
// decisions 2 and 3 read it). batchToJoin is the batch a merge heard now joins, and why in words:
//
// - an assembling or queued batch of its kind it is related to (related.go): it joins it and waits for the
// window, which its coming restarts. Related to two or more, the newer are folded into the oldest, which
// answers all their merges: a merge never waits in two windows, and what it shares with each walks in one.
// - none, and its delivery group has a member still to merge, or mesh-delivery could not say (groupWait, waits
// names why): a new batch, which waits for the window, beside the ones it shares nothing with; its words say
// why.
// - none, and no member of its group is still to merge: a new batch whose window is closed at once. Cut at
// once means only that: walks stay one at a time (ADR 0276 decision 4), so with a walk open the batch is
// queued behind it as any closed batch is.
func batchToJoin(ctx context.Context, inv *inventory.Inventory, heard inventory.BatchedMerge, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, now time.Time, own bool, waits string) (inventory.Plan, string, error) {
edges, err := inv.Dependencies(ctx)
if err != nil {
return inventory.Plan{}, "", err
}
joins, whys, _, err := relatedBatches(ctx, inv, []inventory.BatchedMerge{heard}, own, "", entries, read, edges)
if err != nil {
return inventory.Plan{}, "", err
}
if len(joins) > 0 {
into := joins[0]
why := "related to " + into.ID + " (" + whys[0] + ")"
if len(joins) > 1 {
plans, moves, err := foldInto(ctx, inv, &into, joins[1:])
if err != nil {
return inventory.Plan{}, "", err
}
if err := inv.SavePlansMoving(ctx, plans, moves); err != nil {
return inventory.Plan{}, "", err
}
var folded []string
for i, b := range joins[1:] {
folded = append(folded, b.ID+" ("+whys[i+1]+")")
}
why += ", and to " + strings.Join(folded, ", ") + ", folded into it"
}
into.Delivery.Batch.CutAtOnce = false
return into, why, nil
}
if waits != "" {
// Its own window, beside the batches it shares nothing with: waiting is for its group's members, which
// are related to it by their head and join it (ADR 0306 decision 2).
b, err := newBatch(ctx, inv, heard, now, own, false, waits)
return b, waits + ", it waits for the window in a batch of its own", err
}
b, err := newBatch(ctx, inv, heard, now, own, true, "")
return b, "it shares nothing with a batch assembling and no member of a group is still to merge: it is cut at once", err
}
// relatedBatches are the batches not yet cut of a kind, oldest first, parted into those related to merges (with
// the first reason of each) and the others. Each batch is judged by what its cut would move (ADR 0306 decision
// 3): its merges' moves as they were kept and as they read now — the catalogue may have moved since a merge was
// heard — widened to their dependents, and, for a catalogue batch, every walk waiting for its word, which the
// cut folds in (cutBatch's carry). A merge kept with no moves leaves its batch unknown, related to everything.
// except is a walk not to count as folded: the one being stopped.
func relatedBatches(ctx context.Context, inv *inventory.Inventory, merges []inventory.BatchedMerge, own bool,
except string, entries []inventory.Entry, read map[string][]inventory.ReadRepository,
edges []inventory.Edge) (joins []inventory.Plan, whys []string, others []inventory.Plan, err error) {
batches, err := inv.Batches(ctx)
if err != nil {
return inventory.Plan{}, err
return nil, nil, nil, err
}
var kind []inventory.Plan
for _, b := range batches {
if b.OwnPath() == own {
return b, nil
kind = append(kind, b)
}
}
if len(kind) == 0 {
return nil, nil, nil, nil
}
carry := newFootprint()
if !own {
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch may fold every one.
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, nil, nil, err
}
for _, p := range plans {
if p.Release != nil || !p.Waiting() || p.ID == except {
continue
}
kept, err := inv.MergesOf(ctx, p.ID)
if err != nil {
return nil, nil, nil, err
}
carry = unionFootprint(carry, footprintOfWalk(p, kept))
}
}
mine := footprintAsCut(merges, entries, read, edges)
for _, b := range kind {
held, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return nil, nil, nil, err
}
if b.Delivery == nil {
b.Delivery = &inventory.PlanDelivery{}
}
if b.Delivery.Batch == nil {
b.Delivery.Batch = &inventory.PlanBatch{Own: own}
}
if ok, why := related(mine, unionFootprint(footprintAsCut(held, entries, read, edges), carry), edges); ok {
joins, whys = append(joins, b), append(whys, why)
} else {
others = append(others, b)
}
}
return joins, whys, others, nil
}
// footprintAsCut is a batch's footprint as a cut now would read it: footprintOfBatch, widened by the modules each
// merge moves as the catalogue reads now (planBatch judges each merge again at the cut, so what it moves may
// have changed since it was heard). Both are kept: when in doubt, related.
func footprintAsCut(merges []inventory.BatchedMerge, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, edges []inventory.Edge) footprint {
f := footprintOfBatch(merges, edges)
var moved []string
for _, m := range merges {
s := announcedOf(m).SourceMoved
if s.Owner == "" {
continue
}
s.MergedAt = "" // news, whatever a cut marked seen since (as planBatch reads it)
for _, e := range wouldMove(s, entries, read) {
moved = append(moved, e.Manifest.Module)
}
}
for _, name := range reachableFrom(moved, edges) {
f.modules[name] = true
}
return f
}
// unionFootprint is what two changes reach together.
func unionFootprint(a, b footprint) footprint {
f := newFootprint()
for _, from := range []footprint{a, b} {
for k := range from.repos {
f.repos[k] = true
}
for k := range from.modules {
f.modules[k] = true
}
for k := range from.heads {
f.heads[k] = true
}
}
f.unknown = a.unknown
if f.unknown == "" {
f.unknown = b.unknown
}
return f
}
// foldInto folds batches into the oldest one related to a merge: the plans to write (into first, so no merge
// names a plan the store does not hold) and the merges to move, for one SavePlansMoving. The folded batches name
// the batch that took them over; a stop's closed window and its withheld modules (ADR 0299) go with their merges.
func foldInto(ctx context.Context, inv *inventory.Inventory, into *inventory.Plan, from []inventory.Plan) ([]*inventory.Plan,
[]inventory.MergeMove, error) {
if into.Delivery == nil {
into.Delivery = &inventory.PlanDelivery{}
}
if into.Delivery.Batch == nil {
into.Delivery.Batch = &inventory.PlanBatch{Own: into.OwnPath()}
}
plans := []*inventory.Plan{into}
var moves []inventory.MergeMove
for i := range from {
f := from[i]
merges, err := inv.MergesOf(ctx, f.ID)
if err != nil {
return nil, nil, err
}
for _, m := range merges {
moves = append(moves, inventory.MergeMove{Repository: m.Repository, Commit: m.Commit, Plan: into.ID, Alone: m.Alone})
}
if f.Delivery == nil {
f.Delivery = &inventory.PlanDelivery{}
}
if w := f.Delivery.Batch; w != nil {
into.Delivery.Batch.Closed = into.Delivery.Batch.Closed || w.Closed
into.Delivery.Batch.Withheld = union(into.Delivery.Batch.Withheld, w.Withheld)
sort.Strings(into.Delivery.Batch.Withheld)
}
f.State = inventory.PlanSuperseded
f.Delivery.Batch = nil
f.Delivery.TakenOverBy = into.ID
f.Note = fmt.Sprintf("folded into %s before it was cut: a merge related to both joined it, and %s answers "+
"its merges", into.ID, into.ID)
plans = append(plans, &f)
}
return plans, moves, nil
}
// newBatch is a new batch of a kind, kept before its first merge names it: a merge never names a plan the store
// does not hold. cutAtOnce closes its window at once; waits says why it waits for its group, when it does.
func newBatch(ctx context.Context, inv *inventory.Inventory, first inventory.BatchedMerge, now time.Time, own,
cutAtOnce bool, waits string) (inventory.Plan, error) {
b := inventory.Plan{ID: fmt.Sprintf("plan-%d", now.UnixNano()), Repository: first.Repository, Branch: first.Branch,
Commit: first.Commit, Merged: first.Merged, Created: now, State: inventory.PlanAssembling,
Tiers: [][]string{}, Modules: map[string]*inventory.PlanModule{},
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own}}}
// Kept before its first merge names it: a merge never names a plan the store does not hold.
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own, CutAtOnce: cutAtOnce, Waits: waits}}}
if err := inv.SavePlan(ctx, &b); err != nil {
return inventory.Plan{}, err
}
@@ -367,7 +566,16 @@ func keepBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan,
}
closes, latest := windowOf(merges, b.Created, window, atMost)
b.Delivery.Merges = named
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, Own: b.OwnPath()}
var closed, cutAtOnce bool
var withheld []string
var waits string
if was := b.Delivery.Batch; was != nil {
closed, withheld = was.Closed, was.Withheld // a stop's, which no look takes back (novox/hq ADR 0299)
cutAtOnce = was.CutAtOnce // the merge's, until another joins it (ADR 0304 decision 1)
waits = was.Waits // its group's, as said when it was made (ADR 0306 decision 2)
}
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, Own: b.OwnPath(),
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits}
b.State = inventory.PlanAssembling
if behind != "" && windowClosed(*b, now) {
b.State = inventory.PlanQueued
@@ -396,13 +604,14 @@ func windowOf(merges []inventory.BatchedMerge, created time.Time, window, atMost
return last.Add(window), created.Add(atMost)
}
// windowClosed says a batch's window is closed: no merge for the window's length, or its maximum reached.
// windowClosed says a batch's window is closed: closed by a stop, closed at once for a merge that shares
// nothing (ADR 0304 decision 1), no merge for the window's length, or its maximum reached.
func windowClosed(b inventory.Plan, now time.Time) bool {
if b.Delivery == nil || b.Delivery.Batch == nil {
return true
}
w := b.Delivery.Batch
return !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
return w.Closed || w.CutAtOnce || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
}
// carriedOf is a set of merges as a walk carries them: the latest merge of each repository's branch, and
@@ -478,7 +687,8 @@ func spelledOf(m inventory.BatchedMerge) string {
// namedOf is one merge as a walk or batch names it: its commit, and its pull request and moves as announced.
func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge {
k := announcedOf(m)
return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves}
return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves,
Merged: m.Merged, Heard: m.Heard}
}
// laterMerge says a was merged after b: by the forge's merge time, then by when each was heard.
@@ -610,8 +820,9 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
return err
}
// The oldest batch whose window closed is cut, an own-path one before a catalogue one (it goes first, and
// the catalogue batch queues behind it rather than being cut and overtaken); one of each kind at most is
// open (theOpenBatch).
// the catalogue batch queues behind it rather than being cut and overtaken). Several of a kind may be open
// since a merge that shares nothing is cut at once (ADR 0304 decision 1): walks stay one at a time, so one
// is cut per look, and once its walk started the others are queued behind it.
sort.SliceStable(batches, func(i, j int) bool { return batches[i].OwnPath() && !batches[j].OwnPath() })
for i := range batches {
b := &batches[i]
@@ -728,11 +939,16 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
commits, named := carriedOf(merges)
var moved []string
var members []batchMember
unchanged := map[string]string{}
for _, m := range combinedMerges(merges) {
// **News, whatever was seen since** (review of this change): each merge was judged when it was heard, and
// a cut, a fold or a failed walk since marks its modules seen — so a merge walked alone after a failed
// walk, or folded, would read as history and move nothing. When it was made is not asked again here.
m.MergedAt = ""
// **Unchanged, said** (novox/hq issue 611): read before the merge is acted on, from the same records.
for n, why := range unchangedOfMerge(m, entries, read) {
unchanged[n] = why
}
names, err := movesOfMerge(ctx, inv, m, entries, read)
if err != nil {
return err
@@ -761,7 +977,18 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
}
sort.Strings(also)
ordered := groupOrderEdges(members, entries, read, edges)
var withheld []string
plan := planOfMoves(moved, append(append([]inventory.Edge{}, edges...), ordered...))
// **Withheld, a stopped merge's modules** (novox/hq ADR 0299): a module is built at its branch, which holds the
// merge a stop left out of the walk these merges were walked again from — built here only as a dependent, it
// would deliver that merge. One a merge of this batch moves is built: that merge carries the stopped change,
// and the stop named it.
if batch.Delivery != nil && batch.Delivery.Batch != nil {
if out := withhold(&plan, batch.Delivery.Batch.Withheld, moved); len(out) > 0 {
withheld = out
fmt.Printf(" withheld, a stopped merge's modules built only as dependents: %s\n", strings.Join(out, ", "))
}
}
plan.ID, plan.Revision, plan.Created, plan.Epoch = batch.ID, batch.Revision, now, batch.Epoch
plan.Commits = commits
newest := newestCommit(commits)
@@ -772,10 +999,17 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
}
plan.Delivery.Merges = named
plan.Delivery.Alone = batch.Delivery != nil && batch.Delivery.Alone
plan.Delivery.Withheld = withheld
plan.Delivery.Unchanged = unchangedLeft(unchanged, plan.Modules)
// **Its moments and its class** (novox/hq ADR 0282 decision 6): measured, never acted on.
plan.Times = walkTimesAtCut(*batch, plan, entries, now)
if len(moved) == 0 {
plan.State = inventory.PlanDone
plan.Tiers = [][]string{}
plan.Note = "moved nothing the mesh holds: nothing to walk"
if len(plan.Delivery.Unchanged) > 0 {
plan.Note = "moved nothing the mesh holds: nothing to walk; " + unchangedSaid(plan.Delivery.Unchanged)
}
if err := inv.SavePlan(ctx, &plan); err != nil {
return err
}
@@ -820,6 +1054,121 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil
}
// unchangedSaid names the modules a walk says matched what runs, for its note.
func unchangedSaid(unchanged map[string]string) string {
names := make([]string, 0, len(unchanged))
for n := range unchanged {
names = append(names, n)
}
sort.Strings(names)
return "unchanged, the build the mesh holds is theirs: " + strings.Join(names, ", ")
}
// unchangedLeft is what a walk says matched what runs: every module a merge left unchanged that the walk does not
// build anyway, as another merge's module or a dependent (novox/hq issue 611). Nil when none is.
func unchangedLeft(unchanged map[string]string, builds map[string]*inventory.PlanModule) map[string]string {
out := map[string]string{}
for n, why := range unchanged {
if _, built := builds[n]; !built {
out[n] = why
}
}
if len(out) == 0 {
return nil
}
return out
}
// withhold takes from a plan every module named that its merges do not move, and the tiers left empty: the
// modules taken.
func withhold(plan *inventory.Plan, names, moved []string) []string {
var out []string
for _, n := range names {
if _, in := plan.Modules[n]; in && !slices.Contains(moved, n) {
delete(plan.Modules, n)
out = append(out, n)
}
}
if len(out) == 0 {
return nil
}
var tiers [][]string
for _, t := range plan.Tiers {
var kept []string
for _, n := range t {
if !slices.Contains(out, n) {
kept = append(kept, n)
}
}
if len(kept) > 0 {
tiers = append(tiers, kept)
}
}
if tiers == nil {
tiers = [][]string{}
}
plan.Tiers = tiers
sort.Strings(out)
return out
}
// walkTimesAtCut is a walk's own moments as it is cut (novox/hq ADR 0282 decision 6): when its batch's window
// closed — no merge for the window's length, or its maximum, whichever came first — when it was cut, and its
// class. A merge walked alone had no window.
func walkTimesAtCut(batch, walk inventory.Plan, entries []inventory.Entry, now time.Time) *inventory.PlanTimes {
cut := now
t := &inventory.PlanTimes{Cut: &cut, Class: classOf(walk, entries)}
if batch.Delivery != nil && batch.Delivery.Batch != nil {
w := batch.Delivery.Batch
closed := w.ClosesAt
if !w.AtMost.IsZero() && w.AtMost.Before(closed) {
closed = w.AtMost
}
if w.CutAtOnce && !batch.Created.IsZero() {
closed = batch.Created // closed when it was made (ADR 0304 decision 1)
}
if !closed.IsZero() {
if closed.After(now) {
closed = now
}
t.WindowClosed = &closed
}
}
return t
}
// resolverSeat is the seat of the mesh's resolver: a module claiming it is a core module (ADR 0282 decision 1).
const resolverSeat = "mesh-dns-resolver"
// classOf is a walk's class (novox/hq ADR 0282 decision 1): core when it walks a module on the controller's own
// path or one holding the mesh's resolver, leaf otherwise.
func classOf(walk inventory.Plan, entries []inventory.Entry) string {
modules := make([]string, 0, len(walk.Modules))
for m := range walk.Modules {
modules = append(modules, m)
}
return classOfModules(modules, entries)
}
// classOfModules is the class of modules moved together: core when one is on the controller's own path or holds
// the mesh's resolver, leaf otherwise (ADR 0282 decision 1).
func classOfModules(modules []string, entries []inventory.Entry) string {
resolvers := map[string]bool{}
for _, e := range entries {
for _, c := range e.Manifest.Claims {
if c.Name == resolverSeat {
resolvers[e.Manifest.Module] = true
}
}
}
for _, m := range modules {
if _, own := onTheControllersPath[m]; own || resolvers[m] {
return inventory.ClassCore
}
}
return inventory.ClassLeaf
}
// combinedMerges is a batch's merges as one merge per repository's branch: the latest, with every file the
// merges of it changed and said to be a module's — on a linear trunk the latest contains the others. A file is
// removed when the last merge of the batch that changed it removed it. merges are in the order they were made.
@@ -1044,6 +1393,10 @@ func batchWords(b inventory.Plan, now time.Time) string {
if b.State == inventory.PlanQueued {
return fmt.Sprintf("queued behind %s; grouped: %s; plan not yet calculated", w.Behind, grouped)
}
if w.CutAtOnce {
return fmt.Sprintf("cut at once, sharing nothing with a batch assembling; grouped: %s; plan not yet calculated",
grouped)
}
left := w.ClosesAt.Sub(now)
if w.AtMost.Before(w.ClosesAt) {
left = w.AtMost.Sub(now)
@@ -1051,8 +1404,12 @@ func batchWords(b inventory.Plan, now time.Time) string {
if left < 0 {
left = 0
}
return fmt.Sprintf("assembling: %s left (at the latest %s); grouped: %s; plan not yet calculated",
secondsWords(left), w.AtMost.Local().Format("15:04:05"), grouped)
waits := ""
if w.Waits != "" {
waits = " — " + w.Waits
}
return fmt.Sprintf("assembling: %s left (at the latest %s)%s; grouped: %s; plan not yet calculated",
secondsWords(left), w.AtMost.Local().Format("15:04:05"), waits, grouped)
}
// groupedWords is a batch's commits, each with the earlier merges it answers.
+77 -13
View File
@@ -34,6 +34,7 @@ func windowed(t *testing.T) *stores {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
noGroups(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
@@ -58,19 +59,70 @@ func windowed(t *testing.T) *stores {
// t0 is the moment a test's clock starts: merges are dated by it, and heard after it.
var t0 = time.Now().UTC().Truncate(time.Second)
// catalogueMerge is a merge of the catalogue changing one module's directory, made at a moment.
// catalogueMerge is a merge of the catalogue changing one module's directory, made at a moment, from a pull
// request's head branch of its own, as the forge announces every merge. Whether its head names a delivery group
// with a member still to merge is mesh-delivery's word, which windowed stubs as no group at all (noGroups); a test
// whose merges wait for the window as a group's says so (groupsWaiting).
func catalogueMerge(commit, module string, made time.Time) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit,
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Head: "pr/" + commit, Commit: commit,
MergedAt: made.Format(time.RFC3339Nano), Paths: []string{"modules/" + module + "/module.json"},
ModuleDirs: []string{"modules/" + module}, ModuleDirsSaid: true}
}
// repoMerge is a merge of a repository of one module's own.
// repoMerge is a merge of a repository of one module's own, from a head branch of its own (catalogueMerge).
func repoMerge(repo, commit string, made time.Time) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Commit: commit,
return link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Head: "pr/" + commit, Commit: commit,
MergedAt: made.Format(time.RFC3339Nano), Paths: []string{"main.go"}}
}
// noGroups stubs mesh-delivery's word on groups for a test: no head names a group.
func noGroups(t *testing.T) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) { return nil, false, nil }
t.Cleanup(func() { deliveryGroupOf = was })
}
// groupsWaiting stubs mesh-delivery's word on groups for a test: each head named is a group with a member of another
// repository whose pull request is open (novox/elsewhere, ready), so a merge from it waits for the window (ADR 0306
// decision 2); every other head names no group.
func groupsWaiting(t *testing.T, heads ...string) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(_ context.Context, _ bool, head string) ([]groupMember, bool, error) {
if slices.Contains(heads, head) {
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
return nil, false, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
}
// everyHeadWaits stubs mesh-delivery's word on groups for a test whose merges each wait for the window as they did
// before novox/hq ADR 0304: every head is a group with a member of another repository still to merge.
func everyHeadWaits(t *testing.T) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
}
// fromHead is a merge from a head branch a test names: merges from one head are members of one delivery group,
// related by it (novox/hq ADR 0304 decision 1), so they share a batch whatever else they share.
func fromHead(m link.SourceMoved, head string) link.SourceMoved {
m.Head = head
return m
}
// lone is a merge that names no head: known to be no delivery group's, it waits for no window unless it is
// related to a batch assembling (ADR 0304 decision 1).
func lone(m link.SourceMoved) link.SourceMoved {
m.Head = ""
return m
}
// hear is a merge heard at a moment.
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
@@ -108,6 +160,7 @@ func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
// later commit, which names both merges — the earlier carried by the later.
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
@@ -152,9 +205,10 @@ func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
// Merges of three repositories in one window are one walk, one commit for each.
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/three")
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)),
hear(t, open, fromHead(repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)), "feat/three"),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
@@ -187,9 +241,10 @@ func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
// closes at most ten minutes after its first merge however busy.
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
hear(t, open, repoMerge("two", "c2", t0.Add(80*time.Second)), t0.Add(80*time.Second))
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(80*time.Second)), "feat/both"), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
@@ -200,12 +255,13 @@ func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
}
busy := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, repoMerge(repo, "c"+string(rune('a'+i)), last), last)
hear(t, busy, fromHead(repoMerge(repo, "c"+string(rune('a'+i)), last), "feat/busy"), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
@@ -223,10 +279,11 @@ func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
// what is grouped, and that the plan is not yet calculated. It says so on the bus as plan-moved, too.
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/553b7191claude")
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
inventory.PlanSaved = func(p inventory.Plan) { said = append(said, p) }
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { said = append(said, p) }
t.Cleanup(func() { inventory.PlanSaved = was })
now := time.Now().UTC()
claude := catalogueMerge("553b7191claude", "app", now.Add(-20*time.Second))
@@ -235,7 +292,7 @@ func TestTheAssemblingBatchIsShown(t *testing.T) {
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, repoMerge("one", "a6bc0931one", now), now)
hear(t, open, fromHead(repoMerge("one", "a6bc0931one", now), dunst.Head), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
@@ -360,6 +417,7 @@ func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
// moves — named at once on a walk done — and joins the next batch when it does not.
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
@@ -391,6 +449,7 @@ func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
// first; the first delivered answers the older ones, and the search stops.
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
@@ -459,6 +518,7 @@ func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
@@ -496,9 +556,10 @@ func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
// store, a merge handed over again is not doubled, and the batch is cut when its window closes.
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both")
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
hear(t, open, repoMerge("two", "c2", t0.Add(30*time.Second)), t0.Add(30*time.Second))
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(30*time.Second)), "feat/both"), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
@@ -506,7 +567,7 @@ func TestABatchSurvivesARestart(t *testing.T) {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, repoMerge("one", "c1", t0), t0.Add(60*time.Second)) // the bus hands it over again
hear(t, again, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
@@ -563,6 +624,7 @@ func TestAWaitingWalkNamesTheMergesItAnswers(t *testing.T) {
// nothing, and the merge joins the next batch.
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
@@ -716,6 +778,7 @@ func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
// still waits for the word, so no catalogue delivery skips its turn.
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
@@ -860,6 +923,7 @@ func sameMerge(a, b inventory.PlanMerge) bool {
// its delivery would skip its turn. It joins the next catalogue batch.
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
+4
View File
@@ -48,6 +48,10 @@ func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time)
if f.atMost.Before(f.closed) {
f.closed = f.atMost
}
if w.Batch.CutAtOnce && !b.Created.IsZero() {
// Its window closed when it was made, which is its latest too (ADR 0304 decision 1).
f.atMost, f.closed = b.Created, b.Created
}
}
if f.behind != "" {
if walk, err := inv.PlanByID(ctx, f.behind); err == nil {
+207
View File
@@ -0,0 +1,207 @@
package main
import (
"context"
"errors"
"fmt"
"sort"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/conditions"
)
// **A filling bucket or log is said before it is full** (novox/hq ADR 0297 §6, issue 501).
//
// A module's bucket and its log each have a cap on the bus, and when either is full the bus refuses
// the next write: the module stops doing what it writes for, and nothing said so before. On
// 2026-10-11 the issue tracker's bucket held a sixth of its cap after two days, growing toward a stop
// nobody would have heard coming. So the self-check reads every module's bucket and log, and one at
// fillRaiseAt of its cap or more raises a condition naming the module, the bucket or log, and how full
// it is.
//
// **Cleared by observation below fillClearBelow, not below fillRaiseAt**: a bucket or log hovering at
// its threshold would otherwise be raised and cleared on every run. Between the two, an open condition
// is kept and none is raised.
//
// **One that cannot be read is said, and the rest are still judged.** An unanswered question about one
// stream is no reason to know nothing of the others: it is a finding of its own, naming the bucket or
// log and why, and a fill condition already open for it is kept, because not knowing is not a pass.
// The fill thresholds, in percent of a bucket's or log's cap.
const (
fillRaiseAt = 75
fillClearBelow = 70
)
// The conditions the fill probe raises: one filling, and one that could not be read.
const (
kindBucketOrLogFilling = "bucket-or-log-filling"
kindBucketOrLogUnread = "bucket-or-log-unread"
)
// probeFillID is the probe's id in the registry.
const probeFillID = "D-fill"
// bucketOrLog is one module's bucket or log as the fill probe reads it.
type bucketOrLog struct {
Module string
// Kind is `bucket` or `log`; Name its local name; Stream the stream it is on the bus.
Kind, Name, Stream string
}
// filled is how full one bucket or log is, read from the bus.
type filled struct {
Bytes, Max uint64
}
// percent is how full, in whole percent, rounded down.
func (f filled) percent() uint64 {
if f.Max == 0 {
return 0
}
return f.Bytes * 100 / f.Max
}
// fillKey is where a bucket's or log's fill condition is kept.
func fillKey(s bucketOrLog) string { return conditions.Key(conditions.ScopeBus, s.Stream, "filling") }
// fillObservation is what one bucket's or log's fill says: a finding at fillRaiseAt or above, and,
// while its condition is open, at fillClearBelow or above too; nothing otherwise, which is what clears
// it. The operator's words are the kind's (plain_words.go); the summary and the evidence name the
// module, the bucket or log, and the fill.
func fillObservation(s bucketOrLog, f filled, open bool) (conditions.Observation, bool) {
if f.Max == 0 {
return conditions.Observation{}, false // one with no cap cannot fill
}
// Compared in bytes, not rounded percent: 74.9% is not 75%.
atRaise := f.Bytes*100 >= f.Max*fillRaiseAt
aboveClear := f.Bytes*100 >= f.Max*fillClearBelow
if !atRaise && !(open && aboveClear) {
return conditions.Observation{}, false
}
pct := f.percent()
return conditions.Observation{
Scope: conditions.ScopeBus, ID: s.Stream, Token: "filling", Kind: kindBucketOrLogFilling,
Severity: conditions.Warning,
Summary: fmt.Sprintf("%s's %s %s holds %s of %s, %d%%: at its cap the bus refuses the module's next write",
s.Module, s.Kind, s.Name, mibWords(f.Bytes), mibWords(f.Max), pct),
Said: fmt.Sprintf("module %s, %s %s (stream %s): %d of %d bytes, %d%%", s.Module, s.Kind, s.Name,
s.Stream, f.Bytes, f.Max, pct),
}, true
}
// unreadObservation says one bucket or log could not be read, and why. Asked over the network, so one
// look can be wrong: raised on the second look in a row (confirm.go).
func unreadObservation(s bucketOrLog, why error) conditions.Observation {
return conditions.Observation{
Scope: conditions.ScopeBus, ID: s.Stream, Token: "unread", Kind: kindBucketOrLogUnread,
Severity: conditions.Warning, Confirm: true,
Summary: fmt.Sprintf("%s's %s %s could not be read, so how full it is is not known", s.Module, s.Kind, s.Name),
Said: fmt.Sprintf("module %s, %s %s (stream %s): %v", s.Module, s.Kind, s.Name, s.Stream, why),
}
}
// keptFilling is an open fill condition said again for a bucket or log that could not be read this
// time: not knowing how full it is now is no reason to say it has room. Only ever made from a condition
// read back as open — its own summary and severity, never words made up for it.
func keptFilling(s bucketOrLog, open conditions.Condition, why error) conditions.Observation {
return conditions.Observation{
Scope: conditions.ScopeBus, ID: s.Stream, Token: "filling", Kind: kindBucketOrLogFilling,
Severity: open.Severity, Summary: open.Summary,
Said: fmt.Sprintf("module %s, %s %s (stream %s): not read this time (%v); kept open as it was",
s.Module, s.Kind, s.Name, s.Stream, why),
}
}
// mibWords is a size as a person reads it, in MiB with one decimal.
func mibWords(b uint64) string {
return fmt.Sprintf("%.1f MiB", float64(b)/(1024*1024))
}
// readFill is how full one bucket or log is on the bus; found false when it is not on the bus.
type readFill func(ctx context.Context, s bucketOrLog) (f filled, found bool, err error)
// openFill is the fill condition open under a key, if one is, or why it could not be read.
type openFill func(ctx context.Context, key string) (conditions.Condition, bool, error)
// judgeFills judges every bucket and log on its own: one that cannot be read is said as unread, with
// its fill condition kept only when that condition was read back and is open — when it cannot be read
// either, nothing is said of its fill, which the unread finding already covers. Every other is judged
// by its fill; for one of those, a condition that cannot be read is taken as open, so an unknown never
// clears a fill measured between fillClearBelow and fillRaiseAt.
func judgeFills(ctx context.Context, all []bucketOrLog, read readFill, open openFill) []conditions.Observation {
var out []conditions.Observation
for _, s := range all {
f, found, err := read(ctx, s)
if err != nil {
out = append(out, unreadObservation(s, err))
if c, isOpen, cerr := open(ctx, fillKey(s)); cerr == nil && isOpen {
out = append(out, keptFilling(s, c, err))
}
continue
}
if !found {
continue // not on the bus: created on the controller's next raise, and nothing there can fill
}
_, isOpen, cerr := open(ctx, fillKey(s))
isOpen = isOpen || cerr != nil
if o, said := fillObservation(s, f, isOpen); said {
out = append(out, o)
}
}
sort.SliceStable(out, func(i, j int) bool { return out[i].Key() < out[j].Key() })
return out
}
// declaredBucketsAndLogs is every module's bucket and log the catalogue declares, by its stream.
func declaredBucketsAndLogs(ctx context.Context, d *doctor) ([]bucketOrLog, error) {
buckets, err := d.open.inventory.DeclaredBuckets(ctx)
if err != nil {
return nil, err
}
logs, err := d.open.inventory.DeclaredLogs(ctx)
if err != nil {
return nil, err
}
var out []bucketOrLog
for _, b := range buckets {
out = append(out, bucketOrLog{Module: b.Module, Kind: "bucket", Name: b.Name, Stream: "KV_" + b.Bucket()})
}
for _, l := range logs {
out = append(out, bucketOrLog{Module: l.Module, Kind: "log", Name: l.Name, Stream: l.Stream()})
}
return out, nil
}
// probeFill reads every module's bucket and log on the bus and says each one filling toward its cap,
// and each one that could not be read.
func probeFill(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
all, err := declaredBucketsAndLogs(ctx, d)
if err != nil {
return nil, err
}
js := d.js.Context()
read := func(ctx context.Context, s bucketOrLog) (filled, bool, error) {
if err := ctx.Err(); err != nil {
return filled{}, false, err
}
info, err := js.StreamInfo(s.Stream, nats.Context(ctx))
switch {
case errors.Is(err, nats.ErrStreamNotFound):
return filled{}, false, nil
case err != nil:
return filled{}, false, err
case info.Config.MaxBytes <= 0:
return filled{}, true, nil
}
return filled{Bytes: info.State.Bytes, Max: uint64(info.Config.MaxBytes)}, true, nil
}
open := func(ctx context.Context, key string) (conditions.Condition, bool, error) {
if d.keeper == nil {
return conditions.Condition{}, false, nil
}
return d.keeper.Get(ctx, key)
}
return judgeFills(ctx, all, read, open), nil
}
+165
View File
@@ -0,0 +1,165 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/conditions"
)
// **A filling log or bucket is said at three quarters of its cap and cleared below seven tenths**
// (novox/hq ADR 0297 §6): raised at 75% naming the module, the bucket or log and its fill; not at 74.9%;
// kept between 70% and 75% while it is open, and not raised there when it is not; and cleared — said no
// more — once it reads below 70%, open or not.
func TestAFillingBucketOrLogIsRaisedAtThreeQuartersAndClearedBelowSevenTenths(t *testing.T) {
const mib = 1024 * 1024
log := bucketOrLog{Module: "mesh-issues", Kind: "log", Name: "changes", Stream: "LOG_mesh-issues_changes"}
bucket := bucketOrLog{Module: "mesh-issues", Kind: "bucket", Name: "issues", Stream: "KV_mesh-issues_issues"}
for _, c := range []struct {
name string
of bucketOrLog
bytes uint64
max uint64
open bool
said bool
}{
{"a log at exactly 75%", log, 768 * mib, 1024 * mib, false, true},
{"a bucket at exactly 75%", bucket, 48 * mib, 64 * mib, false, true},
{"a bucket full", bucket, 64 * mib, 64 * mib, false, true},
{"a log just under 75%", log, 768*mib - 1, 1024 * mib, false, false},
{"a bucket at 72%, not open", bucket, 64 * mib * 72 / 100, 64 * mib, false, false},
{"a bucket at 72%, open: kept", bucket, 64 * mib * 72 / 100, 64 * mib, true, true},
{"a log at exactly 70%, open: kept", log, 700 * mib, 1000 * mib, true, true},
{"a log just under 70%, open: cleared", log, 700*mib - 1, 1000 * mib, true, false},
{"a bucket at 10%, open: cleared", bucket, 64 * mib / 10, 64 * mib, true, false},
{"a bucket with no cap", bucket, 100, 0, true, false},
} {
o, said := fillObservation(c.of, filled{Bytes: c.bytes, Max: c.max}, c.open)
if said != c.said {
t.Errorf("%s: said %v, want %v", c.name, said, c.said)
continue
}
if !said {
continue
}
if o.Key() != fillKey(c.of) || o.Kind != kindBucketOrLogFilling || o.Severity != conditions.Warning {
t.Errorf("%s: raised as %s (%s, %s)", c.name, o.Key(), o.Kind, o.Severity)
}
for _, part := range []string{c.of.Module, c.of.Kind + " " + c.of.Name, "%", " of "} {
if !strings.Contains(o.Summary, part) {
t.Errorf("%s: does not name %q: %q", c.name, part, o.Summary)
}
}
if !strings.Contains(o.Said, "bytes") {
t.Errorf("%s: the evidence does not say the fill in bytes: %q", c.name, o.Said)
}
}
o, _ := fillObservation(log, filled{Bytes: 800 * mib, Max: 1024 * mib}, false)
if !strings.Contains(o.Summary, "800.0 MiB of 1024.0 MiB, 78%") {
t.Errorf("the fill is said as %q", o.Summary)
}
}
// **One bucket or log that cannot be read does not stop the others being judged** (review of #231): it is
// said as unread with its reason, a fill condition open for it is kept, and every other bucket and log
// is judged by its own fill.
func TestAnUnreadableBucketOrLogIsSaidAndTheRestAreStillJudged(t *testing.T) {
const mib = 1024 * 1024
broken := bucketOrLog{Module: "a", Kind: "bucket", Name: "broken", Stream: "KV_a_broken"}
brokenOpen := bucketOrLog{Module: "a", Kind: "log", Name: "was-filling", Stream: "LOG_a_was-filling"}
full := bucketOrLog{Module: "b", Kind: "log", Name: "changes", Stream: "LOG_b_changes"}
empty := bucketOrLog{Module: "c", Kind: "bucket", Name: "quiet", Stream: "KV_c_quiet"}
absent := bucketOrLog{Module: "d", Kind: "bucket", Name: "not-yet", Stream: "KV_d_not-yet"}
refusal := errors.New("the bus did not answer")
read := func(_ context.Context, s bucketOrLog) (filled, bool, error) {
switch s {
case broken, brokenOpen:
return filled{}, false, refusal
case full:
return filled{Bytes: 900 * mib, Max: 1000 * mib}, true, nil
case empty:
return filled{Bytes: 1, Max: 64 * mib}, true, nil
}
return filled{}, false, nil
}
open := func(_ context.Context, key string) (conditions.Condition, bool, error) {
if key == fillKey(brokenOpen) {
return conditions.Condition{Key: key, Severity: conditions.Warning,
Summary: "a's log was-filling holds 800.0 MiB of 1024.0 MiB, 78%"}, true, nil
}
return conditions.Condition{}, false, nil
}
got := map[string]conditions.Observation{}
for _, o := range judgeFills(context.Background(), []bucketOrLog{broken, brokenOpen, full, empty, absent}, read, open) {
got[o.Key()] = o
}
for _, s := range []bucketOrLog{broken, brokenOpen} {
o, said := got[conditions.Key(conditions.ScopeBus, s.Stream, "unread")]
if !said || o.Kind != kindBucketOrLogUnread || !o.Confirm || !strings.Contains(o.Said, refusal.Error()) {
t.Errorf("%s could not be read and was said as %+v", s.Stream, o)
}
}
if o, kept := got[fillKey(brokenOpen)]; !kept || !strings.Contains(o.Said, "kept open") {
t.Errorf("an open fill condition of a log not read this time was not kept: %+v", o)
}
if _, said := got[fillKey(broken)]; said {
t.Error("a bucket not read and not filling was said filling")
}
if o, said := got[fillKey(full)]; !said || o.Kind != kindBucketOrLogFilling {
t.Errorf("a log at 90%% beside an unreadable one was not judged: %+v", got)
}
if len(got) != 4 {
t.Errorf("said %d findings, want 4 (two unread, one kept, one filling): %v", len(got), got)
}
}
// Both kinds have plain words of their own, which hold to the plain rule (novox/hq ADR 0253).
func TestAFillingOrUnreadBucketOrLogIsSaidInPlainWords(t *testing.T) {
for _, kind := range []string{kindBucketOrLogFilling, kindBucketOrLogUnread} {
wording, has := plainWordings[kind]
if !has {
t.Errorf("%s has no plain words", kind)
continue
}
if why, ok := conditions.PlainWords(wording(conditions.Observation{Kind: kind})); !ok {
t.Errorf("%s: %s", kind, why)
}
}
}
// **When neither the bucket or log nor its condition can be read, nothing is said of its fill** (second
// review of #231): the unread finding covers it, and no fill is invented for it. For one that is read and
// measured between seven tenths and three quarters, a condition that cannot be read is taken as open, so
// an unknown never clears it.
func TestNothingIsSaidOfAFillWhenNeitherItNorItsConditionCanBeRead(t *testing.T) {
const mib = 1024 * 1024
broken := bucketOrLog{Module: "a", Kind: "log", Name: "changes", Stream: "LOG_a_changes"}
between := bucketOrLog{Module: "b", Kind: "bucket", Name: "issues", Stream: "KV_b_issues"}
read := func(_ context.Context, s bucketOrLog) (filled, bool, error) {
if s == broken {
return filled{}, false, errors.New("the bus did not answer")
}
return filled{Bytes: 72 * mib, Max: 100 * mib}, true, nil
}
open := func(context.Context, string) (conditions.Condition, bool, error) {
return conditions.Condition{}, false, errors.New("the conditions bucket did not answer")
}
got := map[string]conditions.Observation{}
for _, o := range judgeFills(context.Background(), []bucketOrLog{broken, between}, read, open) {
got[o.Key()] = o
}
if o, said := got[fillKey(broken)]; said {
t.Fatalf("a fill was said for a log whose fill and condition could not be read: %+v", o)
}
if _, said := got[conditions.Key(conditions.ScopeBus, broken.Stream, "unread")]; !said {
t.Error("the log that could not be read was not said as unread")
}
if _, kept := got[fillKey(between)]; !kept {
t.Error("a bucket at 72% whose condition could not be read was cleared")
}
if len(got) != 2 {
t.Errorf("said %d findings, want 2: %v", len(got), got)
}
}
+34
View File
@@ -696,6 +696,20 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
}
}
}
// **The list of predefined keys arrives with a build of the catalogue** (novox/hq issue 630): the
// builder reads predefined.json at the catalogue repository's root at the commit built, and it is kept
// before the module is registered, so a preference falling back to a key added in the same commit is
// judged against the list that adds it. Only from the catalogue: another repository names no list.
if len(result.Predefined) > 0 && isCatalogueRepository(result.Repository) {
list, err := catalogue.ParsePredefined(result.Predefined)
if err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
manifest.Module, short(result.Commit), err)
}
if err := inv.SetPredefined(ctx, list, result.Repository, result.Commit); err != nil {
return manifest, kept, err
}
}
if err := inv.RegisterModule(ctx, manifest, recorded); err != nil {
if errors.Is(err, inventory.ErrSuperseded) {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w",
@@ -835,6 +849,9 @@ type answers struct {
// public name on the machine went dark. The holds were correct; they were recorded only in the
// machine's own state file, and the one visible symptom was a count that did not add up.
untaken map[string]map[string]int
// leftOut is, per machine, every module of its set its composition leaves out, and why (novox/hq issue
// 380): assigned and not applied, which every push said only in passing. Not well while there is any.
leftOut map[string][]leftOutModule
// unheld is every module on a machine whose resources are applied through a seat nothing on
// that machine holds (novox/hq ADR 0207), with the modules that could hold it. Reported, not
// refused, until the switch — and while there is any, the mesh is not all well: the order the
@@ -863,6 +880,10 @@ type answers struct {
// applied, is not well: an assignment somebody made is not made yet, or will not be.
pending []inventory.PendingAssignment
pendingUnread string
// lastSent is every machine's last recorded send, by name, with what the machine answered of it (novox/hq
// issue 485); lastSentUnread why it could not be read.
lastSent map[string]inventory.Send
lastSentUnread string
}
// heldBy is every artifact this mesh has built, for a build that may need one as its base.
@@ -1018,3 +1039,16 @@ func buildLog(ctx context.Context, id string) error {
}
return nil
}
// catalogueRepository is the repository the catalogue is: the one whose predefined.json is the list of
// predefined keys (novox/hq issue 630).
const catalogueRepository = "mesh-catalog"
// isCatalogueRepository is whether a repository, as a path on the forge or a URL, is the catalogue.
func isCatalogueRepository(repository string) bool {
trimmed := strings.TrimSuffix(strings.TrimSuffix(strings.TrimSpace(repository), "/"), ".git")
if i := strings.LastIndex(trimmed, "/"); i >= 0 {
trimmed = trimmed[i+1:]
}
return trimmed == catalogueRepository
}
+90 -1
View File
@@ -153,8 +153,97 @@ type forgeFacts struct {
}
// askTheForge asks the forge, through its module's tools on the bus, for a repository's id and its branch's
// protection. A variable so a test needs no forge.
// protection, waiting out a forge not served for a moment (novox/hq issue 486). A variable so a test needs no
// forge.
var askTheForge = func(ctx context.Context, owner, repo, branch string) (forgeFacts, error) {
return forgeWaitedOut(ctx, forgeWait, owner+"/"+repo, func(ctx context.Context) (forgeFacts, error) {
return askTheForgeOnce(ctx, owner, repo, branch)
})
}
// waitOut bounds how long a question is asked again while its answer is "not now": a pause of first, doubling
// to most, for at most bound in all. now and sleep are the clock, so a test waits in no real time.
type waitOut struct {
bound, first, most time.Duration
now func() time.Time
sleep func(context.Context, time.Duration) error
}
// forgeWait is the registration's wait for the forge (novox/hq issue 486). On 2026-10-11 a build's registration
// asked the forge while node-tools on the control-node re-served after a rollout of the tool runner, heard
// "nothing serves gitea.gitea_api", and failed its plan; the forge answered again seconds later. A re-serve takes
// about 10 s, so 60 s waits one out with room to spare. It is a cap on the whole wait, an ask in flight included:
// the registration holds the controller's one receive loop while it waits, and a loop held two minutes reads as
// deaf. The pauses are those the build machine waits the registry with (novox/hq issue 457).
var forgeWait = waitOut{bound: 60 * time.Second, first: time.Second, most: 10 * time.Second,
now: time.Now, sleep: sleepOrDone}
// sleepOrDone pauses for d, or less when ctx ends first.
func sleepOrDone(ctx context.Context, d time.Duration) error {
t := time.NewTimer(d)
defer t.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-t.C:
return nil
}
}
// forgeNotNow is an answer that says the forge's tool is not served now — the tool runner re-serving — not that
// the forge refused. Only that is asked again: a refusal, an unreadable answer or a repository with no id is the
// forge's word, and a tool that did not answer in time has already held the receive loop its whole wait.
func forgeNotNow(err error) bool { return errors.Is(err, link.ErrNothingServes) }
// forgeWaitedOut asks the forge about repository, asking again while it answers "not served now", all of it
// within w's bound, an ask in flight included (novox/hq issue 486). Past the bound it fails, saying it waited,
// for how long, and what the forge last said; when the caller's own context ends, it says that instead.
func forgeWaitedOut(ctx context.Context, w waitOut, repository string,
ask func(context.Context) (forgeFacts, error)) (forgeFacts, error) {
capped, cancel := context.WithTimeout(ctx, w.bound)
defer cancel()
began := w.now()
pause := w.first
for asked := 1; ; asked++ {
facts, err := ask(capped)
if err == nil {
if asked > 1 {
fmt.Printf("the forge answered about %s again, after %s (novox/hq issue 486)\n", repository,
w.now().Sub(began).Round(time.Second))
}
return facts, nil
}
if ctx.Err() != nil {
return forgeFacts{}, err // the caller's own context ended: no wait of ours to speak of
}
gaveUp := func() (forgeFacts, error) {
return forgeFacts{}, fmt.Errorf("waited %s for the forge to answer about %s, asking %d times, and it "+
"did not (novox/hq issue 486): %w", w.now().Sub(began).Round(time.Second), repository, asked, err)
}
if capped.Err() != nil {
return gaveUp() // the bound ended an ask in flight
}
if !forgeNotNow(err) {
return forgeFacts{}, err
}
if w.now().Sub(began)+pause > w.bound {
return gaveUp()
}
fmt.Printf("the forge cannot be asked about %s now (%v); asking again in %s, for at most %s in all "+
"(novox/hq issue 486)\n", repository, err, pause, w.bound)
if serr := w.sleep(capped, pause); serr != nil {
if ctx.Err() != nil {
return forgeFacts{}, fmt.Errorf("stopped waiting for the forge to answer about %s: %w", repository,
ctx.Err())
}
return gaveUp()
}
pause = min(2*pause, w.most)
}
}
// askTheForgeOnce asks the forge once, through its module's tools on the bus.
func askTheForgeOnce(ctx context.Context, owner, repo, branch string) (forgeFacts, error) {
js, err := aBus()
if err != nil {
return forgeFacts{}, err
+48
View File
@@ -2,6 +2,8 @@ package main
import (
"fmt"
"path"
"slices"
"sort"
"strings"
@@ -28,6 +30,14 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
for _, e := range entries {
byName[e.Manifest.Module] = e
}
// The machine holding the bus, whose declaration carries every module's grants (novox/hq issue 490).
holder := ""
for _, e := range entries {
if e.Manifest.BusUsers != "" && e.Manifest.ClaimsSeat(catalogue.BrokerSeat) && len(e.On) > 0 {
holder = e.On[0]
}
}
var granting []string
machines := map[string]*link.MachinePlan{}
machine := func(name string) *link.MachinePlan {
if machines[name] == nil {
@@ -50,6 +60,10 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
machine(on).Receives = append(machine(on).Receives, name)
}
}
if !waits && holder != "" && !(len(e.On) == 1 && e.On[0] == holder) && len(e.On) > 0 &&
!slices.Contains(granting, name) {
granting = append(granting, name)
}
switch {
case (name == "nats" || catalogue.ProvidesBus(e.Manifest)) && len(e.On) > 0:
// Said before the merge (novox/hq issue 336): a new bus build holds every send to the bus's machine
@@ -76,6 +90,13 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
}
}
}
if len(granting) > 0 {
// Said before the merge (novox/hq issue 490): what the walk does when the change alters the bus's
// user list. Whether it does is known only once the builds are registered, so it is said as a rule.
p.Steps = append(p.Steps, fmt.Sprintf("%s: if this changes what the bus's user list says (a new tool, "+
"subject or user), %s is sent that list alone, every build there kept, before %s is judged on its first "+
"machine", grantsStepWord, holder, strings.Join(granting, ", ")))
}
var names []string
for name := range machines {
names = append(names, name)
@@ -85,9 +106,36 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
p.Machines = append(p.Machines, *machines[name])
}
p.Summary = summaryOf(p)
p.Class = changeClassOf(p, entries)
return p
}
// classNone is the class of a change that moves no module: nothing for a walk to move (novox/hq ADR 0313 §2).
const classNone = "none"
// changeClassOf is a change plan's class (novox/hq ADR 0313 §2): that of the slowest-budgeted module it moves,
// itself or as a dependent — core when one is on the controller's own path or holds the mesh's resolver, leaf
// otherwise — and none when it moves nothing. A module it adds a directory for counts by its own class: the
// catalogue names a module's directory by the module, so the directory's last element is its name. The graph holds
// no manifest of it yet, so a new module is core by its name alone (on the controller's own path), never by a
// resolver claim its manifest makes.
func changeClassOf(p link.ChangePlan, entries []inventory.Entry) string {
if len(p.Moved) == 0 && len(p.Dependents) == 0 && len(p.New) == 0 {
return classNone
}
modules := append(append([]string{}, p.Moved...), p.Dependents...)
for _, dir := range p.New {
if name := path.Base(strings.Trim(dir, "/")); name != "." && name != "/" && name != "" {
modules = append(modules, name)
}
}
return classOfModules(modules, entries)
}
// grantsStepWord names the grants step (novox/hq issue 490): the bus's user list sent alone to the machine
// holding the bus ahead of a walk's gate. Not the bus step, which replaces the bus itself.
const grantsStepWord = "grants step"
// busUpgradeNeeded is how a change plan says that merging it holds the bus's machine for a person's step.
const busUpgradeNeeded = "merging this needs a person's bus upgrade"
+63
View File
@@ -47,6 +47,13 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
shelf := catalogue.Shelf{}
faulted := map[string]bool{}
failed := 0
// The list of predefined keys a preference may fall back to (novox/hq issue 630): the predefined.json
// at the root of the repository the manifests are in, found above the first of them.
list, listAt, listErr := predefinedAbove(paths[0])
if listErr != nil {
fmt.Fprintf(out, "%s: %v\n", listAt, listErr)
failed++
}
for _, path := range paths {
raw, err := os.ReadFile(path)
if err != nil {
@@ -75,6 +82,16 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed += len(wrong)
faulted[m.Module] = true
}
// A module that declares a setting names the keys of its mergeable files (novox/hq ADR 0326 §3,
// issue 574): judged where a definition is written, as at registration, never where a stored
// build is read again.
if wrong := catalogue.WrittenSettingsProblems(m); len(wrong) > 0 {
for _, p := range wrong {
fmt.Fprintf(out, "%s: %s\n", path, p)
}
failed += len(wrong)
faulted[m.Module] = true
}
if named := catalogue.InstallationProblems(m); len(named) > 0 {
for _, p := range named {
fmt.Fprintf(out, "%s: %s\n", path, p)
@@ -82,6 +99,20 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed += len(named)
faulted[m.Module] = true
}
// A preference falls back only to a key on the list, with its type (novox/hq issue 630).
if named := catalogue.PredefinedNamed(m); len(named) > 0 {
if listAt == "" {
fmt.Fprintf(out, "note: %s: its preferences fall back to %s, and no %s was found above it; "+
"registration judges them against the list the mesh holds\n", path, strings.Join(named, ", "),
catalogue.PredefinedFile)
} else if wrong := catalogue.PredefinedProblems(m, list); len(wrong) > 0 {
for _, p := range wrong {
fmt.Fprintf(out, "%s: %s\n", path, p)
}
failed += len(wrong)
faulted[m.Module] = true
}
}
shelf[m.Module] = m
}
@@ -179,6 +210,14 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
}
fmt.Fprintf(out, ", keeps state %s", strings.Join(kept, ", "))
}
// And the logs it keeps, with their caps (novox/hq ADR 0297).
if len(m.Logs) > 0 {
kept := make([]string, 0, len(m.Logs))
for _, l := range m.Logs {
kept = append(kept, fmt.Sprintf("%s (%d MiB)", l.Name, l.Cap()))
}
fmt.Fprintf(out, ", keeps log %s", strings.Join(kept, ", "))
}
if len(m.Reads) > 0 {
fmt.Fprintf(out, ", reads %s", strings.Join(m.Reads, ", "))
}
@@ -267,3 +306,27 @@ func manifestsUnder(dir string) ([]string, error) {
sort.Strings(found)
return found, err
}
// predefinedAbove reads the predefined.json found in the directory of a manifest or one above it, up to
// the repository's root (the directory holding .git). The path it was found at is "" when there is none.
func predefinedAbove(manifest string) (catalogue.Predefined, string, error) {
dir, err := filepath.Abs(filepath.Dir(manifest))
if err != nil {
return nil, "", nil
}
for {
at := filepath.Join(dir, catalogue.PredefinedFile)
if raw, err := os.ReadFile(at); err == nil {
list, err := catalogue.ParsePredefined(raw)
return list, at, err
}
if _, err := os.Stat(filepath.Join(dir, ".git")); err == nil {
return nil, "", nil
}
up := filepath.Dir(dir)
if up == dir {
return nil, "", nil
}
dir = up
}
}
+4 -5
View File
@@ -6,8 +6,10 @@ import (
"path/filepath"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"testing"
"github.com/novox/mesh-controller/internal/beside"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The check anybody can run is the check registration runs (novox/hq issue 148, ADR 0037): a manifest
@@ -56,10 +58,7 @@ func TestModuleCheckJudgesBetweenTheManifestsGiven(t *testing.T) {
// The real catalogue passes the command, the way it passes the test that used to be the only check.
func TestModuleCheckPassesTheCatalogue(t *testing.T) {
root := filepath.Join("..", "..", "..", "mesh-catalog", "modules")
if _, err := os.Stat(root); err != nil {
t.Skipf("catalogue sibling not present: %v", err)
}
root := beside.Catalogue(t)
paths, err := manifestsUnder(root)
if err != nil || len(paths) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
+110 -12
View File
@@ -5,6 +5,8 @@ import (
"encoding/json"
"fmt"
"path"
"regexp"
"runtime/debug"
"slices"
"sort"
"strings"
@@ -159,9 +161,7 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
return u, err == nil
})
fmt.Print(planText(plan))
direct := link.Checked{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Commit: p.Commit,
ID: link.NewBuildID(time.Now()), Verdict: "pass", Summary: noModuleTouched,
Gate: &link.CheckLayer{Verdict: "pass", Summary: noModuleTouched}, Plan: &plan}
direct := directVerdict(p, plan)
switch {
case !scope.gated() && !scope.Mesh:
fmt.Printf("%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
@@ -199,6 +199,24 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
return nil
}
// directVerdict is the verdict of a change no build judges — it touches no module, and either the repository is not
// the mesh's or the head holds no merge-check.sh. **It names the trunk commit it holds for** (novox/hq ADR 0323 §3):
// the base branch's head the forge announced with the head, so the merge tool can weigh it against the trunk's head
// and does not refuse it for ever (a verdict naming no trunk is never the trunk's head's). Nothing was built to
// judge: the trunk commit is the one the forge said the change was announced against.
func directVerdict(p link.PullUpdated, plan link.ChangePlan) link.Checked {
trunk := p.BaseSHA
if len(trunk) > 12 {
trunk = trunk[:12]
}
if trunk != "" {
plan.Trunk = trunk
}
return link.Checked{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Commit: p.Commit,
ID: link.NewBuildID(time.Now()), Verdict: "pass", Summary: noModuleTouched,
Gate: &link.CheckLayer{Verdict: "pass", Summary: noModuleTouched}, Plan: &plan, Trunk: trunk}
}
// checkRequestFor is the ask for one pull request's head: the repository as the mesh clones it, the head,
// and what is read beside it.
func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scope checkScope,
@@ -246,9 +264,11 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
if dir == "mesh-controller" {
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if e.Source.Seat != "" {
if lab, err := clone(inventory.Source{Seat: e.Source.Seat, Repository: siblingOf(e.Source.Repository,
"mesh-lab")}); err == nil {
beside["mesh-lab"] = link.CheckedOut{Repository: lab, Ref: refs["mesh-lab"]}
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
if url, err := clone(inventory.Source{Seat: e.Source.Seat, Repository: siblingOf(e.Source.Repository,
sibling)}); err == nil {
beside[sibling] = link.CheckedOut{Repository: url, Ref: refs[sibling]}
}
}
}
}
@@ -271,17 +291,54 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
// catalogue, whose checkout beside is what tests read its files from, at its main, what the next merge
// builds from; and beside the controller its main, for a judge the running controller predates, and the
// lab's main, whose replays every check runs. **One rule, read by the check the controller asks for and by
// the facts snapshot** (Facts.Beside), so a check run by hand clones what the build seat clones.
// the facts snapshot** (Facts.Beside), so a check run by hand clones what the build seat clones. Beside the
// controller also the SDK, checked out at the commit the running controller's go.mod pins (sdkPinned), not
// one a desktop holds (novox/hq issue 449). The checkout only places the clone: the conformance test reads the
// fixtures at the pin of the tree under check, from the clone's history, so a pull request moving the SDK is
// judged against the SDK it moves to (internal/link/conformance_test.go).
func besideRefs(dir, running string) map[string]string {
switch dir {
case "mesh-catalog":
return map[string]string{dir: "main"}
case "mesh-controller":
return map[string]string{dir: running, "mesh-controller-main": "main", "mesh-lab": "main"}
return map[string]string{dir: running, "mesh-controller-main": "main", "mesh-lab": "main",
"mesh-sdk": sdkPinned()}
}
return map[string]string{dir: running}
}
// sdkModule is the Go module of the SDK the controller is built against.
const sdkModule = "git.novox.be/novox/mesh-sdk/go"
// pseudoCommit is the commit a Go pseudo-version names: v0.1.11-0.20261009143344-f047d0a4a970 → f047d0a4a970.
var pseudoCommit = regexp.MustCompile(`-([0-9a-f]{12})$`)
// sdkPinned is the ref of the SDK repository this controller was built from, as its go.mod pins it and its
// build records it: a pseudo-version's commit, or a release's tag (the SDK tags its Go module under go/).
// "main" only for a binary that records no SDK version — a local replace — which a running controller is
// not (novox/hq issue 449).
func sdkPinned() string {
info, ok := debug.ReadBuildInfo()
if !ok {
return "main"
}
for _, dep := range info.Deps {
if dep.Path != sdkModule {
continue
}
if dep.Replace != nil {
dep = dep.Replace
}
if m := pseudoCommit.FindStringSubmatch(dep.Version); m != nil {
return m[1]
}
if strings.HasPrefix(dep.Version, "v") {
return "go/" + dep.Version
}
}
return "main"
}
// siblingOf is another repository of the same owner: novox/mesh-controller → novox/mesh-lab.
func siblingOf(repository, name string) string {
if cut := strings.LastIndex(repository, "/"); cut >= 0 {
@@ -322,6 +379,9 @@ func checkedOf(result link.BuildResult) link.Checked {
case result.Check != nil:
c.Verdict, c.Summary, c.Report = result.Check.Verdict, result.Check.Summary, result.Check.Report
c.Gate, c.RepoCheck = result.Check.Gate, result.Check.RepoCheck
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and the commits of the
// repositories read beside it.
c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took
if c.Gate == nil {
// A build seat from before the layers: its verdict is the gate's.
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
@@ -347,14 +407,20 @@ func checkedOf(result link.BuildResult) link.Checked {
}
if result.Checked != nil {
c.Plan = result.Checked.Plan
if c.Plan != nil && c.Trunk != "" {
// The kept delivery plan records the trunk commit with its other inputs (ADR 0323 §3).
plan := *c.Plan
plan.Trunk = c.Trunk
c.Plan = &plan
}
// A delivery group's composed check (novox/hq ADR 0239): every head it judged, this one first.
if g := result.Checked; g.Group != "" {
c.Group = g.Group
c.Members = append(c.Members, link.CheckedMember{Owner: g.Owner, Repo: g.Repo, Number: g.Number,
Commit: result.Ref})
Commit: result.Ref, Trunk: c.Trunk})
for _, m := range g.Members {
c.Members = append(c.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo, Number: m.Number,
Commit: m.Ref})
Commit: m.Ref, Trunk: memberTrunk(result.Check, m)})
}
}
}
@@ -363,12 +429,40 @@ func checkedOf(result link.BuildResult) link.Checked {
l.Verdict = "error"
}
}
// **A check run again says so** (novox/hq issue 594). The summaries are the lines the pull request
// shows, clipped by the forge to a few words, so they get only a short marker after the verdict's own
// words; the report gets the whole sentence. Layers copied, so the build's own outcome is left as it came.
if mark := link.RunMarker(result.Delivery); mark != "" {
c.Summary += " " + mark
for _, l := range []**link.CheckLayer{&c.Gate, &c.RepoCheck} {
if *l != nil {
layer := **l
layer.Summary += " " + mark
*l = &layer
}
}
c.Report = link.Restarted(result.Delivery) + "\n\n" + c.Report
}
if len(c.Report) > maxCheckReport {
c.Report = "…" + c.Report[len(c.Report)-maxCheckReport:]
}
return c
}
// memberTrunk is the trunk commit the build seat merged one of a group's heads into (novox/hq ADR 0323 §6), empty
// when it said none.
func memberTrunk(outcome *link.CheckOutcome, m link.GroupMember) string {
if outcome == nil {
return ""
}
for _, j := range outcome.Members {
if strings.EqualFold(j.Owner, m.Owner) && strings.EqualFold(j.Repo, m.Repo) && strings.HasPrefix(m.Ref, j.Commit) {
return j.Trunk
}
}
return ""
}
func prefixedAll(prefix string, items []string) []string {
out := make([]string, 0, len(items))
for _, i := range items {
@@ -383,8 +477,12 @@ func sayChecked(ctx context.Context, c link.Checked) {
if c.RepoCheck != nil {
repo = strings.ToUpper(c.RepoCheck.Verdict) + " — " + c.RepoCheck.Summary
}
fmt.Printf("%s: %s/%s#%d at %.8s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
c.Commit, orSomewhere(c.On), strings.ToUpper(c.Verdict), c.Summary, repo)
judged := ""
if c.Trunk != "" {
judged = " merged into trunk " + c.Trunk
}
fmt.Printf("%s: %s/%s#%d at %.8s%s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
c.Commit, judged, orSomewhere(c.On), strings.ToUpper(c.Verdict), c.Summary, repo)
if checkEvents == nil {
return
}
+78
View File
@@ -210,3 +210,81 @@ func TestABuildOffTheTrunkIsNeverPublished(t *testing.T) {
t.Errorf("a build seat that said nothing was refused: %v", err)
}
}
// **A verdict names the trunk commit it judged** (novox/hq ADR 0323 §3, issue 341): the controller's `checked`
// carries the head and the trunk commit the build seat merged it into, the commits of the repositories read beside
// it, how long it took, and the kept delivery plan records the trunk commit with its other inputs. A group's
// composed check names, per member, the trunk commit that head was merged into (§6).
func TestAVerdictNamesTheTrunkCommitItJudged(t *testing.T) {
plan := &link.ChangePlan{Repository: "novox/mesh-host", Base: "main", Head: "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
Summary: "builds mesh-host"}
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 66, Base: "main", Plan: plan}
c := checkedOf(link.BuildResult{ID: "b", Ref: "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", Checked: asked,
Check: &link.CheckOutcome{Verdict: "pass", Took: "4m12s", Head: "dfb1415aaaaa", Trunk: "d28ca95bbbbb",
Gate: &link.CheckLayer{Verdict: "pass"}, RepoCheck: &link.CheckLayer{Verdict: "pass"},
Beside: map[string]string{"mesh-catalog": "0123456789ab", "mesh-lab": "ba9876543210"}}})
if c.Trunk != "d28ca95bbbbb" || c.Commit != "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" {
t.Fatalf("the verdict names head %s and trunk %q", c.Commit, c.Trunk)
}
if c.Beside["mesh-catalog"] != "0123456789ab" || c.Beside["mesh-lab"] != "ba9876543210" {
t.Fatalf("the verdict names the repositories read beside it as %v", c.Beside)
}
if c.Took != "4m12s" {
t.Fatalf("the verdict says it took %q", c.Took)
}
if c.Plan == nil || c.Plan.Trunk != "d28ca95bbbbb" || c.Plan.Head != plan.Head {
t.Fatalf("the kept plan records %+v", c.Plan)
}
if plan.Trunk != "" {
t.Fatal("the plan the controller asked with was changed: the verdict's is a copy")
}
group := &link.CheckRequest{Owner: "novox", Repo: "mesh-controller", Number: 7, Base: "main", Group: "feat/x",
Members: []link.GroupMember{{Owner: "novox", Repo: "mesh-host", Number: 8, Ref: "cccccccccccc0000", Base: "main"}}}
c = checkedOf(link.BuildResult{ID: "g", Ref: "aaaaaaaaaaaa0000", Checked: group, Check: &link.CheckOutcome{
Verdict: "pass", Head: "aaaaaaaaaaaa", Trunk: "111111111111", Gate: &link.CheckLayer{Verdict: "pass"},
Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-host", Commit: "cccccccccccc", Trunk: "222222222222"}}}})
if len(c.Members) != 2 || c.Members[0].Trunk != "111111111111" || c.Members[1].Trunk != "222222222222" ||
c.Members[1].Commit != "cccccccccccc0000" {
t.Fatalf("the composed check names its members %+v", c.Members)
}
}
// **The merge made for a check is never registered** (novox/hq ADR 0323 §2): a build of it — a commit on no branch
// of the forge — is refused, naming the trunk. The rule of ADR 0238 §1 stands unchanged; this holds it for the
// merge commit a check makes.
func TestABuildOfACheckMergeIsRefusedNamingTheTrunk(t *testing.T) {
merge := link.BuildResult{ID: "b", Commit: "4b5c6d7e8f90aabbccddeeff0011223344556677", Trunk: "main"}
err := publishable(merge, "")
if !errors.Is(err, errOffTheTrunk) || !strings.Contains(err.Error(), "not on main") {
t.Fatalf("a build of a check's merge was answered %v", err)
}
}
// **A verdict no build judges names its trunk commit too** (novox/hq ADR 0323 §3): a change touching no module of a
// repository with no merge-check.sh (novox/mesh-tools-go) is answered by the controller itself. Naming no trunk, the
// merge tool refused it, asked again, got the same verdict, and refused for ever.
func TestADirectVerdictNamesTheTrunkTheForgeAnnounced(t *testing.T) {
p := link.PullUpdated{Owner: "novox", Repo: "mesh-tools-go", Number: 3, Base: "main", Commit: "1a2b3c4d5e6f7a8b9c0d",
BaseSHA: "7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b"}
c := directVerdict(p, link.ChangePlan{Repository: "novox/mesh-tools-go", Base: "main", Head: p.Commit})
if c.Trunk != "7a8b9c0d1e2f" || c.Plan == nil || c.Plan.Trunk != "7a8b9c0d1e2f" {
t.Fatalf("the direct verdict names trunk %q, its plan %+v", c.Trunk, c.Plan)
}
if c.Verdict != "pass" || c.Commit != p.Commit {
t.Fatalf("the direct verdict is %+v", c)
}
if c := directVerdict(link.PullUpdated{Owner: "novox", Repo: "x", Commit: "abc"}, link.ChangePlan{}); c.Trunk != "" {
t.Fatalf("a forge that said no base head gave the trunk %q", c.Trunk)
}
}
// A head checked again through mesh-delivery (`delivery check --members`, a recheck) reaches the same direct verdict:
// it names the base head the forge last announced, kept on the member, or the merge tool would refuse it for ever.
func TestARecheckedHeadsDirectVerdictNamesItsTrunk(t *testing.T) {
m := orderMember{ID: "novox/mesh-tools-go@1a2b3c4d5e6f", Repository: "novox/mesh-tools-go", Base: "main",
Head: "1a2b3c4d5e6f7a8b9c0d", Number: 3, BaseSHA: "7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b"}
if c := directVerdict(m.pull(), link.ChangePlan{}); c.Trunk != "7a8b9c0d1e2f" {
t.Fatalf("a rechecked head's direct verdict names trunk %q", c.Trunk)
}
}
+76
View File
@@ -0,0 +1,76 @@
package main
import (
"context"
"encoding/json"
"fmt"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// The looks of the open conditions — how often each was observed, when last, its newest evidence — are
// kept in the controller's store, not the conditions bucket (novox/hq issue 420, ADR 0289 §1): the bucket
// is written when what a condition says changes, never for a look that saw it as it was.
// storeLooks keeps the conditions' looks in the inventory.
type storeLooks struct{ inv *inventory.Inventory }
func (s storeLooks) KeepLooks(ctx context.Context, key string, l conditions.Looks) error {
evidence, err := json.Marshal(l.Evidence)
if err != nil {
return err
}
return s.inv.KeepConditionLooks(ctx, inventory.ConditionLooks{Key: key, Observations: l.Observations,
LastObserved: l.LastObserved, Evidence: evidence})
}
func (s storeLooks) Looked(ctx context.Context, key string) (conditions.Looks, bool, error) {
row, kept, err := s.inv.ConditionLooksOf(ctx, key)
if err != nil || !kept {
return conditions.Looks{}, kept, err
}
l, err := looksOf(row)
return l, err == nil, err
}
func (s storeLooks) AllLooks(ctx context.Context) (map[string]conditions.Looks, error) {
rows, err := s.inv.AllConditionLooks(ctx)
if err != nil {
return nil, err
}
out := make(map[string]conditions.Looks, len(rows))
for key, row := range rows {
l, err := looksOf(row)
if err != nil {
return nil, err
}
out[key] = l
}
return out, nil
}
func (s storeLooks) ForgetLooks(ctx context.Context, key string) error {
return s.inv.ForgetConditionLooks(ctx, key)
}
func looksOf(row inventory.ConditionLooks) (conditions.Looks, error) {
l := conditions.Looks{Observations: row.Observations, LastObserved: row.LastObserved.UTC()}
if len(row.Evidence) > 0 {
if err := json.Unmarshal(row.Evidence, &l.Evidence); err != nil {
return conditions.Looks{}, fmt.Errorf("the evidence kept for the condition %s cannot be read: %w", row.Key, err)
}
}
return l, nil
}
// looksHere is where a process that is not the serving controller reads the conditions' looks: the store,
// opened for it, closed by the function returned. A store that cannot be opened is said on every condition
// read through it, never read as none.
func looksHere(ctx context.Context) (conditions.LookStore, func()) {
inv, err := openInventory(ctx)
if err != nil {
return conditions.NoLooks{Err: err}, func() {}
}
return storeLooks{inv}, inv.Close
}
@@ -0,0 +1,87 @@
package main
import (
"bytes"
"context"
"encoding/json"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/testbus"
)
// **Looks at an unchanged condition write the bucket once, and `conditions` still says how often and when
// it was seen** (novox/hq issue 420): the looks are in the controller's store, and the verb, run as a
// process of its own, reads them from there. On 2026-10-10 one open condition had rewritten the bucket 7 003
// times in two days.
func TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds(t *testing.T) {
inv := inventory.ForTest(t)
bus := testbus.Start(t)
serving, err := broker.Dial(bus.ClientURL())
if err != nil {
t.Fatal(err)
}
defer serving.Close()
if err := serving.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
ctx := t.Context()
store, history, err := conditions.OnTheBus(ctx, serving.Conn())
if err != nil {
t.Fatal(err)
}
k := conditions.NewKeeper(ctx, conditions.Options{Store: store, History: history, Looks: storeLooks{inv}})
defer k.Close(context.Background())
look := conditions.Observation{Scope: conditions.ScopeMachine, ID: "anchor", Kind: "silent", Machine: "anchor",
Severity: conditions.Warning, Summary: "anchor has not been heard from", Source: "S1"}
for range 4 {
if _, err := k.Observe(ctx, look); err != nil {
t.Fatal(err)
}
}
api, err := jetstream.New(serving.Conn())
if err != nil {
t.Fatal(err)
}
stream, err := api.Stream(ctx, "KV_"+broker.ConditionsBucket)
if err != nil {
t.Fatal(err)
}
info, err := stream.Info(ctx)
if err != nil {
t.Fatal(err)
}
if info.State.LastSeq != 1 {
t.Fatalf("four looks at one unchanged condition wrote the bucket %d times, want 1", info.State.LastSeq)
}
// The verb as a process other than the serving controller: it reads the bucket and the store.
keptBefore := conditionsFrom
conditionsFrom = nil
defer func() { conditionsFrom = keptBefore }()
t.Setenv(broker.NATSVar, bus.ClientURL())
t.Setenv(broker.CertificateVar, "")
var shown bytes.Buffer
if err := showCondition(ctx, []string{"machine.anchor.silent", "--json"}, &shown); err != nil {
t.Fatal(err)
}
var c conditions.Condition
if err := json.Unmarshal(shown.Bytes(), &c); err != nil {
t.Fatalf("%v: %s", err, shown.String())
}
if c.Observations != 4 || len(c.Evidence) != 4 || c.LastObserved.IsZero() || c.LooksUnread != "" ||
time.Since(c.LastObserved) > time.Minute {
t.Fatalf("conditions show reads %d observation(s), %d evidence line(s), last %s (%q)", c.Observations,
len(c.Evidence), c.LastObserved, c.LooksUnread)
}
listed, err := openConditions(ctx)
if err != nil || len(listed) != 1 || listed[0].Observations != 4 {
t.Fatalf("conditions lists %+v: %v", listed, err)
}
}
+16 -8
View File
@@ -24,16 +24,18 @@ import (
// **`status` used to be the only place the mesh said it was wrong, and only to whoever asked.** Every
// core failure of research 031 was found by a person looking. A condition is the mesh saying it: raised
// by a watchdog when a signal is late (signals.go), by a probe when an invariant does not hold
// (doctor.go), or by an event a provider sends (standing.go); kept on the bus with since-when and
// evidence; said on the bus as it changes, for the operator's channel to carry; and cleared when an
// (doctor.go), or by an event a provider sends (standing.go); kept on the bus with since-when, and its
// looks — how often, when last, the newest evidence — in the controller's store (condition_looks.go,
// novox/hq issue 420); said on the bus as it changes, for the operator's channel to carry; and cleared when an
// observation says it is resolved. Nobody resolves one by hand. A person who knows silences it, for a
// while, with a reason, and that is recorded as a hand act.
// conditionsFrom is the serving controller's keeper; nil in any other process, which opens its own.
var conditionsFrom *conditions.Keeper
// keeperOn is a keeper over the store on a connection, saying its transitions on that connection.
func keeperOn(ctx context.Context, conn *nats.Conn) (*conditions.Keeper, error) {
// keeperOn is a keeper over the store on a connection, saying its transitions on that connection, with
// the conditions' looks kept in looks (condition_looks.go).
func keeperOn(ctx context.Context, conn *nats.Conn, looks conditions.LookStore) (*conditions.Keeper, error) {
store, history, err := conditions.OnTheBus(ctx, conn)
if err != nil {
return nil, err
@@ -42,7 +44,7 @@ func keeperOn(ctx context.Context, conn *nats.Conn) (*conditions.Keeper, error)
if err != nil {
return nil, err
}
return conditions.NewKeeper(ctx, conditions.Options{Store: store, History: history,
return conditions.NewKeeper(ctx, conditions.Options{Store: store, History: history, Looks: looks,
Teller: link.OverNATS{Conn: conn, JS: js},
Say: func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) },
// What status leads with changed: composed again soon (a nudge outside the serving controller
@@ -58,8 +60,10 @@ func withKeeper(ctx context.Context, f func(*conditions.Keeper) error) error {
if conditionsFrom != nil {
return f(conditionsFrom)
}
looks, closeLooks := looksHere(ctx)
defer closeLooks()
return onTheBus(func(conn *nats.Conn) error {
k, err := keeperOn(ctx, conn)
k, err := keeperOn(ctx, conn, looks)
if err != nil {
return err
}
@@ -82,6 +86,8 @@ func openConditions(ctx context.Context) ([]conditions.Condition, error) {
return nil, fmt.Errorf("this process has no bus to read the conditions from: %w", err)
}
var out []conditions.Condition
looks, closeLooks := looksHere(ctx)
defer closeLooks()
err := onTheBus(func(conn *nats.Conn) error {
store, _, err := conditions.OnTheBus(ctx, conn)
if err != nil {
@@ -89,7 +95,7 @@ func openConditions(ctx context.Context) ([]conditions.Condition, error) {
}
reading, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
out, err = conditions.Read(reading, store)
out, err = conditions.ReadWith(reading, store, looks)
return err
})
return out, err
@@ -252,12 +258,14 @@ func showCondition(ctx context.Context, args []string, w io.Writer) error {
if conditionsFrom != nil {
c, found, err = conditionsFrom.Get(ctx, key)
} else {
looks, closeLooks := looksHere(ctx)
defer closeLooks()
err = onTheBus(func(conn *nats.Conn) error {
store, _, err := conditions.OnTheBus(ctx, conn)
if err != nil {
return err
}
c, found, err = conditions.ReadOne(ctx, store, key)
c, found, err = conditions.ReadOneWith(ctx, store, looks, key)
return err
})
}
+383
View File
@@ -0,0 +1,383 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// configCommand reads and sets config values (novox/hq issue 630). A value resolves from four sources, the
// last set winning: the module's default, mesh config, node config, and the module's own value on a node
// (its module-assignment config). Mesh and node config hold values only for the predefined keys, the
// catalogue's list; a module's own value is set on one node, never for the whole mesh.
//
// config [list] [--node <node>] [--module <module>]
// config set <key> <value> [--node <node> [--module <module>]] --why <text>
// config clear <key> [--node <node> [--module <module>]] --why <text>
// config history <key> [--node <node>]
//
// A set or clear is recorded in the hand-act log with the caller and why, kept in the history, and takes
// effect at the next push of the machines it reaches, which it names — as `settings set` does.
func configCommand(ctx context.Context, args []string) error {
sub := "list"
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
sub, args = args[0], args[1:]
}
set := flag.NewFlagSet("config", flag.ContinueOnError)
node := set.String("node", "", "one node: its node config, or with --module the module's own value there")
module := set.String("module", "", "a module: its own value on --node (module-assignment config)")
acts := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if *module != "" && *node == "" && sub != "list" {
return fmt.Errorf("config: %s names no node, and there is no module layer for the whole mesh — an "+
"assignment is one module on one node (novox/hq issue 630). Give node, or set mesh config for a "+
"predefined key without module. Nothing was changed", *module)
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
switch sub {
case "list":
if len(positionals) > 0 {
return errors.New("config list [--node <node>] [--module <module>]")
}
return configList(ctx, inv, *node, *module)
case "history":
if len(positionals) != 1 || *module != "" {
return errors.New("config history <key> [--node <node>] — a module's own values are in `settings show <module> --node <node> --history`")
}
past, err := inv.ConfigHistory(ctx, positionals[0], *node)
if err != nil {
return err
}
where := configWhere(*node, "")
if len(past) == 0 {
fmt.Printf("%s, %s: no value has been replaced\n", positionals[0], where)
return nil
}
for _, p := range past {
v, _ := json.Marshal(p.Value)
fmt.Printf("%s, %s: %s until %s (%s", positionals[0], where, v,
p.ReplacedAt.Local().Format("2006-01-02 15:04:05"), p.ReplacedBy)
if p.ReplacedWhy != "" {
fmt.Printf(": %s", p.ReplacedWhy)
}
fmt.Printf("); %s, because %s\n", orUnknown(p.SetBy), orUnknown(p.Why))
}
return nil
case "set", "clear":
default:
return fmt.Errorf("config has no %q; it has list, set, clear and history", sub)
}
// set and clear.
if err := acts.require("config " + sub); err != nil {
return err
}
why := strings.TrimSpace(*acts.why)
var key string
var value any
switch {
case sub == "set" && len(positionals) == 2:
key = positionals[0]
if err := json.Unmarshal([]byte(positionals[1]), &value); err != nil {
// A bare word is a string: `config set theme dark` means "dark".
value = positionals[1]
}
if value == nil {
return errors.New("config set takes a value; to remove one, config clear")
}
case sub == "clear" && len(positionals) == 1:
key = positionals[0]
default:
return errors.New("config set <key> <value> [--node <node> [--module <module>]] --why <text>, or " +
"config clear <key> [--node <node> [--module <module>]] --why <text>")
}
if *node != "" {
if _, err := inv.NodeByName(ctx, *node); err != nil {
return err
}
}
where := configWhere(*node, *module)
// The module's own value on one node: a preference it declares, of its type, judged as `settings set`
// judges a layer, its trusted keys refused through a verb exactly as there (novox/hq issue 339, 340).
if *module != "" {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return err
}
m, known := shelf[*module]
if !known {
return fmt.Errorf("%w: %s", inventory.ErrNoSuchModule, *module)
}
p, declared := catalogue.PreferencesOf(m)[key]
if !declared {
return fmt.Errorf("config: %s declares no preference %q; config sets a module's preferences (%s) — "+
"any other setting is the settings verb's. Nothing was changed", *module, key,
preferencesOrNone(sortedPreferenceKeys(m)))
}
if value != nil {
if err := catalogue.ConfigValueProblem(fmt.Sprintf("%s's preference %q", *module, key), p.Type, value); err != nil {
return err
}
}
before, _, err := inv.Layer(ctx, *node, *module)
if err != nil {
return err
}
after := map[string]any{}
for k, v := range before {
after[k] = v
}
if value == nil {
delete(after, key)
} else {
after[key] = value
}
if err := refuseTerminalSettingsThroughAVerb(ctx, inv, before, after, *module, *node); err != nil {
return err
}
acts.record(ctx, "config", configActArgs(sub, positionals, *node, *module))
was, had, err := inv.SetModuleValue(ctx, *node, *module, key, value, setByWords()+": "+why)
if err != nil {
return err
}
sayConfigChange(key, where, value, was, had)
fmt.Printf(" takes effect at the next push: run `plan %s --diff` to see it, `push %s` to send it\n", *node, *node)
return nil
}
// Mesh config or node config: a predefined key only, judged before the act is recorded.
if value != nil {
if err := inv.JudgeConfig(ctx, key, *node, value); err != nil {
return err
}
}
acts.record(ctx, "config", configActArgs(sub, positionals, *node, ""))
before, had := configValueOf(ctx, inv, key, *node)
if value == nil {
removed, err := inv.ClearConfig(ctx, key, *node, setByWords(), why)
if err != nil {
return err
}
if !removed {
fmt.Printf("%s, %s: no value was set; nothing changed\n", key, where)
return nil
}
} else if err := inv.SetConfig(ctx, key, *node, value, setByWords(), why); err != nil {
return err
}
sayConfigChange(key, where, value, before, had)
reached, err := reachedBy(ctx, inv, key, *node)
if err != nil {
return err
}
if len(reached) == 0 {
fmt.Println(" no module assigned there falls back to it yet; nothing to push")
return nil
}
fmt.Printf(" takes effect at the next push of %s: run `push %s` to send it\n", strings.Join(reached, ", "),
strings.Join(reached, " "))
return nil
}
// configList lists the predefined keys with their mesh and node config, then each module's preferences
// with the value on each node it is assigned to and where it comes from.
func configList(ctx context.Context, inv *inventory.Inventory, node, module string) error {
if node != "" {
if _, err := inv.NodeByName(ctx, node); err != nil {
return err
}
}
list, err := inv.Predefined(ctx)
if err != nil {
return err
}
values, err := inv.Config(ctx)
if err != nil {
return err
}
var b strings.Builder
if len(list) == 0 {
fmt.Fprintf(&b, "no predefined keys: the list arrives with a build of the catalogue carrying %s\n",
catalogue.PredefinedFile)
} else {
fmt.Fprintf(&b, "predefined keys (the catalogue's %s), with mesh config and node config:\n", catalogue.PredefinedFile)
}
for _, key := range list.Keys() {
k := list[key]
fmt.Fprintf(&b, " %s (%s): %s\n", key, k.Type, k.Why)
held := false
for _, v := range values {
if v.Key != key || (node != "" && v.Node != "" && v.Node != node) {
continue
}
held = true
shown, _ := json.Marshal(v.Value)
fmt.Fprintf(&b, " %s: %s — %s, because %s\n", configWhere(v.Node, ""), shown, orUnknown(v.SetBy), v.Why)
}
if !held {
b.WriteString(" no mesh or node config: each module's default\n")
}
}
for _, key := range inventory.ConfigKeysSorted(values) {
if _, on := list[key]; on {
continue
}
for _, v := range values {
if v.Key == key {
shown, _ := json.Marshal(v.Value)
fmt.Fprintf(&b, " %s is no longer a predefined key, and %s still holds %s for it: it reaches nothing; "+
"config clear %s%s\n", key, configWhere(v.Node, ""), shown, key, nodeFlag(v.Node))
}
}
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return err
}
var listed []preferencesOf
for _, e := range entries {
m := e.Manifest
if len(catalogue.PreferencesOf(m)) == 0 || (module != "" && m.Module != module) {
continue
}
if node != "" && !containsString(e.On, node) {
continue
}
p := preferencesOf{Manifest: m, On: map[string][]catalogue.SettingSource{}}
for _, n := range e.On {
if node != "" && n != node {
continue
}
p.Nodes = append(p.Nodes, n)
layers, err := inv.SettingsFor(ctx, n, m.Module)
if err != nil {
return err
}
p.On[n] = catalogue.Effective(m, layers)
}
listed = append(listed, p)
}
b.WriteString("\npreferences, and on each node the value and where it comes from (default, mesh config, " +
"node config, or the module on the node):\n")
b.WriteString(describePreferences(listed))
fmt.Print(b.String())
return nil
}
// configWhere is how a config value's place is said.
func configWhere(node, module string) string {
switch {
case module != "":
return module + " on " + node
case node != "":
return catalogue.NodeConfigLayer(node)
}
return catalogue.MeshConfigLayer
}
// configValueOf is the value mesh config (node "") or a node's config holds for a key.
func configValueOf(ctx context.Context, inv *inventory.Inventory, key, node string) (any, bool) {
values, err := inv.Config(ctx)
if err != nil {
return nil, false
}
for _, v := range values {
if v.Key == key && v.Node == node {
return v.Value, true
}
}
return nil, false
}
// reachedBy is the nodes a value of mesh config (node "") or a node's config for a key reaches: those
// running a module whose preference falls back to it.
func reachedBy(ctx context.Context, inv *inventory.Inventory, key, node string) ([]string, error) {
entries, err := inv.Catalogued(ctx)
if err != nil {
return nil, err
}
nodes := map[string]bool{}
for _, e := range entries {
if !containsString(catalogue.PredefinedNamed(e.Manifest), key) {
continue
}
for _, n := range e.On {
if node == "" || n == node {
nodes[n] = true
}
}
}
out := make([]string, 0, len(nodes))
for n := range nodes {
out = append(out, n)
}
sort.Strings(out)
return out, nil
}
func sayConfigChange(key, where string, value, before any, had bool) {
was := "nothing"
if had {
w, _ := json.Marshal(before)
was = string(w)
}
if value == nil {
fmt.Printf("%s, %s: cleared (was %s); the next source down applies\n", key, where, was)
return
}
now, _ := json.Marshal(value)
fmt.Printf("%s, %s: %s (was %s); the value it replaced is kept\n", key, where, now, was)
}
func configActArgs(sub string, positionals []string, node, module string) []string {
args := append([]string{sub}, positionals...)
if node != "" {
args = append(args, "--node", node)
}
if module != "" {
args = append(args, "--module", module)
}
return args
}
func sortedPreferenceKeys(m catalogue.Manifest) []string {
declared := catalogue.PreferencesOf(m)
keys := make([]string, 0, len(declared))
for k := range declared {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
func orUnknown(s string) string {
if strings.TrimSpace(s) == "" {
return "not kept"
}
return s
}
func preferencesOrNone(keys []string) string {
if len(keys) == 0 {
return "it declares none"
}
return strings.Join(keys, ", ")
}
+187
View File
@@ -0,0 +1,187 @@
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The config verb's line: a listing without key, a set or clear with why, and a module value refused
// without a node (novox/hq issue 630).
func TestTheConfigVerbsLine(t *testing.T) {
for _, c := range []struct {
args map[string]any
want string
}{
{map[string]any{}, "config list"},
{map[string]any{"node": "g14", "module": "dunst"}, "config list --node g14 --module dunst"},
{map[string]any{"key": "font-size", "value": "12", "why": "bigger"}, "config set font-size 12 --why bigger"},
{map[string]any{"key": "font-size", "value": "13", "node": "g14", "why": "dense"},
"config set font-size 13 --node g14 --why dense"},
{map[string]any{"key": "font-size", "value": "14", "node": "g14", "module": "dunst", "why": "x"},
"config set font-size 14 --node g14 --module dunst --why x"},
{map[string]any{"key": "font-size", "clear": "true", "node": "g14", "why": "back"},
"config clear font-size --node g14 --why back"},
{map[string]any{"key": "font-size", "history": "true"}, "config history font-size"},
} {
argv, err := argvFor("config", c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%v: %v %v, want %s", c.args, argv, err, c.want)
}
}
for want, args := range map[string]map[string]any{
"no module layer for the whole mesh": {"key": "font-size", "value": "12", "module": "dunst", "why": "x"},
"why": {"key": "font-size", "value": "12"},
"names its key": {"value": "12"},
"not both": {"key": "font-size", "value": "12", "clear": "true", "why": "x"},
} {
if _, err := argvFor("config", args); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%v: %v, want %q", args, err, want)
}
}
}
// End to end through the verb, as the operator's acceptance reads it: cleared → the default; mesh font-size,
// then the node's, then dunst's own on the node — each in the node's dunstrc as composed (novox/hq issue 630).
func TestConfigReachesTheComposedFileFromEachSource(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
if err := open.inventory.SetPredefined(ctx, catalogue.Predefined{
"font-size": {Type: catalogue.TypeNumber, Why: "the size text is drawn at"}}, "", ""); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "dunst", Version: "1",
Preferences: map[string]catalogue.Preference{
"font-size": {Type: catalogue.TypeNumber, Default: float64(10), Predefined: "font-size", Why: "100 DPI"},
},
Resources: []map[string]any{{"id": "dunstrc", "type": "file", "path": "/etc/dunst/dunstrc", "mode": "0644", "trusted": false,
"content": "font = Inter ${setting:font-size}\n"}}})
for _, n := range []string{"laptop", "anchor"} {
if _, err := assign(ctx, open, n, "dunst"); err != nil {
t.Fatal(err)
}
}
dunstrc := func(node string) string {
t.Helper()
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
declared, problems, err := composedAndValidated(ctx, open, node, gens, Allocating)
if err != nil {
t.Fatalf("%s does not compose: %v %v", node, err, problems)
}
for _, r := range declared.Resources {
if r["path"] == "/etc/dunst/dunstrc" {
return fmt.Sprint(r["content"])
}
}
t.Fatalf("%s has no dunstrc: %v", node, problems)
return ""
}
want := func(node, size string) {
t.Helper()
if got := dunstrc(node); got != "font = Inter "+size+"\n" {
t.Fatalf("%s's dunstrc: %q, want size %s", node, got, size)
}
}
verb := func(args map[string]any) error {
t.Helper()
return throughVerb(t, "config", args)
}
want("laptop", "10")
if err := verb(map[string]any{"key": "font-size", "value": "11", "why": "a larger default"}); err != nil {
t.Fatal(err)
}
want("laptop", "11")
want("anchor", "11")
if err := verb(map[string]any{"key": "font-size", "value": "13", "node": "laptop", "why": "a dense screen"}); err != nil {
t.Fatal(err)
}
want("laptop", "13")
want("anchor", "11")
if err := verb(map[string]any{"key": "font-size", "value": "15", "node": "laptop", "module": "dunst",
"why": "dunst alone larger"}); err != nil {
t.Fatal(err)
}
want("laptop", "15")
listing := stdoutOf(t, func() error { return verb(map[string]any{}) })
for _, said := range []string{"font-size (number): the size text is drawn at", "mesh config: 11",
"node config (laptop): 13", "laptop: 15 (dunst on laptop)", "anchor: 11 (mesh config)",
"falls back to the predefined key font-size"} {
if !strings.Contains(listing, said) {
t.Errorf("the listing does not say %q:\n%s", said, listing)
}
}
// Cleared, each falls back to the next source down.
for _, step := range []struct {
args map[string]any
size string
}{
{map[string]any{"key": "font-size", "clear": "true", "node": "laptop", "module": "dunst", "why": "x"}, "13"},
{map[string]any{"key": "font-size", "clear": "true", "node": "laptop", "why": "x"}, "11"},
{map[string]any{"key": "font-size", "clear": "true", "why": "x"}, "10"},
} {
if err := verb(step.args); err != nil {
t.Fatal(err)
}
want("laptop", step.size)
}
// Refused: a key not on the list for mesh config, a module value without a node, a type not the key's.
if err := verb(map[string]any{"key": "colour", "value": "blue", "why": "x"}); err == nil ||
!strings.Contains(err.Error(), "predefined") {
t.Fatalf("mesh config for a key not on the list: %v", err)
}
if err := runLine(t, []string{"config", "set", "font-size", "12", "--module", "dunst", "--why", "x"}); err == nil ||
!strings.Contains(err.Error(), "no module layer for the whole mesh") {
t.Fatalf("a module value without a node: %v", err)
}
if err := verb(map[string]any{"key": "font-size", "value": `"large"`, "node": "laptop", "module": "dunst",
"why": "x"}); err == nil || !strings.Contains(err.Error(), "is a number") {
t.Fatalf("a string for a number: %v", err)
}
if err := verb(map[string]any{"key": "nothing", "value": "1", "node": "laptop", "module": "dunst",
"why": "x"}); err == nil || !strings.Contains(err.Error(), "declares no preference") {
t.Fatalf("a module value for a key it does not declare: %v", err)
}
}
// The module check reads the catalogue's predefined.json at the repository's root and refuses a preference
// falling back to a key not on it (novox/hq issue 630).
func TestTheModuleCheckJudgesPreferencesAgainstTheCataloguesList(t *testing.T) {
root := t.TempDir()
for _, dir := range []string{".git", "modules/dunst"} {
if err := os.MkdirAll(filepath.Join(root, dir), 0o755); err != nil {
t.Fatal(err)
}
}
manifest := filepath.Join(root, "modules/dunst/module.json")
write := func(path, body string) {
t.Helper()
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
write(manifest, `{"module": "dunst", "version": "1",
"preferences": {"font-size": {"type": "number", "default": 10, "predefined": "font-size", "why": "100 DPI"}},
"resources": [{"id": "rc", "type": "file", "path": "/etc/rc", "mode": "0644", "trusted": false,
"content": "font ${setting:font-size}"}]}`)
write(filepath.Join(root, "predefined.json"), `{"theme": {"type": "string", "why": "light or dark"}}`)
var out strings.Builder
if err := moduleCheck([]string{manifest}, &out); err == nil || !strings.Contains(out.String(), "not on the catalogue's list") {
t.Fatalf("a key not on the list passed: %v\n%s", err, out.String())
}
write(filepath.Join(root, "predefined.json"), `{"font-size": {"type": "number", "why": "the size text is drawn at"}}`)
out.Reset()
if err := moduleCheck([]string{manifest}, &out); err != nil {
t.Fatalf("a key on the list was refused: %v\n%s", err, out.String())
}
}
+215
View File
@@ -0,0 +1,215 @@
package main
import (
"context"
"fmt"
"sort"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
)
// **No send but a person's `push <node>` and a gate's own put-back moves the mesh's own core to a build older
// than the one its machine was last sent** (novox/hq issue 201). Every send made through sendToEach keeps it — a
// plan's, a release's, a healer's, a rotation's, and the bus step too, which is a person's act but composes under
// sendKeeps like the rest. On 2026-10-02 two merges a minute apart made two plans;
// the controller's plan rolled the new controller, which widened its seat's row, and the node-engine's plan then
// sent the control node the controller build before it. The older binary read the row it could not serve and the
// mesh lost its voice for ten minutes — the thing that would have corrected it was the thing that was down. ADR
// 0185 took the outage out of it (a controller behind its row serves what it can); the race is closed here,
// without needing its trigger: whatever made the mesh's record older, the send keeps the build.
//
// **Kept, not refused.** The send composes the controller or the node-engine at the build the machine was last
// sent, as a recorded module is kept (ADR 0245), and carries everything else: no other module, secret, healer or
// walk waits on it, and nothing is retried. The kept move is no move for a gate either: ungatedIn and gatedSend
// leave it out (withoutKept), so a gate judges the other moves of its send. A plan whose own build is the older one finds its gate superseded by
// the build kept there (issue 352) and ends. The mesh's record stays older, and a warning condition says so until
// a send carries the record — a person's `push <node>` accepts the older build — or the record moves forward: a
// newer merge, or `rebuild mesh-controller`.
//
// Older is by the builds' registration order — when each was asked (novox/hq issue 219) — never by the commits'
// dates. So a retry or a rebuild of an old commit, asked anew, counts as newer: it is registered as the module's
// current build by the same order, and the guard does not stop that move. A build that failed its gate is never
// ahead: the machine was sent it and the record was put back before it (gateFailed), and if the put-back's own
// send was refused, the next send completes it rather than keeping the failed build there. A gate's own put-back
// carries its modules back whatever the order (sendScope.puttingBack); the controller's is held to the store's
// schema already (issue 352).
// kindCoreDowngrade is a core module kept at the build its machine was last sent, over an older record.
const kindCoreDowngrade = "core-downgrade-kept"
// coreDowngradeSource is the condition's source: the sends, which raise and clear it.
const coreDowngradeSource = "sends"
// downgradeGuarded is the core components a send may not move backwards: the controller, which reads the
// store and its seat's row, and the node-engine, which reads what the controller declares.
var downgradeGuarded = []string{lease.ComponentController, lease.ComponentEngine}
// askedBuild is what the order needs of one build of a module: when it was asked, and whether it failed its gate.
type askedBuild struct {
asked time.Time
failedGate bool
}
// buildAsked is the newest build of a module from a commit that worked — when it was asked (or recorded, when
// the request time is not known), and whether it failed its gate — and false when the records hold none. A
// variable, so a test reads its own order.
var buildAsked = func(ctx context.Context, inv *inventory.Inventory, module, commit string) (askedBuild, bool, error) {
builds, err := inv.Builds(ctx, module, askedLookBack)
if err != nil {
return askedBuild{}, false, err
}
for _, b := range builds {
if b.Worked() && sameCommit(b.Commit, commit) {
failed, err := inv.GateFailed(ctx, b.ID)
if err != nil {
return askedBuild{}, false, err
}
return askedBuild{asked: b.AskedOrAt(), failedGate: failed}, true, nil
}
}
return askedBuild{}, false, nil
}
// askedLookBack is how many of a module's newest builds are read for when a build was asked.
const askedLookBack = 50
// coreDowngrade is one core module a machine was last sent a build of that is newer than the mesh's record.
type coreDowngrade struct {
module, node, component string
// sent is the build the machine was last sent, record the one the mesh holds.
sent, record string
sentAsked, recordAsked time.Time
}
func (d coreDowngrade) String() string {
return fmt.Sprintf("%s was last sent %s %s, and the mesh holds the older %s (asked %s, before %s): a send there "+
"keeps %s; `push %s` sends the older build, and a newer merge or `rebuild %s` moves the record forward "+
"(novox/hq issue 201)", d.node, d.module, short(d.sent), short(d.record),
d.recordAsked.UTC().Format(time.RFC3339), d.sentAsked.UTC().Format(time.RFC3339), short(d.sent), d.node,
d.module)
}
// coreDowngrades is, for one machine, each guarded core module whose record is older than the build the machine
// was last sent, leaving out what a gate's own put-back puts back and a sent build that failed its gate. A move
// whose either build the records do not hold is not proved older and is left to the gate that judges it. Pure
// but for asked.
func coreDowngrades(node string, sent map[string]string, record map[string]string, scope sendScope,
asked func(module, commit string) (askedBuild, bool, error)) ([]coreDowngrade, error) {
var out []coreDowngrade
for module, was := range sent {
component := coreComponent(module)
now := record[module]
if !guarded(component) || was == "" || now == "" || sameCommit(was, now) {
continue
}
if scope.puttingBack && scope.modules[module] {
continue
}
from, known, err := asked(module, was)
if err != nil {
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(was), err)
}
if !known || from.failedGate {
continue
}
to, known, err := asked(module, now)
if err != nil {
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(now), err)
}
if known && to.asked.Before(from.asked) {
out = append(out, coreDowngrade{module: module, node: node, component: component, sent: was, record: now,
sentAsked: from.asked, recordAsked: to.asked})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].module < out[j].module })
return out, nil
}
func guarded(component string) bool {
for _, c := range downgradeGuarded {
if c == component {
return true
}
}
return false
}
// coreKept is, for a send that keeps (sendKeeps), each guarded core module of a machine to keep at the build it
// was last sent, module → commit, each said and raised as a condition.
func coreKept(ctx context.Context, inv *inventory.Inventory, node string, sent map[string]string,
current map[string]inventory.CurrentBuild) (map[string]string, error) {
record := map[string]string{}
for m, c := range current {
record[m] = c.Commit
}
found, err := coreDowngrades(node, sent, record, scopeOf(ctx), func(module, commit string) (askedBuild, bool, error) {
return buildAsked(ctx, inv, module, commit)
})
if err != nil || len(found) == 0 {
return nil, err
}
out := map[string]string{}
for _, d := range found {
out[d.module] = d.sent
sayCoreDowngrade(ctx, d)
}
return out, nil
}
// coreDowngradeObservation is a kept core build as a condition: `core.<component>.<machine>.<kind>`.
func coreDowngradeObservation(component, node, summary string) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeCore, ID: component + "." + node, Kind: kindCoreDowngrade,
Token: kindCoreDowngrade, Machine: node, Severity: conditions.Warning, Source: coreDowngradeSource,
Summary: summary}
}
func sayCoreDowngrade(ctx context.Context, d coreDowngrade) {
fmt.Printf("KEPT: %s\n", d)
keeper := conditionsFrom
if keeper == nil {
return
}
if _, err := keeper.Observe(ctx, coreDowngradeObservation(d.component, d.node, d.String())); err != nil {
fmt.Printf("the condition for %s kept on %s could not be raised: %v\n", d.module, d.node, err)
}
}
// settleCoreDowngrades clears, after a send to a machine is away, the condition of each guarded core module the
// send carried at the mesh's record — a person's push that accepted the older build, or a send after the record
// moved forward — or did not carry at all. A send that kept the newer build leaves it open.
func settleCoreDowngrades(ctx context.Context, inv *inventory.Inventory, node string, carried map[string]string) {
keeper := conditionsFrom
if keeper == nil || carried == nil {
return
}
var open []string
for _, c := range downgradeGuarded {
if _, found, err := keeper.Get(ctx, coreDowngradeObservation(c, node, "").Key()); err == nil && found {
open = append(open, c)
}
}
if len(open) == 0 {
return
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
fmt.Printf("whether %s still keeps a core build over the mesh's record cannot be read: %v\n", node, err)
return
}
for module, now := range current {
component := coreComponent(module)
if !guarded(component) {
continue
}
if was, has := carried[module]; has && !sameCommit(was, now.Commit) {
continue
}
key := coreDowngradeObservation(component, node, "").Key()
if _, err := keeper.Clear(ctx, key, node+" was sent the "+component+" the mesh holds"); err != nil {
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
}
}
}
+38
View File
@@ -169,6 +169,44 @@ func TestAShrinkOfMoreThanHalfIsUrgent(t *testing.T) {
}
}
// THE FALSE ALARM (issue 368), replayed through the store D13 reads: an agent's home moved from the
// operator's own home (94.7 MB) to the agent account's fresh one (490 B), and `data-shrank` was raised
// for data that was never lost. A moved item is read against its new path only, so nothing is raised —
// and a genuine shrink at the new path, a week of history later, still is.
func TestAMovedPathIsNoShrinkAndAShrinkThereStillIs(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
shelf := shelfFor(t, houseManifest)
declared := []inventory.DeclaredData{{Module: "house", Item: "config", Class: "irreplaceable", Owned: true}}
start := time.Now().Add(-6 * time.Hour)
measure := func(at time.Time, path string, size int64) []conditions.Observation {
t.Helper()
if _, err := inv.RecordData(ctx, "home", declared, map[string]map[string]inventory.Measurement{"house": {
"config": {Path: path, Size: bytesOf(size), MeasuredAt: when(at), LastWrite: when(at),
LastBackup: when(at)}}}, "", at); err != nil {
t.Fatal(err)
}
records, err := inv.Data(ctx)
if err != nil {
t.Fatal(err)
}
peaks, err := inv.DataPeaks(ctx, at.Add(-shrinkWindow))
if err != nil {
t.Fatal(err)
}
return dataFindings(records, peaks, shelf, nil, nil, at)
}
measure(start, "/home/operator/.claude", 94_700_000)
if got := findingsByKind(measure(start.Add(10*time.Minute), "/home/agent/.claude", 490)); got[kindDataShrank].Kind != "" {
t.Fatalf("a moved path raised a shrink: %+v", got[kindDataShrank])
}
measure(start.Add(2*time.Hour), "/home/agent/.claude", 300<<20)
got := findingsByKind(measure(start.Add(4*time.Hour), "/home/agent/.claude", 1<<20))[kindDataShrank]
if got.Severity != conditions.Urgent || !strings.Contains(got.Summary, "shrank") {
t.Fatalf("a genuine shrink at the new path was not raised: %+v", got)
}
}
// Data said to be written all the time and not written; data with no backup or an old one — urgent when
// irreplaceable, a warning when valuable; and a new item given its bound before it is said.
func TestQuietDataAndMissingBackupsAreSaidByClass(t *testing.T) {
+5 -1
View File
@@ -144,11 +144,15 @@ func actOnDeadLetter(ctx context.Context, on *busHandles, act string, id uint64,
}
switch act {
case "deliver":
_, to, err := link.DeliverAgain(on.js, id)
delivered, to, err := link.DeliverAgain(on.js, id)
if err != nil {
return nil, err
}
answer["delivered_on"] = to
if delivered.Original != "" {
// What became of the ask in its seat's queue (novox/hq issue 334): removed, or left, and why.
answer["original"] = delivered.Original
}
answer["done"] = fmt.Sprintf("dead letter %d was delivered again to %s, and nobody else; it is no longer kept",
id, consumerWho(d.Stream, d.Consumer))
case "drop":
+600 -26
View File
@@ -10,6 +10,7 @@ import (
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
@@ -82,6 +83,38 @@ func waitingNote(p inventory.Plan) string {
p.Delivery.Awaits, p.ID)
}
// waitingSaid is what a walk waiting for its word says when the delivery's owner said what holds it (novox/hq
// issue 529): a delivery held for a person, its reason and the verb that releases it — the person's decision, not
// a hand start — or the deliveries not at their turn. Told nothing, it is waitingNote.
func waitingSaid(p inventory.Plan, lines []stalledLine) string {
if len(lines) == 0 {
return waitingNote(p)
}
var held, unheld, other []string
for _, l := range lines {
switch l.State {
case "held":
held = append(held, l.Says)
case "unheld":
unheld = append(unheld, l.Says)
default:
other = append(other, l.Says)
}
}
switch {
case len(held) > 0:
return fmt.Sprintf("published; its walk waits for %s's word, and a delivery it carries is held for a person: %s",
p.Delivery.Awaits, strings.Join(held, "; "))
case len(other) > 0:
// Not at its turn: it goes when its turn comes. A hand start here would skip its group's order (review of
// novox/hq issue 529), so none is offered.
return fmt.Sprintf("published; its walk waits for %s's word: %s", p.Delivery.Awaits, strings.Join(other, "; "))
}
// Held by nothing: nothing will give it its word, so the hand start is the way on.
return fmt.Sprintf("published; its walk waits for %s's word: %s — `plans go %s --why …` starts it by hand",
p.Delivery.Awaits, strings.Join(unheld, "; "), p.ID)
}
// letGo gives a waiting walk its word: by the delivery's owner, or a person. The next advance asks its
// first tier. Refused for a walk that does not wait.
func letGo(ctx context.Context, inv *inventory.Inventory, id, by, why string) (inventory.Plan, error) {
@@ -117,28 +150,419 @@ func letGo(ctx context.Context, inv *inventory.Inventory, id, by, why string) (i
// stopWalk ends a walk on its delivery's word: failed, said as stopped by whom, why. What it asked still
// builds and registers; nothing further is asked or sent.
func stopWalk(ctx context.Context, inv *inventory.Inventory, id, by, why string) (inventory.Plan, error) {
p, _, err := stopWalkWithout(ctx, inv, id, by, why, nil, time.Now().UTC())
return p, err
}
// stopAnswer is what a stop that leaves merges out says, as data (novox/hq ADR 0299): the record stopped, the
// batch that walks its other merges again with those merges, every merge left out, the modules the stopped walk
// already sent that stay on the machines named, the modules the batch's walk withholds, and every merge elsewhere
// that will deliver a stopped change when it is walked — a later merge of its repository, or one moving its modules.
type stopAnswer struct {
Stopped string `json:"stopped"`
WalkedAgainBy string `json:"walked_again_by,omitempty"`
WalkedAgain []string `json:"walked_again,omitempty"`
LeftOut []inventory.PlanLeftOut `json:"left_out,omitempty"`
Kept []inventory.PlanKept `json:"kept,omitempty"`
Withheld []string `json:"withheld,omitempty"`
StillCarriedBy []string `json:"still_carried_by,omitempty"`
}
// stopWalkWithout ends a walk on its delivery's word and, given merges to leave out, walks every other merge it
// answered again without them (novox/hq issue 459, ADR 0299): one walk carries every merge of its batch, so ending
// it for one group member ended every other delivery on it.
//
// - Left out are the merges named, each later merge of their repository's branch, which contains one, and each
// merge moving a module a left-out merge moves: a module is built at its branch, which holds the stopped merge.
// - The others join the open batch of their kind — merges heard while the walk ran wait there, and an older
// commit never walks after a newer one — or a new batch when none is open; its window is closed, and its walk
// withholds the left-out merges' modules where it would build them only as dependents.
// - Given a batch not yet cut, the merges left out leave it for a stopped record of their own, and the batch
// goes on without them; with nothing left in it, the batch itself is stopped.
// - The stopped record, the batch and every merge moved are written in one transaction (decision 7).
//
// With nothing left to walk, or no merge named, the walk is only ended. A merge named that the record does not
// answer refuses the stop whole.
func stopWalkWithout(ctx context.Context, inv *inventory.Inventory, id, by, why string, without []string,
now time.Time) (inventory.Plan, stopAnswer, error) {
said := stopAnswer{Stopped: id}
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return inventory.Plan{}, err
return inventory.Plan{}, said, err
}
defer release()
p, err := inv.PlanByID(ctx, id)
if err != nil {
return inventory.Plan{}, err
return inventory.Plan{}, said, err
}
if !p.Open() {
return p, fmt.Errorf("%s is already %s", p.ID, p.State)
batch := p.Batch()
if !p.Open() && !(batch && len(without) > 0) {
return p, said, fmt.Errorf("%s is already %s", p.ID, p.State)
}
if p.Delivery == nil {
p.Delivery = &inventory.PlanDelivery{}
var merges, again []inventory.BatchedMerge
if len(without) > 0 {
if merges, err = inv.MergesOf(ctx, p.ID); err != nil {
return p, said, err
}
if again, said.LeftOut, err = leaveOut(merges, without); err != nil {
return p, said, fmt.Errorf("%s is not stopped: %w", p.ID, err)
}
}
p.Delivery.Stopped, p.Delivery.StoppedWhy = by, why
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("stopped by %s at tier %d: %s", by, p.Tier, why)
if err := inv.SavePlan(ctx, &p); err != nil {
return p, err
stopped := func(r *inventory.Plan) {
if r.Delivery == nil {
r.Delivery = &inventory.PlanDelivery{}
}
r.Delivery.Stopped, r.Delivery.StoppedWhy, r.Delivery.LeftOut = by, why, said.LeftOut
r.State = inventory.PlanFailed
r.Note = fmt.Sprintf("stopped by %s at tier %d: %s", by, r.Tier, why)
}
return p, nil
withheld := movesOf(leftOutMerges(merges, said.LeftOut))
said.Withheld = withheld
if said.StillCarriedBy, err = stillCarried(ctx, inv, p.ID, merges, said.LeftOut, withheld); err != nil {
return p, said, err
}
if batch {
return p, said, stopInBatch(ctx, inv, &p, merges, again, &said, stopped, now)
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return p, said, err
}
running := map[string][]string{}
for _, e := range entries {
running[e.Manifest.Module] = e.On
}
said.Kept = keptOf(p, movesOf(again), running)
stopped(&p)
p.Delivery.Kept = said.Kept
if len(again) == 0 {
if len(said.LeftOut) > 0 {
p.Note += fmt.Sprintf("; left out %s, nothing else to walk", mergesLeftOutWords(said.LeftOut))
}
return p, said, inv.SavePlan(ctx, &p)
}
target, folded, foldMoves, err := batchToWalkAgain(ctx, inv, p.ID, again, now)
if err != nil {
return p, said, fmt.Errorf("%s is not stopped: %w", p.ID, err)
}
target.Delivery.Batch.Closed = true
target.Delivery.Batch.Withheld = union(target.Delivery.Batch.Withheld, withheld)
sort.Strings(target.Delivery.Batch.Withheld)
said.WalkedAgainBy = target.ID
var moves []inventory.MergeMove
for _, m := range again {
said.WalkedAgain = append(said.WalkedAgain, m.Repository+"@"+m.Commit)
moves = append(moves, inventory.MergeMove{Repository: m.Repository, Commit: m.Commit, Plan: target.ID, Alone: m.Alone})
}
p.Delivery.WalkedAgainBy = target.ID
p.Note += fmt.Sprintf("; its other merges are walked again by %s: %s", target.ID, strings.Join(said.WalkedAgain, ", "))
if len(said.LeftOut) > 0 {
p.Note += "; left out " + mergesLeftOutWords(said.LeftOut)
}
if len(said.Kept) > 0 {
p.Note += "; " + keptWords(said.Kept)
}
// The batch first, so no merge names a plan the store does not hold; all of it, or nothing (decision 7) — the
// batches folded into it too.
if err := inv.SavePlansMoving(ctx, append([]*inventory.Plan{&target, &p}, folded...),
append(moves, foldMoves...)); err != nil {
return p, said, fmt.Errorf("%s is not stopped, and none of its merges moved: %w", p.ID, err)
}
// What the batch holds and names now: written again by the cutter's next look should this fail.
if err := keepBatch(ctx, inv, &target, now, ""); err != nil {
fmt.Printf(" %s holds its merges; its record is written again at the next look: %v\n", target.ID, err)
}
fmt.Printf("%s stopped by %s; %s walks its other merges again: %s\n", p.ID, by, target.ID,
strings.Join(said.WalkedAgain, ", "))
return p, said, nil
}
// stopInBatch leaves merges out of a batch not yet cut (ADR 0299): they go to a stopped record of their own, which
// names the batch as walking the others, and the batch withholds their modules; a batch left with none is stopped
// itself.
func stopInBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, merges, again []inventory.BatchedMerge,
said *stopAnswer, stopped func(*inventory.Plan), now time.Time) error {
if len(again) == 0 {
stopped(b)
b.Delivery.Batch = nil
b.Note = "a batch whose every merge was left out by a stop: " + b.Note
return inv.SavePlan(ctx, b)
}
left := leftOutMerges(merges, said.LeftOut)
first := left[0]
r := inventory.Plan{ID: fmt.Sprintf("plan-%d", now.UnixNano()), Repository: first.Repository, Branch: first.Branch,
Commit: first.Commit, Merged: first.Merged, Created: now, Tiers: [][]string{},
Modules: map[string]*inventory.PlanModule{}}
if r.ID == b.ID {
r.ID += "-left"
}
stopped(&r)
r.Delivery.WalkedAgainBy = b.ID
r.Note += fmt.Sprintf("; left out of %s before it was walked: %s; %s walks the others", b.ID,
mergesLeftOutWords(said.LeftOut), b.ID)
b.Delivery.Batch.Withheld = union(b.Delivery.Batch.Withheld, said.Withheld)
sort.Strings(b.Delivery.Batch.Withheld)
var moves []inventory.MergeMove
for _, m := range left {
moves = append(moves, inventory.MergeMove{Repository: m.Repository, Commit: m.Commit, Plan: r.ID, Alone: m.Alone})
}
for _, m := range again {
said.WalkedAgain = append(said.WalkedAgain, m.Repository+"@"+m.Commit)
}
said.Stopped, said.WalkedAgainBy = r.ID, b.ID
if err := inv.SavePlansMoving(ctx, []*inventory.Plan{&r, b}, moves); err != nil {
return fmt.Errorf("%s is not changed, and none of its merges moved: %w", b.ID, err)
}
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
fmt.Printf(" %s holds its merges; its record is written again at the next look: %v\n", b.ID, err)
}
fmt.Printf("%s: %s left out by a stop, kept by %s\n", b.ID, mergesLeftOutWords(said.LeftOut), r.ID)
return nil
}
// batchToWalkAgain is the batch a stopped walk's merges join (novox/hq ADR 0299): a batch of their kind not yet cut
// that they are related to (ADR 0304 decision 1, ADR 0306 decision 3), or a new one, not yet written. Related to
// two or more, the newer are folded into the oldest: the plans and merge moves that fold them, to be written in
// the stop's one act. The kind is read from what the merges moved; the stopped walk itself is no walk a cut folds.
func batchToWalkAgain(ctx context.Context, inv *inventory.Inventory, stopping string, again []inventory.BatchedMerge,
now time.Time) (inventory.Plan, []*inventory.Plan, []inventory.MergeMove, error) {
own := false
for _, n := range movesOf(again) {
if _, on := onTheControllersPath[n]; on {
own = true
}
}
entries, read, edges, err := theGraph(ctx, inv)
if err != nil {
return inventory.Plan{}, nil, nil, err
}
joins, _, _, err := relatedBatches(ctx, inv, again, own, stopping, entries, read, edges)
if err != nil {
return inventory.Plan{}, nil, nil, err
}
if len(joins) > 0 {
into := joins[0]
if len(joins) == 1 {
return into, nil, nil, nil
}
plans, moves, err := foldInto(ctx, inv, &into, joins[1:])
if err != nil {
return inventory.Plan{}, nil, nil, err
}
return into, plans[1:], moves, nil
}
first := again[0]
return inventory.Plan{ID: fmt.Sprintf("plan-%d", now.UnixNano()), Repository: first.Repository, Branch: first.Branch,
Commit: first.Commit, Merged: first.Merged, Created: now, State: inventory.PlanAssembling,
Tiers: [][]string{}, Modules: map[string]*inventory.PlanModule{},
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own}}}, nil, nil, nil
}
// leaveOut splits a walk's merges by the merges named to leave out, each `<repository>@<commit>` (a commit's
// first characters will do): the merges walked again, and those left out — each named one; each later merge of
// its repository's branch, which contains it; and each merge moving a module a left-out merge moves, which is
// built at its branch and so holds the stopped merge — until none more is. A name the walk does not answer is
// refused.
func leaveOut(merges []inventory.BatchedMerge, without []string) ([]inventory.BatchedMerge, []inventory.PlanLeftOut, error) {
var stopped []inventory.BatchedMerge
for _, w := range without {
repository, commit, ok := strings.Cut(strings.TrimSpace(w), "@")
if !ok || repository == "" || len(commit) < 7 {
return nil, nil, fmt.Errorf("%q names no merge: <repository>@<commit>", w)
}
i := slices.IndexFunc(merges, func(m inventory.BatchedMerge) bool {
return strings.EqualFold(m.Repository, repository) && strings.HasPrefix(m.Commit, commit)
})
if i < 0 {
return nil, nil, fmt.Errorf("it answers no merge %s@%s", repository, commit)
}
stopped = append(stopped, merges[i])
}
same := func(a, b inventory.BatchedMerge) bool {
return strings.EqualFold(a.Repository, b.Repository) && a.Commit == b.Commit
}
out := map[int]inventory.PlanLeftOut{}
for i, m := range merges {
if slices.ContainsFunc(stopped, func(s inventory.BatchedMerge) bool { return same(s, m) }) {
out[i] = inventory.PlanLeftOut{Repository: m.Repository, Commit: m.Commit}
}
}
for grew := true; grew; {
grew = false
for i, m := range merges {
if _, left := out[i]; left {
continue
}
for j, o := range out {
l := merges[j]
contains := o.Contains
if contains == "" {
contains = l.Commit
}
if strings.EqualFold(l.Repository, m.Repository) && l.Branch == m.Branch && laterMerge(m, l) {
out[i] = inventory.PlanLeftOut{Repository: m.Repository, Commit: m.Commit, Contains: contains}
} else if shared := sharedMoves(m, l); len(shared) > 0 {
out[i] = inventory.PlanLeftOut{Repository: m.Repository, Commit: m.Commit, Contains: contains,
Shares: shared}
} else {
continue
}
grew = true
break
}
}
}
var again []inventory.BatchedMerge
var left []inventory.PlanLeftOut
for i, m := range merges {
if o, is := out[i]; is {
left = append(left, o)
} else {
again = append(again, m)
}
}
sort.SliceStable(left, func(i, j int) bool { return left[i].Contains == "" && left[j].Contains != "" })
return again, left, nil
}
// sharedMoves are the modules two kept merges both moved when they were heard.
func sharedMoves(a, b inventory.BatchedMerge) []string {
var out []string
theirs := announcedOf(b).Moves
for _, n := range announcedOf(a).Moves {
if slices.Contains(theirs, n) && !slices.Contains(out, n) {
out = append(out, n)
}
}
sort.Strings(out)
return out
}
// leftOutMerges are the kept merges a stop left out.
func leftOutMerges(merges []inventory.BatchedMerge, out []inventory.PlanLeftOut) []inventory.BatchedMerge {
var left []inventory.BatchedMerge
for _, m := range merges {
if slices.ContainsFunc(out, func(o inventory.PlanLeftOut) bool {
return strings.EqualFold(o.Repository, m.Repository) && o.Commit == m.Commit
}) {
left = append(left, m)
}
}
return left
}
// movesOf is every module kept merges moved when they were heard, sorted.
func movesOf(merges []inventory.BatchedMerge) []string {
var out []string
for _, m := range merges {
out = union(out, announcedOf(m).Moves)
}
sort.Strings(out)
return out
}
// keptOf are the modules a stopped walk sent that no merge walked again moves, and the machines each reached:
// those keep running the stopped walk's build. A module sent to its first machines reached those, and the rest it
// was sent to after them; one sent with no first machine (its policy sends it to all together, or a later tier is
// built by it) reached every machine running it.
func keptOf(p inventory.Plan, walkedAgain []string, running map[string][]string) []inventory.PlanKept {
var out []inventory.PlanKept
for name, m := range p.Modules {
if m == nil || slices.Contains(walkedAgain, name) {
continue
}
var machines []string
if m.FirstAt != nil || m.SentAt != nil {
machines = union(machines, m.First)
}
if m.SentAt != nil && len(m.First) == 0 {
machines = union(machines, running[name])
}
if m.Gate != nil && m.Gate.Since != nil {
machines = union(machines, m.Gate.Machines)
}
for machine := range m.Rest {
machines = union(machines, []string{machine})
}
if len(machines) == 0 {
continue
}
sort.Strings(machines)
out = append(out, inventory.PlanKept{Module: name, Machines: machines})
}
sort.Slice(out, func(i, j int) bool { return out[i].Module < out[j].Module })
return out
}
// keptWords is what a stopped walk leaves on the machines, as a person reads it.
func keptWords(kept []inventory.PlanKept) string {
var words []string
for _, k := range kept {
words = append(words, k.Module+" on "+strings.Join(k.Machines, ", "))
}
return "left running the stopped walk's build, which nothing walks again: " + strings.Join(words, "; ")
}
// stillCarried are the merges outside the stopped record, in a batch or an open walk, that will deliver a stopped
// change when they are walked: a later merge of its repository's branch, which contains it, or one that moves a
// module it moves. Holding a merged change back for good is a revert's (ADR 0299).
func stillCarried(ctx context.Context, inv *inventory.Inventory, id string, merges []inventory.BatchedMerge,
out []inventory.PlanLeftOut, withheld []string) ([]string, error) {
var named []inventory.BatchedMerge
for _, m := range leftOutMerges(merges, out) {
if slices.ContainsFunc(out, func(o inventory.PlanLeftOut) bool { return o.Commit == m.Commit && o.Contains == "" }) {
named = append(named, m)
}
}
if len(named) == 0 {
return nil, nil
}
open, err := inv.OpenPlans(ctx)
if err != nil {
return nil, err
}
batches, err := inv.Batches(ctx)
if err != nil {
return nil, err
}
var carried []string
for _, p := range append(open, batches...) {
if p.ID == id {
continue
}
ms, err := inv.MergesOf(ctx, p.ID)
if err != nil {
return nil, err
}
for _, m := range ms {
later := slices.ContainsFunc(named, func(s inventory.BatchedMerge) bool {
return strings.EqualFold(s.Repository, m.Repository) && s.Branch == m.Branch && laterMerge(m, s)
})
shares := slices.ContainsFunc(announcedOf(m).Moves, func(n string) bool { return slices.Contains(withheld, n) })
if later || shares {
carried = append(carried, fmt.Sprintf("%s@%s (in %s)", m.Repository, m.Commit, p.ID))
}
}
}
sort.Strings(carried)
return carried, nil
}
// mergesLeftOutWords is the merges a stop left out, as a person reads them: each that contains a stopped one, or
// shares its modules, says so.
func mergesLeftOutWords(out []inventory.PlanLeftOut) string {
var words []string
for _, o := range out {
s := o.Repository + "@" + short(o.Commit)
switch {
case len(o.Shares) > 0:
s += " (which moves " + strings.Join(o.Shares, ", ") + " as " + short(o.Contains) + " does)"
case o.Contains != "":
s += " (which contains " + short(o.Contains) + ")"
}
words = append(words, s)
}
return strings.Join(words, ", ")
}
// orderMember is one member of a group, as mesh-delivery says it.
@@ -156,6 +580,9 @@ type orderMember struct {
CloneURL string `json:"clone_url,omitempty"`
// After are the repositories its pull request says it goes after (`after: <repository>`).
After []string `json:"after,omitempty"`
// BaseSHA is the base branch's head the forge last announced the head with (novox/hq ADR 0323): the trunk commit
// a verdict no build judges names when the head is checked again through mesh-delivery.
BaseSHA string `json:"base_sha,omitempty"`
}
// pull is the member as the forge announced it.
@@ -165,7 +592,7 @@ func (m orderMember) pull() link.PullUpdated {
if base == "" {
base = "main"
}
return link.PullUpdated{Owner: owner, Repo: repo, Number: m.Number, Base: base, Commit: m.Head,
return link.PullUpdated{Owner: owner, Repo: repo, Number: m.Number, Base: base, Commit: m.Head, BaseSHA: m.BaseSHA,
CloneURL: m.CloneURL, Paths: m.Paths, PathsTruncated: m.PathsTruncated, Removed: m.Removed,
ModuleDirs: m.ModuleDirs, ModuleDirsSaid: m.ModuleDirsSaid}
}
@@ -466,6 +893,8 @@ func deliveryCommand(ctx context.Context, args []string) error {
group := set.String("group", "", "a delivery group's id")
by := set.String("by", catalogue.DeliverySeat, "who says it")
why := set.String("why", "", "why")
without := set.String("without", "", "merges a stop leaves out, <repository>@<commit>, comma-separated: the walk's "+
"other merges are walked again without them (novox/hq ADR 0299)")
limit := set.Int("n", 50, "how many ended walks to answer beside the open ones")
planID := set.String("plan", "", "one walk")
positionals, err := parseAround(set, rest)
@@ -523,7 +952,7 @@ func deliveryCommand(ctx context.Context, args []string) error {
if *group == "" && len(members) == 1 {
// One head, checked again as the forge's announcement would have it (a recheck): the controller's
// own path for a pull request, run because the delivery's owner asked.
if err := sayingOnTheBus(ctx, func() error { return (following{open}).PullUpdated(ctx, members[0].pull()) }); err != nil {
if err := sayingOnTheBus(ctx, inv, func() error { return (following{open}).PullUpdated(ctx, members[0].pull()) }); err != nil {
return err
}
return answer(map[string]any{"rechecked": members[0].ID})
@@ -541,7 +970,7 @@ func deliveryCommand(ctx context.Context, args []string) error {
return errors.New("delivery go <plan> [--by <who>] [--why <text>]")
}
var p inventory.Plan
if err := sayingOnTheBus(ctx, func() (err error) {
if err := sayingOnTheBus(ctx, inv, func() (err error) {
p, err = letGo(ctx, inv, positionals[0], *by, strings.TrimSpace(*why))
return err
}); err != nil {
@@ -552,15 +981,21 @@ func deliveryCommand(ctx context.Context, args []string) error {
return nil
case "stop":
if len(positionals) != 1 || strings.TrimSpace(*why) == "" {
return errors.New("delivery stop <plan> --why <text> [--by <who>]")
return errors.New("delivery stop <plan> --why <text> [--by <who>] [--without <repository>@<commit>,…]")
}
var p inventory.Plan
if err := sayingOnTheBus(ctx, func() (err error) {
p, err = stopWalk(ctx, inv, positionals[0], *by, strings.TrimSpace(*why))
var said stopAnswer
if err := sayingOnTheBus(ctx, inv, func() (err error) {
p, said, err = stopWalkWithout(ctx, inv, positionals[0], *by, strings.TrimSpace(*why), splitList(*without),
time.Now().UTC())
return err
}); err != nil {
return err
}
if *without != "" {
// What it left out and what it walks again, as data: the delivery's owner reads it (ADR 0299).
return answer(said)
}
fmt.Printf("%s stopped by %s at tier %d of %d; what was asked still builds and registers, nothing further "+
"is asked or sent\n", p.ID, *by, p.Tier, len(p.Tiers))
return nil
@@ -596,12 +1031,81 @@ func deliveryCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
// Each walk's phases, with the rest's reports (novox/hq ADR 0282 decision 6).
now := time.Now()
for i := range walks {
walks[i].Phases = walks[i].WalkPhases(now, appliedFrom(ctx, inv))
}
return answer(map[string]any{"held": deliverySeatHeld(entries), "walks": walks,
"own-path": sortedKeysOf(ownPathWords())})
}
return fmt.Errorf("delivery %s: plan, order, check, go, stop or walks", sub)
}
// appliedFrom answers a machine's own report of the send a walk made to it from the controller's `apply` durations
// (novox/hq issue 582); a lookup that fails is a report not read, which leaves the walk's end unknown rather than wrong.
func appliedFrom(ctx context.Context, inv *inventory.Inventory) inventory.AppliedLookup {
return func(node string, send inventory.SentDeclaration, walkSent time.Time) (inventory.AppliedReport, bool) {
r, ok, err := inv.FirstAppliedAfter(ctx, node, send, walkSent)
if err != nil {
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, walkSent.Format(time.RFC3339), err)
return inventory.AppliedReport{}, false
}
return r, ok
}
}
// recordWalkPhases keeps, once, each phase of every walk ended lately whose end is known, as a duration of kind
// walk-phase per class (novox/hq ADR 0282 decision 6): what `durations` summarises. A walk whose end is unknown
// past ApplySilentAfter keeps its measured phases without its total.
func recordWalkPhases(ctx context.Context, inv *inventory.Inventory, now time.Time) error {
recent, err := inv.RecentPlans(ctx, 30)
if err != nil {
return err
}
for _, p := range recent {
if p.State != inventory.PlanDone || p.Release != nil || now.Sub(p.Updated) > 2*time.Hour {
continue
}
anyKept, totalKept, err := inv.WalkPhasesKept(ctx, p.ID)
if err != nil {
return err
}
if totalKept {
continue
}
ph := p.WalkPhases(now, appliedFrom(ctx, inv))
if ph != nil && ph.End == nil && anyKept {
continue // kept without its end; kept again only once its end is known
}
if ph == nil || (ph.End == nil && now.Sub(p.Updated) < inventory.ApplySilentAfter+time.Minute) {
continue
}
class := ph.Class
if class == "" {
class = "unclassed"
}
for _, x := range ph.Phases {
if x.State != inventory.PhaseMeasured || x.Start == nil {
continue
}
if err := inv.RecordDuration(ctx, inventory.Duration{Kind: inventory.DurationWalkPhase,
Subject: class + "/" + x.Name, Ref: fmt.Sprintf("%s/%s/%d", p.ID, x.Name, x.Tier), Started: *x.Start,
Took: time.Duration(x.TookMS) * time.Millisecond, Detail: p.Named()}); err != nil {
return err
}
}
if ph.End != nil && ph.From != nil {
if err := inv.RecordDuration(ctx, inventory.Duration{Kind: inventory.DurationWalkPhase,
Subject: class + "/total", Ref: p.ID + "/total", Started: *ph.From,
Took: time.Duration(ph.TotalMS) * time.Millisecond, Detail: ph.Said}); err != nil {
return err
}
}
}
return nil
}
// ownPathWords is the controller's own path as words, for an answer.
func ownPathWords() map[string]string { return onTheControllersPath }
@@ -684,7 +1188,7 @@ func askGroupCheck(ctx context.Context, open *stores, group string, members []or
return "", fmt.Errorf("%s cannot be cloned for the group's check: %w", m.ID, err)
}
request.Check.Members = append(request.Check.Members, link.GroupMember{Owner: mp.Owner, Repo: mp.Repo,
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths})
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths, Base: mp.Base})
}
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
@@ -723,12 +1227,19 @@ func shelfOf(entries []inventory.Entry) map[string]catalogue.Manifest {
// delivery it walks reads its steps from. Never in the way of the walk: said beside it, and a save that could
// not be said is logged — the delivery's owner, which also asks for the walks it follows, finds it by
// comparison. Records arriving out of order are told apart by the plan's revision.
func sayPlanMoved(ctx context.Context, bus link.Bus, p inventory.Plan) {
go publishPlanMoved(ctx, bus, p)
//
// On the save that starts the walk and the one that ends it, the walk also says what it touches, its bound and,
// at the end, its outcome (novox/hq ADR 0308 decision 1, walk_said.go), read from facts; nil facts say only what
// the walk recorded.
func sayPlanMoved(ctx context.Context, bus link.Bus, facts walkFacts, p inventory.Plan) {
go publishPlanMoved(ctx, bus, facts, p)
}
func publishPlanMoved(ctx context.Context, bus link.Bus, p inventory.Plan) {
body, err := json.Marshal(p)
func publishPlanMoved(ctx context.Context, bus link.Bus, facts walkFacts, p inventory.Plan) {
// Its phases so far (novox/hq ADR 0282 decision 6): the rest's reports come after the walk ends, and are
// read by whoever asks for the walk (`delivery walks`).
p.Phases = p.WalkPhases(time.Now(), nil)
body, err := json.Marshal(planMovedBody{Plan: p, walkSaid: sayWalk(ctx, facts, p)})
if err != nil {
return
}
@@ -743,7 +1254,7 @@ func publishPlanMoved(ctx context.Context, bus link.Bus, p inventory.Plan) {
// controller's, or a connection of the command's own — a verb runs as a command of its own, and what it kept
// would otherwise be said by nobody until the delivery's owner read the walks back. Without a bus the command
// still runs; what it did is found by comparison.
func sayingOnTheBus(ctx context.Context, f func() error) error {
func sayingOnTheBus(ctx context.Context, facts walkFacts, f func() error) error {
if checkEvents != nil {
return f()
}
@@ -754,9 +1265,10 @@ func sayingOnTheBus(ctx context.Context, f func() error) error {
return err
}
bus := link.OverNATS{Conn: conn, JS: js}
was := inventory.PlanSaved
checkEvents = bus
inventory.PlanSaved = func(p inventory.Plan) { publishPlanMoved(ctx, bus, p) }
defer func() { checkEvents, inventory.PlanSaved = nil, nil }()
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { publishPlanMoved(ctx, bus, facts, p) }
defer func() { checkEvents, inventory.PlanSaved = nil, was }()
ran = true
return f()
})
@@ -766,3 +1278,65 @@ func sayingOnTheBus(ctx context.Context, f func() error) error {
}
return err
}
// sayingWhatAVerbKept is PlanSaved for a process that is not the serving controller: each walk the verb keeps is
// said as `plan-moved` before the save returns, with the facts of the inventory that kept it, on the bus the verb
// reaches. It reaches the bus on the first save and never for a verb that keeps no walk. When the bus cannot be
// reached, a save that started or ended a walk says so on the verb's error output: its mark is kept and no later
// save says it, so the router holds what the walk touched until its bound, or not at all for a start. done lets
// the bus go and takes the hook back.
func sayingWhatAVerbKept(ctx context.Context) (said func(*inventory.Inventory, inventory.Plan), done func()) {
var (
mu sync.Mutex
reached bool
bus link.Bus
closing func()
failed error
)
said = func(inv *inventory.Inventory, p inventory.Plan) {
mu.Lock()
defer mu.Unlock()
if !reached {
reached = true
if handActConn != nil {
bus, failed = overNATS(handActConn)
} else if js, err := aBus(); err != nil {
failed = err
} else if bus, failed = overNATS(js.Conn()); failed == nil {
closing = js.Close
} else {
js.Close()
}
}
if bus == nil {
switch p.WalkSaid {
case inventory.WalkStarted:
fmt.Fprintf(os.Stderr, "%s: its start is kept and could NOT be said as plan-moved (%v): the router "+
"does not hold what it touches\n", p.ID, failed)
case inventory.WalkEnded:
fmt.Fprintf(os.Stderr, "%s: its end is kept and could NOT be said as plan-moved (%v): the router holds "+
"what it touched until its bound\n", p.ID, failed)
}
return
}
publishPlanMoved(ctx, bus, inv, p)
}
done = func() {
mu.Lock()
defer mu.Unlock()
inventory.PlanSaved = nil
if closing != nil {
closing()
}
}
return said, done
}
// overNATS is the bus a connection carries, with its JetStream.
func overNATS(conn *nats.Conn) (link.Bus, error) {
js, err := conn.JetStream()
if err != nil {
return nil, err
}
return link.OverNATS{Conn: conn, JS: js}, nil
}
+61 -4
View File
@@ -37,12 +37,41 @@ type stalledLine struct {
// Number is the pull request's, for a line of a head the forge never announced (novox/hq issue 347).
Number int `json:"number,omitempty"`
State string `json:"state"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
// Walk is the walk a line is about: a waiting walk no live delivery holds, or, asked with `walks`, each walk
// waiting for the owner's word and what holds it (novox/hq issues 362, 529).
Walk string `json:"walk,omitempty"`
// Waiting are the merge checks a line of the state `unanswered` waits on, as mesh-delivery says each:
// "mesh/merge-gate pending since <UTC time>", "mesh/repo-check never set" (novox/hq issue 438).
Waiting []string `json:"waiting,omitempty"`
// Checks are the same merge checks as data, which the controller words in the operator's own time (novox/hq
// issue 443): a time inside a finished sentence cannot be said again in another zone. A mesh-delivery from before
// says none, and Waiting is said as it reads.
Checks []waitingCheck `json:"checks,omitempty"`
// Held is the state of the delivery of a line of the state `unanswered`, ready or rejected, when one is held
// (novox/hq issues 441, 451): its check answered once, so a recheck is the step. Empty for a head with no
// delivery, and from a mesh-delivery that does not say it yet; the words are never read for it.
Held string `json:"held,omitempty"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
}
// waitingCheck is one merge check a stalled line waits on, as mesh-delivery gives it: its context, its state —
// "pending", or "never-set" — and, for a pending one, since when in RFC 3339, empty when the forge did not say.
type waitingCheck struct {
Context string `json:"context"`
State string `json:"state"`
Since string `json:"since,omitempty"`
}
// wordsNow is the time the words are said at, which says whether a time needs its day; a seam a test replaces.
var wordsNow = time.Now
// wordsZone is the zone a stalled line's times are said in: the controller's local zone, as every other time in its
// messages. A seam a test replaces, so that no test writes time.Local, which every goroutine of the package reads.
var wordsZone = func() *time.Location { return time.Local }
// operatorsOnly is whether the table leaves H2 nothing to do for the line: the state is the operator's.
func (l stalledLine) operatorsOnly() bool { return l.H2 == "" || strings.HasPrefix(l.H2, "none") }
@@ -103,6 +132,34 @@ func deliveriesStalled(ctx context.Context, conn *nats.Conn, held bool) ([]stall
return lines, nil
}
// walkWaits are, by walk, the lines mesh-delivery says of a walk waiting for its word: what holds it.
type walkWaits map[string][]stalledLine
// walksWaiting asks the delivery's owner what holds each walk waiting for its word (novox/hq issue 529): its
// `stalled` asked with `walks`, so a line of `plans` names a held delivery, its reason and the verb that releases
// it, instead of pointing at a hand start. Nil when no holder is on record or none answers: the line then reads as
// before. An owner from before this answers its stalled lines, of which only those naming a walk are kept.
func walksWaiting(ctx context.Context, conn *nats.Conn, held bool) walkWaits {
if !held {
return nil
}
raw, err := askDeliveryOwner(ctx, conn, "stalled", map[string]any{"walks": true})
if err != nil {
return nil
}
var lines []stalledLine
if json.Unmarshal(raw, &lines) != nil {
return nil
}
out := walkWaits{}
for _, l := range lines {
if l.Walk != "" {
out[l.Walk] = append(out[l.Walk], l)
}
}
return out
}
// stalledObservations are the conditions of what is stalled.
func stalledObservations(lines []stalledLine) []conditions.Observation {
out := make([]conditions.Observation, 0, len(lines))
@@ -119,6 +119,13 @@ func TestTheBusMachineIsBehindByItsUserListAlone(t *testing.T) {
if userListBehind("", "") {
t.Fatal("a machine sent no list reads as behind")
}
// A list that could not be composed is behind: not known is never "not behind" (the review of hq issue 353).
if !listBehind("", errors.New("the plan cannot be worked out"), digestOf([]byte(list))) {
t.Fatal("a list nobody could compose reads as current")
}
if listBehind(list, nil, digestOf([]byte(list))) || !listBehind(list, nil, "") {
t.Fatal("listBehind does not read the digest as userListBehind does")
}
}
// The machine holding the bus goes first: its declaration carries the user list the new grants are
+18 -3
View File
@@ -74,8 +74,12 @@ type probe struct {
// probeRegistry is the registry, in to-be 45's order. **The registry is the design's live form**: a
// probe added to a design is a row added here.
var probeRegistry = []probe{
{ID: "D1", Asserts: "every machine's declaration composes, and passes the node-engine's validation",
From: "issues 236, 263, 275", Kind: "declaration-refused", Raises: []string{kindAwaitingPush}, Phase: 1,
// D1 also says every module assigned and left out of a machine's declaration (novox/hq issue 380), from the
// resolution it already makes: needs-operator for a setting nobody gave, left-out for any other cause.
{ID: "D1", Asserts: "every machine's declaration composes, and passes the node-engine's validation; no module " +
"assigned to a machine is left out of its declaration unsaid",
From: "issues 236, 263, 275, 380", Kind: "declaration-refused",
Raises: []string{kindAwaitingPush, kindLeftOut, kindNeedsOperator}, Phase: 1,
run: probeDeclarations},
{ID: "D2", Asserts: "every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6",
From: "issue 262", Kind: "resolver-wrong", Phase: 1, run: probeResolvers},
@@ -116,6 +120,11 @@ var probeRegistry = []probe{
{ID: probeDeliveriesID, Asserts: "no delivery is held past its state's bound unsaid: mesh-delivery's " +
"`stalled`, each with the transition its table lets healer H2 take", From: "ADR 0239",
Kind: kindDeliveryStalled, Phase: 3, run: probeDeliveries},
// The delivery budgets (novox/hq ADR 0282 decision 7): the newest delivery of each class within its budget,
// read from mesh-delivery's `times`; a measurement said, never a delivery held.
{ID: probeBudgetsID, Asserts: "the newest delivery of a leaf module ran on every machine within three minutes of " +
"its merge, and of a core module within ten: mesh-delivery's `times`", From: "ADR 0282, ADR 0304",
Kind: kindOverBudget, Phase: 3, run: probeBudgets},
// A client of the bus reconnecting in a loop (novox/hq issue 327), from the server's record of closed
// connections, which the bus's own module reads.
{ID: probeReconnectsID, Asserts: "no user of the bus had its connection dropped more than twelve times in the " +
@@ -131,6 +140,12 @@ var probeRegistry = []probe{
{ID: "D-root", Asserts: "no agent can become root without a person on a machine where the router or a channel " +
"proving its sender runs: not by its own account, and not through a tool that runs its command as an account " +
"that can", From: "ADR 0259 §8", Kind: kindRootNotFree, Phase: 2, run: probeAgentRoot},
// A module's bucket or log filling toward its cap (novox/hq ADR 0297 §6): said at three quarters, cleared
// below seven tenths, so one at its threshold is not raised and cleared on every run. One that cannot be
// read is said on its own, and every other is still judged.
{ID: probeFillID, Asserts: "every module's bucket and log holds less than three quarters of its cap; one " +
"said filling is cleared once it holds less than seven tenths", From: "ADR 0297, issue 501",
Kind: kindBucketOrLogFilling, Raises: []string{kindBucketOrLogUnread}, Phase: 1, run: probeFill},
{ID: "DW", Asserts: "the watchdogs of the signals table ran within three of their intervals",
From: "ADR 0227 rule 6: the watchers are watched", Kind: "watchdogs-silent", Phase: 1, run: probeWatchdogs},
// The core's health definitions (novox/hq to-be 45 §8, ADR 0236): what a core component's new build is
@@ -636,7 +651,7 @@ func localDoctor(ctx context.Context) (*doctor, func(), error) {
open.Close()
return nil, nil, err
}
k, err := keeperOn(ctx, js.Conn())
k, err := keeperOn(ctx, js.Conn(), storeLooks{open.inventory})
if err != nil {
js.Close()
open.Close()
+42
View File
@@ -263,6 +263,37 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
f.Controller.Commit = c.Commit
}
// The list of predefined keys, and mesh and node config (novox/hq issue 630).
predefined, err := inv.Predefined(ctx)
if err != nil {
return snapshot.Facts{}, err
}
for key, k := range predefined {
if f.Predefined == nil {
f.Predefined = map[string]snapshot.PredefinedKey{}
}
f.Predefined[key] = snapshot.PredefinedKey{Type: k.Type, Why: k.Why}
}
config, err := inv.Config(ctx)
if err != nil {
return snapshot.Facts{}, err
}
nodeConfig := map[string]map[string]any{}
meshConfig := map[string]any{}
for _, c := range config {
if c.Node == "" {
meshConfig[c.Key] = c.Value
continue
}
if nodeConfig[c.Node] == nil {
nodeConfig[c.Node] = map[string]any{}
}
nodeConfig[c.Node][c.Key] = c.Value
}
if len(meshConfig) > 0 {
f.Config = scrub.Values(meshConfig)
}
gens, gensErr := generators(ctx, open)
hostShelf := shelf[hostModule]
meshWide := map[string]bool{}
@@ -280,6 +311,9 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
if n.HostVersion != "" {
engines[n.HostVersion] = true
}
if values := nodeConfig[n.Name]; len(values) > 0 {
m.Config = scrub.Values(values)
}
m.System = systemOf(hostShelf, n.HostVersion)
m.Libc = libcOf(m.System)
reported, err := inv.DescribedOf(ctx, n.Name)
@@ -345,6 +379,10 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
return snapshot.Facts{}, err
}
for _, layer := range layers {
if layer.Config {
// Mesh and node config are said once, beside the machine and the mesh (novox/hq issue 630).
continue
}
if layer.From == catalogue.MeshWideLayer {
if !meshWide[module] {
meshWide[module] = true
@@ -514,6 +552,10 @@ func composedAndValidated(ctx context.Context, open *stores, node string, gens m
if declared.Epoch, err = open.inventory.SentEpoch(ctx, record.ID); err != nil {
return sendable{}, nil, err
}
// And the generation it was last sent, as the would-send is (novox/hq issue 234).
if declared.Generation, err = open.inventory.SentGeneration(ctx, record.ID); err != nil {
return sendable{}, nil, err
}
body, err := declared.Body()
if err != nil {
return sendable{}, nil, err
+1 -1
View File
@@ -31,7 +31,7 @@ func aMeshWithSecrets(t *testing.T) (*stores, []string) {
}
}
secrets := []string{"Hunter2-Is-Not-A-Password-9f8e7d", "0123456789abcdefABCDEF0123456789zz"}
if err := open.inventory.SetSettings(ctx, "", "files", map[string]any{
if err := open.inventory.KeepSettings(ctx, "", "files", map[string]any{
"admin_password": secrets[0], "upstream": "10.77.0.9", "hub": "anchor"}); err != nil {
t.Fatal(err)
}
@@ -0,0 +1,66 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A provider whose wait fails the controller's check lists who waits on it (novox/hq issue 450).
//
// A wait that does not check out is judged unhealthy (ADR 0283 decision 3), so the provider raises its unhealthy
// condition and its consumers are held under it (ADR 0240 rule 5). What is stored is what the machine said, the
// wait unchecked: read as said, the provider seems to wait for the operator, and the held consumers were never
// listed on the condition that is open.
// refusedWait is a wait for a secret the database's manifest does not declare: it fails the check.
var refusedWait = inventory.Wait{Part: "postgres-database", Secret: "certificate", What: "the database's certificate"}
// judgedRefused is the provider's resource as the controller judges it: unhealthy, saying why.
func judgedRefused() inventory.ResourceHealth {
r := waitingDatabaseFor(refusedWait)
r.State, r.Waits = link.StateUnhealthy, nil
r.Reason = "says it waits for the secret certificate, which db does not declare"
return r
}
// The consumer's statement arrives after the provider's, so it is the consumer's judging (sayWaiters) that must list
// it at the provider's unhealthy condition, made urgent by who waits on it.
func TestAProviderWhoseWaitFailedItsCheckListsWhoWaitsOnIt(t *testing.T) {
k, _ := withConditionsInMemory(t)
stored := waitingDatabaseFor(refusedWait)
open := judgeBoth(t, k, []inventory.ResourceHealth{judgedRefused()},
map[string]inventory.NodeHealth{"anchor": {Node: "anchor", Resources: []inventory.ResourceHealth{stored}}},
shopFailingBeside(stored))
provider := conditionOf(open, moduleUnhealthyKey("db", "anchor"))
if len(open) != 1 || provider == nil {
t.Fatalf("a provider whose wait failed its check and a consumer of it raised %v; want the provider's "+
"unhealthy alone", openKeysOf(open))
}
if said := provider.Evidence[0].Said; !strings.Contains(said, "shop on laptop") {
t.Fatalf("the provider's unhealthy condition does not list shop on laptop as waiting on it: %s", said)
}
if provider.Severity != conditions.Urgent {
t.Fatalf("the provider's unhealthy condition is %s with a consumer waiting on it; want urgent", provider.Severity)
}
}
// A provider that waits for a secret already given is unhealthy to its consumers too, before its own condition opens:
// they are held under it as under any unhealthy provider, never as waiting for the operator, and its condition, when
// open, is said as unhealthy with who waits on it.
func TestAProviderWaitingForASecretAlreadyGivenIsUnhealthyToItsConsumers(t *testing.T) {
given := holdingOf(nil, shopFailingBeside(waitingDatabase()), shopOnTheDatabase)
given.waits.given["db@anchor"] = map[string]time.Time{"licence": time.Now().Add(-time.Hour)}
by, held := given.heldWith("laptop", "shop", []inventory.ResourceHealth{failingConsumer("shop")})
if !held || by.provider != theDatabase || len(by.waits) > 0 {
t.Fatalf("shop under a database waiting for a licence already given: held %v under %v with waits %v; want "+
"held under db on anchor as unhealthy, no waits", held, by.provider, by.waits)
}
if _, uncovered := given.waitingUncovered("laptop", "shop", []inventory.ResourceHealth{failingConsumer("shop")}); uncovered {
t.Fatalf("a provider whose wait failed its check is said as waiting for another part")
}
}
+54
View File
@@ -0,0 +1,54 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A change plan says its class** (novox/hq ADR 0313 §2): the issue tracker holds a fix to the budget of the
// slowest-budgeted module the fix moves, and a fix that moves no module has no fix budget. The plan says core,
// leaf or none, so mesh-delivery can pass it on with the fix's delivery.
func TestAChangePlanSaysTheClassOfWhatItMoves(t *testing.T) {
const repo = "http://forge.internal:20000/novox/mesh-catalog.git"
entry := func(name, path string) inventory.Entry {
e := fromRepo(name, repo, path)
e.Source.BuiltFrom = "old"
e.On = []string{"anchor"}
return e
}
gitea := entry("gitea", "modules/gitea")
resolver := entry("dnsmasq", "modules/dnsmasq")
resolver.Manifest.Claims = []catalogue.Claim{{Name: resolverSeat}}
delivery := entry(catalogue.DeliverySeat, "modules/mesh-delivery")
webApp := entry("photos", "modules/photos")
entries := []inventory.Entry{gitea, resolver, delivery, webApp}
// photos stands on gitea: a change to gitea moves photos as its dependent.
edges := []inventory.Edge{{From: "photos", To: "gitea", Kind: inventory.EdgeStandsOn}}
policy := func(string) (inventory.Upgrade, bool) { return inventory.Upgrade{RollOut: true}, true }
plan := func(paths ...string) link.ChangePlan {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "head", Paths: paths}
return changePlanOf("novox/mesh-catalog", "main", "head", reachOfMerge(m, entries, nil, edges), entries, policy)
}
for _, c := range []struct {
why string
paths []string
want string
}{
{"a leaf module and its dependent", []string{"modules/gitea/index.ts"}, inventory.ClassLeaf},
{"the delivery's owner, on the controller's own path", []string{"modules/mesh-delivery/cmd/x.go"}, inventory.ClassCore},
{"the mesh's resolver", []string{"modules/dnsmasq/module.json"}, inventory.ClassCore},
{"a leaf and a core module together: the slowest-budgeted", []string{"modules/gitea/index.ts", "modules/dnsmasq/module.json"}, inventory.ClassCore},
{"nothing any module's build reads", []string{"README.md"}, "none"},
{"a new leaf module alone", []string{"modules/weather/module.json"}, inventory.ClassLeaf},
{"a new module on the controller's own path, by its own class", []string{"next/mesh-delivery/module.json"}, inventory.ClassCore},
{"a new leaf module beside a core one moved", []string{"modules/weather/module.json", "modules/dnsmasq/module.json"}, inventory.ClassCore},
} {
if got := plan(c.paths...).Class; got != c.want {
t.Errorf("%s: class %q, want %q", c.why, got, c.want)
}
}
}
+148
View File
@@ -0,0 +1,148 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// A wait on a clock of its own: each pause moves it on, and nothing waits in real time.
func aFakeWait(bound time.Duration) (waitOut, *[]time.Duration) {
clock := time.Date(2026, 10, 11, 11, 52, 2, 0, time.UTC)
var paused []time.Duration
return waitOut{bound: bound, first: time.Second, most: 10 * time.Second,
now: func() time.Time { return clock },
sleep: func(_ context.Context, d time.Duration) error {
paused = append(paused, d)
clock = clock.Add(d)
return nil
}}, &paused
}
// notServed is what the bus says while node-tools re-serves, in the words seen on 2026-10-11.
func notServed() error {
return fmt.Errorf("cannot ask gitea.gitea_api: %w", link.ErrNothingServes)
}
// A registration's forge question that lands while the tool runner re-serves is asked again, and the forge's
// answer, seconds later, is the registration's (novox/hq issue 486).
func TestTheForgeNotServedForAMomentIsWaitedOut(t *testing.T) {
w, paused := aFakeWait(60 * time.Second)
asked := 0
facts, err := forgeWaitedOut(t.Context(), w, "novox/mesh-catalog", func(context.Context) (forgeFacts, error) {
asked++
if asked <= 2 {
return forgeFacts{}, notServed()
}
return forgeFacts{ID: 42, Guarded: true}, nil
})
if err != nil {
t.Fatalf("a forge not served for two asks failed the registration: %v", err)
}
if facts.ID != 42 || !facts.Guarded || asked != 3 {
t.Fatalf("the forge's answer was %+v after %d asks, want its answer on the third", facts, asked)
}
if want := []time.Duration{time.Second, 2 * time.Second}; fmt.Sprint(*paused) != fmt.Sprint(want) {
t.Fatalf("paused %v, want %v", *paused, want)
}
}
// A tool that did not answer within its wait has held the receive loop that long already: not asked again.
func TestTheForgeNotAnsweringInTimeIsNotAskedAgain(t *testing.T) {
w, paused := aFakeWait(60 * time.Second)
asked := 0
_, err := forgeWaitedOut(t.Context(), w, "novox/x", func(context.Context) (forgeFacts, error) {
asked++
return forgeFacts{}, fmt.Errorf("gitea did not answer: %w", link.ErrNoAnswer)
})
if !errors.Is(err, link.ErrNoAnswer) || asked != 1 || len(*paused) != 0 {
t.Fatalf("a forge that did not answer in time was asked %d times, paused %v: %v", asked, *paused, err)
}
}
// An ask that hangs past the bound is cut off at the bound, not at its own wait, and the failure still says it
// waited: the receive loop is held no longer than the bound (novox/hq issue 486).
func TestAnAskHangingPastTheBoundIsCutOffAtIt(t *testing.T) {
w := waitOut{bound: 200 * time.Millisecond, first: 10 * time.Millisecond, most: 50 * time.Millisecond,
now: time.Now, sleep: sleepOrDone}
asked := 0
began := time.Now()
_, err := forgeWaitedOut(t.Context(), w, "novox/mesh-catalog", func(ctx context.Context) (forgeFacts, error) {
asked++
if asked == 1 {
return forgeFacts{}, notServed()
}
select { // link.Ask's own wait is 30 s; the bound must end it first
case <-ctx.Done():
return forgeFacts{}, fmt.Errorf("gitea did not answer: %w", link.ErrNoAnswer)
case <-time.After(30 * time.Second):
return forgeFacts{ID: 1, Guarded: true}, nil
}
})
if took := time.Since(began); took > 2*time.Second {
t.Fatalf("the wait took %s, past its bound of %s", took, w.bound)
}
if err == nil || !strings.Contains(err.Error(), "waited") || !strings.Contains(err.Error(), "asking 2 times") ||
!strings.Contains(err.Error(), "issue 486") {
t.Fatalf("the bound's end does not say it waited: %v", err)
}
}
// Past the bound the registration fails, saying it waited, for how long, and what the forge last said.
func TestTheForgeNotServedPastTheBoundFailsSayingItWaited(t *testing.T) {
w, paused := aFakeWait(60 * time.Second)
_, err := forgeWaitedOut(t.Context(), w, "novox/mesh-catalog", func(context.Context) (forgeFacts, error) {
return forgeFacts{}, notServed()
})
if err == nil {
t.Fatal("a forge never served was read as an answer")
}
var waited time.Duration
for _, p := range *paused {
waited += p
}
if waited > 60*time.Second || waited < 40*time.Second {
t.Fatalf("waited %s in all, want close to and within the bound of 60s", waited)
}
for _, want := range []string{"waited " + waited.String(), "novox/mesh-catalog", "nothing serves", "issue 486"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the failure does not say %q: %v", want, err)
}
}
if !errors.Is(err, link.ErrNothingServes) {
t.Errorf("the failure lost what the forge last said: %v", err)
}
}
// A refusal is the forge's word: never asked again.
func TestARefusalFromTheForgeIsNeverAskedAgain(t *testing.T) {
w, paused := aFakeWait(60 * time.Second)
asked := 0
_, err := forgeWaitedOut(t.Context(), w, "novox/x", func(context.Context) (forgeFacts, error) {
asked++
return forgeFacts{}, errors.New("gitea.gitea_api: 404 Not Found")
})
if err == nil || asked != 1 || len(*paused) != 0 {
t.Fatalf("a refusal was asked %d times, paused %v: %v", asked, *paused, err)
}
}
// A registration whose own context ends stops waiting at once.
func TestAWaitForTheForgeEndsWithItsContext(t *testing.T) {
w, _ := aFakeWait(60 * time.Second)
ctx, cancel := context.WithCancel(t.Context())
w.sleep = func(ctx context.Context, d time.Duration) error { cancel(); return sleepOrDone(ctx, d) }
asked := 0
_, err := forgeWaitedOut(ctx, w, "novox/x", func(context.Context) (forgeFacts, error) {
asked++
return forgeFacts{}, notServed()
})
if !errors.Is(err, context.Canceled) || asked != 1 {
t.Fatalf("asked %d times after the context ended: %v", asked, err)
}
}
+147 -22
View File
@@ -27,8 +27,10 @@ import (
// **"Reported applied" is not enough.** ADR 0218 sent a module to one machine first and the rest once
// that machine reported it applied. A build that applies and then does nothing, crashes, serves no
// tools, or breaks the machine's word to the mesh passed that test. Now the first machine is judged by
// the component's health — the core's health definitions, or a module's own — passing three times over
// at least two minutes, within ten minutes of the apply. Only then are the rest sent.
// the component's health — the core's health definitions, or a module's own — within ten minutes of the
// apply: a core module passing three times over at least two minutes, a leaf module once, as soon as the
// node-engine states it healthy (novox/hq ADR 0282 decision 4 as ADR 0304 decision 3 changed it). Only then
// are the rest sent.
//
// **A failing gate stops the plan and puts the previous build back there.** The build is marked failed
// at its gate — registration refuses it and nothing sends it again on its own — the module's registered
@@ -43,11 +45,38 @@ var (
gateSettle = 2 * time.Minute
// gateBound is how long after the apply the build may take to become healthy.
gateBound = 10 * time.Minute
// gatePasses is how many consecutive judgings must find it healthy, gateEvery apart at the least.
gatePasses = 3
gateEvery = 40 * time.Second
// gatePasses is how many consecutive judgings must find a core module healthy, gateEvery apart at the
// least, after gateSettle. gateLeafPasses is a leaf module's: one, taken once the node-engine states every
// resource of it healthy by its own check, with no settle (novox/hq ADR 0282 decision 4 as ADR 0304
// decision 3 changed it, issue 524).
gatePasses = 3
gateLeafPasses = 1
gateLeafSettles = false
gateEvery = 40 * time.Second
)
// coreModule is whether a module is judged as the core (novox/hq ADR 0282 decision 1): one on the controller's
// own path, or one holding the mesh's resolver. Every other module is a leaf.
func coreModule(module string, m catalogue.Manifest) bool {
if _, own := onTheControllersPath[module]; own {
return true
}
for _, c := range m.Claims {
if c.Name == resolverSeat {
return true
}
}
return coreComponent(module) != ""
}
// gateWants is how many passes a module needs, and whether it must stay healthy gateSettle first.
func gateWants(module string, m catalogue.Manifest) (passes int, settles bool) {
if coreModule(module, m) {
return gatePasses, true
}
return gateLeafPasses, gateLeafSettles
}
// gateProbe is the registry's row for the gate's verdicts and the witnesses' rollbacks: its conditions
// are raised by a plan as it judges, and kept or cleared by the probe on every run.
const gateProbe = "DG"
@@ -121,10 +150,13 @@ type gateFacts struct {
health map[string]inventory.NodeHealth
healthErr error
// heldOn is, per "<module>@<machine>", the provider its findings are held under (ADR 0240 rule 5).
heldOn map[string]string
heldOn map[string]heldReading
// groupsAdded is, per module, whether the move judged puts an account in a group its previous build did
// not (issue 318 review): the only move whose wait for a new login is excused.
groupsAdded map[string]bool
// waits is what the controller holds to check a module's wait for the operator (novox/hq ADR 0283): the
// manifest of each module's build judged, and the secrets given on each machine.
waits operatorWaitFacts
// sent is, per machine, the declaration the gate's own send carried there (novox/hq issue 352): a
// report is held against it, never against the send made last. sentBuilds is what each machine was
// last sent of every module, and judged the commit of each module this gate judges: a machine last
@@ -134,6 +166,29 @@ type gateFacts struct {
commits map[string]string
}
// heldReading is the provider a module's findings are held under, as "<module> on <machine>", and, when that
// provider only waits for the operator for the part the module needs, its wait in one sentence (novox/hq issue
// 405): the module's gate then reads as a wait for a person (ADR 0254), a pass carrying the wait, as ADR 0283
// decision 4 reads the waiting provider itself. A walk never waits on the operator's secret, there or here.
type heldReading struct {
on, waits string
}
// heldReadings is, per "<module>@<machine>" in every machine's newest statement, what its findings are held under.
func heldReadings(hold *holding) map[string]heldReading {
out := map[string]heldReading{}
for machine := range hold.healths {
for module, by := range hold.heldModules(machine) {
reading := heldReading{on: by.provider.Module + " on " + by.provider.Node}
if len(by.waits) > 0 {
reading.waits = operatorWaitSaid(by.provider.Module, by.provider.Node, by.waits)
}
out[module+"@"+machine] = reading
}
}
return out
}
// reportedOn says a machine's last report is on what the gate sent it (novox/hq issue 352): on that
// declaration, or one it was sent after it — or, for a gate kept before sends were kept on it, on the
// declaration last sent. On 2026-10-09 a release's gate read the control node's report against a newer
@@ -199,12 +254,7 @@ var gatherGateFacts = func(ctx context.Context, open *stores, component string)
// Whose findings wait on an unhealthy provider (ADR 0240 rule 5): their gates wait, not fail.
if f.healthErr == nil && f.openErr == nil {
if hold, err := readHolding(ctx, inv, f.open); err == nil {
f.heldOn = map[string]string{}
for machine := range f.health {
for module, p := range hold.heldModules(machine) {
f.heldOn[module+"@"+machine] = p.Module + " on " + p.Node
}
}
f.heldOn = heldReadings(hold)
}
}
if theLease != nil {
@@ -257,10 +307,11 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
firstLine(f.openErr.Error())
}
for _, c := range f.open {
// A wait for a person's new login, or for a directory used as found to be handed over, is the module's
// reading, not a fault raised since the send: the gate reads it from the statement below (ADR 0254,
// novox/hq issue 339).
if c.Source == gateProbe || c.OpenAt(since) || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound {
// A wait for a person's new login, for a directory used as found to be handed over, or for the
// operator's secret or setting, is the module's reading, not a fault raised since the send: the gate
// reads it from the statement below (ADR 0254, novox/hq issue 339, ADR 0283).
if c.Source == gateProbe || c.OpenAt(since) || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound ||
c.Kind == kindNeedsOperator {
continue
}
onIt := c.Subject.Machine == machine || slices.Contains(c.Subject.Also, machine) ||
@@ -477,6 +528,7 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
return "", err
}
facts.groupsAdded = movesAddingGroups(ctx, open.inventory, g, pairs, shelf)
facts.waits = gateWaitFacts(ctx, open.inventory, g, pairs, shelf, facts.health)
facts.sent = g.Sent
facts.commits, facts.sentBuilds = judgedCommits(g, pairs), map[string]map[string]string{}
// A module this gate put back at once (putBackBroken) was sent its earlier build by the gate itself:
@@ -585,8 +637,24 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
}
}
settled := now.Sub(*g.Since) >= gateSettle
// What the send needs is what its most demanding module needs: a send carrying a core module is judged
// as the core.
wants, settles := 0, false
for _, m := range modules {
p, s := gateWants(m, shelf[m])
wants, settles = max(wants, p), settles || s
}
if len(modules) == 0 {
// A send judged on no machine of its modules (its first machine not running them) has no class to
// be judged by: it is judged as the core, never passed at once without a reading.
wants, settles = gatePasses, true
}
g.Wants = wants
// A module's own pass is its own class's: a leaf beside a core module that fails keeps its build after
// its one reading, and only what did not pass is put back (issue 318's rule, by class since issue 524).
passedAlone := func(m string) bool {
return settled && reading[m] <= healthPerson && g.Healthy[m] >= gatePasses
p, s := gateWants(m, shelf[m])
return (settled || !s) && reading[m] <= healthPerson && g.Healthy[m] >= p
}
// fail decides the send failed: what passed on its own keeps its pass, everything else is put back.
fail := func(why string) {
@@ -603,6 +671,7 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
pastBound := now.Sub(*g.Since) > gateBound
switch {
case worst == healthBroken:
g.Read(now, false, g.BrokenWhy)
var judging []string
for _, m := range modules {
if reading[m] != healthBroken && !passedAlone(m) {
@@ -621,8 +690,10 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
// Waiting on a provider that is unhealthy: not a pass, and not a failure at the bound either —
// the provider's own condition says what is wrong (ADR 0240 rule 5).
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
g.Read(now, false, why)
case worst == healthNotYet:
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
g.Read(now, false, why)
if pastBound {
fail(fmt.Sprintf("not healthy within %s of its apply: %s", gateBound, why))
}
@@ -630,7 +701,8 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
// Healthy, or waiting for a person (ADR 0254): a pass, the wait carried along in the verdict.
g.Passes++
g.LastPass, g.Last, g.Failing = &now, "", nil
if g.Passes >= gatePasses && settled {
g.Read(now, true, "")
if g.Passes >= wants && (settled || !settles) {
decide(g, inventory.GatePassed, fmt.Sprintf("healthy %d times over %s", g.Passes,
now.Sub(*g.Since).Round(time.Second))+waitsSaid(g.Waits), now)
}
@@ -747,6 +819,47 @@ func movesAddingGroups(ctx context.Context, inv *inventory.Inventory, g *invento
return out
}
// gateWaitFacts reads what checks the waits for the operator of the modules a gate judges (novox/hq ADR 0283): the
// manifest of the build judged — the one it moves to, else the catalogue's — and the secrets given on each machine
// that says a module of them waits.
func gateWaitFacts(ctx context.Context, inv *inventory.Inventory, g *inventory.PlanGate, pairs []judged,
shelf map[string]catalogue.Manifest, health map[string]inventory.NodeHealth) operatorWaitFacts {
var f operatorWaitFacts
judgedManifests := map[string]catalogue.Manifest{}
byMachine := map[string][]string{}
for _, j := range pairs {
waiting := false
for _, r := range health[j.node].Resources {
if r.Module == j.module && r.State == link.StateWaiting {
waiting = true
}
}
if !waiting {
continue
}
byMachine[j.node] = append(byMachine[j.node], j.module)
if _, done := judgedManifests[j.module]; done {
continue
}
to := g.To
for _, c := range g.Carried {
if c.Module == j.module {
to = c.To
break
}
}
if m, found, err := inv.ManifestAt(ctx, j.module, to); err == nil && found {
judgedManifests[j.module] = m
} else if m, known := shelf[j.module]; known {
judgedManifests[j.module] = m
}
}
for machine, modules := range byMachine {
readWaitFacts(ctx, inv, machine, modules, judgedManifests, &f)
}
return f
}
// decide sets a gate's verdict.
func decide(g *inventory.PlanGate, verdict, why string, now time.Time) {
g.Verdict, g.Why, g.JudgedAt = verdict, why, &now
@@ -926,7 +1039,7 @@ func gateFailed(ctx context.Context, open *stores, p *inventory.Plan, module str
}
return
}
sent, err := sendRollout(withScope(ctx, sendScope{modules: map[string]bool{module: true}}), open, g.Machines)
sent, err := sendRollout(withScope(ctx, sendScope{modules: map[string]bool{module: true}, puttingBack: true}), open, g.Machines)
if err != nil {
notBack(fmt.Sprintf("its registered build is back at %s, and sending it to %s was refused: %v — `push %s` "+
"sends it", short(previous.Commit), strings.Join(g.Machines, ", "), err, g.Machines[0]))
@@ -1019,7 +1132,7 @@ func sendRollbacks(ctx context.Context, open *stores, p *inventory.Plan, b *roll
}
inv := open.inventory
sort.Strings(b.machines)
sent, err := sendRollout(withScope(ctx, sendScope{modules: b.modules}), open, b.machines)
sent, err := sendRollout(withScope(ctx, sendScope{modules: b.modules, puttingBack: true}), open, b.machines)
var back []string
for _, r := range b.pending {
if err != nil {
@@ -1272,9 +1385,9 @@ func gateLine(g *inventory.PlanGate) string {
if g.Why != "" {
line += ": " + g.Why
} else if g.Last != "" {
line += fmt.Sprintf(" (%d of %d passes; wanting: %s)", g.Passes, gatePasses, g.Last)
line += fmt.Sprintf(" (%d of %d passes; wanting: %s)", g.Passes, gateWanted(g), g.Last)
} else if g.Verdict == "" {
line += fmt.Sprintf(" (%d of %d passes)", g.Passes, gatePasses)
line += fmt.Sprintf(" (%d of %d passes)", g.Passes, gateWanted(g))
}
if g.Verdict == "" && len(g.Waits) > 0 {
line += waitsSaid(g.Waits)
@@ -1285,6 +1398,10 @@ func gateLine(g *inventory.PlanGate) string {
if g.Rollback != "" {
line += "; " + g.Rollback
}
// Before the send it judges (novox/hq issue 490).
if said := grantsSaid(g.Grants); said != "" {
line += "; " + said
}
return line
}
@@ -1313,3 +1430,11 @@ func witnessObservation(node string, r lease.Rollback) conditions.Observation {
Summary: fmt.Sprintf("the witness on %s judged the %s %s not healthy %s at %s: %s", node, r.Component,
short(r.From), what, r.At.UTC().Format(time.RFC3339), r.Why)}
}
// gateWanted is the passes a gate's send needs, three where the gate was judged before it was said.
func gateWanted(g *inventory.PlanGate) int {
if g.Wants > 0 {
return g.Wants
}
return gatePasses
}
@@ -53,6 +53,7 @@ func healthyContainer(module string) inventory.ResourceHealth {
// tools' witness put its build back at once, and the send waits for the healthy module's own verdict — a pass,
// counted over the gate's spacing — before it says its own. The broken one alone is put back.
func TestAFaultBeforeTheSettleTimeWaitsForTheModuleBesideIt(t *testing.T) {
judgeLeavesAsTheCore(t)
open := aMesh(t)
ctx := t.Context()
withGateBounds(t, 150*time.Millisecond, 20*time.Millisecond, 10*time.Second)
@@ -284,6 +285,7 @@ func TestAProviderWaitingForALoginSaysWhoWaitsOnIt(t *testing.T) {
// tools are marked and put back in the very judging that found them broken — their registered build is the
// one before, and the laptop is sent it — while `app` goes on being judged, recovers, and keeps its own pass.
func TestABrokenModuleIsPutBackAtOnceAndTheOneBesideItGetsItsOwnVerdict(t *testing.T) {
judgeLeavesAsTheCore(t)
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
+56 -1
View File
@@ -153,6 +153,7 @@ func (g *gateMesh) plan(t *testing.T) inventory.Plan {
// registered and sent to it again — and never reaches the second machine; it is marked, said as a
// condition and an event, never registered or rolled back again.
func TestABuildThatFailsItsGateIsRolledBackOnItsFirstMachineAndGoesNoFurther(t *testing.T) {
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
@@ -255,7 +256,7 @@ func TestABuildThatPassesItsGateRollsEverywhereUnattended(t *testing.T) {
t.Fatalf("sent %v", g.sent)
}
gate := p.Modules["app"].Gate
if p.State != inventory.PlanDone || gate == nil || gate.Verdict != inventory.GatePassed || gate.Passes < gatePasses {
if p.State != inventory.PlanDone || gate == nil || gate.Verdict != inventory.GatePassed || gate.Passes < gateWanted(gate) {
t.Fatalf("the plan is %s (%s), its gate %+v", p.State, p.Note, gate)
}
if v, found, err := g.open.inventory.GateOf(ctx, "build-2"); err != nil || !found || v.Verdict != inventory.GatePassed {
@@ -266,8 +267,52 @@ func TestABuildThatPassesItsGateRollsEverywhereUnattended(t *testing.T) {
}
}
// A leaf module passes its gate on one reading, as soon as its first machine is healthy, without waiting out the
// core's settle time (novox/hq ADR 0282 decision 4 as ADR 0304 decision 3 changed it, issue 524).
func TestALeafModulePassesOnOneHealthyReading(t *testing.T) {
g := aGateMesh(t)
ctx := t.Context()
gateSettle = time.Hour // a core module could not pass for an hour
advancePlans(ctx, g.open) // the first machine is sent the new build
advancePlans(ctx, g.open) // one healthy judging
p := g.plan(t)
gate := p.Modules["app"].Gate
if gate == nil || gate.Verdict != inventory.GatePassed || gate.Passes != 1 || gate.Wants != 1 {
t.Fatalf("a leaf module's gate after one healthy judging: %+v (plan %s: %s)", gate, p.State, p.Note)
}
if !strings.Contains(gate.Why, "healthy 1 times") {
t.Errorf("the verdict says %q", gate.Why)
}
}
// Which modules are judged as the core: those on the controller's own path and the resolver's holder, three
// readings after the settle time; every other module one (novox/hq ADR 0282 decision 1, ADR 0304 decision 3).
func TestAModuleIsJudgedByItsClass(t *testing.T) {
for _, c := range []struct {
module string
claims []catalogue.Claim
passes int
settles bool
}{
{"app", nil, 1, false},
{"nzbget", nil, 1, false},
{"mesh-controller", nil, 3, true},
{"mesh-host", nil, 3, true},
{"node-tools", nil, 3, true},
{"nats", nil, 3, true},
{catalogue.DeliverySeat, nil, 3, true},
{"dnsmasq", []catalogue.Claim{{Name: resolverSeat}}, 3, true},
} {
p, s := gateWants(c.module, catalogue.Manifest{Module: c.module, Claims: c.claims})
if p != c.passes || s != c.settles {
t.Errorf("%s: %d passes, settles %v; want %d, %v", c.module, p, s, c.passes, c.settles)
}
}
}
// A first machine that never becomes healthy fails its gate at the bound, and is put back.
func TestABuildNeverHealthyFailsAtTheBound(t *testing.T) {
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
advancePlans(ctx, g.open)
@@ -593,3 +638,13 @@ func TestAWitnessesVerdictIsAConditionWhileItsReportsSayIt(t *testing.T) {
t.Fatalf("a verdict the reports no longer carry is still said: %+v %v", obs, err)
}
}
// judgeLeavesAsTheCore makes a test's leaf modules judged as a core module is, three readings after the settle
// time, for a test of the gate's other rules (a fault, the bound, a supersession) that needs a judging still
// open after a healthy reading.
func judgeLeavesAsTheCore(t *testing.T) {
t.Helper()
wasPasses, wasSettles := gateLeafPasses, gateLeafSettles
gateLeafPasses, gateLeafSettles = gatePasses, true
t.Cleanup(func() { gateLeafPasses, gateLeafSettles = wasPasses, wasSettles })
}
+300
View File
@@ -0,0 +1,300 @@
package main
import (
"context"
"fmt"
"io"
"os"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The assignment generation a declaration was composed from, and the record of every send (novox/hq
// issue 234).
//
// On 2026-10-04 a declaration newer in sequence than every other named four fewer modules than the
// assignments held, and a machine applied it and undeclared all four. The sequence orders arrival and
// cannot tell a later send that carries an older view of the assignments. So a declaration now carries
// the generation of the assignments it was composed from — a counter the store raises in the same
// transaction as every assignment change — and a machine refuses one older than it applied, unless it is
// a gate's put-back. And every send is written down with who sent it, from which generation and naming
// which modules: the controller logged no send then, and which process sent the stale declaration could
// not be read back from anything the mesh kept.
// kindOlderGeneration is S20's kind: a machine refused a send for the generation it was composed from.
const kindOlderGeneration = "older-generation"
// generationRefusalsSaid is how long a refused send is said after its refusal: an hour, as an advisory is.
const generationRefusalsSaid = advisoryQuiet
// stamp gives a composed declaration the order allotted to it before it was composed: its sequence, and
// the epoch and generation when the machine reads them — and keeps the generation and acting epoch for the
// record of the send whether or not the machine is sent them.
//
// **No put-back mark is sent.** A gate puts a machine back by composing it again (sendRollout), so its
// declaration is allotted here like any other and carries the generation as it stands — never older than
// what the machine applied. The node-engine reads a `put_back` key (mesh-host#82); this controller never
// needs to send one.
func (o order) stamp(d *sendable) {
d.Sequence, d.Epoch = o.sequence, o.epoch
d.composedFrom, d.actingEpoch = o.generation, o.acting
d.Generation = 0
if o.readsGeneration && o.generation > 0 {
d.Generation = o.generation
}
}
// sentRecord is what the record of a send keeps beside its digest.
type sentRecord struct {
sequence int64
epoch uint64
generation int64
// toldGeneration is the generation on the wire, which the would-send is stamped with: zero for a machine
// whose node-engine has not said it reads one.
toldGeneration int64
sender string
modules []string
}
// sentRecordOf is a composed send's record: its sender is the caller this process acts for.
func sentRecordOf(ctx context.Context, d sendable) sentRecord {
return sentRecord{sequence: d.Sequence, epoch: d.actingEpoch, generation: d.composedFrom,
toldGeneration: d.Generation, sender: senderOf(callerOf(ctx)), modules: d.modules}
}
// callerOf is who asked for what this process does: the seat call's caller when ctx belongs to one, the
// caller the controller ran this command for, or the account at the shell.
func callerOf(ctx context.Context) string {
if c := link.CallerIn(ctx); c != "" {
return c
}
return link.Caller()
}
// senderOf is a send's sender in words: who asked, and the process and build that composed it — the
// answer issue 234 could not find, because two controllers and a one-shot push were all sending then.
func senderOf(caller string) string {
host, _ := os.Hostname()
return fmt.Sprintf("%s (pid %d on %s, build %s)", caller, os.Getpid(), host, version)
}
// namedModules are the modules a declaration names: its machine's set, less what was left out of it.
func namedModules(plan catalogue.Resolution, leftOut map[string]string) []string {
out := make([]string, 0, len(plan.Modules))
for _, m := range plan.Modules {
if _, left := leftOut[m.Module]; !left {
out = append(out, m.Module)
}
}
sort.Strings(out)
return out
}
// heardGenerationRefusal keeps a machine's refusal of a send for its generation, so S20 names its sender
// and says it (novox/hq issue 234): on the send it refused or — when the mesh has no record of sending that
// sequence — as a refusal of its own, naming the sender as unknown. Either way S20 is raised: a refusal the
// mesh cannot attribute is the louder fact, not a quieter one.
//
// **And raises the mesh's counter past what the machine applied, when the refused send was composed from
// the counter as it stands** (counterBehind): then the sender's view was not stale — the counter is behind
// the machine, which is a store put back from a backup — and every send after would be refused for ever. A
// stale sender's generation is below the counter, and the counter is left alone for it.
//
// Nothing is sent from here. This runs in the controller's receive loop, and a push from it would hold that
// loop, and the machine's hold, for as long as the push takes — the deaf controller of issues 184 and 185.
// S20 names `push <node>` for that case instead.
func heardGenerationRefusal(ctx context.Context, inv *inventory.Inventory, report link.Report) {
r := report.OlderGeneration
if r == nil || inv == nil || report.Node == "" {
return
}
node, err := inv.NodeByName(ctx, report.Node)
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-controller: %s refused a send for its generation, and the machine cannot be "+
"read, so it is not raised: %v\n", report.Node, err)
return
}
var raised int64
if now, err := inv.AssignmentGeneration(ctx); err != nil {
fmt.Fprintf(os.Stderr, "mesh-controller: %s refused a send for its generation, and the mesh's own cannot be "+
"read: %v\n", report.Node, err)
} else if counterBehind(*r, now) {
if raised, err = inv.RaiseAssignmentGeneration(ctx, r.Applied); err != nil {
fmt.Fprintf(os.Stderr, "mesh-controller: the assignment generation (%d) is behind what %s applied (%d), "+
"and could not be raised: %v\n", now, report.Node, r.Applied, err)
raised = 0
} else {
fmt.Fprintf(os.Stderr, "mesh-controller: the assignment generation was %d, behind what %s applied (%d) — "+
"a store put back from a backup — and is raised to %d; `push %s` sends it what the mesh holds now\n",
now, report.Node, r.Applied, raised, report.Node)
}
}
send, found, err := inv.RefusedSend(ctx, node.ID, report.Sequence, r.Applied, raised)
if err == nil && !found {
send, err = inv.RecordUnrecordedRefusal(ctx, inventory.Send{Node: node.ID, Sequence: report.Sequence,
Epoch: report.Epoch, Generation: r.Generation, Digest: report.Declared, RefusedApplied: r.Applied,
CounterRaisedTo: raised})
}
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-controller: %s refused send %d for its generation, and the refusal could not be "+
"kept, so it is not raised: %v\n", report.Node, report.Sequence, err)
return
}
fmt.Fprintf(os.Stderr, "mesh-controller: %s refused send %d from %s: composed from assignment generation %d, "+
"and it applied %d\n", report.Node, report.Sequence, send.Sender, r.Generation, r.Applied)
}
// counterBehind says a refusal shows the mesh's counter behind the machine rather than a stale sender: the
// refused send carried the counter as it stands now, and the machine applied more than that.
func counterBehind(r link.GenerationRefusal, now int64) bool {
return r.Generation >= now && r.Applied > r.Generation
}
// watchGenerationRefusals is S20: every send a machine refused within the hour for the generation it was
// composed from, naming who sent it (novox/hq issue 234). One per machine, the newest refusal.
func watchGenerationRefusals(f *signalFacts) []conditions.Observation {
seen := map[string]bool{}
var out []conditions.Observation
for _, s := range f.refusedSends {
if s.RefusedAt == nil || f.now.Sub(*s.RefusedAt) > generationRefusalsSaid || seen[s.NodeName] {
continue
}
seen[s.NodeName] = true
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: s.NodeName, Kind: kindOlderGeneration,
Machine: s.NodeName, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s refused sequence %d from %s: it was composed from assignment generation %d, and "+
"%s applied generation %d — a sender composing from a view of the assignments the mesh has moved "+
"past; it named %s", s.NodeName, s.Sequence, s.Sender, s.Generation, s.NodeName, s.RefusedApplied,
modulesWords(s.Modules)),
Said: fmt.Sprintf("refused at %s", s.RefusedAt.UTC().Format(time.RFC3339)),
Headline: fmt.Sprintf("%s refused an out-of-date update", s.NodeName),
Explanation: fmt.Sprintf("Something sent %s an update made from an older list of what runs there. %s "+
"refused it, so nothing was removed. Nothing for you to do unless it repeats.", s.NodeName, s.NodeName),
Resolved: fmt.Sprintf("%s has had no out-of-date update for an hour", s.NodeName)}
if s.CounterRaisedTo > 0 {
// Not a stale sender: the mesh's counter was behind the machine (a store put back from a backup) and
// was raised; nothing has sent the machine its declaration since, so a person is asked to.
o.Summary = fmt.Sprintf("%s refused sequence %d from %s: it was composed from assignment generation %d, "+
"the mesh's own, and %s applied generation %d — the mesh's counter was behind the machine (a store "+
"put back from a backup?) and is raised to %d; `push %s` sends it what the mesh holds now",
s.NodeName, s.Sequence, s.Sender, s.Generation, s.NodeName, s.RefusedApplied, s.CounterRaisedTo,
s.NodeName)
o.Explanation = fmt.Sprintf("Needs you: push %s. The mesh's record was older than %s, so %s refused its "+
"update and kept what it had. The record is repaired; a push sends the update again.",
s.NodeName, s.NodeName, s.NodeName)
}
out = append(out, o)
}
return out
}
// modulesWords is a send's modules as a sentence says them.
func modulesWords(modules []string) string {
if len(modules) == 0 {
return "no module the mesh recorded"
}
return strings.Join(modules, ", ")
}
// writeLastSend says what a machine was last told, by whom and from which generation (novox/hq issue 234), and
// what it answered (issue 485): nothing when the mesh has not recorded a send to it.
func writeLastSend(ctx context.Context, w io.Writer, inv *inventory.Inventory, node string) error {
s, found, err := inv.LastSend(ctx, node)
if err != nil || !found {
return err
}
fmt.Fprintf(w, "%s:\n", node)
for _, line := range lastSendLines(s) {
fmt.Fprintln(w, line)
}
return nil
}
// lastSendLines is a machine's last send in full, as `node show` and `plan` say it (novox/hq issue 485): when,
// by whom, under which lease epoch, from which assignment generation, naming which modules, and what the
// machine answered of it — the refusal in its own words.
func lastSendLines(s inventory.Send) []string {
const stamp = "2006-01-02 15:04:05"
epoch := "under no lease epoch"
if s.Epoch > 0 {
epoch = fmt.Sprintf("under lease epoch %d", s.Epoch)
}
generation := "no assignment generation recorded"
if s.Generation > 0 {
generation = fmt.Sprintf("assignment generation %d", s.Generation)
}
lines := []string{
fmt.Sprintf(" last sent sequence %d at %s", s.Sequence, s.SentAt.Local().Format(stamp)),
fmt.Sprintf(" by %s", s.Sender),
fmt.Sprintf(" %s, composed from %s", epoch, generation),
fmt.Sprintf(" naming %s", modulesWords(s.Modules)),
}
answer := func(words string) { lines = append(lines, " its answer "+words) }
switch a := s.Answer; {
case s.RefusedAt != nil:
// Before the report: a refusal for the generation is kept on the send itself and is the machine's
// answer to this send in particular.
answer(fmt.Sprintf("refused it at %s: it had applied assignment generation %d, newer than this one",
s.RefusedAt.Local().Format(stamp), s.RefusedApplied))
case a.Outcome == "" || a.At == nil:
answer("the machine has not reported on it yet")
case a.Outcome == inventory.OutcomeApplied:
answer("applied it at " + a.At.Local().Format(stamp))
case a.Outcome == inventory.OutcomeRefused:
answer("refused it at " + a.At.Local().Format(stamp) + ", in its words:")
for _, line := range strings.Split(strings.TrimRight(a.Refused, "\n"), "\n") {
lines = append(lines, " "+strings.TrimSpace(line))
}
default:
answer(fmt.Sprintf("%s at %s: %d resource(s) failed — `status` says which", a.Outcome,
a.At.Local().Format(stamp), a.Failed))
}
return lines
}
// sendAnswerWord is what a machine answered of a send, in a word or few, as status says it in one line.
func sendAnswerWord(s inventory.Send) string {
switch a := s.Answer; {
case s.RefusedAt != nil:
return fmt.Sprintf("refused: it had applied generation %d", s.RefusedApplied)
case a.Outcome == "" || a.At == nil:
return "not reported on yet"
case a.Outcome == inventory.OutcomeFailed:
return fmt.Sprintf("failed (%d resource(s))", a.Failed)
default:
return a.Outcome
}
}
// printLastSends says, one line per machine, when it was last sent a declaration, by whom, from which assignment
// generation and what it answered (novox/hq issue 485). `node show <node>` says the send in full.
func printLastSends(nodes []inventory.Node, sends map[string]inventory.Send, unread string) {
if unread != "" {
fmt.Printf("what each machine was last sent could NOT be read: %s\n\n", unread)
return
}
if len(nodes) == 0 {
return
}
fmt.Println("last sent, per machine:")
for _, n := range nodes {
s, found := sends[n.Name]
if !found {
fmt.Printf(" %-12s no send recorded\n", n.Name)
continue
}
generation := "no generation"
if s.Generation > 0 {
generation = fmt.Sprintf("generation %d", s.Generation)
}
fmt.Printf(" %-12s %s by %s, %s — %s\n", n.Name, s.SentAt.Local().Format("2006-01-02 15:04"), s.Sender,
generation, sendAnswerWord(s))
}
fmt.Printf("\n `node show <node>` says a machine's last send in full\n\n")
}
+148
View File
@@ -0,0 +1,148 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The assignment generation a declaration was composed from (novox/hq issue 234).
func bodyKeys(t *testing.T, s sendable) map[string]any {
t.Helper()
raw, err := s.Body()
if err != nil {
t.Fatal(err)
}
var keys map[string]any
if err := json.Unmarshal(raw, &keys); err != nil {
t.Fatal(err)
}
return keys
}
// **The generation goes on the wire only to a machine whose node-engine said it reads one**: an older
// node-engine decodes strictly and refuses an unknown key, whole. No put-back mark is ever sent: a gate
// composes its put-back afresh, with the generation as it stands.
func TestTheGenerationIsSentOnlyToAMachineThatReadsOne(t *testing.T) {
resources := []map[string]any{{"id": "a", "type": "file", "path": "/etc/a", "content": "x\n"}}
reads := order{sequence: 12, epoch: 57, generation: 40, readsGeneration: true, acting: 57}
var told sendable
told.Resources = resources
reads.stamp(&told)
keys := bodyKeys(t, told)
if keys["generation"] != float64(40) || keys["sequence"] != float64(12) {
t.Fatalf("a machine that reads a generation was sent %v", keys)
}
if _, there := keys["put_back"]; there {
t.Fatalf("a put-back mark was sent: %v", keys)
}
if told.composedFrom != 40 || told.actingEpoch != 57 {
t.Errorf("the send does not keep what it was composed from for its record: %+v", told)
}
older := order{sequence: 12, epoch: 57, generation: 40, readsGeneration: false, acting: 57}
var untold sendable
untold.Resources = resources
older.stamp(&untold)
if _, there := bodyKeys(t, untold)["generation"]; there {
t.Fatal("a machine whose node-engine never said it reads a generation was sent one")
}
if untold.composedFrom != 40 {
t.Errorf("the generation it was composed from is recorded whether or not it was sent: %+v", untold)
}
}
// **The sender is named**: the caller of the seat call when there is one, else the shell's account, and the
// process and build that composed it.
func TestASendNamesItsSender(t *testing.T) {
said := senderOf("g14.node-tools, through the mesh-controller seat")
if !strings.Contains(said, "g14.node-tools") || !strings.Contains(said, "pid ") || !strings.Contains(said, "build ") {
t.Fatalf("the sender reads %q", said)
}
}
// refusedSend is a send a machine refused for its generation at a moment.
func refusedSend(at time.Time) inventory.Send {
return inventory.Send{NodeName: "anchor", Sequence: 12, Epoch: 57, Generation: 38, RefusedApplied: 40,
Sender: "a one-shot push by jochen at a shell on anchor (pid 4242 on anchor, build 2026.10.11)",
Modules: []string{"docker"}, SentAt: at.Add(-time.Second), RefusedAt: &at, Recorded: true}
}
// **A refused send is raised naming its sender** (novox/hq issue 234): who sent it, from which generation,
// against which the machine applied, and its sequence — the facts that took a morning to look for.
func TestARefusedSendIsRaisedNamingItsSender(t *testing.T) {
now := time.Date(2026, 10, 11, 12, 0, 0, 0, time.UTC)
f := calm(now)
f.refusedSends = []inventory.Send{refusedSend(now.Add(-time.Minute))}
got := watchGenerationRefusals(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
o := got[0]
if o.Key() != "machine.anchor.older-generation" || o.Machine != "anchor" {
t.Errorf("raised as %s about %q", o.Key(), o.Machine)
}
for _, want := range []string{"a one-shot push by jochen", "generation 38", "generation 40", "sequence 12"} {
if !strings.Contains(o.Summary, want) {
t.Errorf("the summary does not say %q: %s", want, o.Summary)
}
}
if o.Headline == "" || o.Explanation == "" || o.Resolved == "" {
t.Errorf("the condition is not worded for the operator: %+v", o)
}
}
// **A refusal of a send the mesh has no record of is raised too, its sender said to be unknown** — never
// only a line on stderr.
func TestARefusalOfAnUnrecordedSendIsRaisedSayingTheSenderIsUnknown(t *testing.T) {
now := time.Date(2026, 10, 11, 12, 0, 0, 0, time.UTC)
f := calm(now)
s := refusedSend(now.Add(-time.Minute))
s.Recorded, s.Sender, s.Modules = false, "a sender the mesh has no record of (no send of this sequence was recorded)", nil
f.refusedSends = []inventory.Send{s}
got := watchGenerationRefusals(f)
if len(got) != 1 || !strings.Contains(got[0].Summary, "no record of") {
t.Fatalf("an unattributed refusal was not raised as one: %+v", got)
}
}
// **When the refusal showed the mesh's counter behind the machine, the condition asks for a push** — it was
// raised, and nothing has sent the machine its declaration since — and does not say there is nothing to do.
func TestACounterBehindTheMachineAsksForAPush(t *testing.T) {
now := time.Date(2026, 10, 11, 12, 0, 0, 0, time.UTC)
f := calm(now)
s := refusedSend(now.Add(-time.Minute))
s.Generation, s.RefusedApplied, s.CounterRaisedTo = 12, 40, 41
f.refusedSends = []inventory.Send{s}
got := watchGenerationRefusals(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
if !strings.Contains(got[0].Summary, "`push anchor`") || !strings.Contains(got[0].Explanation, "push anchor") ||
strings.Contains(got[0].Explanation, "Nothing for you to do") {
t.Errorf("a counter behind the machine does not ask for a push: %s / %s", got[0].Summary, got[0].Explanation)
}
}
// **Only a refusal of the counter as it stands is the counter behind**: a stale sender's generation is below
// it, and the counter is left alone for that.
func TestOnlyARefusalOfTheCounterAsItStandsRaisesIt(t *testing.T) {
for _, c := range []struct {
refused, applied, now int64
behind bool
}{
{refused: 12, applied: 40, now: 12, behind: true}, // the store was put back: the mesh says 12, the machine had 40
{refused: 38, applied: 40, now: 41, behind: false}, // a stale sender: the counter is already past
{refused: 38, applied: 40, now: 40, behind: false}, // a stale sender: the counter is where the machine is
} {
r := link.GenerationRefusal{Generation: c.refused, Applied: c.applied}
if got := counterBehind(r, c.now); got != c.behind {
t.Errorf("refused %d, applied %d, counter %d: behind %v, want %v", c.refused, c.applied, c.now, got, c.behind)
}
}
}
@@ -0,0 +1,198 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/identity"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The grants step on a store (novox/hq issue 490): the machine holding the bus is sent its new user list,
// and nothing else — not the new build of a module it runs, not a module newly assigned there.
//
// The restic shape: anchor holds the bus and runs restic at c1, as does laptop; restic's c2 gives it a
// health tool. The grants step's send to anchor composes restic at c1, carries the user list that grants
// laptop's node-engine c2's tool, and records that anchor still runs c1. A module newly assigned to anchor
// makes the step fail, said, rather than install it unjudged.
func TestTheGrantsStepSendsTheUserListAndNothingElse(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
start := time.Now().Add(-time.Hour)
build := func(module, commit string, at time.Time, manifest map[string]any) link.BuildResult {
manifest["module"], manifest["version"] = module, "1"
raw, _ := json.Marshal(manifest)
return link.BuildResult{ID: link.NewBuildID(at), Repository: "novox/mesh-catalog", Path: "modules/" + module,
On: "anchor", Module: module, Commit: commit, Manifest: raw,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
for _, b := range []link.BuildResult{
build("nats", "n1", start, natsFixture()),
build("restic", "c1", start.Add(time.Second), resticFixture(false)),
build("wallpaper", "w1", start.Add(2*time.Second), wallpaperFixture()),
} {
if _, _, err := takeIn(ctx, inv, asTheOperator(t, inv, b)); err != nil {
t.Fatal(err)
}
}
for _, a := range [][2]string{{"anchor", "nats"}, {"anchor", "restic"}, {"laptop", "restic"}} {
if _, err := inv.Assign(ctx, a[0], a[1]); err != nil {
t.Fatal(err)
}
}
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.MintBusPassword(ctx, inventory.BusUser{Username: "node." + n, Kind: inventory.BusNode, Node: n}); err != nil {
t.Fatal(err)
}
}
wasEpoch := epochForActs
epochForActs = func(context.Context) (uint64, error) { return 7, nil }
t.Cleanup(func() { epochForActs = wasEpoch })
// A send to anchor composed as sendToEach composes it: numbered, planned, declared, stamped.
compose := func(under context.Context) (catalogue.Resolution, sendable) {
t.Helper()
numbered, err := allot(under, inv, "anchor")
if err != nil {
t.Fatal(err)
}
plan, settings, err := planFor(under, open, "anchor")
if err != nil {
t.Fatal(err)
}
gens, err := generators(under, open)
if err != nil {
t.Fatal(err)
}
declared, err := declarationWith(under, open, "anchor", plan, settings, gens, Allocating)
if err != nil {
t.Fatal(err)
}
numbered.stamp(&declared)
return plan, declared
}
record := func(declared sendable) {
t.Helper()
body, err := declared.Body()
if err != nil {
t.Fatal(err)
}
if _, err := recordSent(ctx, inv, "anchor", body, declared.Builds, declared.Epoch,
sentRecordOf(ctx, declared)); err != nil {
t.Fatal(err)
}
}
// Kept for anchor alone. laptop is never recorded as sent, on purpose: composing anchor resolves laptop
// too (who is on the private network), and a step that kept every machine's builds refused anchor's
// composition for want of laptop's last send (repo-check of #230 at 0f853e93).
kept := keepingEveryBuild(keepingRecorded(ctx), "anchor")
// What anchor was last sent: everything at the builds of before the merge, as an ordinary send sends it.
_, before := compose(keepingRecorded(ctx))
record(before)
plan, declared := compose(kept)
if only, err := grantsOnlyIn(ctx, open, "anchor", plan, declared); err != nil || only != "" {
t.Fatalf("with nothing changed, the step reads %q, %v", only, err)
}
// The merge: restic's c2 gives it a health tool.
if _, _, err := takeIn(ctx, inv, build("restic", "c2", start.Add(time.Minute),
resticFixture(true))); err != nil {
t.Fatal(err)
}
// The step's send to anchor: restic as anchor runs it, c1, and the user list granting c2's tool.
generation, err := inv.AssignmentGeneration(ctx)
if err != nil {
t.Fatal(err)
}
plan, declared = compose(kept)
for _, m := range plan.Modules {
if m.Module == "restic" && len(m.Tools) > 0 {
t.Fatalf("the grants step composed restic's new build on anchor: tools %v", m.Tools)
}
}
if declared.Builds["restic"] != "c1" {
t.Fatalf("the step's send records it carries restic %q; want c1, which anchor runs", declared.Builds["restic"])
}
if !strings.Contains(declared.BusUsers, "mesh.mod.restic.tool.backup_health.laptop") {
t.Errorf("the step's declaration does not grant laptop's node-engine restic's new tool:\n%s", declared.BusUsers)
}
// The generation it carries is the current one (novox/hq issue 234), and the step does not raise it.
if declared.composedFrom != generation {
t.Errorf("the step's send was composed from generation %d; the mesh is at %d", declared.composedFrom, generation)
}
// The guard, on the very declaration the send sends: only the user list differs from the last send.
if only, err := grantsOnlyIn(ctx, open, "anchor", plan, declared); err != nil || only != "" {
t.Fatalf("the step reads as changing more than the user list: %q, %v", only, err)
}
// And the gate's refusal still reads the step's send: it moves nothing there.
if names, err := ungatedIn(kept, open, []string{"anchor"}, ""); err != nil || strings.Join(names, ",") != "anchor" {
t.Fatalf("the step's send is refused as a move: %v, %v", names, err)
}
// Recorded as the send records it: anchor still runs restic c1, its gate still to come there.
record(declared)
if sent, _, err := inv.SentBuilds(ctx, "anchor"); err != nil || sent["restic"] != "c1" {
t.Fatalf("after the step anchor reads as sent restic %q (%v); want c1", sent["restic"], err)
}
if after, err := inv.AssignmentGeneration(ctx); err != nil || after != generation {
t.Fatalf("the step moved the assignment generation from %d to %d (%v)", generation, after, err)
}
// A module newly assigned to anchor: the step would install it, unjudged, so it is refused unsent.
if _, err := inv.Assign(ctx, "anchor", "wallpaper"); err != nil {
t.Fatal(err)
}
plan, declared = compose(kept)
only, err := grantsOnlyIn(ctx, open, "anchor", plan, declared)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(only, "wallpaper newly assigned") || !strings.Contains(only, "+wallpaper") {
t.Fatalf("a new assignment on the bus's machine reads %q; want the step refused, naming it", only)
}
// And the send itself refuses it, unsent: judged where it is composed, before the bus is dialled.
identity.ForTest(t)
if _, err := sendToEach(kept, open, []string{"anchor"}); !errors.Is(err, errNotGrantsOnly) ||
!strings.Contains(err.Error(), "wallpaper") {
t.Fatalf("the grants step's send of a new assignment was not refused by the send: %v", err)
}
}
// fixtureImage is the image every fixture container of the grants step's tests runs.
var fixtureImage = "registry.invalid:5000/restic/backup@sha256:" + strings.Repeat("b", 64)
// resticFixture is restic's manifest in the shape of mesh-catalog #210: at c2 (withTool) its backup judged by
// its new tool `backup_health`, beside a check of another kind it does not run itself — a tool check alone
// is refused (ADR 0227 rule 8, ADR 0240 rule 2) — and at c1 neither.
func resticFixture(withTool bool) map[string]any {
backup := map[string]any{"id": "backup", "type": "container", "name": "restic-backup", "image": fixtureImage}
measure := map[string]any{"id": "measure", "type": "container", "name": "restic-measure", "image": fixtureImage}
m := map[string]any{"resources": []any{backup, measure}}
if withTool {
backup["health"] = map[string]any{"kind": catalogue.HealthTool, "tool": "backup_health"}
measure["health"] = map[string]any{"kind": "runtime"}
m["tools"] = []string{"backup_health"}
}
return m
}
// natsFixture is the bus's module: it holds mesh-broker and is sent the user list.
func natsFixture() map[string]any {
return map[string]any{"bus-users": "/var/lib/nats-module/conf/accounts.conf",
"claims": []any{map[string]any{"name": catalogue.BrokerSeat, "scope": catalogue.ScopeMesh}}}
}
// wallpaperFixture is a module the bus's machine is newly assigned.
func wallpaperFixture() map[string]any {
return map[string]any{"resources": []any{
map[string]any{"id": "paper", "type": "container", "name": "wallpaper", "image": fixtureImage}}}
}
+226
View File
@@ -0,0 +1,226 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The grants reach the bus before the gate (novox/hq issue 490).
//
// On 2026-10-11 a merge gave restic, assigned to every machine, a new health tool. Its walk sent restic to
// its first machine and judged it there; that machine's node-engine asked the new tool and the bus refused
// it, because the grant was in the bus's user list, which only the machine holding the bus carries — and
// that machine runs restic too, so it was left out of the send while restic's new build waited there for
// the very gate that could not pass. A person pushed it by hand, carrying restic's new build there unjudged.
// aSend is one send the walk made: to which machines, and whether every build there was kept.
type aSend struct {
names []string
keepsAll bool
judged []string
}
// sendsRecorded replaces the walk's send and the reading of the bus's user list for one test: the machine
// holding the bus is novox, and its list is behind as behind says.
func sendsRecorded(t *testing.T, holder string, behind bool, refuse error) *[]aSend {
t.Helper()
var sends []aSend
wasSend, wasBehind := sendRollout, brokerBehindOf
t.Cleanup(func() { sendRollout, brokerBehindOf = wasSend, wasBehind })
brokerBehindOf = func(_ context.Context, _ *stores, names []string) (string, bool, error) {
if slices.Contains(names, holder) {
return holder, false, nil
}
return holder, behind, nil
}
sendRollout = func(ctx context.Context, _ *stores, names []string) ([]string, error) {
s := aSend{names: append([]string(nil), names...), keepsAll: len(names) == 1 && everyBuildKept(ctx, names[0])}
for n := range scopeOf(ctx).judged {
s.judged = append(s.judged, n)
}
sends = append(sends, s)
if refuse != nil && s.keepsAll {
return nil, refuse
}
return names, nil
}
return &sends
}
// The restic shape: a new tool on a module on every machine, its first machine ace, the bus on novox.
func TestAWalkSendsTheGrantsToTheBusBeforeTheFirstMachineIsJudged(t *testing.T) {
sends := sendsRecorded(t, "novox", true, nil)
sent, grants, err := sendJudged(t.Context(), nil, "ace")
if err != nil {
t.Fatal(err)
}
if len(*sends) != 2 {
t.Fatalf("the walk made %d send(s), want the grants step and then the first machine: %+v", len(*sends), *sends)
}
first, then := (*sends)[0], (*sends)[1]
if strings.Join(first.names, ",") != "novox" || !first.keepsAll || len(first.judged) != 0 {
t.Errorf("the first send is %+v, want novox alone, every build there kept, judging nothing", first)
}
if strings.Join(then.names, ",") != "ace" || then.keepsAll || strings.Join(then.judged, ",") != "ace" {
t.Errorf("the second send is %+v, want ace, judged", then)
}
if strings.Join(sent, ",") != "ace" {
t.Errorf("the gate's machines are %v, want ace alone: the bus's machine is not judged", sent)
}
if grants == nil || grants.Node != "novox" || grants.At == nil || grants.Failed != "" {
t.Fatalf("the gate keeps %+v as its grants step", grants)
}
line := gateLine(&inventory.PlanGate{Machines: sent, Grants: grants})
if !strings.Contains(line, "grants step: novox sent the bus's user list first, its builds kept") {
t.Errorf("plans says the gate as %q", line)
}
}
// Nothing more when there is nothing to carry: the list unchanged, or the first machine holds the bus.
func TestAWalkTakesNoGrantsStepWhenTheListIsAlreadyThere(t *testing.T) {
for _, c := range []struct {
name string
node string
behind bool
}{{"the list unchanged", "ace", false}, {"the first machine holds the bus", "novox", true}} {
t.Run(c.name, func(t *testing.T) {
sends := sendsRecorded(t, "novox", c.behind, nil)
_, grants, err := sendJudged(t.Context(), nil, c.node)
if err != nil {
t.Fatal(err)
}
if len(*sends) != 1 || (*sends)[0].keepsAll || grants != nil {
t.Errorf("the walk made %+v with grants %+v, want the judged send alone", *sends, grants)
}
})
}
}
// A grants step that cannot be sent is said on the gate, and the walk goes on as it did before.
func TestAGrantsStepThatFailsIsSaidAndTheSendGoesOn(t *testing.T) {
sends := sendsRecorded(t, "novox", true, errors.New("a bus upgrade waits for a person"))
sent, grants, err := sendJudged(t.Context(), nil, "ace")
if err != nil {
t.Fatal(err)
}
if strings.Join(sent, ",") != "ace" || len(*sends) != 2 {
t.Errorf("the walk made %+v", *sends)
}
if grants == nil || grants.Failed == "" || grants.At != nil {
t.Fatalf("the gate keeps %+v", grants)
}
if line := gateLine(&inventory.PlanGate{Machines: sent, Grants: grants}); !strings.Contains(line,
"grants step to novox FAILED, sent without it: a bus upgrade waits for a person") {
t.Errorf("plans says the gate as %q", line)
}
}
// A step whose send would change more than the user list is refused unsent by the send (sendToEach): the
// gate says it FAILED with what differed, and the walk goes on.
func TestAGrantsStepThatWouldCarryMoreIsNotSent(t *testing.T) {
refusal := fmt.Errorf("%w: its send would change more than the bus's user list: -postgres.login-n8n", errNotGrantsOnly)
sends := sendsRecorded(t, "novox", true, refusal)
sent, grants, err := sendJudged(t.Context(), nil, "ace")
if err != nil {
t.Fatal(err)
}
if strings.Join(sent, ",") != "ace" || len(*sends) != 2 {
t.Errorf("the walk made %+v, want the refused step and then ace's judged send", *sends)
}
if grants == nil || grants.At != nil || !strings.Contains(grants.Failed, "-postgres.login-n8n") {
t.Fatalf("the gate keeps %+v", grants)
}
}
// The step's exemption is the list's content alone: its path or mode moving is a change like any other.
func TestOnlyTheUserListsContentIsExempt(t *testing.T) {
list := "nats.bus-users"
file := func(fields map[string]any) sentResource {
r := map[string]any{"id": list, "type": "file", "path": "/conf/accounts.conf", "mode": "0600", "content": "a"}
for k, v := range fields {
r[k] = v
}
body, _ := json.Marshal(map[string]any{"resources": []any{r}})
s, err := summarize(body)
if err != nil {
t.Fatal(err)
}
return s[0]
}
was := file(nil)
if other := otherThanTheList(diffSent([]sentResource{was}, []sentResource{file(map[string]any{"content": "b"})}), list); len(other) != 0 {
t.Errorf("the list's new content reads as more: %v", other)
}
for _, f := range []map[string]any{{"mode": "0644"}, {"path": "/elsewhere"}, {"content": "b", "owner": "nats"}} {
other := otherThanTheList(diffSent([]sentResource{was}, []sentResource{file(f)}), list)
if len(other) != 1 || !strings.HasPrefix(other[0], "~"+list) {
t.Errorf("the list's resource changed by %v reads %v; want it refused", f, other)
}
}
}
// The step keeps the builds of the machine it sends alone: the others its composition resolves are composed
// as any send composes them.
func TestTheGrantsStepKeepsOneMachinesBuilds(t *testing.T) {
ctx := keepingEveryBuild(t.Context(), "novox")
if !everyBuildKept(ctx, "novox") || everyBuildKept(ctx, "ace") || everyBuildKept(t.Context(), "novox") {
t.Error("the grants step keeps the builds of a machine it does not send")
}
}
// The store test's manifests are manifests the controller takes in: checked here, where no store is needed,
// so a fixture the module rules refuse fails before the store test is run (repo-check of #230 at 325575b2).
func TestTheGrantsStepFixturesAreManifests(t *testing.T) {
for name, m := range map[string]map[string]any{"nats": natsFixture(), "restic-c1": resticFixture(false),
"restic-c2": resticFixture(true), "wallpaper": wallpaperFixture()} {
m["module"], m["version"] = strings.SplitN(name, "-", 2)[0], "1"
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
if _, err := catalogue.ParseManifest(raw); err != nil {
t.Errorf("%s: %v", name, err)
}
}
}
// The delivery plan says the step before the merge.
func TestAChangePlanSaysTheGrantsStep(t *testing.T) {
const catalogue_ = "http://forge.internal:20000/novox/mesh-catalog.git"
entry := func(name string, on ...string) inventory.Entry {
e := fromRepo(name, catalogue_, "modules/"+name)
e.Source.BuiltFrom = "old"
e.On = on
return e
}
nats := entry("nats", "novox")
nats.Manifest.BusUsers = "/etc/nats/users.conf"
nats.Manifest.Claims = []catalogue.Claim{{Name: catalogue.BrokerSeat, Scope: catalogue.ScopeMesh}}
restic := entry("restic", "ace", "novox", "shanks")
only := entry("bus-tools", "novox")
entries := []inventory.Entry{nats, restic, only}
rollsOut := func(string) (inventory.Upgrade, bool) { return inventory.Upgrade{RollOut: true}, true }
plan := func(paths ...string) link.ChangePlan {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "head", Paths: paths}
return changePlanOf("novox/mesh-catalog", "main", "head", reachOfMerge(m, entries, nil, nil), entries, rollsOut)
}
text := planText(plan("modules/restic/module.json"))
if !strings.Contains(text, "grants step: if this changes what the bus's user list says (a new tool, subject or "+
"user), novox is sent that list alone, every build there kept, before restic is judged on its first machine") {
t.Errorf("the change plan does not say the grants step:\n%s", text)
}
// A module only on the bus's machine goes there with the list in its own send: no step of its own.
if text := planText(plan("modules/bus-tools/module.json")); strings.Contains(text, "grants step") {
t.Errorf("a module on the bus's machine alone reads:\n%s", text)
}
}
+158
View File
@@ -0,0 +1,158 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group still assembles** (novox/hq ADR 0306 decision 2): a merge whose delivery group has a member
// not yet merged waits for the merge window, whether or not a batch is assembling, so the group's members walk in
// one batch, in their order (ADR 0276 decision 5). Which groups exist and where each member stands is
// mesh-delivery's record (ADR 0239): a group is the pull requests of two repositories or more sharing a head
// branch name, which is its id. The controller asks it, through its `groups` verb, when a merge from a head is
// heard; a member whose pull request is open (proposed, checked, ready or rejected) is still to merge. When
// mesh-delivery does not answer within groupAskWithin, or answers something unreadable, the merge waits: the safe
// error is to wait, and its batch says why.
// groupAskWithin is how long mesh-delivery is given to say a merge's group, while the merge is being heard.
var groupAskWithin = 3 * time.Second
// openPullStates are the states of a delivery whose pull request is open and not merged (mesh-delivery's table):
// a member in one of them is still to merge. Closed unmerged is stopped, final, and waits for nothing.
var openPullStates = map[string]bool{"proposed": true, "checked": true, "ready": true, "rejected": true}
// groupMember is one member of a delivery group as mesh-delivery's `groups` says it: `<owner/repo>@<commit> (state)`.
type groupMember struct {
Repository, State string
}
// deliveryGroupOf is what mesh-delivery says of the group a head names: its members, and whether the head names a
// group at all; an error when it cannot be asked or did not answer. A seam a test replaces.
var deliveryGroupOf = func(ctx context.Context, held bool, head string) ([]groupMember, bool, error) {
if !held {
return nil, false, fmt.Errorf("no holder of %s is on record to say it", catalogue.DeliverySeat)
}
serving := servingBus.Load()
if serving == nil {
return nil, false, errors.New("this controller is not on the bus")
}
asking, cancel := context.WithTimeout(ctx, groupAskWithin)
defer cancel()
raw, err := askDeliveryOwner(asking, serving.Conn(), "groups", map[string]any{})
if err != nil {
if errors.Is(err, link.ErrNothingServes) {
return nil, false, fmt.Errorf("nothing serves %s.groups", catalogue.DeliverySeat)
}
return nil, false, err
}
var lines []struct {
ID string `json:"id"`
Members []string `json:"members"`
}
if err := json.Unmarshal(raw, &lines); err != nil {
return nil, false, fmt.Errorf("%s.groups answered something unreadable: %w", catalogue.DeliverySeat, err)
}
for _, l := range lines {
if l.ID != head {
continue
}
var members []groupMember
for _, s := range l.Members {
members = append(members, memberOf(s))
}
return members, true, nil
}
return nil, false, nil
}
// memberOf reads one member as `groups` says it: `novox/mesh-sdk@d41d255f28ad (held)`.
func memberOf(s string) groupMember {
id, state := strings.TrimSpace(s), ""
if i := strings.LastIndex(id, " ("); i >= 0 && strings.HasSuffix(id, ")") {
id, state = id[:i], id[i+2:len(id)-1]
}
repository, _, _ := strings.Cut(id, "@")
return groupMember{Repository: repository, State: state}
}
// groupWait says whether a merge heard now waits for the window for its delivery group, and why in words; nothing
// when it names no head, its head names no group, or every other member is merged or closed. own is the merge's
// kind (ownPath).
//
// A member of another repository whose merge this controller already keeps in a batch not yet cut of the same
// kind is merged, whatever mesh-delivery says yet (it marks a member merged once its walk opened): the merge joins
// that batch by their head. One kept in a batch of the other kind keeps the merge waiting, whatever mesh-delivery
// says (review of #250): kinds never share a batch, so the merge's batch is a new one, and cut at once it would
// walk before that member's batch is cut and break the group's order (ADR 0276 decision 5). Waiting, the two
// batches are cut in the order their windows close, an own-path one first at one look, as cutBatchesHeld cuts
// every pair of kinds.
//
// Asked before the plans are held: a slow answer holds up no walk.
func groupWait(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, held, own bool) (string, error) {
if m.Head == "" {
return "", nil
}
same, other, err := keptFromHead(ctx, inv, m.Head, own)
if err != nil {
return "", err
}
if len(other) > 0 {
return "group " + m.Head + " has a member waiting in a batch of the other kind: " + strings.Join(other, ", "), nil
}
members, isGroup, err := deliveryGroupOf(ctx, held, m.Head)
if err != nil {
return fmt.Sprintf("%s did not say whether group %s has a member still to merge (%v): it waits for the "+
"window, the safe side", catalogue.DeliverySeat, m.Head, err), nil
}
if !isGroup {
return "", nil
}
var open []string
for _, g := range members {
if strings.EqualFold(g.Repository, m.Owner+"/"+m.Repo) || same[strings.ToLower(g.Repository)] ||
!openPullStates[g.State] {
continue
}
open = append(open, g.Repository+" ("+g.State+")")
}
if len(open) == 0 {
return "", nil
}
return "group " + m.Head + " has a member still to merge: " + strings.Join(open, ", "), nil
}
// keptFromHead are the merges from a head that batches not yet cut hold: the repositories (lower case) of those in
// a batch of the kind own says, and, in words, each of the other kind with its batch.
func keptFromHead(ctx context.Context, inv *inventory.Inventory, head string, own bool) (map[string]bool, []string, error) {
batches, err := inv.Batches(ctx)
if err != nil {
return nil, nil, err
}
same := map[string]bool{}
var other []string
for _, b := range batches {
merges, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return nil, nil, err
}
for _, k := range merges {
if announcedOf(k).Head != head {
continue
}
if b.OwnPath() == own {
same[strings.ToLower(k.Repository)] = true
} else {
other = append(other, fmt.Sprintf("%s@%s in %s", k.Repository, short(k.Commit), b.ID))
}
}
}
return same, other, nil
}
+5
View File
@@ -84,8 +84,13 @@ var handActVerbs = []handActVerb{
"deliberate test, never a repair", DecidedFor: []string{causeDrill}},
// A drill: something broken on purpose to see the mesh raise and clear it. A person's test, never a
// repair, however often it is run.
// A credential a provider's software issued, taken from the provider and kept for its consumers (novox/hq ADR
// 0305): the mesh working as decided, however often the software changes it.
{Verb: handActIssued, Decision: "a credential the provider's software issued, kept for its consumers (ADR 0305)"},
{Verb: "hand-act drill", Decision: "a drill is a person's deliberate test of the mesh, never a repair"},
// A person's decisions by design.
// A config value is what a person chooses the mesh's modules use (novox/hq issue 630), never a repair.
{Verb: "config", Decision: "a config value is a person's choice of what the modules use, not a repair (issue 630)"},
{Verb: "retire approve", Decision: "nothing is retired past its bound without a person (ADR 0230)"},
{Verb: "retire reject", Decision: "keeping a consumer active is a person's word (ADR 0230)"},
{Verb: "cleanup delete", Decision: "nothing retired is deleted without a person (ADR 0230)"},
+1 -1
View File
@@ -55,7 +55,7 @@ func TestARepairByHandCarriesItsReason(t *testing.T) {
want string
}{
{"push", map[string]any{"node": "anchor", "why": "stuck", "cause": "sent-not-reported"},
"push anchor --wait 0 --why stuck --cause sent-not-reported"},
"push anchor --wait 0s --why stuck --cause sent-not-reported"},
{"plans", map[string]any{"close": "plan-1", "why": "the report will not come"},
"plans close plan-1 --why the report will not come"},
{"plans", map[string]any{"retry": "plan-1"}, "plans retry plan-1"},
+51
View File
@@ -167,3 +167,54 @@ func TestASetuidSearchAsksOnlyAKnownNodeAndFailsOnARefusal(t *testing.T) {
t.Fatalf("a start: %v, printed %q", err, out.String())
}
}
// The directory judgement's line asks only a known node, one name and nothing else, fails on a refusal, and says it
// changed nothing (novox/hq issue 358).
func TestADirectoryJudgementAsksOnlyAKnownNodeAndSaysItIsReadOnly(t *testing.T) {
t.Setenv(link.CallerVar, "jo through mesh-cli on anchor")
asked := 0
ask := func(answer link.HandOverAnswer) func(node, by string) (link.HandOverAnswer, error) {
return func(node, by string) (link.HandOverAnswer, error) {
asked++
if node != "novox" || by != "jo through mesh-cli on anchor" {
t.Fatalf("asked %q %q", node, by)
}
return answer, nil
}
}
known := func(string) error { return nil }
var out bytes.Buffer
for _, args := range [][]string{nil, {"novox", "extra"}, {"-x"}} {
if err := judgeDirectoriesAsked(args, known, ask(link.HandOverAnswer{Said: "x"}), &out); err == nil || asked != 0 {
t.Fatalf("%v was asked: %v", args, err)
}
}
if err := judgeDirectoriesAsked([]string{"novox"}, func(string) error { return errors.New("no node called novox") },
ask(link.HandOverAnswer{Said: "x"}), &out); err == nil || asked != 0 {
t.Fatalf("an unknown node: %v", err)
}
if err := judgeDirectoriesAsked([]string{"novox"}, known, ask(link.HandOverAnswer{Refused: "no; nothing was judged"}),
&out); err == nil || !strings.Contains(err.Error(), "novox refused") || out.Len() != 0 {
t.Fatalf("a refusal: %v, printed %q", err, out.String())
}
said := "2 directories declared: 1 placed, 1 would be refused\n/var/lib/a: placed\n/var/lib/b: would be refused: …"
if err := judgeDirectoriesAsked([]string{"novox"}, known, ask(link.HandOverAnswer{Said: said}), &out); err != nil ||
out.String() != "novox (read-only: nothing on novox was changed)\n"+said+"\n" {
t.Fatalf("a judgement: %v, printed %q", err, out.String())
}
}
// The judgement is a seat verb an agent may call — it reads, and changes nothing on the node — and runs exactly
// `node judge-directories <node>` (novox/hq issue 358).
func TestTheDirectoryJudgementIsAVerbThatOnlyReads(t *testing.T) {
argv, err := argvFor("judge-directories", map[string]any{"node": "novox"})
if err != nil || strings.Join(argv, " ") != "node judge-directories novox" {
t.Fatalf("the verb runs %v, %v", argv, err)
}
if _, err := argvFor("judge-directories", map[string]any{}); err == nil {
t.Fatal("the verb ran with no node")
}
if !readingVerbs["judge-directories"] {
t.Fatal("the judgement is not counted a reading verb")
}
}
+1 -1
View File
@@ -128,7 +128,7 @@ func (r *recordedDelivery) grant(context.Context, []readyNode) error { return ni
func (r *recordedDelivery) declare(ctx context.Context, s readyNode, body []byte) (string, error) {
r.declared = append(r.declared, s.node)
return recordSent(ctx, r.inv, s.node, body, s.declared.Builds, s.declared.Epoch)
return recordSent(ctx, r.inv, s.node, body, s.declared.Builds, s.declared.Epoch, sentRecordOf(ctx, s.declared))
}
// aResolver is a module built from a repository, at a commit, with something on the machine that
+259
View File
@@ -0,0 +1,259 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
)
// novox/hq issue 201: on 2026-10-02 a plan sent the control node a controller older than the one running there,
// whose seat row the newer one had already widened; the older binary refused the row and the mesh went quiet for
// ten minutes. No send but a person's push and a gate's own put-back moves the controller or the node-engine back
// behind the build its machine was last sent: the send keeps it there, and carries the rest.
// askedIn answers a build from a fixed table: module@commit → when it was asked; failed names those that failed
// their gate.
func askedIn(table map[string]time.Time, failed ...string) func(module, commit string) (askedBuild, bool, error) {
return func(module, commit string) (askedBuild, bool, error) {
at, known := table[module+"@"+commit]
b := askedBuild{asked: at}
for _, f := range failed {
if f == module+"@"+commit {
b.failedGate = true
}
}
return b, known, nil
}
}
func TestACoreModuleIsKeptOverAnOlderRecord(t *testing.T) {
old, now := t0, t0.Add(time.Minute)
table := map[string]time.Time{
"mesh-controller@c1": old, "mesh-controller@c2": now,
"mesh-host@h1": old, "mesh-host@h2": now,
"app@a1": old, "app@a2": now,
}
sent := map[string]string{"mesh-controller": "c2", "mesh-host": "h2", "app": "a2"}
record := map[string]string{"mesh-controller": "c1", "mesh-host": "h1", "app": "a1"}
found, err := coreDowngrades("anchor", sent, record, sendScope{judged: map[string]bool{"anchor": true}}, askedIn(table))
if err != nil {
t.Fatal(err)
}
if len(found) != 2 || found[0].module != "mesh-controller" || found[0].component != lease.ComponentController ||
found[0].sent != "c2" || found[1].module != "mesh-host" || found[1].component != lease.ComponentEngine {
t.Fatalf("the controller and the node-engine over an older record are kept, and nothing else: %+v", found)
}
if said := found[0].String(); !strings.Contains(said, "anchor was last sent mesh-controller c2") ||
!strings.Contains(said, "`push anchor`") || !strings.Contains(said, "`rebuild mesh-controller`") {
t.Fatalf("it says the machine, the build kept and the two ways out: %s", said)
}
// The record forward, the same build, or a build the records do not hold: nothing kept.
for _, c := range [][2]string{{"c1", "c2"}, {"c2", "c2"}, {"c2", "c9"}, {"c9", "c1"}} {
found, err := coreDowngrades("anchor", map[string]string{"mesh-controller": c[0]},
map[string]string{"mesh-controller": c[1]}, sendScope{}, askedIn(table))
if err != nil || len(found) != 0 {
t.Fatalf("sent %s, record %s: no downgrade proved, nothing kept: %+v %v", c[0], c[1], found, err)
}
}
// A sent build that failed its gate is never ahead: the record put back before it is sent.
if found, _ := coreDowngrades("anchor", sent, record, sendScope{}, askedIn(table, "mesh-controller@c2")); len(found) != 1 ||
found[0].module != "mesh-host" {
t.Fatalf("a controller that failed its gate was kept over the build put back: %+v", found)
}
// A gate's own put-back carries its own modules back, and nothing else.
putBack := sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true}
if found, _ := coreDowngrades("anchor", sent, record, putBack, askedIn(table)); len(found) != 1 || found[0].module != "mesh-host" {
t.Fatalf("a put-back of the controller carries it back, and not the node-engine: %+v", found)
}
if found, _ := coreDowngrades("anchor", sent, record, sendScope{modules: map[string]bool{"mesh-controller": true}},
askedIn(table)); len(found) != 2 {
t.Fatalf("a send for the controller that is no put-back carries it back: %+v", found)
}
}
// core201 is a mesh whose control node, anchor, was sent the newer controller c2 while the mesh's record of the
// controller is the build before it, c1, which passed its gate when it rolled out.
func core201(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
old, now := time.Now().Add(-2*time.Hour), time.Now().Add(-time.Minute)
record := func(commit string, asked time.Time) inventory.Build {
manifest, _ := json.Marshal(catalogue.Manifest{Module: "mesh-controller", Version: "1"})
b := inventory.Build{ID: "build-mesh-controller-" + commit, Module: "mesh-controller", Commit: commit,
Repository: "novox/mesh-controller", Manifest: manifest, Asked: asked, At: asked,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:" + commit + "0000000000000000"}}}
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
return b
}
before := record("c1aaaaaaaaaa", old)
if err := inv.RecordGate(ctx, inventory.GateVerdict{Build: before.ID, Module: "mesh-controller", Commit: before.Commit,
Machines: []string{"anchor"}, Verdict: inventory.GatePassed}); err != nil {
t.Fatal(err)
}
record("c2bbbbbbbbbb", now)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"}, inventory.Source{
Repository: "novox/mesh-controller", BuiltFrom: "c2bbbbbbbbbb", Head: "c2bbbbbbbbbb", Asked: now}); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"mesh-controller": "c2bbbbbbbbbb"}); err != nil {
t.Fatal(err)
}
if err := inv.RestoreModule(ctx, before); err != nil {
t.Fatal(err)
}
wasMoves := machineMoves
machineMoves = func(ctx context.Context, open *stores, f moveFacts, node string, all bool) ([]inventory.CarriedMove, error) {
modules, err := open.inventory.Assigned(ctx, node)
if err != nil {
return nil, err
}
sent, known, err := open.inventory.SentBuilds(ctx, node)
if err != nil {
return nil, err
}
return f.moves(node, modules, sent, known, all), nil
}
t.Cleanup(func() { machineMoves = wasMoves })
return open
}
// TestReplay201 replays the race on a store: the control node runs the newer controller and the mesh's record is
// the build before it. A plan's send there — the node-engine's plan of 2026-10-02 — goes, keeps the newer
// controller, and says so as a condition; the gate's own put-back carries it back; a send that carries the record,
// as a person's push does, clears the condition.
func TestReplay201(t *testing.T) {
open := core201(t)
ctx := t.Context()
k, _ := withConditionsInMemory(t)
judged := withScope(ctx, sendScope{judged: map[string]bool{"anchor": true}})
if kept, err := ungatedIn(judged, open, []string{"anchor"}, ""); err != nil || len(kept) != 1 {
t.Fatalf("a plan's send to the machine is made, the rest of it with it: %v %v", kept, err)
}
keeps, err := recordedKept(keepingRecorded(judged), open, "anchor")
if err != nil || keeps["mesh-controller"] != "c2bbbbbbbbbb" {
t.Fatalf("the send keeps the newer controller there: %v %v", keeps, err)
}
shelf := map[string]catalogue.Manifest{"mesh-controller": {Module: "mesh-controller", Version: "old"}}
if _, err := keepRecorded(keepingRecorded(judged), open, "anchor", shelf); err != nil || shelf["mesh-controller"].Version != "1" {
t.Fatalf("the send composes the kept build's manifest: %+v %v", shelf, err)
}
key := coreDowngradeObservation(lease.ComponentController, "anchor", "").Key()
if c, found, err := k.Get(ctx, key); err != nil || !found || !strings.Contains(c.Summary, "c2bbbbbb") {
t.Fatalf("the condition %s is raised, saying the build kept: %+v %v %v", key, c, found, err)
}
putBack := keepingRecorded(withScope(ctx, sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true}))
if keeps, err := recordedKept(putBack, open, "anchor"); err != nil || len(keeps) != 0 {
t.Fatalf("the gate's own put-back of the controller carries it back: %v %v", keeps, err)
}
settleCoreDowngrades(ctx, open.inventory, "anchor", map[string]string{"mesh-controller": "c2bbbbbbbbbb"})
if _, found, _ := k.Get(ctx, key); !found {
t.Fatal("a send that kept the newer controller cleared the condition")
}
settleCoreDowngrades(ctx, open.inventory, "anchor", map[string]string{"mesh-controller": "c1aaaaaaaaaa"})
if _, found, _ := k.Get(ctx, key); found {
t.Fatal("a send carrying the mesh's record — a person's push — leaves the condition open")
}
}
// The review of #273: a gate failed the newer controller, put the record back, and the put-back's own send was
// refused. The next automatic send — a healer's, which judges nothing — completes the put-back: a build that
// failed its gate is never kept.
func TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend(t *testing.T) {
open := core201(t)
ctx := t.Context()
k, _ := withConditionsInMemory(t)
if err := open.inventory.RecordGate(ctx, inventory.GateVerdict{Build: "build-mesh-controller-c2bbbbbbbbbb",
Module: "mesh-controller", Commit: "c2bbbbbbbbbb", Previous: "c1aaaaaaaaaa", Machines: []string{"anchor"},
Verdict: inventory.GateFailed, Rollback: inventory.NotRolledBack}); err != nil {
t.Fatal(err)
}
healer := withScope(ctx, sendScope{})
if kept, err := ungatedIn(healer, open, []string{"anchor"}, ""); err != nil || len(kept) != 1 {
t.Fatalf("the healer's send is made: %v %v", kept, err)
}
if keeps, err := recordedKept(keepingRecorded(healer), open, "anchor"); err != nil || len(keeps) != 0 {
t.Fatalf("the failed controller was kept over the build put back: %v %v", keeps, err)
}
if open, _ := k.Open(ctx); len(open) != 0 {
t.Fatalf("a completed put-back raised a condition: %+v", open)
}
}
// Review 91 of #273: a gated send to a machine whose controller is kept over an older record leaves the kept
// controller out of what its gate judges. Judged as moving back to the record, the gate would read the send's own
// kept controller as another build sent there after it (issue 352) and be superseded, judging nothing.
func TestAGateJudgesTheOtherMovesBesideAKeptCore(t *testing.T) {
open := core201(t)
ctx := t.Context()
withConditionsInMemory(t)
inv := open.inventory
old, now := time.Now().Add(-2*time.Hour), time.Now().Add(-time.Minute)
for _, b := range []struct {
commit string
at time.Time
}{{"a1aaaaaaaaaa", old}, {"a2bbbbbbbbbb", now}} {
manifest, _ := json.Marshal(catalogue.Manifest{Module: "app", Version: "1"})
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-app-" + b.commit, Module: "app", Commit: b.commit,
Repository: "novox/mesh-catalog", Path: "modules/app", Manifest: manifest, Asked: b.at, At: b.at,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:app-" + b.commit}}}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "app", Version: "1"}, inventory.Source{
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/app", BuiltFrom: b.commit, Head: b.commit,
Asked: b.at}); err != nil {
t.Fatal(err)
}
}
if _, err := inv.Assign(ctx, "anchor", "app"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor-2",
map[string]string{"mesh-controller": "c2bbbbbbbbbb", "app": "a1aaaaaaaaaa"}); err != nil {
t.Fatal(err)
}
// The send as it is made: the controller kept at c2, the app moved to the record.
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
return names, open.inventory.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor-3",
map[string]string{"mesh-controller": "c2bbbbbbbbbb", "app": "a2bbbbbbbbbb"})
}
t.Cleanup(func() { sendRollout = wasSend })
moves, sent, _, err := gatedSend(ctx, open, "anchor", nil)
if err != nil || len(sent) != 1 {
t.Fatalf("the gated send is made: %v %v", sent, err)
}
if len(moves) != 1 || moves[0].Module != "app" || moves[0].To != "a2bbbbbbbbbb" {
t.Fatalf("the gate judges the app's move and not the kept controller: %+v", moves)
}
builds, _, err := inv.SentBuilds(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
g := &inventory.PlanGate{Carried: moves}
f := gateFacts{commits: judgedCommits(g, nil), sentBuilds: map[string]map[string]string{"anchor": builds}}
for _, m := range []string{"app", "mesh-controller"} {
if why, superseded := f.supersededOn(m, "anchor"); superseded {
t.Fatalf("the gate is superseded by its own send: %s", why)
}
}
}
+4 -1
View File
@@ -25,6 +25,7 @@ import (
// another send reaches anchor, unreported; the gate still passes. And a send that moves a judged module
// to another build supersedes the judging: no verdict, nothing put back.
func TestReplay352(t *testing.T) {
judgeLeavesAsTheCore(t)
t.Run("a newer send to the judged machine does not unreport the gate's", testANewerSendDoesNotUnreportTheGatesOwn)
t.Run("a send that moves the module supersedes the judging", testASendThatMovesTheModuleSupersedesTheJudging)
}
@@ -66,10 +67,11 @@ func testANewerSendDoesNotUnreportTheGatesOwn(t *testing.T) {
// A plan's own first send waits while a release judges the same module on that machine with another build.
func TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
advancePlans(ctx, b.open) // the release judges app c2 on anchor
_, _, err := gatedSend(ctx, b.open, "anchor", []inventory.CarriedMove{{Module: "app", Node: "anchor", From: "c2", To: "c3", Build: "build-app-c3"}})
_, _, _, err := gatedSend(ctx, b.open, "anchor", []inventory.CarriedMove{{Module: "app", Node: "anchor", From: "c2", To: "c3", Build: "build-app-c3"}})
if !errors.Is(err, errWalkedElsewhere) || !strings.Contains(err.Error(), "release-") {
t.Fatalf("a newer build of a judged module was sent under the release's gate: %v", err)
}
@@ -80,6 +82,7 @@ func TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere(t *testing.T) {
// A merge plan's judging is superseded the same way: another send moved its module on the first machine.
func TestAPlansJudgingIsSupersededByASendThatMovesItsModule(t *testing.T) {
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
+433
View File
@@ -0,0 +1,433 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 459, ADR 0299: since the merge window one walk carries every merge of its batch, so a stop of one
// group member through `delivery-stop` ended the walk of every other delivery in it. The stop now names the merges
// to leave out: the walk is ended, and every other merge it answered is walked again in a fresh batch without them
// — the stopped merge, and each later merge of its repository's branch, which contains it and ends with it.
// endsItsLine ends, as the test ends, a line the controller's output left open (a walk's ask prints " tier N: " and
// the asked build the rest, which the tests' fake ask never prints): `go test -v` would put the test's "--- PASS"
// after it, and the store-tests step of merge-check.sh, which reads that marker at a line's start, would say the
// test never ran.
func endsItsLine(t *testing.T) {
t.Cleanup(func() { fmt.Println() })
}
// leaved is a kept merge of a repository, made at a moment.
func leaved(repository, commit string, made time.Time) inventory.BatchedMerge {
return inventory.BatchedMerge{Repository: repository, Branch: "main", Commit: commit, Merged: made, Heard: made}
}
// The stopped merge is left out, and so is each later merge of its repository's branch, naming the merge it
// contains; an earlier merge of that repository and every other repository's merge are walked again.
func TestLeftOutAreTheStoppedMergeAndTheMergesThatContainIt(t *testing.T) {
c1 := leaved("novox/mesh-catalog", "c1aaaaaaaaaa", t0)
c2 := leaved("novox/mesh-catalog", "c2bbbbbbbbbb", t0.Add(10*time.Second))
c3 := leaved("novox/mesh-catalog", "c3cccccccccc", t0.Add(20*time.Second))
o1 := leaved("novox/one", "o1dddddddddd", t0.Add(5*time.Second))
again, out, err := leaveOut([]inventory.BatchedMerge{c1, o1, c2, c3}, []string{"novox/Mesh-Catalog@c2bbbbbbbbbb"})
if err != nil {
t.Fatal(err)
}
var walked []string
for _, m := range again {
walked = append(walked, m.Commit)
}
if !reflect.DeepEqual(walked, []string{c1.Commit, o1.Commit}) {
t.Fatalf("walked again: %v; want the earlier catalogue merge and the other repository's", walked)
}
want := []inventory.PlanLeftOut{{Repository: "novox/mesh-catalog", Commit: c2.Commit},
{Repository: "novox/mesh-catalog", Commit: c3.Commit, Contains: c2.Commit}}
if !reflect.DeepEqual(out, want) {
t.Fatalf("left out: %+v; want %+v", out, want)
}
// A merge the walk does not answer is refused, naming it: nothing ends.
if _, _, err := leaveOut([]inventory.BatchedMerge{c1, o1}, []string{"novox/mesh-catalog@ffffffffffff"}); err == nil ||
!strings.Contains(err.Error(), "ffffffffffff") {
t.Fatalf("a merge the walk does not answer was left out: %v", err)
}
if _, _, err := leaveOut([]inventory.BatchedMerge{c1}, []string{"no-commit"}); err == nil {
t.Fatal("a merge without its commit was taken")
}
}
// moving is a kept merge that moved modules when it was heard.
func moving(m inventory.BatchedMerge, modules ...string) inventory.BatchedMerge {
m.Event, _ = json.Marshal(keptMerge{Moves: modules})
return m
}
// A merge of another repository that moves a module the stopped merge moves is left out too, naming the modules
// it shares: the module is built at its branch, which holds the stopped merge (review of #232). And so on, until
// none more is: a merge sharing a module with that one is left out as well.
func TestAMergeSharingAStoppedMergesModuleIsLeftOut(t *testing.T) {
stopped := moving(leaved("novox/mesh-catalog", "c1aaaaaaaaaa", t0), "app")
reader := moving(leaved("novox/one", "o1bbbbbbbbbb", t0.Add(time.Second)), "app", "one")
next := moving(leaved("novox/two", "t1cccccccccc", t0.Add(2*time.Second)), "one")
free := moving(leaved("novox/three", "h1dddddddddd", t0.Add(3*time.Second)), "three")
again, out, err := leaveOut([]inventory.BatchedMerge{stopped, reader, next, free}, []string{"novox/mesh-catalog@c1aaaaaaa"})
if err != nil {
t.Fatal(err)
}
if len(again) != 1 || again[0].Commit != free.Commit {
t.Fatalf("walked again: %+v; want the merge that shares nothing alone", again)
}
want := []inventory.PlanLeftOut{{Repository: "novox/mesh-catalog", Commit: stopped.Commit},
{Repository: "novox/one", Commit: reader.Commit, Contains: stopped.Commit, Shares: []string{"app"}},
{Repository: "novox/two", Commit: next.Commit, Contains: stopped.Commit, Shares: []string{"one"}}}
if !reflect.DeepEqual(out, want) {
t.Fatalf("left out %+v; want %+v", out, want)
}
}
// A walk withholds a module a stopped merge moved where it would build it only as a dependent; one a merge of its
// own batch moves is built, and empty tiers go.
func TestAWalkWithholdsAStoppedMergesDependents(t *testing.T) {
plan := inventory.Plan{Tiers: [][]string{{"one"}, {"app"}, {"notes", "app2"}},
Modules: map[string]*inventory.PlanModule{"one": {}, "app": {}, "notes": {}, "app2": {}}}
got := withhold(&plan, []string{"app", "notes", "absent"}, []string{"notes"})
if !reflect.DeepEqual(got, []string{"app"}) || !reflect.DeepEqual(plan.Tiers, [][]string{{"one"}, {"notes", "app2"}}) {
t.Fatalf("withheld %v, tiers %v", got, plan.Tiers)
}
if _, in := plan.Modules["app"]; in {
t.Fatal("a withheld module is still in the plan")
}
}
// The seat verb passes the merges to leave out to the command.
func TestDeliveryStopTakesTheMergesToLeaveOut(t *testing.T) {
got, err := argvFor("delivery-stop", map[string]any{"plan": "plan-1", "why": "w", "by": "jochen",
"without": "novox/mesh-lab@b1,novox/mesh-lab@b2"})
want := []string{"delivery", "stop", "plan-1", "--why", "w", "--by", "mesh-delivery for jochen",
"--without", "novox/mesh-lab@b1,novox/mesh-lab@b2"}
if err != nil || !reflect.DeepEqual(got, want) {
t.Fatalf("delivery-stop with without → %v %v, wanted %v", got, err, want)
}
}
// A walk of the catalogue's merge and another repository's is stopped without the catalogue's: it ends stopped,
// names the batch that walks the other merge again and the merge it left out, and that batch is cut into a walk
// that builds the other repository's module alone.
func TestAStopWithoutAMergeWalksTheOthersAgain(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("no open walk to stop: %+v", ws)
}
stopped := ws[0]
p, said, err := stopWalkWithout(ctx, open.inventory, stopped.ID, "mesh-delivery for jochen", "its group stopped",
[]string{"novox/mesh-catalog@" + cat.Commit}, t0.Add(3*time.Minute))
if err != nil {
t.Fatal(err)
}
if p.State != inventory.PlanFailed || p.Delivery.Stopped == "" || p.Delivery.WalkedAgainBy == "" ||
!reflect.DeepEqual(p.Delivery.LeftOut, []inventory.PlanLeftOut{{Repository: "novox/mesh-catalog", Commit: cat.Commit}}) {
t.Fatalf("the stopped walk reads %s %+v", p.State, p.Delivery)
}
if said.WalkedAgainBy != p.Delivery.WalkedAgainBy || !reflect.DeepEqual(said.WalkedAgain, []string{"novox/one@" + one.Commit}) {
t.Fatalf("the stop answered %+v", said)
}
again, err := open.inventory.MergesOf(ctx, p.Delivery.WalkedAgainBy)
if err != nil || len(again) != 1 || again[0].Commit != one.Commit {
t.Fatalf("the fresh batch answers %+v (%v); want the other repository's merge alone", again, err)
}
if left, _ := open.inventory.MergesOf(ctx, stopped.ID); len(left) != 1 || left[0].Commit != cat.Commit {
t.Fatalf("the stopped walk still answers %+v; want the left-out merge alone", left)
}
cutAt(t, open, t0.Add(4*time.Minute))
fresh, err := open.inventory.PlanByID(ctx, p.Delivery.WalkedAgainBy)
if err != nil || fresh.Batch() || !fresh.Open() {
t.Fatalf("the fresh batch was not cut into a walk: %s %v", fresh.State, err)
}
if _, in := fresh.Modules["one"]; !in {
t.Fatalf("the fresh walk does not build one: %v", fresh.Modules)
}
if _, in := fresh.Modules["app"]; in {
t.Fatalf("the fresh walk builds app, whose merge was left out: %v", fresh.Modules)
}
if len(fresh.Delivery.Merges) != 1 || fresh.Delivery.Merges[0].Commit != one.Commit {
t.Fatalf("the fresh walk answers %+v", fresh.Delivery.Merges)
}
// A stopped walk starts no search of its own (ADR 0276 decision 3 is for a failed one).
cutAt(t, open, t0.Add(5*time.Minute))
if alone, _ := open.inventory.AloneMerges(ctx); len(alone) != 0 {
t.Fatalf("a merge of the stopped walk waits to be walked alone: %+v", alone)
}
}
// A later merge of the stopped merge's repository contains it: it is left out too, named with the merge it
// contains, and only the other repository's merge is walked again. With nothing left to walk, the walk is only
// ended. A merge the walk does not answer is refused, and nothing ends.
func TestAStopWithoutAMergeEndsTheMergesThatContainIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
first := catalogueMerge("cat1aaaaaaaa", "app", t0)
later := catalogueMerge("cat2cccccccc", "notes", t0.Add(10*time.Second))
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), first.Head)
hear(t, open, first, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
hear(t, open, later, t0.Add(11*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
walk := ws[0]
if _, _, err := stopWalkWithout(ctx, open.inventory, walk.ID, "mesh-delivery for jochen", "x",
[]string{"novox/mesh-catalog@ffffffffffff"}, t0.Add(3*time.Minute)); err == nil {
t.Fatal("a stop leaving out a merge the walk does not answer was taken")
}
if p, _ := open.inventory.PlanByID(ctx, walk.ID); !p.Open() {
t.Fatalf("a refused stop ended the walk: %s", p.State)
}
p, said, err := stopWalkWithout(ctx, open.inventory, walk.ID, "mesh-delivery for jochen", "its group stopped",
[]string{"novox/mesh-catalog@" + first.Commit}, t0.Add(3*time.Minute))
if err != nil {
t.Fatal(err)
}
want := []inventory.PlanLeftOut{{Repository: "novox/mesh-catalog", Commit: first.Commit},
{Repository: "novox/mesh-catalog", Commit: later.Commit, Contains: first.Commit}}
if !reflect.DeepEqual(p.Delivery.LeftOut, want) || !reflect.DeepEqual(said.LeftOut, want) {
t.Fatalf("left out %+v, said %+v; want %+v", p.Delivery.LeftOut, said.LeftOut, want)
}
if !strings.Contains(p.Note, later.Commit[:8]) || !strings.Contains(p.Note, p.Delivery.WalkedAgainBy) {
t.Fatalf("the stopped walk's note does not name the merge ended with it and the fresh batch: %q", p.Note)
}
again, _ := open.inventory.MergesOf(ctx, p.Delivery.WalkedAgainBy)
if len(again) != 1 || again[0].Commit != one.Commit {
t.Fatalf("walked again: %+v; want the other repository's merge alone", again)
}
// Nothing left once the other repository's merge is left out too: the fresh walk is stopped, and nothing more.
cutAt(t, open, t0.Add(4*time.Minute))
fresh, _ := open.inventory.PlanByID(ctx, p.Delivery.WalkedAgainBy)
q, said, err := stopWalkWithout(ctx, open.inventory, fresh.ID, "mesh-delivery for jochen", "that one too",
[]string{"novox/one@" + one.Commit}, t0.Add(5*time.Minute))
if err != nil || q.State != inventory.PlanFailed || q.Delivery.WalkedAgainBy != "" || said.WalkedAgainBy != "" {
t.Fatalf("a stop leaving nothing to walk: %s %+v %+v %v", q.State, q.Delivery, said, err)
}
if _, bs := walks(t, open); len(bs) != 0 {
t.Fatalf("a batch of nothing was made: %+v", bs)
}
}
// A merge heard while the walk ran waits in the open batch, at a later commit of the same repository. The merge
// walked again joins that batch, never a batch of its own that would walk the older commit after the newer one
// (review of #232): its window closes, and its walk is at the newer commit, answering both.
func TestAStopWalksTheOthersAgainInTheOpenBatch(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one1 := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one1, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
walk := ws[0]
one2 := repoMerge("one", "one2cccccccc", t0.Add(3*time.Minute))
hear(t, open, one2, t0.Add(3*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 {
t.Fatalf("the merge heard while the walk ran is in %d batches", len(bs))
}
waiting := bs[0]
p, said, err := stopWalkWithout(ctx, open.inventory, walk.ID, "mesh-delivery for jochen", "its group stopped",
[]string{"novox/mesh-catalog@" + cat.Commit}, t0.Add(3*time.Minute+10*time.Second))
if err != nil {
t.Fatal(err)
}
if p.Delivery.WalkedAgainBy != waiting.ID || said.WalkedAgainBy != waiting.ID {
t.Fatalf("walked again by %q (said %q); want the open batch %s", p.Delivery.WalkedAgainBy, said.WalkedAgainBy, waiting.ID)
}
if _, bs := walks(t, open); len(bs) != 1 {
t.Fatalf("%d batches; want the open one alone", len(bs))
}
b, _ := open.inventory.PlanByID(ctx, waiting.ID)
if b.Delivery.Batch == nil || !b.Delivery.Batch.Closed || !slices.Contains(b.Delivery.Batch.Withheld, "app") {
t.Fatalf("the open batch reads %+v; want its window closed and app withheld", b.Delivery.Batch)
}
// Its window closed by the stop, not by the clock: cut at once.
cutAt(t, open, t0.Add(3*time.Minute+11*time.Second))
fresh, _ := open.inventory.PlanByID(ctx, waiting.ID)
if fresh.Batch() || fresh.CommitOf("novox/one") != one2.Commit {
t.Fatalf("the batch is %s at %q; want it cut, walking the newer commit", fresh.State, fresh.CommitOf("novox/one"))
}
var answered []string
for _, m := range fresh.Delivery.Merges {
answered = append(answered, m.Commit+">"+m.Carried)
}
if !slices.Contains(answered, one1.Commit+">"+one2.Commit) || !slices.Contains(answered, one2.Commit+">") {
t.Fatalf("the walk answers %v; want the earlier merge carried by the later", answered)
}
}
// A stop names what the stopped walk already sent and nothing walks again, on which machines; and a later merge of
// the stopped merge's repository that waits in a batch, which will deliver the stopped change. A stopped walk
// that walked its merges again is refused a retry, saying why.
func TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one1 := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one1, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
walk := ws[0]
sent := t0.Add(150 * time.Second)
for name, m := range walk.Modules {
if name == "app" || name == "one" {
m.First, m.FirstAt = []string{"anchor"}, &sent
}
}
if err := open.inventory.SavePlan(ctx, &walk); err != nil {
t.Fatal(err)
}
cat2 := catalogueMerge("cat2dddddddd", "notes", t0.Add(3*time.Minute))
hear(t, open, cat2, t0.Add(3*time.Minute))
_, said, err := stopWalkWithout(ctx, open.inventory, walk.ID, "mesh-delivery for jochen", "its group stopped",
[]string{"novox/mesh-catalog@" + cat.Commit}, t0.Add(3*time.Minute+10*time.Second))
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(said.Kept, []inventory.PlanKept{{Module: "app", Machines: []string{"anchor"}}}) {
t.Fatalf("kept %+v; want app on anchor alone (one is walked again)", said.Kept)
}
p, _ := open.inventory.PlanByID(ctx, walk.ID)
if !reflect.DeepEqual(p.Delivery.Kept, said.Kept) || !strings.Contains(p.Note, "app on anchor") {
t.Fatalf("the stopped walk's record does not name what stays: %+v %q", p.Delivery.Kept, p.Note)
}
if len(said.StillCarriedBy) != 1 || !strings.Contains(said.StillCarriedBy[0], cat2.Commit) {
t.Fatalf("still carried by %v; want the later catalogue merge waiting in the batch", said.StillCarriedBy)
}
if _, err := retryPlan(ctx, open, walk.ID); err == nil || !strings.Contains(err.Error(), "was stopped without") {
t.Fatalf("a walk stopped without some merges was retried, or refused for another reason: %v", err)
}
}
// A batch not yet cut is stopped without a merge: the merge goes to a stopped record of its own, the batch goes on
// with the others and withholds the stopped merge's modules; with every merge left out, the batch is stopped.
func TestAStopWithoutAMergeLeavesItOutOfABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
_, bs := walks(t, open)
b := bs[0]
r, said, err := stopWalkWithout(ctx, open.inventory, b.ID, "mesh-delivery for jochen", "its group stopped",
[]string{"novox/mesh-catalog@" + cat.Commit}, t0.Add(10*time.Second))
if err != nil {
t.Fatal(err)
}
if said.WalkedAgainBy != b.ID || said.Stopped == b.ID {
t.Fatalf("said %+v; want a stopped record of its own, and the batch walking the others", said)
}
if left, _ := open.inventory.MergesOf(ctx, said.Stopped); len(left) != 1 || left[0].Commit != cat.Commit {
t.Fatalf("the stopped record holds %+v", left)
}
if stopped, _ := open.inventory.PlanByID(ctx, said.Stopped); stopped.State != inventory.PlanFailed ||
stopped.Delivery.Stopped == "" || stopped.Delivery.WalkedAgainBy != b.ID {
t.Fatalf("the stopped record reads %s %+v", stopped.State, stopped.Delivery)
}
_ = r
cutAt(t, open, t0.Add(3*time.Minute))
w, _ := open.inventory.PlanByID(ctx, b.ID)
if _, in := w.Modules["app"]; in || w.Batch() {
t.Fatalf("the batch's walk is %s and builds %v; want it cut without app", w.State, w.Modules)
}
// A batch whose every merge is left out is stopped itself.
hear(t, open, catalogueMerge("cat3eeeeeeee", "notes", t0.Add(4*time.Minute)), t0.Add(4*time.Minute))
_, bs = walks(t, open)
last := bs[0]
if _, said, err := stopWalkWithout(ctx, open.inventory, last.ID, "mesh-delivery for jochen", "x",
[]string{"novox/mesh-catalog@cat3eeeeeeee"}, t0.Add(4*time.Minute+time.Second)); err != nil || said.Stopped != last.ID {
t.Fatalf("a batch left with nothing: %+v %v", said, err)
}
if p, _ := open.inventory.PlanByID(ctx, last.ID); p.State != inventory.PlanFailed || p.Batch() {
t.Fatalf("the emptied batch is %s", p.State)
}
}
// The stopped walk, the batch and the merges moved are one act (ADR 0299 decision 7): a plan written by somebody
// else since it was read refuses all of it, and no merge moves, no plan is written.
func TestSavingPlansAndMovingMergesIsOneAct(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
endsItsLine(t)
ctx := t.Context()
hear(t, open, catalogueMerge("cat1aaaaaaaa", "app", t0), t0.Add(time.Second))
_, bs := walks(t, open)
stale := bs[0]
moved := stale
moved.Note = "written by somebody else"
if err := open.inventory.SavePlan(ctx, &moved); err != nil {
t.Fatal(err)
}
fresh := inventory.Plan{ID: "plan-fresh", Repository: "novox/mesh-catalog", Branch: "main", Commit: "x", Created: t0,
State: inventory.PlanAssembling, Tiers: [][]string{}, Modules: map[string]*inventory.PlanModule{}}
err := open.inventory.SavePlansMoving(ctx, []*inventory.Plan{&fresh, &stale},
[]inventory.MergeMove{{Repository: "novox/mesh-catalog", Commit: "cat1aaaaaaaa", Plan: "plan-fresh"}})
if !errors.Is(err, inventory.ErrPlanMoved) {
t.Fatalf("a stale plan in the act: %v", err)
}
if _, err := open.inventory.PlanByID(ctx, "plan-fresh"); err == nil {
t.Fatal("the new plan was written though the act was refused")
}
if m, _, _ := open.inventory.MergeOf(ctx, "novox/mesh-catalog", "cat1aaaaaaaa"); m.Plan != stale.ID {
t.Fatalf("the merge moved to %q though the act was refused", m.Plan)
}
}
// What a stopped walk leaves on the machines names every machine its sends reached (review of #232): a module sent
// with no first machine reached every machine running it; one sent first reached those, and the rest it was sent
// to; a module a merge walked again moves is sent again, and is not named.
func TestKeptNamesEveryMachineASendReached(t *testing.T) {
at := t0
p := inventory.Plan{Modules: map[string]*inventory.PlanModule{
"together": {SentAt: &at},
"first": {First: []string{"anchor"}, FirstAt: &at, Rest: map[string]inventory.SentDeclaration{"laptop": {}}},
"again": {SentAt: &at},
"unsent": {},
}}
got := keptOf(p, []string{"again"}, map[string][]string{"together": {"laptop", "anchor"}, "again": {"anchor"}})
want := []inventory.PlanKept{{Module: "first", Machines: []string{"anchor", "laptop"}},
{Module: "together", Machines: []string{"anchor", "laptop"}}}
if !reflect.DeepEqual(got, want) {
t.Fatalf("kept %+v; want %+v", got, want)
}
}
+102
View File
@@ -0,0 +1,102 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 529: a walk waiting because its delivery is held for a person said only "its walk waits for
// mesh-delivery's word — `plans go … --why …` starts it by hand", pointing a reader at a hand start instead of at
// the person's decision. The line names what mesh-delivery says holds the walk: the held delivery, its reason and
// `mesh-delivery.release`; or the delivery not at its turn.
func waitingWalk(now time.Time) inventory.Plan {
return inventory.Plan{ID: "plan-1791793277221379309", Repository: "novox/mesh-sdk", Branch: "main",
Commit: "9747f3e1b1ac", State: inventory.PlanBuilding, Tiers: [][]string{{"mesh-sdk"}},
Created: now.Add(-9 * time.Minute), Modules: map[string]*inventory.PlanModule{},
Delivery: &inventory.PlanDelivery{Awaits: "mesh-delivery"}}
}
func TestAWalkWaitingOnAHeldDeliveryNamesTheHold(t *testing.T) {
now := time.Date(2026, 10, 12, 8, 35, 0, 0, time.UTC)
p := waitingWalk(now)
held := "novox/mesh-sdk@d41d255f28ad held since 2026-10-12T08:23:00Z: its group adr-0307-state-history's composed " +
"check did not pass for the heads that merged — a person decides: `mesh-delivery.release` (id " +
"novox/mesh-sdk@d41d255f28ad, why) lets it go, `mesh-delivery.stop` ends it"
waits := walkWaits{p.ID: {{ID: "novox/mesh-sdk@d41d255f28ad", State: "held", Walk: p.ID, Says: held}}}
line := planLineSaid(p, now, pauseView{}, tierAtLeast, nil, waits)
for _, want := range []string{"held for a person", "composed check did not pass", "mesh-delivery.release"} {
if !strings.Contains(line, want) {
t.Fatalf("the line does not say %q: %q", want, line)
}
}
if strings.Contains(line, "starts it by hand") {
t.Fatalf("a held walk's line points at a hand start: %q", line)
}
// Told nothing — an older mesh-delivery, or none answering — it reads as before.
if line := planLineSaid(p, now, pauseView{}, tierAtLeast, nil, nil); !strings.Contains(line, "starts it by hand") {
t.Fatalf("told nothing, the line reads %q", line)
}
}
func TestAWalkWaitingOnADeliveryNotAtItsTurnSaysSo(t *testing.T) {
now := time.Date(2026, 10, 12, 8, 57, 0, 0, time.UTC)
p := waitingWalk(now)
waits := walkWaits{p.ID: {{ID: "novox/mesh-sdk@9747f3e1b1ac", State: "published", Walk: p.ID,
Says: "not let go yet: novox/mesh-tools@3f0c1a6cec18 (published) carried by it not at its turn"}}}
line := planLineSaid(p, now, pauseView{}, tierAtLeast, nil, waits)
if !strings.Contains(line, "not at its turn") || !strings.Contains(line, "novox/mesh-tools@3f0c1a6cec18") {
t.Fatalf("the line reads %q", line)
}
// No hand start offered: it would skip the group's order (review of issue 529).
if strings.Contains(line, "plans go") {
t.Fatalf("a walk not at its turn points at a hand start: %q", line)
}
}
// A walk nothing holds — the nzbgeek case, its merges' deliveries ended as nothing-to-move — is said so, and the hand
// start is the way on.
func TestAWalkHeldByNothingSaysWhyAndTheHandStart(t *testing.T) {
now := time.Date(2026, 10, 11, 20, 30, 0, 0, time.UTC)
p := waitingWalk(now)
waits := walkWaits{p.ID: {{ID: "novox/mesh-media-catalog@cdcdcdcd0000", State: "unheld", Walk: p.ID,
Says: "held by nothing: its merges' deliveries ended as nothing-to-move (novox/mesh-media-catalog@abababab0000), " +
"yet the walk moves nzbgeek, nzbget — nothing gives it its word"}}}
line := planLineSaid(p, now, pauseView{}, tierAtLeast, nil, waits)
if !strings.Contains(line, "ended as nothing-to-move") || !strings.Contains(line, "plans go "+p.ID) {
t.Fatalf("the line reads %q", line)
}
}
// What holds each waiting walk is asked of mesh-delivery's `stalled` with `walks`, a verb the controller's grant
// already names; only lines naming a walk are kept, by walk.
func TestWhatHoldsTheWaitingWalksIsAskedOfTheDeliveryOwner(t *testing.T) {
was := askDeliveryOwner
defer func() { askDeliveryOwner = was }()
var asked map[string]any
askDeliveryOwner = func(_ context.Context, _ *nats.Conn, verb string, args map[string]any) (json.RawMessage, error) {
if verb != "stalled" {
t.Fatalf("asked %s", verb)
}
asked = args
return json.Marshal([]stalledLine{{ID: "novox/a@1", State: "held", Walk: "plan-1", Says: "held: why"},
{ID: "novox/b@2", State: "held", Says: "a delivery past its bound, no walk"}})
}
got := walksWaiting(t.Context(), nil, true)
if asked["walks"] != true {
t.Fatalf("stalled was asked with %v", asked)
}
if len(got) != 1 || len(got["plan-1"]) != 1 || got["plan-1"][0].Says != "held: why" {
t.Fatalf("the waiting walks are %+v", got)
}
if got := walksWaiting(t.Context(), nil, false); got != nil {
t.Fatalf("with no holder on record it asked: %+v", got)
}
}
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"reflect"
"sort"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq issue 611: mesh-controller #293 changed one test file. Its delivery plan, made while another walk built
// build-agent and route-proxy (issue 606), read them whole and named them; at the cut their recorded build sources
// held none of its files, so its walk moved nothing and its record named no module. mesh-delivery could not tell
// "matched what runs" from "never built" and stopped the delivery, asking the operator about a change already on
// every machine. The walk now says, per module, that its build matched what runs (delivery.unchanged, ADR 0267 §5).
// The shape of #293: a test file of the controller's repository. The controller, built from the root, and the two
// modules packaging the repository each recorded a build source holding none of it: each is said unchanged, with
// why; a module of another repository is not named.
func TestAMergeSaysTheModulesWhoseBuildSourceItLeftUnchanged(t *testing.T) {
const catalogueRepo = "http://forge.internal:20000/novox/mesh-catalog.git"
const controllerRepo = "http://forge.internal:20000/novox/mesh-controller.git"
entries := []inventory.Entry{
fromRepo("mesh-controller", controllerRepo, ""),
fromRepo("build-agent", catalogueRepo, "modules/build-agent"),
fromRepo("route-proxy", catalogueRepo, "modules/route-proxy"),
fromRepo("gitea", catalogueRepo, "modules/gitea"),
}
read := map[string][]inventory.ReadRepository{
"mesh-controller": {{Own: true, Paths: []string{"go.mod", "module.json", "cmd/mesh-controller/", "internal/builder/"}}},
"build-agent": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"go.mod", "cmd/mesh-builder/", "internal/builder/"}},
{Own: true, Paths: []string{"modules/build-agent/Dockerfile", "modules/build-agent/module.json"}},
},
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"go.mod", "examples/route-proxy/"}},
{Own: true, Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
},
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main", Commit: "8f1aec0",
Paths: []string{"internal/builder/replay363_test.go"}}
if moved := reachOfMerge(m, entries, read, sharedRepositoryEdges).Moved(); len(moved) != 0 {
t.Fatalf("the fixture moves %v: a test file is in no build source", moved)
}
got := unchangedOfMerge(m, entries, read)
var names []string
for n, why := range got {
names = append(names, n)
if !strings.Contains(why, "holds none of") || !strings.Contains(why, "internal/builder/replay363_test.go") ||
!strings.Contains(why, "the build the mesh holds is its build") {
t.Errorf("%s is said unchanged as %q", n, why)
}
}
if want := []string{"build-agent", "mesh-controller", "route-proxy"}; !reflect.DeepEqual(sortedOf(names), want) {
t.Fatalf("said unchanged: %v; want %v", sortedOf(names), want)
}
// A module whose build source holds a changed file is moved, never said unchanged.
touched := m
touched.Paths = []string{"internal/builder/builder.go"}
if got := unchangedOfMerge(touched, entries, read); got["mesh-controller"] != "" || got["build-agent"] != "" {
t.Errorf("a module the merge moves is said unchanged: %v", got)
}
if got := unchangedOfMerge(touched, entries, read); got["route-proxy"] == "" {
t.Errorf("the route proxy, whose build source holds none of it, is not said unchanged: %v", got)
}
// With no build source recorded, the module is read whole and moved: nothing says it unchanged.
unsaid := map[string][]inventory.ReadRepository{"build-agent": {{Repository: "novox/mesh-controller", Ref: "main"}}}
if got := unchangedOfMerge(m, entries, unsaid); len(got) != 0 {
t.Errorf("modules with no recorded build source are said unchanged: %v", got)
}
// Files not all said: every module is moved, none said unchanged.
truncated := m
truncated.PathsTruncated = true
if got := unchangedOfMerge(truncated, entries, read); len(got) != 0 {
t.Errorf("a merge whose files were not all said says %v unchanged", got)
}
}
// A module the walk builds anyway — another merge's, or a dependent — is not said unchanged: the modules map keeps
// meaning what the walk builds, and the two never name one module.
func TestAWalkSaysUnchangedOnlyWhatItDoesNotBuild(t *testing.T) {
unchanged := map[string]string{"build-agent": "why a", "route-proxy": "why b"}
got := unchangedLeft(unchanged, map[string]*inventory.PlanModule{"build-agent": {}})
if !reflect.DeepEqual(got, map[string]string{"route-proxy": "why b"}) {
t.Fatalf("left %v", got)
}
if got := unchangedLeft(map[string]string{"x": "y"}, map[string]*inventory.PlanModule{"x": {}}); got != nil {
t.Fatalf("left %v; want none", got)
}
}
// Against the store, the path a real merge takes: a merge whose module recorded a build source that holds none
// of its files is cut into a walk done with nothing to walk, whose record names the module unchanged and why.
func TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns(t *testing.T) {
open := windowed(t)
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-one", Repository: "novox/one", Ref: "main",
Module: "one", Commit: "c0", On: "builder", Asked: t0.Add(-time.Hour),
Sources: []inventory.BuildSource{{Paths: []string{"main.go", "go.mod"}}}}); err != nil {
t.Fatal(err)
}
readme := repoMerge("one", "one1aaaaaaaa", t0)
readme.Paths = []string{"README.md"}
hear(t, open, readme, t0.Add(time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("walks: %+v", ws)
}
w := ws[0]
if w.State != inventory.PlanDone || len(w.Tiers) != 0 || len(w.Modules) != 0 {
t.Fatalf("the walk reads %s, tiers %v, modules %v; want done with nothing to walk", w.State, w.Tiers, w.Modules)
}
if w.Delivery == nil || !strings.Contains(w.Delivery.Unchanged["one"], "holds none of the 1 changed file(s)") {
t.Fatalf("the walk does not say one matched what runs: %+v", w.Delivery)
}
if !strings.Contains(w.Note, "unchanged") || !strings.Contains(w.Note, "one") {
t.Errorf("the walk's note does not name it: %q", w.Note)
}
// A merge that changes the build source is walked, and says nothing unchanged.
code := repoMerge("one", "one2bbbbbbbb", t0.Add(3*time.Minute))
hear(t, open, code, t0.Add(3*time.Minute+time.Second))
cutAt(t, open, t0.Add(6*time.Minute))
ws, _ = walks(t, open)
if len(ws) != 2 {
t.Fatalf("walks: %+v", ws)
}
if _, built := ws[0].Modules["one"]; !built || ws[0].Delivery == nil || len(ws[0].Delivery.Unchanged) != 0 {
t.Fatalf("the walk of a change to main.go reads modules %v, delivery %+v", ws[0].Modules, ws[0].Delivery)
}
}
func sortedOf(names []string) []string {
out := append([]string{}, names...)
sort.Strings(out)
return out
}
+1 -1
View File
@@ -77,7 +77,7 @@ func TestASendIsRecordedEvenWhenTheSenderIsBeingCancelled(t *testing.T) {
}
cancel() // the sender is going away: its context is cancelled between the send and the record
body := []byte(`{"declaration":1,"resources":[]}`)
digest, err := recordSent(ctx, inv, "anchor", body, nil, 0)
digest, err := recordSent(ctx, inv, "anchor", body, nil, 0, sentRecord{sender: "a test"})
if err != nil {
// NodeByName on the cancelled context may itself refuse; the record must still be possible
// through the detached context, so look the node up again on a live one.
+272
View File
@@ -0,0 +1,272 @@
package main
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/secrets"
)
// `secret issued <provision>`: the credential a provider's software issued, taken from the provider and kept as
// the provision's one credential for every consumer (novox/hq ADR 0305, issue 526).
//
// **The value comes from the provider, never from whoever asks.** A module served by its machine's runtime calls
// under the runtime's one bus account, which carries that machine's agents' calls too, so a value handed to a
// verb could be anybody's. This line carries none: it asks the provider's own tool, on the provider's machine, in
// the controller's own name — the subject that names its caller, which the bus lets the controller alone publish
// (issue 365) — with a sealing key made for this one call. The tool answers only the controller, the value sealed
// to that key; the controller opens it, seals it to every holder and forgets it, as the desk's value is (ADR 0277
// §23). Asking again fetches the same value again, and a value every holder already holds is not sealed anew.
//
// **What this trusts: the provider's machine.** The answer is taken from whoever answers the provider's tool
// subject on the provider's machine first. On that machine its runtime serves the subject for every module it
// carries, so anything that runtime runs, or anything else granted that subject there, could answer before the
// provider does and hand the controller a value of its own. ADR 0305 accepts that by trusting the provider's
// machine, as the provider itself is trusted; the bus keeps every other machine from answering it.
// handActIssued is the verb a change of an issued value is recorded under: the mesh working as decided.
const handActIssued = "secret issued"
// issuedAsk is how long the provider's tool is given to answer.
const issuedAsk = 20 * time.Second
// issuedProvider is one assignment of a module whose offer says its software issues the provision's credential.
type issuedProvider struct {
Node, Module, Tool string
}
// issuedTaker is the line's work, with what it reaches handed in: the store, the bus and the sends.
type issuedTaker struct {
// providers are the assignments to ask, and whether any module offers the provision so at all.
providers func(provision string) ([]issuedProvider, bool, error)
ask func(p issuedProvider, args map[string]any) (link.Answer, error)
keep func(provision, provider, value string) (inventory.IssuedKept, error)
record func(act link.HandAct) error
send func(machines []string) error
}
// take asks every provider of the provision for its software's value and keeps each; it answers what it did, in
// words that never carry the value or its fingerprint.
func (t issuedTaker) take(provision string) (string, error) {
providers, offered, err := t.providers(provision)
if err != nil {
return "", err
}
if !offered {
return "", fmt.Errorf("no module offers %q with a credential its software issues (\"credential\": "+
"{\"issued-by\": \"software\", \"tool\": …}, ADR 0305), so there is nothing to take from a provider; "+
"nothing was asked", provision)
}
if len(providers) == 0 {
return fmt.Sprintf("no consumer holds %s from a provider whose software issues it yet, so there is nothing "+
"to seal; a consumer that binds holds a value the mesh made until the provider next asks", provision), nil
}
var said []string
var failed []string
for _, p := range providers {
words, err := t.takeFrom(provision, p)
if err != nil {
failed = append(failed, fmt.Sprintf("%s on %s: %v", p.Module, p.Node, err))
continue
}
said = append(said, words)
}
out := strings.Join(said, "\n")
if len(failed) > 0 {
return out, fmt.Errorf("%s was not taken from every provider:\n %s", provision, strings.Join(failed, "\n "))
}
return out, nil
}
// takeFrom is one provider: asked, opened, kept, recorded and sent when it changed.
func (t issuedTaker) takeFrom(provision string, p issuedProvider) (string, error) {
public, private, err := secrets.Keypair()
if err != nil {
return "", fmt.Errorf("no key could be made to take the value: %w", err)
}
answer, err := t.ask(p, map[string]any{"provision": provision, "seal_to": public})
if err != nil {
return "", fmt.Errorf("its tool %s could not be asked: %w", p.Tool, err)
}
if answer.Error != "" {
return "", fmt.Errorf("its tool %s refused: %s", p.Tool, answer.Error)
}
var given struct {
Sealed string `json:"sealed"`
}
if err := unmarshalAnswer(answer, &given); err != nil || given.Sealed == "" {
return "", fmt.Errorf("its tool %s answered no sealed value", p.Tool)
}
opened, err := secrets.Open(private, given.Sealed)
if err != nil {
// Never the value, never what failed to open: only that it was not sealed to this call.
return "", fmt.Errorf("its tool %s answered a value not sealed to this call; nothing was taken", p.Tool)
}
value := strings.TrimSpace(string(opened))
for i := range opened {
opened[i] = 0
}
if value == "" {
return "", fmt.Errorf("its tool %s answered an empty value; nothing was taken", p.Tool)
}
kept, err := t.keep(provision, p.Node, value)
value = ""
if err != nil {
return "", err
}
holders := make([]string, 0, len(kept.Holders))
for _, h := range kept.Holders {
holders = append(holders, fmt.Sprintf("%s on %s%s", h.ConsumerModule, h.Consumer, asLocal(h.Local)))
}
words := ""
if len(kept.Unsealable) > 0 {
var names []string
for _, h := range kept.Unsealable {
names = append(names, fmt.Sprintf("%s on %s", h.ConsumerModule, h.Consumer))
}
words = fmt.Sprintf("\n not sealed to %s: no sealing key on that machine", strings.Join(names, ", "))
}
if !kept.Changed {
return fmt.Sprintf("%s from %s on %s: every holder already holds the value its software issued (%s); "+
"nothing was sent%s", provision, p.Module, p.Node, strings.Join(holders, ", "), words), nil
}
sealedTo := make([]string, 0, len(kept.Sealed))
for _, h := range kept.Sealed {
sealedTo = append(sealedTo, fmt.Sprintf("%s on %s%s", h.ConsumerModule, h.Consumer, asLocal(h.Local)))
}
recorded := ""
if err := t.record(link.HandAct{Verb: handActIssued, Args: []string{provision},
Why: fmt.Sprintf("%s on %s gave the credential its software issued for %s; sealed to %s", p.Module, p.Node,
provision, strings.Join(sealedTo, ", ")),
Cause: "issued-by-software"}); err != nil {
recorded = fmt.Sprintf("\n this was NOT recorded in the hand-act log, and is done anyway: %v", err)
}
if err := t.send(kept.Machines); err != nil {
return "", fmt.Errorf("%s from %s on %s is sealed to %s and could not be sent: %w; `push --behind` sends it",
provision, p.Module, p.Node, strings.Join(sealedTo, ", "), err)
}
return fmt.Sprintf("%s from %s on %s: the value its software issued is sealed to %s; sent %s, and each of "+
"them starts again on it%s%s", provision, p.Module, p.Node, strings.Join(sealedTo, ", "),
strings.Join(kept.Machines, ", "), words, recorded), nil
}
// issuedProvidersOf are the assignments to ask for a provision: every machine providing it to a pair the mesh
// holds, and on it the assigned module whose offer says its software issues the credential (ADR 0305 §1).
func issuedProvidersOf(ctx context.Context, inv *inventory.Inventory, provision string) ([]issuedProvider, bool, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, false, err
}
if !offersIssued(shelf, provision) {
return nil, false, nil
}
holders, err := inv.HoldersOf(ctx, provision, "")
if err != nil {
return nil, true, err
}
nodes := map[string]bool{}
for _, h := range holders {
nodes[h.Provider] = true
}
names := make([]string, 0, len(nodes))
for n := range nodes {
names = append(names, n)
}
sort.Strings(names)
var out []issuedProvider
for _, node := range names {
assigned, err := inv.Assigned(ctx, node)
if err != nil {
return nil, true, err
}
out = append(out, issuedOn(shelf, node, assigned, provision)...)
}
return out, true, nil
}
// offersIssued is whether any module offers the provision with a credential its software issues.
func offersIssued(shelf map[string]catalogue.Manifest, provision string) bool {
for _, m := range shelf {
if _, issued := m.IssuedCredentialOf(provision); issued {
return true
}
}
return false
}
// issuedOn are the modules assigned to one machine whose offer says their software issues the provision's
// credential, each with the tool that answers it.
func issuedOn(shelf map[string]catalogue.Manifest, node string, assigned []string, provision string) []issuedProvider {
var out []issuedProvider
for _, module := range assigned {
if tool, issued := shelf[module].IssuedCredentialOf(provision); issued {
out = append(out, issuedProvider{Node: node, Module: module, Tool: tool})
}
}
return out
}
// issuedRefusesRotation refuses `rotate` of a provision its provider's software issues (ADR 0305 §9): a value the
// mesh made would replace one the software holds, and every consumer would hold a dead credential until the
// provider next asks.
func issuedRefusesRotation(shelf map[string]catalogue.Manifest, provision string) error {
if !offersIssued(shelf, provision) {
return nil
}
return fmt.Errorf("%s's credential is issued by its provider's software, which the mesh cannot make or "+
"rotate (ADR 0305): change it in that software, and `secret issued %s` takes the value it holds now. "+
"Nothing was rotated", provision, provision)
}
// secretIssued is `secret issued <provision>`, on this controller's stores and bus.
func secretIssued(ctx context.Context, args []string) error {
if len(args) != 1 || args[0] == "" || strings.HasPrefix(args[0], "-") {
return errors.New("secret issued <provision>")
}
provision := args[0]
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
t := issuedTaker{
providers: func(provision string) ([]issuedProvider, bool, error) {
return issuedProvidersOf(ctx, open.inventory, provision)
},
ask: func(p issuedProvider, args map[string]any) (link.Answer, error) {
var answer link.Answer
err := onTheBus(func(conn *nats.Conn) error {
var err error
answer, err = link.AskModuleToolOnAs(ctx, conn, broker.ControllerName, p.Module, p.Tool, p.Node,
args, issuedAsk)
return err
})
return answer, err
},
keep: func(provision, provider, value string) (inventory.IssuedKept, error) {
return open.inventory.KeepIssuedSecret(ctx, provision, provider, value)
},
record: func(act link.HandAct) error {
return onTheBus(func(conn *nats.Conn) error {
_, err := link.RecordHandAct(ctx, conn, act)
return err
})
},
send: func(machines []string) error { return sendTo(ctx, open, machines) },
}
words, err := t.take(provision)
if words != "" {
fmt.Println(words)
}
return err
}
+218
View File
@@ -0,0 +1,218 @@
package main
import (
"encoding/json"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/secrets"
)
// The `issued` verb runs `secret issued <provision>` and nothing else, and that line is the one `secret` line
// beside rotate and the desk's that a verb may run (novox/hq ADR 0305 §3, §10, issue 526).
func TestTheIssuedVerbRunsTheOneLineThatCarriesNoValue(t *testing.T) {
argv, err := argvFor("issued", map[string]any{"provision": "plex-api"})
if err != nil {
t.Fatal(err)
}
if strings.Join(argv, " ") != "secret issued plex-api" {
t.Fatalf("the verb composed %q", argv)
}
if _, err := argvFor("issued", map[string]any{}); err == nil {
t.Fatal("the verb ran with no provision")
}
if _, err := argvFor("issued", map[string]any{"provision": "plex-api", "secret": "a value"}); err == nil {
t.Fatal("the verb took a value beside the provision")
}
for _, line := range [][]string{
{"secret", "issued", "plex-api", "--from", "/tmp/x"},
{"secret", "issued", "--from"},
{"secret", "issued"},
{"secret", "accept", "ace", "kometa", "plex-api", "--provider", "ace"},
{"secret", "recover", "ace", "kometa", "plex-api"},
} {
if err := terminalOnly(line); err == nil {
t.Errorf("%q ran through a verb", line)
}
}
if err := terminalOnly([]string{"secret", "issued", "plex-api"}); err != nil {
t.Fatalf("the issued line was refused through its verb: %v", err)
}
found := false
for _, v := range catalogue.ControllerVerbs {
if v.Name == "issued" {
found = true
}
}
if !found {
t.Fatal("the controller's seat does not declare the issued verb")
}
}
// A provider's tool answering the value sealed to the call's key: what a provider does (ADR 0305 §2).
func issuedSealed(t *testing.T, value string) func(issuedProvider, map[string]any) (link.Answer, error) {
return func(_ issuedProvider, args map[string]any) (link.Answer, error) {
key, _ := args["seal_to"].(string)
if key == "" {
t.Fatal("the provider was asked without a key to seal to")
}
if _, carried := args["value"]; carried {
t.Fatal("the ask carried a value")
}
sealed, err := secrets.Seal(key, []byte(value))
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(map[string]string{"sealed": sealed})
return link.Answer{Result: raw}, nil
}
}
type issuedRecorder struct {
kept []string
acts []link.HandAct
sent [][]string
again bool // the value is held by every holder already
}
func (r *issuedRecorder) taker(t *testing.T, ask func(issuedProvider, map[string]any) (link.Answer, error)) issuedTaker {
return issuedTaker{
providers: func(provision string) ([]issuedProvider, bool, error) {
return []issuedProvider{{Node: "ace", Module: "plex", Tool: "plex_issued_credential"}}, true, nil
},
ask: ask,
keep: func(provision, provider, value string) (inventory.IssuedKept, error) {
r.kept = append(r.kept, value)
holders := []inventory.Holder{{Provision: provision, Consumer: "ace", ConsumerModule: "kometa", Provider: provider},
{Provision: provision, Consumer: "ace", ConsumerModule: "tautulli", Provider: provider}}
if r.again {
return inventory.IssuedKept{Holders: holders}, nil
}
return inventory.IssuedKept{Changed: true, Holders: holders, Sealed: holders, Machines: []string{"ace"}}, nil
},
record: func(act link.HandAct) error { r.acts = append(r.acts, act); return nil },
send: func(machines []string) error { r.sent = append(r.sent, machines); return nil },
}
}
// The value the provider's tool sealed to the call is kept, recorded and sent, and the answer never says it.
func TestAnIssuedValueIsTakenSealedFromTheProviderAndSent(t *testing.T) {
var r issuedRecorder
words, err := r.taker(t, issuedSealed(t, "the-servers-own-token")).take("plex-api")
if err != nil {
t.Fatal(err)
}
if len(r.kept) != 1 || r.kept[0] != "the-servers-own-token" {
t.Fatalf("the provider's value was not kept: %d kept", len(r.kept))
}
if len(r.sent) != 1 || strings.Join(r.sent[0], ",") != "ace" {
t.Fatalf("the holders' machines were not sent: %v", r.sent)
}
if len(r.acts) != 1 || r.acts[0].Verb != handActIssued {
t.Fatalf("the change was not recorded in the hand-act log: %+v", r.acts)
}
for _, said := range []string{words, r.acts[0].Why, strings.Join(r.acts[0].Args, " ")} {
if strings.Contains(said, "the-servers-own-token") || strings.Contains(said, inventory.IssuedStamp("the-servers-own-token")) {
t.Fatalf("the value or its fingerprint was said: %q", said)
}
}
if !strings.Contains(words, "kometa on ace") || !strings.Contains(words, "tautulli on ace") {
t.Fatalf("the answer does not name the holders: %q", words)
}
}
// The same value again writes nothing, records nothing and sends nobody (ADR 0305 §5).
func TestAnIssuedValueEveryHolderHoldsSendsNobody(t *testing.T) {
r := issuedRecorder{again: true}
words, err := r.taker(t, issuedSealed(t, "the-servers-own-token")).take("plex-api")
if err != nil {
t.Fatal(err)
}
if len(r.sent) != 0 || len(r.acts) != 0 {
t.Fatalf("an unchanged value was sent or recorded: %v %v", r.sent, r.acts)
}
if !strings.Contains(words, "nothing was sent") {
t.Fatalf("the answer does not say nothing was sent: %q", words)
}
}
// A value not sealed to the call's key is not taken, nor is a refusal, nor an empty answer (ADR 0305 §2).
func TestOnlyAValueSealedToTheCallIsTaken(t *testing.T) {
other, _, _ := secrets.Keypair()
elsewhere, _ := secrets.Seal(other, []byte("somebody's value"))
raw, _ := json.Marshal(map[string]string{"sealed": elsewhere})
plain, _ := json.Marshal(map[string]string{"value": "the-servers-own-token"})
for name, ask := range map[string]func(issuedProvider, map[string]any) (link.Answer, error){
"sealed to another key": func(issuedProvider, map[string]any) (link.Answer, error) { return link.Answer{Result: raw}, nil },
"answered in the clear": func(issuedProvider, map[string]any) (link.Answer, error) { return link.Answer{Result: plain}, nil },
"refused": func(issuedProvider, map[string]any) (link.Answer, error) {
return link.Answer{Error: "the controller alone"}, nil
},
"not reached": func(issuedProvider, map[string]any) (link.Answer, error) {
return link.Answer{}, errors.New("503 no responders")
},
"sealed empty": issuedSealed(t, " "),
} {
var r issuedRecorder
if _, err := r.taker(t, ask).take("plex-api"); err == nil {
t.Errorf("%s: taken", name)
}
if len(r.kept) != 0 || len(r.sent) != 0 {
t.Errorf("%s: something was kept or sent", name)
}
}
}
// A provision no offer says its software issues is refused before anything is asked.
func TestAProvisionNoOfferSaysIsIssuedIsRefused(t *testing.T) {
asked := false
taker := issuedTaker{
providers: func(string) ([]issuedProvider, bool, error) { return nil, false, nil },
ask: func(issuedProvider, map[string]any) (link.Answer, error) {
asked = true
return link.Answer{}, nil
},
}
if _, err := taker.take("postgres-database"); err == nil || !strings.Contains(err.Error(), "ADR 0305") {
t.Fatalf("expected a refusal naming the record, got %v", err)
}
if asked {
t.Fatal("a provider was asked for a provision no offer says its software issues")
}
}
// Which assignments are asked: the modules on the providing machine whose offer says so, with their tool.
func TestTheProvidersAskedAreTheOffersThatSaySo(t *testing.T) {
shelf := map[string]catalogue.Manifest{
"plex": {Module: "plex", Provides: []catalogue.Offer{{Name: "plex-api", Credential: &catalogue.OfferCredential{
IssuedBy: catalogue.IssuedBySoftware, Tool: "plex_issued_credential"}}}},
"jellyfin": {Module: "jellyfin", Provides: []catalogue.Offer{{Name: "plex-api"}}},
}
got := issuedOn(shelf, "ace", []string{"jellyfin", "plex", "kometa"}, "plex-api")
if len(got) != 1 || got[0] != (issuedProvider{Node: "ace", Module: "plex", Tool: "plex_issued_credential"}) {
t.Fatalf("the providers asked: %+v", got)
}
if err := issuedRefusesRotation(shelf, "plex-api"); err == nil || !strings.Contains(err.Error(), "secret issued plex-api") {
t.Fatalf("rotate of an issued provision was not refused with the way to take it: %v", err)
}
if err := issuedRefusesRotation(shelf, "postgres-database"); err != nil {
t.Fatalf("rotate of an ordinary provision was refused: %v", err)
}
}
// A change of an issued value is the mesh working as decided, never a repair (ADR 0305 §8).
func TestAnIssuedValueIsADecisionInTheHandActLog(t *testing.T) {
for _, v := range handActVerbs {
if v.Verb == handActIssued {
if v.Decision == "" {
t.Fatal("secret issued counts as a repair")
}
return
}
}
t.Fatal("secret issued is not in the hand-act table")
}
+160
View File
@@ -0,0 +1,160 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 485: the mesh records every send — sender, lease epoch, generation, modules — and only `plan`
// said it. `status` says per machine when it was last sent a declaration and by whom, in one line; `node show`
// says the last send in full and what the machine answered; and `status --json` carries it as fields, since
// the store's reader may not read the controller's tables and these verbs are the only window onto them.
func aLastSend(answer inventory.SendAnswer) inventory.Send {
sent := time.Date(2026, 10, 11, 18, 40, 5, 0, time.Local)
return inventory.Send{NodeName: "laptop", Sequence: 12, Epoch: 57,
Sender: "a one-shot push by jochen at a shell on anchor", Generation: 40, Digest: "d12",
Modules: []string{"docker", "sudo"}, SentAt: sent, Recorded: true, Answer: answer}
}
func TestStatusSaysEachMachinesLastSendInOneLine(t *testing.T) {
at := time.Date(2026, 10, 11, 18, 41, 0, 0, time.Local)
asked := answers{
nodes: []inventory.Node{{Name: "anchor", LastSeen: time.Now()}, {Name: "laptop", LastSeen: time.Now()}},
lastSent: map[string]inventory.Send{"laptop": aLastSend(inventory.SendAnswer{
Outcome: inventory.OutcomeApplied, At: &at})},
}
shown := printed(t, func() error { return printStatus(asked) })
var laptop, anchor []string
for _, line := range strings.Split(shown, "\n") {
fields := strings.Fields(line)
if len(fields) > 0 && fields[0] == "laptop" {
laptop = append(laptop, line)
}
if len(fields) > 0 && fields[0] == "anchor" {
anchor = append(anchor, line)
}
}
if len(laptop) != 1 || !strings.Contains(laptop[0], "2026-10-11 18:40") ||
!strings.Contains(laptop[0], "by a one-shot push by jochen at a shell on anchor") ||
!strings.Contains(laptop[0], "generation 40") || !strings.Contains(laptop[0], "applied") {
t.Fatalf("status does not say laptop's last send in one line:\n%s", shown)
}
if len(anchor) != 1 || !strings.Contains(anchor[0], "no send recorded") {
t.Fatalf("status does not say anchor has no send recorded, in one line:\n%s", shown)
}
}
func TestStatusSaysTheLastSendsCouldNotBeRead(t *testing.T) {
asked := answers{nodes: []inventory.Node{{Name: "laptop", LastSeen: time.Now()}},
lastSentUnread: "the store went away"}
if shown := printed(t, func() error { return printStatus(asked) }); !strings.Contains(shown, "the store went away") {
t.Fatalf("a record of sends that could not be read is not said:\n%s", shown)
}
if asked.well() {
t.Error("a mesh whose sends could not be read is called well")
}
}
func TestNodeShowSaysTheLastSendInFull(t *testing.T) {
at := time.Date(2026, 10, 11, 18, 41, 0, 0, time.Local)
refused := strings.Join(lastSendLines(aLastSend(inventory.SendAnswer{Outcome: inventory.OutcomeRefused,
Refused: "unknown field \"generation\"\nsecond line", At: &at})), "\n")
for _, want := range []string{"sequence 12", "2026-10-11 18:40:05", "a one-shot push by jochen at a shell on anchor",
"lease epoch 57", "generation 40", "docker, sudo", "refused", "unknown field \"generation\""} {
if !strings.Contains(refused, want) {
t.Errorf("node show's last send does not say %q:\n%s", want, refused)
}
}
if applied := strings.Join(lastSendLines(aLastSend(inventory.SendAnswer{Outcome: inventory.OutcomeApplied,
At: &at})), "\n"); !strings.Contains(applied, "applied it at 2026-10-11 18:41:00") {
t.Errorf("an applied send is not said applied:\n%s", applied)
}
if waiting := strings.Join(lastSendLines(aLastSend(inventory.SendAnswer{})), "\n"); !strings.Contains(waiting,
"not reported on it yet") {
t.Errorf("a send not reported on is not said so:\n%s", waiting)
}
stale := aLastSend(inventory.SendAnswer{})
when := at
stale.RefusedAt, stale.RefusedApplied = &when, 44
if s := strings.Join(lastSendLines(stale), "\n"); !strings.Contains(s, "refused it at 2026-10-11 18:41:00") ||
!strings.Contains(s, "generation 44") {
t.Errorf("a send refused for its generation is not said so:\n%s", s)
}
}
func TestStatusJSONCarriesEachMachinesLastSend(t *testing.T) {
at := time.Date(2026, 10, 11, 18, 41, 0, 0, time.UTC)
asked := answers{nodes: []inventory.Node{{Name: "anchor"}, {Name: "laptop"}},
lastSent: map[string]inventory.Send{"laptop": aLastSend(inventory.SendAnswer{Outcome: inventory.OutcomeRefused,
Refused: "unknown field", At: &at})}}
body, err := statusAsJSON(asked)
if err != nil {
t.Fatal(err)
}
var got struct {
LastSent []map[string]any `json:"lastSent"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
if len(got.LastSent) != 1 {
t.Fatalf("status --json carries %d last sends, want laptop's only:\n%s", len(got.LastSent), body)
}
s := got.LastSent[0]
if s["node"] != "laptop" || s["sequence"] != float64(12) || s["epoch"] != float64(57) ||
s["generation"] != float64(40) || s["sender"] != "a one-shot push by jochen at a shell on anchor" ||
s["digest"] != "d12" || s["sentAt"] == nil || s["answer"] != "refused" || s["refused"] != "unknown field" ||
s["answeredAt"] == nil {
t.Fatalf("laptop's last send reads %v", s)
}
if modules, _ := s["modules"].([]any); len(modules) != 2 || modules[0] != "docker" {
t.Fatalf("the modules it named read %v", s["modules"])
}
}
// And through the store: a send recorded and refused by the machine is what `node show` and `status --json`
// read back.
func TestNodeShowAndStatusReadTheLastSendFromTheStore(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSentWith(ctx, record.ID, "d12", nil, 57, 40, inventory.Send{Sequence: 12,
Epoch: 57, Sender: "a one-shot push by jochen at a shell on anchor", Generation: 40, Digest: "d12",
Modules: []string{"docker", "sudo"}}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.RecordDoing(ctx, record.ID, inventory.Doing{Outcome: inventory.OutcomeRefused,
Declared: "d12", Refused: "unknown field \"generation\""}); err != nil {
t.Fatal(err)
}
shown := printed(t, func() error { return showNode(ctx, open, "laptop") })
for _, want := range []string{"a one-shot push by jochen at a shell on anchor", "lease epoch 57", "generation 40",
"docker, sudo", "unknown field \"generation\""} {
if !strings.Contains(shown, want) {
t.Errorf("node show laptop does not say %q:\n%s", want, shown)
}
}
body := printed(t, func() error { return statusFor(ctx, open, true) })
var got struct {
LastSent []struct {
Node string `json:"node"`
Sender string `json:"sender"`
Answer string `json:"answer"`
} `json:"lastSent"`
}
if err := json.Unmarshal([]byte(body), &got); err != nil {
t.Fatalf("%v:\n%s", err, body)
}
if len(got.LastSent) != 1 || got.LastSent[0].Node != "laptop" || got.LastSent[0].Answer != "refused" ||
got.LastSent[0].Sender != "a one-shot push by jochen at a shell on anchor" {
t.Fatalf("status --json's last sends read %+v", got.LastSent)
}
}
+11 -4
View File
@@ -55,8 +55,14 @@ func TestTwoControllersOneActs(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
// Each renews five times and looks again ten times as often as a serving controller does, so the
// handover and the loss below are seen in a second, not in a renewal of five (novox/hq issue 591): what
// is checked is the order of events, which does not change with the interval. Each renewal still has a
// serving controller's five seconds to answer (acting.go), so a slow bus does not lose A the lease.
const renewEvery, poll = lease.RenewEvery / 5, lease.Poll / 10
// Controller A takes the lease.
a := &actor{}
a := &actor{renewEvery: renewEvery, poll: poll}
aCtx, stopA := context.WithCancel(ctx)
defer stopA()
lostA, err := a.serveUnderTheLease(aCtx, inv, url)
@@ -69,7 +75,7 @@ func TestTwoControllersOneActs(t *testing.T) {
}
// Controller B starts while A holds it, and waits — acting on nothing meanwhile.
b := &actor{}
b := &actor{renewEvery: renewEvery, poll: poll}
bCtx, stopB := context.WithCancel(ctx)
defer stopB()
tookB := make(chan (<-chan struct{}), 1)
@@ -82,10 +88,11 @@ func TestTwoControllersOneActs(t *testing.T) {
}
tookB <- lost
}()
// B looks fifteen times, across one of A's renewals at least.
select {
case <-tookB:
t.Fatal("B took the lease while A held it")
case <-time.After(3 * time.Second):
case <-time.After(15 * poll):
}
if _, err := b.epoch(ctx); !errors.Is(err, lease.ErrNotHeld) {
t.Fatalf("B, waiting, may act: %v", err)
@@ -125,7 +132,7 @@ func TestTwoControllersOneActs(t *testing.T) {
}
select {
case <-lostB:
case <-time.After(2 * lease.RenewEvery):
case <-time.After(2 * renewEvery):
t.Fatal("B was not told it lost the lease")
}
if _, err := b.epoch(ctx); !errors.Is(err, lease.ErrNotHeld) {
+222
View File
@@ -0,0 +1,222 @@
package main
import (
"errors"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
// A module assigned to a machine and left out of its declaration is said (novox/hq issue 380).
//
// **An omission is a finding, never a refusal of the push** (ADR 0163, rule 6): the machine is sent everything
// else, and the module's held things are kept. Until issue 380 the only place it showed was `plan`: nfs-server was
// assigned to the home server for days, every send left it out for a setting nobody gave, and the operator believed
// it ran. So the self-check (D1) judges every machine as the next send would (Resolution.LeftOutBecause, the send's
// own judgement) and raises a condition on that machine for each module it leaves out:
//
// - a setting nobody gave (catalogue.UnsetSettingError) is the operator's to give: kind needs-operator, naming
// the module, the setting and the command that sets it;
// - any other cause is said as a module not working is: kind left-out, a warning with the reason as evidence.
//
// Never urgent and never escalated by age (as ADR 0283 decision 5): nothing the machine ran was undone. It clears
// on the first run that no longer finds it — the module composed again, or no longer assigned. `status` and
// `node show` list the same modules under "assigned, not applied" with the same reason.
//
// A module left out because its stored manifest has a key this controller does not know is not raised here: the
// catalogue's own unknown-field condition says it once for the whole mesh (ADR 0262); it is still listed.
const (
// probeLeftOutID is the self-check's probe that raises and clears these conditions: D1, which already
// resolves every machine (probeDeclarations), so the judgement costs no resolution of its own.
probeLeftOutID = "D1"
// kindLeftOut is a module left out for any cause but a setting nobody gave; it is also every such condition's
// token, whichever its kind, so a cause that changes is the same condition said anew.
kindLeftOut = "left-out"
)
// leftOutModule is one module of a machine's set that its declaration leaves out, and why.
type leftOutModule struct {
Module string
// Setting is the setting nobody gave, when that is the cause.
Setting string
// Why is the declaration's own reason, whole.
Why string
// Unread is a stored manifest this controller cannot read whole (said by the catalogue's condition).
Unread bool
}
// leftOutOf is every module of a machine's resolution that a push would leave out, sorted. Pure: the judgement
// the push makes (catalogue.Resolution.LeftOutBecause), nothing allocated.
func leftOutOf(plan catalogue.Resolution, settings catalogue.SettingsBy, adopted bool) []leftOutModule {
because := plan.LeftOutBecause(settings, adopted)
out := make([]leftOutModule, 0, len(because))
for module, why := range because {
l := leftOutModule{Module: module, Why: oneLine(why.Error())}
var unset *catalogue.UnsetSettingError
var unread *catalogue.UnreadManifestError
switch {
case errors.As(why, &unset):
l.Setting = unset.Setting
case errors.As(why, &unread):
l.Unread = true
}
out = append(out, l)
}
sort.Slice(out, func(i, j int) bool { return out[i].Module < out[j].Module })
return out
}
// settingCommand is the command that gives a module's setting on one machine.
func settingCommand(module, setting, node string) string {
return fmt.Sprintf("`settings set %s '{%q: …}' --node %s`", module, setting, node)
}
// reason is why a module is left out, as `status`, `node show` and the condition's summary say it: for a setting
// nobody gave, the setting and the command that gives it; otherwise the declaration's own words.
func (l leftOutModule) reason(node string) string {
if l.Setting != "" {
return fmt.Sprintf("nothing sets its setting %q, so every send leaves it out of its declaration — %s sets it", l.Setting,
settingCommand(l.Module, l.Setting, node))
}
return "every send leaves it out of its declaration: " + l.Why
}
// leftOutObservations are the conditions a machine's left-out modules raise, one each.
func leftOutObservations(node string, left []leftOutModule) []conditions.Observation {
var out []conditions.Observation
for _, l := range left {
if l.Unread {
continue
}
out = append(out, leftOutObservation(node, l))
}
return out
}
// leftOutObservation is one module left out of one machine's declaration: needs-operator for a setting nobody
// gave, left-out otherwise; a warning either way, the operator's to resolve.
func leftOutObservation(node string, l leftOutModule) conditions.Observation {
o := conditions.Observation{Scope: conditions.ScopeModule, ID: l.Module + "." + node, Token: kindLeftOut,
Kind: kindLeftOut, Machine: node, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s is assigned to %s and not applied: %s", l.Module, node, l.reason(node)),
Said: l.Why}
w := leftOutWords(l.Module, node, l.Setting)
if l.Setting != "" {
o.Kind = kindNeedsOperator
} else {
// The words of why may name a path or an address, which the operator's channel withholds: the
// summary sends them to the evidence, and `plan` says them in full.
o.Summary = fmt.Sprintf("%s is assigned to %s and not applied: every send leaves it out of its declaration, "+
"because what is set for it does not fit its manifest — the evidence and `plan %s` say why", l.Module, node, node)
}
o.Headline, o.Explanation, o.Needs, o.Resolved = w.Headline, w.Explanation, w.Needs, w.Resolved
return o
}
// leftOutWords is what the operator reads of a module left out (ADR 0253): plain, the act named.
func leftOutWords(module, node, setting string) words {
w := words{
Headline: fmt.Sprintf("%s is not applied on %s", module, node),
Explanation: fmt.Sprintf("%s is assigned to %s, and every send to %s leaves it out of the declaration because what "+
"is set for it does not fit its manifest. Nothing of it changes there; the rest of %s is sent as usual.",
module, node, node, node),
Needs: fmt.Sprintf("read why in the details, then change what is set for %s or unassign it.", module),
Resolved: fmt.Sprintf("%s on %s is no longer left out", module, node),
}
if setting != "" {
w.Explanation = fmt.Sprintf("%s is assigned to %s, and every send to %s leaves it out of the declaration because "+
"nothing sets its setting %s. Nothing of it runs there until it is set; the rest of %s is sent as usual.",
module, node, node, setting, node)
w.Needs = fmt.Sprintf("set %s for %s on %s, or approve it when it is proposed to you.", setting, module, node)
// A setting's name that is not plain (a dotted key) is in the summary instead.
if _, ok := conditions.PlainWords(w, node); !ok {
w.Explanation = fmt.Sprintf("%s is assigned to %s, and every send to %s leaves it out of the declaration because a "+
"setting it needs is not set. Nothing of it runs there until it is set; the details name it.",
module, node, node)
w.Needs = fmt.Sprintf("set what %s needs on %s; the details name the setting.", module, node)
}
}
return w
}
// notAppliedLines is a machine's "assigned, not applied" as `node show` prints it: one line a module, with the
// reason its condition says.
func notAppliedLines(node string, left []leftOutModule) []string {
if len(left) == 0 {
return nil
}
lines := []string{"", " assigned, not applied:"}
for _, l := range left {
lines = append(lines, fmt.Sprintf(" %-22s %s", l.Module, l.reason(node)))
}
return lines
}
// machineNotApplied is one module assigned to a machine and left out of its declaration, in `status --json`.
type machineNotApplied struct {
Node string `json:"node"`
Module string `json:"module"`
Setting string `json:"setting,omitempty"`
Reason string `json:"reason"`
// UnreadManifest is a module left out because this controller cannot read its manifest whole: it raises no
// condition of its own on the machine, since the catalogue's unknown-field condition says it once (ADR 0262).
UnreadManifest bool `json:"unread-manifest,omitempty"`
}
// notApplied is every machine's left-out modules, in a stated order, as `status --json` carries them.
func notApplied(left map[string][]leftOutModule) []machineNotApplied {
names := make([]string, 0, len(left))
for name := range left {
names = append(names, name)
}
sort.Strings(names)
var out []machineNotApplied
for _, name := range names {
for _, l := range left[name] {
out = append(out, machineNotApplied{Node: name, Module: l.Module, Setting: l.Setting, Reason: l.reason(name),
UnreadManifest: l.Unread})
}
}
return out
}
// printNotApplied is status's "assigned, not applied", every machine's.
func printNotApplied(left map[string][]leftOutModule) {
rows := notApplied(left)
if len(rows) == 0 {
return
}
unread := 0
for _, r := range rows {
if r.UnreadManifest {
unread++
}
}
// Which raise a condition is said, not implied (review of #223): one whose manifest this controller cannot
// read is listed here and raises none of its own on the machine — the catalogue's condition says it.
raises := "each raises a condition on its machine"
switch {
case unread == len(rows):
raises = "none raises a condition on its machine: this controller cannot read their manifests, which the " +
"catalogue's own condition says"
case unread > 0:
raises += fmt.Sprintf(", except the %d whose manifest this controller cannot read, which the catalogue's own "+
"condition says", unread)
}
fmt.Printf("%d module(s) assigned, not applied — every send leaves them out of their declaration; %s:\n",
len(rows), raises)
for _, r := range rows {
fmt.Printf(" %-12s %-22s %s\n", r.Node, r.Module, r.Reason)
}
fmt.Println()
}
// isLeftOutCondition is whether a condition is a module left out of its declaration, which the machine's health
// statements neither raise nor clear: by its token, which every one carries whatever its kind.
func isLeftOutCondition(c conditions.Condition) bool {
return c.Subject.Scope == conditions.ScopeModule && strings.HasSuffix(c.Key, "."+kindLeftOut)
}
+259
View File
@@ -0,0 +1,259 @@
package main
import (
"context"
"encoding/json"
"os"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq issue 380: nfs-server was assigned to the home server for days and every send left it out — "nfs-server
// has a file that says ${setting:shares}, and nothing sets shares for it" — and nothing but `plan` said so. The
// manifest is the catalogue's own at the commit that added it (mesh-catalog 48fba44), which still says
// ${setting:shares}; the reason below is the one the live push printed.
// leftOutNFS is the resolution of a machine assigned that nfs-server, with the settings given.
func leftOutNFS(t *testing.T) catalogue.Resolution {
t.Helper()
raw, err := os.ReadFile("testdata/left-out/nfs-server.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
return catalogue.Resolution{Modules: []catalogue.Manifest{m}}
}
// sharesGiven is the operator's setting for it on the machine.
func sharesGiven(value string) catalogue.SettingsBy {
return catalogue.SettingsBy{"nfs-server": {{From: "anchor", Values: map[string]any{"shares": value}}}}
}
func TestAModuleLeftOutForASettingNobodyGaveNeedsTheOperatorNamingTheSettingAndTheCommand(t *testing.T) {
left := leftOutOf(leftOutNFS(t), nil, false)
if len(left) != 1 || left[0].Module != "nfs-server" || left[0].Setting != "shares" {
t.Fatalf("left out: %+v", left)
}
if !strings.Contains(left[0].Why, `nothing sets "shares" for it`) {
t.Fatalf("the reason is not the push's own: %s", left[0].Why)
}
obs := leftOutObservations("anchor", left)
if len(obs) != 1 {
t.Fatalf("raised %+v", obs)
}
o := obs[0]
if o.Key() != "module.nfs-server.anchor.left-out" || o.Kind != kindNeedsOperator || o.Machine != "anchor" ||
o.Severity != conditions.Warning || o.Resolver != conditions.ResolverOperator {
t.Fatalf("raised %s as %s, %s, by %s, on %q", o.Key(), o.Kind, o.Severity, o.Resolver, o.Machine)
}
for _, want := range []string{"nfs-server", "anchor", `"shares"`, "`settings set nfs-server '{\"shares\": …}' --node anchor`"} {
if !strings.Contains(o.Summary, want) {
t.Errorf("the summary does not name %s: %s", want, o.Summary)
}
}
if o.Said != left[0].Why {
t.Errorf("the evidence is not the reason the send gives: %s", o.Said)
}
plainExample(t, o, "nfs-server is not applied on anchor",
"Needs you: set shares for nfs-server on anchor, or approve it when it is proposed to you. nfs-server is assigned to "+
"anchor, and every send to anchor leaves it out of the declaration because nothing sets its setting shares. "+
"Nothing of it runs there until it is set; the rest of anchor is sent as usual.")
}
func TestAModuleLeftOutForAnotherCauseIsAWarningWithTheReasonAsEvidence(t *testing.T) {
// A setting stored that its manifest can no longer take: one with a line break (issue 339).
left := leftOutOf(leftOutNFS(t), sharesGiven("library=/srv/library\nmedia=/srv/media"), false)
if len(left) != 1 || left[0].Setting != "" {
t.Fatalf("left out: %+v", left)
}
obs := leftOutObservations("anchor", left)
if len(obs) != 1 {
t.Fatalf("raised %+v", obs)
}
o := obs[0]
if o.Key() != "module.nfs-server.anchor.left-out" || o.Kind != kindLeftOut || o.Severity != conditions.Warning {
t.Fatalf("raised %s as %s, %s", o.Key(), o.Kind, o.Severity)
}
if !strings.Contains(o.Said, "holds a line break") || strings.Contains(o.Summary, "/srv/") {
t.Fatalf("the reason is not the evidence, or the summary carries it to the channel:\n%s\n%s", o.Summary, o.Said)
}
plainExample(t, o, "nfs-server is not applied on anchor",
"Needs you: read why in the details, then change what is set for nfs-server or unassign it. nfs-server is assigned to "+
"anchor, and every send to anchor leaves it out of the declaration because what is set for it does not fit "+
"its manifest. Nothing of it changes there; the rest of anchor is sent as usual.")
}
// The self-check raises it, keeps it a warning however long it stands, and clears it when the module composes
// again or is no longer assigned; a machine's health statement neither clears nor raises it.
func TestALeftOutModulesConditionClearsWhenItComposesAgainOrIsUnassigned(t *testing.T) {
plan, settings := leftOutNFS(t), catalogue.SettingsBy(nil)
withProbes(t, probe{ID: probeLeftOutID, Asserts: "the test's", Kind: kindLeftOut, Phase: 1,
run: func(context.Context, *doctor) ([]conditions.Observation, error) {
return leftOutObservations("anchor", leftOutOf(plan, settings, false)), nil
}})
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store})
defer k.Close(context.Background())
d := &doctor{keeper: k, teller: &conditions.Told{}, host: "anchor"}
key := "module.nfs-server.anchor.left-out"
openOnes := func() map[string]conditions.Condition {
t.Helper()
open, err := k.Open(t.Context())
if err != nil {
t.Fatal(err)
}
out := map[string]conditions.Condition{}
for _, c := range open {
out[c.Key] = c
}
return out
}
d.runOnce(t.Context(), "a test")
c, raised := openOnes()[key]
if !raised || c.Kind != kindNeedsOperator || c.Severity != conditions.Warning {
t.Fatalf("a module left out raised %+v", openOnes())
}
// A statement from the machine that says nothing of it — the module runs nothing there — leaves it open.
if err := judgeModuleHealth(t.Context(), nil, k, "anchor", map[string][]inventory.ResourceHealth{}, nil,
time.Now().Add(72*time.Hour)); err != nil {
t.Fatal(err)
}
if _, still := openOnes()[key]; !still {
t.Fatal("a health statement that says nothing of the module cleared its left-out condition")
}
d.runOnce(t.Context(), "a test")
if c := openOnes()[key]; c.Severity != conditions.Warning {
t.Fatalf("after a health statement and days, it is %+v", openOnes())
}
// The setting given: it composes, and the condition clears.
settings = sharesGiven("library=/srv/library")
d.runOnce(t.Context(), "a test")
if _, still := openOnes()[key]; still {
t.Fatalf("composed again, still open: %+v", openOnes())
}
// Left out again, then unassigned: no longer in the machine's set, and it clears.
settings = nil
d.runOnce(t.Context(), "a test")
if _, raised := openOnes()[key]; !raised {
t.Fatal("left out again and not raised")
}
plan = catalogue.Resolution{}
d.runOnce(t.Context(), "a test")
if _, still := openOnes()[key]; still {
t.Fatalf("unassigned, still open: %+v", openOnes())
}
}
func TestTheLeftOutProbeIsInTheRegistry(t *testing.T) {
for _, p := range probeRegistry {
if p.ID == probeLeftOutID {
if p.run == nil || !slices.Contains(p.Raises, kindLeftOut) || !slices.Contains(p.Raises, kindNeedsOperator) {
t.Fatalf("%+v", p)
}
return
}
}
t.Fatalf("no probe %s: a module left out is said nowhere", probeLeftOutID)
}
// status and node show list it under "assigned, not applied" with the reason the condition says, and status is
// not well while there is one.
func TestStatusAndNodeListAModuleAssignedAndNotApplied(t *testing.T) {
left := map[string][]leftOutModule{"anchor": leftOutOf(leftOutNFS(t), nil, false)}
reason := leftOutObservations("anchor", left["anchor"])[0].Summary
asked := answers{leftOut: left}
if asked.well() {
t.Fatal("a mesh with a module assigned and not applied is called well")
}
shown := printed(t, func() error { return printStatus(asked) })
if !strings.Contains(shown, "1 module(s) assigned, not applied") || !strings.Contains(shown, "nfs-server") ||
!strings.Contains(shown, "`settings set nfs-server '{\"shares\": …}' --node anchor` sets it") {
t.Fatalf("status says:\n%s", shown)
}
if !strings.Contains(reason, left["anchor"][0].reason("anchor")) {
t.Fatalf("status and the condition say different reasons:\n%s\n%s", shown, reason)
}
body, err := statusAsJSON(asked)
if err != nil {
t.Fatal(err)
}
var doc struct {
NotApplied []machineNotApplied `json:"not-applied"`
}
if err := json.Unmarshal(body, &doc); err != nil || len(doc.NotApplied) != 1 ||
doc.NotApplied[0].Setting != "shares" || doc.NotApplied[0].Node != "anchor" {
t.Fatalf("status --json: %v %s", err, body)
}
lines := strings.Join(notAppliedLines("anchor", left["anchor"]), "\n")
if !strings.Contains(lines, "assigned, not applied:") || !strings.Contains(lines, "nfs-server") ||
!strings.Contains(lines, `nothing sets its setting "shares"`) {
t.Fatalf("node show says:\n%s", lines)
}
}
// The skip is this probe's alone (review of #223): a part waiting for the operator (ADR 0283) keeps its own
// needs-operator condition, `module.<m>.<node>.needs-operator`, which still clears on the first statement that no
// longer names it — beside a left-out condition on the same machine, which stays.
func TestAWaitsNeedsOperatorStillClearsWhenItsStatementStopsNamingItBesideALeftOutOne(t *testing.T) {
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store})
defer k.Close(context.Background())
left := leftOutObservations("anchor", leftOutOf(leftOutNFS(t), nil, false))
if err := k.Reconcile(t.Context(), probeLeftOutID, left); err != nil {
t.Fatal(err)
}
waiting := map[string][]inventory.ResourceHealth{"notes": {{Module: "notes", Resource: "server", State: link.StateWaiting,
Waits: []inventory.Wait{{Setting: "domain", What: "the domain it serves"}}}}}
if err := judgeModuleHealth(t.Context(), nil, k, "anchor", waiting, map[string]int{"notes": 2}, time.Now()); err != nil {
t.Fatal(err)
}
waitKey, leftKey := needsOperatorKey("notes", "anchor"), "module.nfs-server.anchor.left-out"
open := func() map[string]conditions.Condition {
t.Helper()
list, err := k.Open(t.Context())
if err != nil {
t.Fatal(err)
}
out := map[string]conditions.Condition{}
for _, c := range list {
out[c.Key] = c
}
return out
}
if c, raised := open()[waitKey]; !raised || c.Kind != kindNeedsOperator {
t.Fatalf("the wait raised %+v", open())
}
if c := open()[leftKey]; c.Kind != kindNeedsOperator {
t.Fatalf("the left-out condition is not open as needs-operator: %+v", open())
}
// D1 runs again and still finds nfs-server left out: its reconcile clears only what D1 raised, never the wait,
// which the machine's statement raised.
if err := k.Reconcile(t.Context(), probeLeftOutID, left); err != nil {
t.Fatal(err)
}
if _, still := open()[waitKey]; !still {
t.Fatalf("D1's reconcile cleared the wait's needs-operator: %+v", open())
}
if err := judgeModuleHealth(t.Context(), nil, k, "anchor", map[string][]inventory.ResourceHealth{}, nil, time.Now()); err != nil {
t.Fatal(err)
}
if _, still := open()[waitKey]; still {
t.Fatalf("the statement no longer names the wait, and its needs-operator is still open: %+v", open())
}
if _, still := open()[leftKey]; !still {
t.Fatalf("the left-out condition was cleared by a health statement: %+v", open())
}
}
+67
View File
@@ -0,0 +1,67 @@
package main
import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// localNetworksOf is a machine's attached networks judged (novox/hq ADR 0340): which count as its local
// network, from what it last said of itself and the links it said face outside it. A machine that has
// said none judges to nothing, and a port reached from `local-network` is then admitted from the private
// network only, and said so.
func localNetworksOf(ctx context.Context, inv *inventory.Inventory, node string) ([]catalogue.LocalNetwork, error) {
account, had, err := inv.MachineOf(ctx, node)
if err != nil {
return nil, err
}
outward, err := inv.OutwardLinksOf(ctx, node)
if err != nil {
return nil, err
}
return judgedNetworks(account, had, outward, overlay.Interface), nil
}
// judgedNetworks is the judging alone, from what was read.
func judgedNetworks(account inventory.MachineAccount, had bool, outward []string, tunnel string) []catalogue.LocalNetwork {
if !had || account.Machine == nil {
return nil
}
attached := make([]catalogue.AttachedNetwork, 0, len(account.Machine.Networks))
for _, n := range account.Machine.Networks {
attached = append(attached, catalogue.AttachedNetwork{Link: n.Link, Network: n.Network})
}
return catalogue.JudgeLocalNetworks(attached, outward, tunnel)
}
// localNetworkLines is what `node` shows of a machine's attached networks (ADR 0340 §8): each one, with
// whether it counts as the local network and why, or why none is shown.
func localNetworkLines(account inventory.MachineAccount, had bool, judged []catalogue.LocalNetwork) []string {
out := []string{" its local network (what a port reached from local-network is admitted from):"}
switch {
case !had || account.Machine == nil:
return append(out, " not said — this machine has not reported what it is attached to")
case len(account.Machine.Networks) == 0:
why := account.Machine.Unknown["networks"]
if why == "" {
why = "its node-engine is older than the one that says it"
}
return append(out, " not said — "+why)
}
counted := 0
for _, j := range judged {
verdict := "not counted"
if j.Counts {
verdict = "counts"
counted++
}
out = append(out, fmt.Sprintf(" %-22s on %-10s %s: %s", j.Network, j.Link, verdict, j.Why))
}
if counted == 0 {
out = append(out, " none counts: a port reached from local-network is admitted from the private network only")
}
return out
}
+65
View File
@@ -0,0 +1,65 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// **node shows what the local network resolved to** (novox/hq ADR 0340 §8): every attached network,
// whether it counts, and why the others do not — the public subnet named as never counted.
func TestNodeShowsWhichAttachedNetworksCountAndWhy(t *testing.T) {
account := inventory.MachineAccount{Architecture: "amd64", Kernel: "linux", Machine: &inventory.MadeOf{
Networks: []inventory.AttachedNetwork{
{Link: "eth0", Network: "192.168.1.10/24"},
{Link: "eth1", Network: "203.0.113.7/24"},
{Link: "wg-mesh", Network: "10.10.0.4/24"},
}}}
judged := judgedNetworks(account, true, []string{"eth0", "eth1"}, "wg-mesh")
text := strings.Join(localNetworkLines(account, true, judged), "\n")
for _, want := range []string{
"192.168.1.0/24", "counts: in a private range (RFC 1918, 192.168.0.0/16)",
"203.0.113.0/24", "not counted: public, never counted",
"10.10.0.0/24", "the private network's own link",
} {
if !strings.Contains(text, want) {
t.Errorf("node does not say %q:\n%s", want, text)
}
}
}
// A machine that said none, or whose node-engine is older, is said as that — never as an empty list
// that reads like a machine on no network.
func TestNodeSaysWhenNoNetworkWasSaid(t *testing.T) {
older := inventory.MachineAccount{Machine: &inventory.MadeOf{}}
if text := strings.Join(localNetworkLines(older, true, nil), "\n"); !strings.Contains(text, "older than the one that says it") {
t.Errorf("an older node-engine is not said:\n%s", text)
}
public := inventory.MachineAccount{Machine: &inventory.MadeOf{
Networks: []inventory.AttachedNetwork{{Link: "eth0", Network: "198.51.100.9/24"}}}}
judged := judgedNetworks(public, true, []string{"eth0"}, "wg-mesh")
if text := strings.Join(localNetworkLines(public, true, judged), "\n"); !strings.Contains(text, "none counts") {
t.Errorf("a machine on a public network only is not said to have no local network:\n%s", text)
}
if text := strings.Join(localNetworkLines(inventory.MachineAccount{}, false, nil), "\n"); !strings.Contains(text, "not said") {
t.Errorf("a machine that never reported is not said:\n%s", text)
}
}
// The preview of a flip says what a port from the local network keeps: the local networks it stays
// open to, or that the machine reported none (ADR 0340 §6).
func TestTheFlipPreviewSaysWhatALocalNetworkPortKeeps(t *testing.T) {
rules := []catalogue.Rule{{Port: 1883, Protocol: "tcp", From: catalogue.FromLocalNetwork, Because: []string{"mosquitto"}}}
at := inventory.Reach{Protocol: "tcp", Address: "0.0.0.0", Port: 1883}
with := derivedFilter{rules: rules, mesh: []string{"10.10.0.1"}, outwardLinks: []string{"eth0"},
local: []string{"192.168.1.0/24"}}
if fate := with.fate(at); !strings.Contains(fate, "192.168.1.0/24") {
t.Errorf("the preview says %q, without the local network it stays open to", fate)
}
with.local = nil
if fate := with.fate(at); !strings.Contains(fate, "reported none") {
t.Errorf("the preview says %q, without saying no local network was reported", fate)
}
}
+398
View File
@@ -0,0 +1,398 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// The tests of novox/hq issue 525's window skip: a merge waits for the merge window only when a batch assembling
// holds a merge related to it, or its delivery group may have a member still to merge (ADR 0304 decision 1, ADR
// 0306 decisions 2 and 3). Walks stay one at a time.
// A merge from no head that shares nothing, with no walk open, is cut into its walk when it is heard: no window.
func TestALoneMergeSharingNothingIsCutAtOnce(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, lone(repoMerge("one", "c1", t0)), t0)
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 || !ws[0].Open() || ws[0].Commit != "c1" {
t.Fatalf("a lone merge sharing nothing: %d walk(s) %+v, %d batch(es)", len(ws), ws, len(bs))
}
if ws[0].Times == nil || ws[0].Times.WindowClosed == nil || !ws[0].Times.WindowClosed.Equal(t0) {
t.Fatalf("its window closed at %+v; want when it was heard, %s", ws[0].Times, t0)
}
}
// A merge related to the batch assembling joins it and waits for the window, which its coming restarts: by its
// repository, by a module in common (a dependent of what the batch moves), by an edge of the module graph between
// their modules (one that widens nothing), or by its delivery group's head.
func TestARelatedMergeJoinsTheBatchAndWaitsForTheWindow(t *testing.T) {
ctx := t.Context()
for _, c := range []string{"same repository", "a module in common", "an edge of the graph", "the same group head"} {
t.Run(c, func(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c1")
asksWithPaths(t)
inv := open.inventory
first := repoMerge("one", "c1", t0)
second := lone(repoMerge("two", "c2", t0.Add(10*time.Second)))
why := map[string]string{"same repository": "both change novox/one", "a module in common": "both move two",
"an edge of the graph": "two depends on builder (built-by)",
"the same group head": "both are members of group " + first.Head}[c]
switch c {
case "same repository":
second = lone(repoMerge("one", "c2", t0.Add(10*time.Second)))
case "a module in common":
// two is built on one: a merge of one moves two too.
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "two", Version: "1",
Build: &catalogue.Build{On: []catalogue.BuildsOn{{Arg: "BASE", Module: "one"}}}},
inventory.Source{Repository: "novox/two", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
case "an edge of the graph":
// two is built by the build machine: an edge that orders and widens nothing.
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "builder", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-build-machine", Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/builder", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
first = repoMerge("builder", "c1", t0)
case "the same group head":
second = fromHead(repoMerge("two", "c2", t0.Add(10*time.Second)), first.Head)
}
hear(t, open, first, t0)
said := captured(t, func() error {
return (following{open: open}).hearMerge(ctx, second, t0.Add(10*time.Second))
})
if !strings.Contains(said, "("+why+")") {
t.Fatalf("the merge was not said related by %q: %s", why, said)
}
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the related merge: %d walk(s), batches %+v", len(ws), bs)
}
if bs[0].Delivery.Batch.CutAtOnce {
t.Fatal("a batch a related merge joined is cut at once")
}
cutAt(t, open, t0.Add(99*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window the related merge restarted closed: %+v", ws)
}
cutAt(t, open, t0.Add(100*time.Second))
ws, bs = walks(t, open)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
})
}
}
// A merge from a head whose delivery group has a member still to merge waits for the window, even with no batch
// assembling, and says why; its sibling joins the same batch by their head (ADR 0306 decision 2). A merge from a
// head that names no group shares nothing and is cut at once, as one from no head is.
func TestAMergeOfAGroupWaitsEvenAlone(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
groupsWaiting(t, "feat/x")
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/x"), t0)
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 1 || bs[0].Delivery.Batch.CutAtOnce || windowClosed(bs[0], t0) {
t.Fatalf("a group's merge alone: %d walk(s), batches %+v", len(ws), bs)
}
if words := batchWords(bs[0], t0); !strings.HasPrefix(words, "assembling: 90 s left") ||
!strings.Contains(words, "group feat/x has a member still to merge: novox/elsewhere (ready)") {
t.Fatalf("it reads %q", words)
}
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(5*time.Second)), "feat/x"), t0.Add(5*time.Second))
if _, bs := walks(t, open); len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the group's second member did not join the first: %+v", bs)
}
cutAt(t, open, t0.Add(94*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before its window closed: %+v", ws)
}
cutAt(t, open, t0.Add(95*time.Second))
ws, bs = walks(t, open)
if len(ws) != 1 || len(bs) != 0 || ws[0].CommitOf("novox/one") != "c1" || ws[0].CommitOf("novox/two") != "c2" {
t.Fatalf("at its window: %d walk(s) %+v, %d batch(es)", len(ws), ws, len(bs))
}
}
// A merge from a head that names no group, sharing nothing, with no walk open, is cut at once.
func TestAHeadedMergeInNoGroupIsCutAtOnce(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "fix/alone"), t0)
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 || !ws[0].Open() {
t.Fatalf("a headed merge in no group: %d walk(s), %d batch(es)", len(ws), len(bs))
}
}
// When mesh-delivery cannot say a head's group, the merge waits for the window, the safe side, and its batch says
// why.
func TestAGroupNobodyCanSayWaits(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return nil, false, errors.New("nothing serves mesh-delivery.groups")
}
t.Cleanup(func() { deliveryGroupOf = was })
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/x"), t0)
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 1 || bs[0].Delivery.Batch.CutAtOnce {
t.Fatalf("a merge whose group nobody could say: %d walk(s), batches %+v", len(ws), bs)
}
if words := batchWords(bs[0], t0); !strings.Contains(words, "mesh-delivery did not say whether group feat/x has a "+
"member still to merge (nothing serves mesh-delivery.groups): it waits for the window, the safe side") {
t.Fatalf("it reads %q", words)
}
}
// A member counts as still to merge only while its pull request is open, and not once this controller keeps its
// merge: a group whose other members are merged or closed makes nobody wait.
func TestAGroupsMergedOrClosedMembersMakeNobodyWait(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
members := []groupMember{{Repository: "novox/one", State: "ready"}, {Repository: "novox/two", State: "published"},
{Repository: "novox/three", State: "stopped"}}
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) { return members, true, nil }
t.Cleanup(func() { deliveryGroupOf = was })
m := fromHead(repoMerge("one", "c1", t0), "feat/x")
if why, err := groupWait(ctx, open.inventory, m, true, false); err != nil || why != "" {
t.Fatalf("the group's other members are merged or closed, yet: %q %v", why, err)
}
members = append(members, groupMember{Repository: "novox/four", State: "checked"})
if why, _ := groupWait(ctx, open.inventory, m, true, false); !strings.Contains(why, "novox/four (checked)") {
t.Fatalf("an open member: %q", why)
}
// novox/four's merge is kept in a batch already: merged, whatever mesh-delivery says yet.
hear(t, open, fromHead(repoMerge("two", "c4", t0), "feat/x"), t0)
members[3].Repository = "novox/two"
if why, _ := groupWait(ctx, open.inventory, m, true, false); why != "" {
t.Fatalf("a member this controller keeps the merge of still waits: %q", why)
}
if got := memberOf("novox/mesh-sdk@d41d255f28ad (held)"); got != (groupMember{Repository: "novox/mesh-sdk", State: "held"}) {
t.Fatalf("a member reads %+v", got)
}
}
// A merge related to two batches assembling folds the newer into the older, which answers every merge of both
// and names nothing twice; the folded one names the batch that took it over.
func TestAMergeRelatedToTwoBatchesFoldsThem(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/a", "feat/b")
asksWithPaths(t)
ctx := t.Context()
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/a"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(5*time.Second)), "feat/b"), t0.Add(5*time.Second))
_, bs := walks(t, open)
if len(bs) != 2 {
t.Fatalf("two merges sharing nothing: %d batch(es)", len(bs))
}
older, newer := bs[1], bs[0] // newest first
// Of one's repository and from two's head: related to both.
hear(t, open, fromHead(repoMerge("one", "c3", t0.Add(10*time.Second)), "feat/b"), t0.Add(10*time.Second))
_, bs = walks(t, open)
if len(bs) != 1 || bs[0].ID != older.ID || len(bs[0].Delivery.Merges) != 3 {
t.Fatalf("after a merge related to both: batches %+v; want %s answering three merges", bs, older.ID)
}
folded, err := open.inventory.PlanByID(ctx, newer.ID)
if err != nil || folded.State != inventory.PlanSuperseded || folded.Delivery == nil ||
folded.Delivery.TakenOverBy != older.ID {
t.Fatalf("the newer batch reads %s %+v (%v)", folded.State, folded.Delivery, err)
}
if left, _ := open.inventory.MergesOf(ctx, newer.ID); len(left) != 0 {
t.Fatalf("the folded batch still answers %+v", left)
}
cutAt(t, open, t0.Add(100*time.Second))
ws, bs := walks(t, open)
var cut []inventory.Plan
for _, w := range ws {
if w.State != inventory.PlanSuperseded {
cut = append(cut, w)
}
}
if len(cut) != 1 || len(bs) != 0 || cut[0].ID != older.ID || cut[0].CommitOf("novox/one") != "c3" ||
cut[0].CommitOf("novox/two") != "c2" {
t.Fatalf("the folded batch's walk: %+v, %d batch(es)", cut, len(bs))
}
}
// With a walk open, a batch cut at once still waits for it, said queued behind it (walks stay one at a time, ADR
// 0276 decision 4); it is cut once that walk ended, with no window to wait for.
func TestACutAtOnceBatchQueuesBehindAnOpenWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c1")
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("no walk open: %+v", ws)
}
first := ws[0]
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("two", "c2", heard)), heard)
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID ||
!bs[0].Delivery.Batch.CutAtOnce {
t.Fatalf("a lone merge during a walk: %d walk(s), batches %+v", len(ws), bs)
}
if !strings.HasPrefix(batchWords(bs[0], heard), "queued behind "+first.ID) {
t.Fatalf("it reads %q", batchWords(bs[0], heard))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, heard.Add(time.Second))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut once the walk ended: %+v %+v", ws, bs)
}
if ws[0].Times == nil || ws[0].Times.WindowClosed == nil || !ws[0].Times.WindowClosed.Equal(heard) {
t.Fatalf("its window closed at %+v; want when its merge was heard, %s", ws[0].Times, heard)
}
}
// A stopped walk's other merges are walked again in a batch they are related to, and a batch of their own when
// they share nothing with the one assembling (ADR 0299, ADR 0304 decision 1): never in a batch of another
// change, whose window the stop would close.
func TestAStopWalksTheOthersAgainInABatchOfTheirOwnWhenTheyShareNothing(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
walk := ws[0]
hear(t, open, repoMerge("two", "two1cccccccc", t0.Add(3*time.Minute)), t0.Add(3*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 {
t.Fatalf("the merge heard while the walk ran is in %d batches", len(bs))
}
other := bs[0]
_, said, err := stopWalkWithout(ctx, open.inventory, walk.ID, "mesh-delivery for jochen", "its group stopped",
[]string{"novox/mesh-catalog@" + cat.Commit}, t0.Add(3*time.Minute+10*time.Second))
if err != nil {
t.Fatal(err)
}
if said.WalkedAgainBy == "" || said.WalkedAgainBy == other.ID {
t.Fatalf("walked again by %q; want a batch of its own, not %s", said.WalkedAgainBy, other.ID)
}
b, _ := open.inventory.PlanByID(ctx, other.ID)
if b.Delivery.Batch == nil || b.Delivery.Batch.Closed || len(b.Delivery.Merges) != 1 {
t.Fatalf("the other change's batch reads %+v; want it untouched by the stop", b.Delivery)
}
}
// A catalogue batch's cut folds in every walk waiting for its delivery's word (cutBatch's carry), so what that
// walk moves is the batch's too: a lone merge related to the waiting walk joins the batch assembling and waits for
// its window, rather than being cut at once as sharing nothing with what the batch's own merges move.
func TestALoneMergeRelatedToAWalkTheCutFoldsJoinsTheBatch(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c3")
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
hear(t, open, repoMerge("three", "c3", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
hear(t, open, lone(repoMerge("one", "c2", t0.Add(2*time.Minute+5*time.Second))), t0.Add(2*time.Minute+5*time.Second))
_, bs := walks(t, open)
if len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 || bs[0].Delivery.Batch.CutAtOnce {
t.Fatalf("a lone merge related to the waiting walk: batches %+v", bs)
}
}
// A delivery group with a member of each kind (review of #250): a merge on the controller's own path and a
// catalogue merge never share a batch, so the later member cannot join the earlier one's. Kept in a batch of the
// other kind, the earlier member keeps the later one waiting, whatever mesh-delivery says yet — never cut at once
// to walk before it. The two batches are then cut in the order their windows close (cutBatchesHeld: the oldest
// closed batch first, an own-path one first at one look, and the other queued behind a walk that started): the
// earlier member's window closes first, so it walks first, in both orders.
func TestAGroupAcrossBothKindsKeepsItsOrder(t *testing.T) {
for _, ownFirst := range []bool{true, false} {
name := "catalogue member first"
if ownFirst {
name = "controller member first"
}
t.Run(name, func(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
was := deliveryGroupOf
// mesh-delivery still says both members ready: it marks one merged only once its walk opened.
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return []groupMember{{Repository: "novox/mesh-controller", State: "ready"},
{Repository: "novox/mesh-catalog", State: "ready"}}, true, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
own := fromHead(repoMerge("mesh-controller", "k1", t0), "feat/both-kinds")
cat := fromHead(catalogueMerge("cat1aaaaaaaa", "app", t0), "feat/both-kinds")
first, second := cat, own
if ownFirst {
first, second = own, cat
}
second.MergedAt = t0.Add(10 * time.Second).Format(time.RFC3339Nano)
hear(t, open, first, t0)
hear(t, open, second, t0.Add(10*time.Second))
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 2 {
t.Fatalf("both members heard: %d walk(s), %d batch(es); want two batches, nothing cut", len(ws), len(bs))
}
later := bs[0] // newest first
if later.Delivery.Batch.CutAtOnce || !strings.Contains(later.Delivery.Batch.Waits,
"has a member waiting in a batch of the other kind: novox/"+first.Repo+"@") {
t.Fatalf("the later member's batch reads %+v; want it waiting for the earlier member", later.Delivery.Batch)
}
// The earlier member's window closes first: its batch alone is cut.
cutAt(t, open, t0.Add(90*time.Second))
ws, bs = walks(t, open)
if len(ws) != 1 || !ws[0].Open() || ws[0].CommitOf("novox/"+first.Repo) != first.Commit || len(bs) != 1 ||
bs[0].ID != later.ID {
t.Fatalf("at the earlier window: walks %+v, batches %+v; want the earlier member alone cut", ws, bs)
}
earlier := ws[0]
// The later member's window closes while the earlier walk runs: it queues behind it.
cutAt(t, open, t0.Add(100*time.Second))
if _, bs := walks(t, open); len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != earlier.ID {
t.Fatalf("the later member's batch does not queue behind the earlier walk: %+v", bs)
}
earlier.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &earlier); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(101*time.Second))
got, err := open.inventory.PlanByID(ctx, later.ID)
if err != nil || got.Batch() || got.CommitOf("novox/"+second.Repo) != second.Commit {
t.Fatalf("the later member was not walked after the earlier: %s %v", got.State, err)
}
})
}
}
+135
View File
@@ -0,0 +1,135 @@
package main
import (
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// machineLines is what a machine said it is made of (novox/hq issue 160), as `node` shows it: every fact in
// one block with its age, the architecture and kernel the profile always carried among them, and each fact
// the machine could not determine said as unknown with its why — never as a zero, and never left out.
func machineLines(a inventory.MachineAccount, had bool, now time.Time) []string {
if !had {
return []string{" what it is made of: never said — this machine has not reported a profile"}
}
m := a.Machine
if m == nil {
return []string{
" what it is made of: not said — its node-engine is older than the one that says it",
" " + factLine("architecture", orUnsaid(a.Architecture)),
" " + factLine("kernel", orUnsaid(a.Kernel)),
}
}
age := roughly(now.Sub(m.ObservedAt)) + " ago"
if m.ObservedAt.IsZero() {
// Said with no time: its age is not known, and is never read as two thousand years.
age = "at a time it did not say (age unknown)"
}
out := []string{fmt.Sprintf(" what it is made of, as it said %s:", age)}
add := func(name, value string) { out = append(out, " "+factLine(name, value)) }
unknown := map[string]string{}
for k, v := range m.Unknown {
unknown[k] = v
}
// said is the value, or the fact's own reason it is unknown; a fact neither said nor unknown is a
// node-engine that broke its own rule, and is said so rather than shown blank.
said := func(fact, value string) string {
if value != "" {
delete(unknown, fact)
return value
}
if why, ok := unknown[fact]; ok {
delete(unknown, fact)
return "unknown: " + why
}
return "unknown: the machine said neither it nor why"
}
add("architecture", orUnsaid(a.Architecture))
release := said("kernel release", m.KernelRelease)
if m.KernelRelease == "" {
release = "(release " + release + ")"
}
add("kernel", orUnsaid(a.Kernel)+" "+release)
dist := said("distribution", m.Distribution)
if m.Distribution != "" {
dist += " " + said("version", m.Version)
}
add("distribution", dist)
add("processor", said("processor", m.Processor))
add("cores", said("cores", countWords(m.Cores)))
add("threads", said("threads", countWords(m.Threads)))
add("memory", said("memory", bytesWords(m.MemoryBytes)))
add("virtual", said("virtual or physical", m.Virtual))
up := ""
if m.BootedAt != nil {
up = fmt.Sprintf("%s, since %s", roughly(now.Sub(*m.BootedAt)), m.BootedAt.Local().Format("2006-01-02 15:04"))
}
add("up", said("uptime", up))
add("timezone", said("timezone", m.Timezone))
if len(m.Filesystems) == 0 {
add("filesystems", said("filesystems", ""))
} else {
for i, f := range m.Filesystems {
name := ""
if i == 0 {
name = "filesystems"
}
add(name, fmt.Sprintf("%s %s, %s free (%s %s)", f.Mount, bytesWords(f.SizeBytes),
bytesWords(f.FreeBytes), f.Type, f.Device))
}
if why, ok := unknown["filesystems"]; ok {
add("", "and some unknown: "+why)
delete(unknown, "filesystems")
}
}
add("public address", said("public address", ""))
// Any other fact a newer node-engine named unknown, so nothing it said is dropped.
rest := make([]string, 0, len(unknown))
for k := range unknown {
rest = append(rest, k)
}
sort.Strings(rest)
for _, k := range rest {
add(k, "unknown: "+unknown[k])
}
return out
}
func factLine(name, value string) string { return fmt.Sprintf("%-15s %s", name, value) }
func orUnsaid(s string) string {
if strings.TrimSpace(s) == "" {
return "not said"
}
return s
}
// countWords is a number a machine said, or nothing when it said none (its unknown says why).
func countWords(n int) string {
if n <= 0 {
return ""
}
return fmt.Sprint(n)
}
// bytesWords is an amount of bytes a person reads, in binary units; nothing when none was said.
func bytesWords(n int64) string {
if n <= 0 {
return ""
}
units := []string{"B", "KiB", "MiB", "GiB", "TiB", "PiB"}
v, u := float64(n), 0
for v >= 1024 && u < len(units)-1 {
v /= 1024
u++
}
if u == 0 {
return fmt.Sprintf("%d B", n)
}
return fmt.Sprintf("%.1f %s", v, units[u])
}
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// Defends novox/hq issue 160: `node` shows every fact a machine said it is made of, in one block with its
// age, the architecture and kernel among them; and a fact it could not determine as unknown with why.
func TestNodeShowsWhatAMachineIsMadeOfWithItsAge(t *testing.T) {
now := time.Date(2026, 10, 12, 15, 0, 0, 0, time.UTC)
booted := now.Add(-75 * time.Hour)
a := inventory.MachineAccount{Architecture: "amd64", Kernel: "linux", Machine: &inventory.MadeOf{
ObservedAt: now.Add(-3 * time.Hour), KernelRelease: "7.2.8-arch1-2", Distribution: "Arch Linux",
Version: "rolling", Processor: "AMD Ryzen 9 8945HS", Cores: 8, Threads: 16, MemoryBytes: 32891494400,
Virtual: "physical", BootedAt: &booted, Timezone: "Europe/Brussels",
Filesystems: []inventory.MountedFilesystem{{Mount: "/", Device: "/dev/nvme0n1p3", Type: "ext4",
SizeBytes: 987978522624, FreeBytes: 31830110208}},
Unknown: map[string]string{"public address": "how it learns it is not decided yet"},
}}
got := strings.Join(machineLines(a, true, now), "\n")
t.Log("\n" + got)
for _, want := range []string{
"what it is made of, as it said 3h ago:",
"architecture amd64",
"kernel linux 7.2.8-arch1-2",
"distribution Arch Linux rolling",
"processor AMD Ryzen 9 8945HS",
"cores 8",
"threads 16",
"memory 30.6 GiB",
"virtual physical",
"up 3d, since",
"timezone Europe/Brussels",
"filesystems / 920.1 GiB, 29.6 GiB free (ext4 /dev/nvme0n1p3)",
"public address unknown: how it learns it is not decided yet",
} {
if !strings.Contains(got, want) {
t.Errorf("missing %q", want)
}
}
}
func TestAFactAMachineCouldNotReadIsShownUnknownNeverZero(t *testing.T) {
now := time.Date(2026, 10, 12, 15, 0, 0, 0, time.UTC)
a := inventory.MachineAccount{Architecture: "arm64", Kernel: "linux", Machine: &inventory.MadeOf{
ObservedAt: now, Unknown: map[string]string{"memory": "MemTotal not found in /proc/meminfo",
"cores": "/proc/cpuinfo says no core id", "kernel release": "permission denied"}}}
got := strings.Join(machineLines(a, true, now), "\n")
t.Log("\n" + got)
for _, want := range []string{
"memory unknown: MemTotal not found in /proc/meminfo",
"cores unknown: /proc/cpuinfo says no core id",
"kernel linux (release unknown: permission denied)",
// Neither said nor named unknown: the node-engine broke its rule, and that is shown.
"timezone unknown: the machine said neither it nor why",
} {
if !strings.Contains(got, want) {
t.Errorf("missing %q", want)
}
}
if strings.Contains(got, " 0 ") || strings.Contains(got, "0 B") {
t.Error("a fact the machine could not read was shown as a zero")
}
}
func TestAnOlderNodeEngineStillShowsArchitectureAndKernel(t *testing.T) {
got := strings.Join(machineLines(inventory.MachineAccount{Architecture: "amd64", Kernel: "linux"}, true,
time.Now()), "\n")
if !strings.Contains(got, "older than the one that says it") || !strings.Contains(got, "architecture amd64") ||
!strings.Contains(got, "kernel linux") {
t.Errorf("%s", got)
}
if got := strings.Join(machineLines(inventory.MachineAccount{}, false, time.Now()), "\n"); !strings.Contains(got, "never said") {
t.Errorf("%s", got)
}
}
func TestAnAccountWithoutItsTimeSaysAgeUnknown(t *testing.T) {
got := machineLines(inventory.MachineAccount{Architecture: "amd64", Kernel: "linux",
Machine: &inventory.MadeOf{KernelRelease: "7.2"}}, true, time.Now())[0]
if !strings.Contains(got, "age unknown") || strings.Contains(got, "d ago") {
t.Errorf("%s", got)
}
}
+12 -1
View File
@@ -63,6 +63,15 @@ func run() error {
// 45 §6), so the next controller takes it at once rather than after its age.
defer theLease.release()
// Every walk a verb keeps is said on the bus, whichever verb kept it (novox/hq ADR 0308 decision 1): the save
// that starts or ends a walk keeps its mark, and no later save says it again. The serving controller says its
// own saves.
if args[0] != "serve" {
said, done := sayingWhatAVerbKept(ctx)
inventory.PlanSaved = said
defer done()
}
switch args[0] {
case "build":
return buildCommand(ctx, args[1:])
@@ -162,6 +171,8 @@ func run() error {
return adoptCommand(ctx, args[1:])
case "settings":
return settingsCommand(ctx, args[1:])
case "config":
return configCommand(ctx, args[1:])
case "secret":
return secretCommand(ctx, args[1:])
case "operator":
@@ -311,7 +322,7 @@ func usage() {
the self-check: the last verdict, a run now, the probes, the signals' ages
healers [--days N] [--json] the healers, what they did lately, and their brake (to-be 45 §7)
durations [--kind K] [--days N] [--json]
apply, heartbeat, plan-tier and build durations, per machine or module
apply, heartbeat, plan-tier, build and walk-phase durations
collection [--json] kept archives held/unheld by a manifest, and what the sweep may let go
builder issue <name> a broker account for a build machine, scoped to build work,
delivered as the builder module's broker secret (module add it first)
+60 -5
View File
@@ -296,6 +296,24 @@ func judgeChange(ctx context.Context, in mergeCheckInput) (mergeVerdict, error)
}
v.Failures = append(v.Failures, failures...)
}
// The list of predefined keys a head's catalogue tree carries, which the change composes with
// (novox/hq issue 630); the mesh's own list otherwise.
var changedList catalogue.Predefined
for _, h := range heads {
if h.Tree == "" {
continue
}
raw, err := os.ReadFile(filepath.Join(h.Tree, catalogue.PredefinedFile))
if err != nil {
continue
}
list, err := catalogue.ParsePredefined(raw)
if err != nil {
v.Failures = append(v.Failures, err.Error())
continue
}
changedList = list
}
if len(in.group) > 0 {
say("a delivery group: %d head(s) composed together", len(heads))
}
@@ -319,7 +337,7 @@ func judgeChange(ctx context.Context, in mergeCheckInput) (mergeVerdict, error)
// Both compositions, each in a store of its own.
say("composing every machine as the mesh is")
before, _, notes, err := composeEveryMachine(ctx, in, base, sources, nil)
before, _, notes, err := composeEveryMachine(ctx, in, base, sources, nil, nil)
if err != nil {
return v, fmt.Errorf("the mesh as the snapshot says it is could not be raised: %w", err)
}
@@ -332,7 +350,7 @@ func judgeChange(ctx context.Context, in mergeCheckInput) (mergeVerdict, error)
}
}
sort.Strings(added)
after, tried, notes, err := composeEveryMachine(ctx, in, change, sources, added)
after, tried, notes, err := composeEveryMachine(ctx, in, change, sources, added, changedList)
if err != nil {
return v, fmt.Errorf("the mesh with the change could not be raised: %w", err)
}
@@ -748,13 +766,13 @@ func standInBuild(m catalogue.Manifest) (catalogue.Manifest, error) {
// grant to compose — and answers each machine's composition. Notes are what of the snapshot could not
// be raised: a refusal to register a module, keep a setting or a pin.
func composeEveryMachine(ctx context.Context, in mergeCheckInput, shelf map[string]catalogue.Manifest,
sources map[string]inventory.Source, trials []string) (map[string]mergeComposed, map[string]mergeTrial, []string, error) {
sources map[string]inventory.Source, trials []string, list catalogue.Predefined) (map[string]mergeComposed, map[string]mergeTrial, []string, error) {
open, drop, err := throwawayStores(ctx, in.admin)
if err != nil {
return nil, nil, nil, err
}
defer drop()
notes, err := raiseFromFacts(ctx, open, in.facts, shelf, sources)
notes, err := raiseFromFacts(ctx, open, in.facts, shelf, sources, list)
if err != nil {
return nil, nil, notes, err
}
@@ -950,9 +968,22 @@ func throwawayStores(ctx context.Context, admin string) (*stores, func(), error)
// for each standing in for the one it holds, its assignments, the seats it holds, its pins and its
// settings, and a stand-in for every secret a person gave.
func raiseFromFacts(ctx context.Context, open *stores, f snapshot.Facts, shelf map[string]catalogue.Manifest,
sources map[string]inventory.Source) ([]string, error) {
sources map[string]inventory.Source, list catalogue.Predefined) ([]string, error) {
inv := open.inventory
var notes []string
// The list of predefined keys before any module, which registration judges its preferences against
// (novox/hq issue 630): the change's own when its catalogue tree carries one, the mesh's otherwise.
if list == nil && len(f.Predefined) > 0 {
list = catalogue.Predefined{}
for key, k := range f.Predefined {
list[key] = catalogue.PredefinedKey{Type: k.Type, Why: k.Why}
}
}
if len(list) > 0 {
if err := inv.SetPredefined(ctx, list, "", ""); err != nil {
notes = append(notes, fmt.Sprintf("the list of predefined keys is not kept: %s", oneLine(err.Error())))
}
}
names := make([]string, 0, len(shelf))
for name := range shelf {
names = append(names, name)
@@ -1045,6 +1076,20 @@ func raiseFromFacts(ctx context.Context, open *stores, f snapshot.Facts, shelf m
}
}
}
// Mesh and node config, as the mesh holds them (novox/hq issue 630).
for _, key := range sortedAnyKeys(f.Config) {
if err := inv.SetConfig(ctx, key, "", f.Config[key], "the facts snapshot", "raised from the facts snapshot"); err != nil {
notes = append(notes, fmt.Sprintf("the mesh config of %s is not kept: %s", key, oneLine(err.Error())))
}
}
for _, m := range f.Machines {
for _, key := range sortedAnyKeys(m.Config) {
if err := inv.SetConfig(ctx, key, m.Name, m.Config[key], "the facts snapshot", "raised from the facts snapshot"); err != nil {
notes = append(notes, fmt.Sprintf("%s's node config of %s is not kept: %s", m.Described(), key,
oneLine(err.Error())))
}
}
}
for _, s := range f.Settings {
if err := inv.KeepSettings(ctx, "", s.Module, standInPaths(s.Values)); err != nil {
notes = append(notes, fmt.Sprintf("the mesh's settings of %s are not kept: %s", s.Module, oneLine(err.Error())))
@@ -1312,3 +1357,13 @@ func firstOr(items, otherwise []string) string {
}
return "it says nothing more"
}
// sortedAnyKeys is a map's keys, sorted.
func sortedAnyKeys(in map[string]any) []string {
keys := make([]string, 0, len(in))
for k := range in {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
+25 -3
View File
@@ -390,9 +390,10 @@ func TestAWithheldPathIsStoodInForByAPath(t *testing.T) {
// controller's ask and by the facts — and finds the repository it checks from its origin.
func TestACheckByHandClonesWhatTheSeatClones(t *testing.T) {
for dir, refs := range map[string]map[string]string{
"mesh-catalog": {"mesh-catalog": "main"},
"mesh-host": {"mesh-host": "c0ffee"},
"mesh-controller": {"mesh-controller": "c0ffee", "mesh-controller-main": "main", "mesh-lab": "main"},
"mesh-catalog": {"mesh-catalog": "main"},
"mesh-host": {"mesh-host": "c0ffee"},
"mesh-controller": {"mesh-controller": "c0ffee", "mesh-controller-main": "main", "mesh-lab": "main",
"mesh-sdk": sdkPinnedByGoMod(t)},
} {
got := besideRefs(dir, "c0ffee")
for d, ref := range refs {
@@ -412,3 +413,24 @@ func TestACheckByHandClonesWhatTheSeatClones(t *testing.T) {
}
}
}
// sdkPinnedByGoMod is the SDK commit this tree's go.mod pins, read from the file rather than the build, so
// sdkPinned is held to what the repository says (novox/hq issue 449).
func sdkPinnedByGoMod(t *testing.T) string {
t.Helper()
raw, err := os.ReadFile(filepath.Join("..", "..", "go.mod"))
if err != nil {
t.Fatal(err)
}
for _, line := range strings.Split(string(raw), "\n") {
fields := strings.Fields(line)
if len(fields) >= 2 && fields[0] == sdkModule {
if m := pseudoCommit.FindStringSubmatch(fields[1]); m != nil {
return m[1]
}
return "go/" + fields[1]
}
}
t.Fatalf("go.mod requires no %s", sdkModule)
return ""
}
+185 -42
View File
@@ -74,9 +74,21 @@ func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Ke
kept := inventory.ResourceHealth{Module: r.Module, Resource: r.Resource, Kind: r.Kind, Target: r.Target,
State: r.State, Reason: r.Reason, Since: r.Since, Streak: r.Streak, Restarts: r.Restarts,
Check: r.Check, Needs: r.Needs, Account: r.Account, Root: r.Root}
for _, w := range r.Waits {
kept.Waits = append(kept.Waits, inventory.Wait{Part: w.Part, Secret: w.Secret, Setting: w.Setting, Provision: w.Provision, What: w.What})
}
resources = append(resources, kept)
if r.State == link.StateUnhealthy && r.Module != "" {
unhealthy[r.Module] = append(unhealthy[r.Module], kept)
}
// **A wait for the operator is checked before it is excused** (novox/hq ADR 0283): a waiting resource whose
// wait does not check out is judged unhealthy, saying why; one that does is kept beside the unhealthy ones, so
// judgeModuleHealth can say it as needs-operator. What is stored is what the machine said.
var wf operatorWaitFacts
if mods := waitingModules(resources); len(mods) > 0 {
readWaitFacts(ctx, inv, node, mods, nil, &wf)
}
for _, r := range checkWaiting(node, resources, wf) {
if (r.State == link.StateUnhealthy || r.State == link.StateWaiting) && r.Module != "" {
unhealthy[r.Module] = append(unhealthy[r.Module], r)
}
}
streaks := map[string]int{}
@@ -135,16 +147,17 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
}
standing := map[string]conditions.Condition{}
for _, c := range open {
if (c.Kind == kindModuleUnhealthy || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound) &&
c.Subject.Machine == node {
// A module left out of the composition is the self-check's to raise and clear, never a statement's
// (novox/hq issue 380): its needs-operator is not cleared for not being in what the machine runs.
if (c.Kind == kindModuleUnhealthy || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound ||
c.Kind == kindNeedsOperator) &&
c.Subject.Machine == node && !isLeftOutCondition(c) {
standing[c.Key] = c
}
}
var hold *holding
if inv != nil {
if hold, err = readHolding(ctx, inv, open); err != nil {
return err
}
hold, err := readHoldingFor(ctx, inv, open)
if err != nil {
return err
}
var problems []string
modules := make([]string, 0, len(unhealthy))
@@ -159,6 +172,39 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
heldOn := map[string]string{}
providers := map[catalogue.Chosen]bool{}
for _, m := range modules {
// **A part that waits for the operator is said as that** (novox/hq ADR 0283): its waits already checked,
// the operator's, never urgent, its words naming the act.
if waits, waiting := operatorWait(m, unhealthy[m]); waiting {
o := needsOperatorObservation(m, node, waits, unhealthy[m])
// **A provider waiting for the operator says who waits on it** (novox/hq issue 405), as one waiting for a
// login does: its consumers are held under it.
if hold != nil {
sayWaitingOn(&o, hold.waitersOn(catalogue.Chosen{Node: node, Module: m}))
}
seen[o.Key()] = true
became[m] = kindNeedsOperator
if _, isOpen := standing[o.Key()]; streaks[m] < moduleUnhealthyAfter && !isOpen {
continue
}
if _, err := k.Observe(ctx, o); err != nil {
problems = append(problems, err.Error())
}
continue
}
// **A wait for the operator beside anything else is said too** (novox/hq issue 405): a module with a checked
// wait beside a new login owed, a directory used as found or a fault of its own is said as that, and the
// operator's secret or setting it waits for was not mentioned until the other cleared. Its needs-operator
// condition stands beside the other, on the same looks; what follows judges the rest without the wait.
if waits, rest := besideAWait(m, unhealthy[m]); len(waits) > 0 {
o := needsOperatorObservation(m, node, waits, waitingOf(m, unhealthy[m]))
seen[o.Key()] = true
if _, isOpen := standing[o.Key()]; streaks[m] >= moduleUnhealthyAfter || isOpen {
if _, err := k.Observe(ctx, o); err != nil {
problems = append(problems, err.Error())
}
}
unhealthy[m] = rest
}
// **A directory used as found is said as that** (novox/hq issue 339): the operator's to hand over at the
// machine, never urgent — nothing is broken by the wait that a person was not told of — and its own kind,
// so the gate never reads it as a fault of the build that happened to be sent beside it.
@@ -195,13 +241,24 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
}
o := moduleUnhealthyObservation(m, node, unhealthy[m])
if hold != nil {
if p, held := hold.heldUnder(node, m, unhealthy[m]); held {
// Held under the provider's condition: nothing of its own, and the provider's says it waits.
heldOn[o.Key()] = p.Module + " on " + p.Node
if by, held := hold.heldWith(node, m, unhealthy[m]); held {
// Held under the provider's condition: nothing of its own, and the provider's says it waits. The
// clearing line says which the provider is: unhealthy, or waiting for the operator (issue 405).
p := by.provider
heldOn[o.Key()] = p.Module + " on " + p.Node + ", which is unhealthy"
if len(by.waits) > 0 {
heldOn[o.Key()] = p.Module + " on " + p.Node + ", which waits for you"
}
providers[p] = true
continue
}
sayWaitingOn(&o, hold.waitersOn(catalogue.Chosen{Node: node, Module: m}))
// A provider that waits for the operator for a part this finding cannot be matched to does not hold it
// (novox/hq issue 405): raised as its own, and saying the provider waits, so neither is hidden.
if p, waiting := hold.waitingUncovered(node, m, unhealthy[m]); waiting {
o.Said += fmt.Sprintf("; %s on %s waits for you, but not for anything %s is known to need, so %s's "+
"fault is said on its own", p.Module, p.Node, m, m)
}
}
seen[o.Key()] = true
became[m] = kindModuleUnhealthy
@@ -228,8 +285,11 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
if c.Kind == kindUsedAsFound {
why = fmt.Sprintf("%s says no directory of %s is used as found any more", node, module)
}
if c.Kind == kindNeedsOperator {
why = fmt.Sprintf("%s says %s no longer waits for the operator", node, module)
}
if on, held := heldOn[key]; held {
why = fmt.Sprintf("what %s finds on %s waits on %s, which is unhealthy: held under its condition", module, node, on)
why = fmt.Sprintf("what %s finds on %s waits on %s: held under its condition", module, node, on)
}
// **A condition that became the other kind** is not "working again" (issue 318 review): its clearing
// line says what it became.
@@ -238,6 +298,12 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
case c.Kind == kindModuleUnhealthy && became[module] == kindReloginNeeded:
why = fmt.Sprintf("%s on %s now waits only for a new login", module, node)
resolved = fmt.Sprintf("%s on %s now waits only for a new login", module, node)
case c.Kind == kindModuleUnhealthy && became[module] == kindNeedsOperator:
why = fmt.Sprintf("%s on %s now only waits for the operator", module, node)
resolved = fmt.Sprintf("%s on %s now only waits for you", module, node)
case c.Kind == kindNeedsOperator && became[module] == kindModuleUnhealthy:
why = fmt.Sprintf("%s on %s no longer only waits for the operator, and is not healthy", module, node)
resolved = fmt.Sprintf("What %s on %s waited for is given, and it still does not work", module, node)
case c.Kind == kindReloginNeeded && became[module] == kindModuleUnhealthy:
why = fmt.Sprintf("%s on %s no longer waits for a new login, and is not healthy", module, node)
resolved = fmt.Sprintf("The new login on %s is done, and %s still does not work", node, module)
@@ -262,36 +328,74 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
// sayWaiters observes a provider's open condition again, with who waits on it, from its machine's newest
// statement. Nothing when its condition is not open: it is raised by its own statements, on its own looks.
func sayWaiters(ctx context.Context, k *conditions.Keeper, hold *holding, p catalogue.Chosen, now time.Time) error {
var raisedAt *conditions.Condition
for i, c := range hold.open {
if c.Key == moduleUnhealthyKey(p.Module, p.Node) || c.Key == reloginKey(p.Module, p.Node) {
raisedAt = &hold.open[i]
}
}
if raisedAt == nil {
return nil
}
// Its statement as it was judged, its waits checked (novox/hq issue 450): a wait that failed the check is
// unhealthy, so the condition built here is the one its own statement raised, and who waits on it is listed.
var rs []inventory.ResourceHealth
for _, r := range hold.healths[p.Node].Resources {
if r.Module == p.Module && r.State == link.StateUnhealthy {
for _, r := range hold.checked(p.Node) {
if r.Module == p.Module && (r.State == link.StateUnhealthy || r.State == link.StateWaiting) {
rs = append(rs, r)
}
}
if len(rs) == 0 {
return nil
}
o := moduleUnhealthyObservation(p.Module, p.Node, rs)
if said, waits := personWait(p.Module, p.Node, rs); waits {
o = reloginObservation(p.Module, p.Node, said, operatorOn(ctx, hold.inv, p.Node), rs)
// The condition its own statement says it under: needs-operator while it only waits for the operator (novox/hq
// issue 405), else that for the rest of it, a wait beside it said on its own.
var o conditions.Observation
if waits, waiting := operatorWait(p.Module, rs); waiting {
o = needsOperatorObservation(p.Module, p.Node, waits, rs)
} else {
_, rest := besideAWait(p.Module, rs)
o = moduleUnhealthyObservation(p.Module, p.Node, rest)
if said, waits := personWait(p.Module, p.Node, rest); waits {
o = reloginObservation(p.Module, p.Node, said, operatorOn(ctx, hold.inv, p.Node), rest)
}
}
if o.Key() != raisedAt.Key {
return nil // its own statement says it next
raised := false
for _, c := range hold.open {
raised = raised || c.Key == o.Key()
}
if !raised {
return nil // not open yet, or open as another kind: its own statement says it next
}
sayWaitingOn(&o, hold.waitersOn(p))
_, err := k.Observe(ctx, o)
return err
}
// besideAWait is, for a module with something not healthy beside a part waiting for the operator, the waits (checked
// already) and the rest without the waiting parts (novox/hq issue 405); no waits when nothing waits, or when the
// module only waits (operatorWait says that whole). Pure.
func besideAWait(module string, rs []inventory.ResourceHealth) ([]inventory.Wait, []inventory.ResourceHealth) {
if _, only := operatorWait(module, rs); only {
return nil, rs
}
var waits []inventory.Wait
var rest []inventory.ResourceHealth
for _, r := range rs {
if r.Module == module && r.State == link.StateWaiting {
waits = append(waits, r.Waits...)
continue
}
rest = append(rest, r)
}
if len(waits) == 0 {
return nil, rs
}
return waits, rest
}
// waitingOf is a module's waiting resources: the evidence of its wait.
func waitingOf(module string, rs []inventory.ResourceHealth) []inventory.ResourceHealth {
var out []inventory.ResourceHealth
for _, r := range rs {
if r.Module == module && r.State == link.StateWaiting {
out = append(out, r)
}
}
return out
}
// reloginKey is a module's relogin-needed condition on a machine.
func reloginKey(module, node string) string {
return conditions.Key(conditions.ScopeModule, module+"."+node, kindReloginNeeded)
@@ -363,12 +467,15 @@ func waitingAccounts(module string, rs []inventory.ResourceHealth) []string {
}
// sayWaitingOn adds to a module's condition the consumers held under it (to-be 48 §6): urgent while anyone
// waits on it, whether it is not working or waits for a new login.
// waits on it, whether it is not working or waits for a new login. **A wait for the operator's secret or setting
// stays a warning** (ADR 0283 decision 5, novox/hq issue 405): who waits on it is listed, and nothing escalates it.
func sayWaitingOn(o *conditions.Observation, waiters []string) {
if len(waiters) == 0 {
return
}
o.Severity = conditions.Urgent
if o.Kind != kindNeedsOperator {
o.Severity = conditions.Urgent
}
o.Said += "; " + waitingWords(waiters)
o.Summary += fmt.Sprintf("; %d consumer(s) wait on it", len(waiters))
o.Explanation += fmt.Sprintf(" %d module(s) that depend on it wait for it.", len(waiters))
@@ -390,9 +497,11 @@ func moduleUnhealthyObservation(module, node string, rs []inventory.ResourceHeal
continue
}
words = append(words, fmt.Sprintf("its %s %s %s", r.Kind, r.Resource, reasonWords(r)))
said = append(said, fmt.Sprintf("%s (%s %s): %s, %d look(s) in a row, %d restart(s) counted, since %s",
r.Resource, r.Kind, r.Target, orNotSaid(r.Reason), r.Streak, r.Restarts,
r.Since.UTC().Format("2006-01-02 15:04:05 MST")))
// The restart count runs from when the engine began watching it, not from the state's since (novox/hq
// issue 458).
said = append(said, fmt.Sprintf("%s (%s %s): %s, since %s, %d look(s) in a row, %s",
r.Resource, r.Kind, r.Target, orNotSaid(r.Reason), r.Since.UTC().Format("2006-01-02 15:04:05 MST"),
r.Streak, restartsWords(r.Restarts)))
}
return conditions.Observation{Scope: conditions.ScopeModule, ID: module + "." + node, Token: "unhealthy",
Kind: kindModuleUnhealthy, Machine: node, Severity: conditions.Warning, Source: sourceHealth,
@@ -406,6 +515,15 @@ func moduleUnhealthyObservation(module, node string, rs []inventory.ResourceHeal
Resolved: fmt.Sprintf("%s works again on %s", module, node)}
}
// restartsWords is a restart count as what it is (novox/hq issue 458). The engine reports its own count
// (mesh-host's liveness judge, kept.Counted): every rise of systemd's NRestarts or the runtime's RestartCount
// after a start's grace, so only restarts by the Restart= setting or the restart policy. It keeps the count
// across a recreate, a start by hand and its own restarts, and forgets it only when the resource is no longer
// declared. So it counts automatic restarts since the engine began watching it, never since a state's "since".
func restartsWords(n int) string {
return fmt.Sprintf("%d automatic restart(s) since the node-engine began watching it", n)
}
// reasonWords is why a resource is unhealthy, as a person reads it.
func reasonWords(r inventory.ResourceHealth) string {
// An account waiting for a new login (ADR 0252) is said in the mesh's words, not the engine's.
@@ -414,12 +532,17 @@ func reasonWords(r inventory.ResourceHealth) string {
}
switch r.Reason {
case "restarting":
return fmt.Sprintf("keeps restarting (%d restart(s) counted)", r.Restarts)
return fmt.Sprintf("keeps restarting (%s)", restartsWords(r.Restarts))
case "down":
return "is not running"
case "":
return "is unhealthy"
}
// A wait for the operator that did not check out is said as the controller found it (ADR 0283): names of
// secrets and settings only, never what the check itself said.
if strings.HasPrefix(r.Reason, waitRefusedPrefix) {
return r.Reason
}
// What a declared check found says an endpoint, a path or an address: evidence, never the summary the
// operator's channel carries (ADR 0234 §6). The summary names the check.
if r.Check != "" {
@@ -511,7 +634,9 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
if h.HeardAt.Before(since) {
return healthNotYet, fmt.Sprintf("%s has not said how what %s runs is since it was sent", machine, module)
}
wait, waits := personWait(module, machine, h.Resources)
// **A wait for the operator is checked first** (novox/hq ADR 0283): one that does not check out is unhealthy.
resources := checkWaiting(machine, h.Resources, f.waits)
wait, waits := personWait(module, machine, resources)
// **Only a build whose own send put the account in a new group is excused** (issue 318 review): read from
// what the controller sent, never from when the machine says the wait began — that time is the engine's
// memory, reset by its restart and moved by a change of words. A build that adds no account group cannot
@@ -519,11 +644,18 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
if waits && !f.groupsAdded[module] {
waits = false
}
var found []string
for _, r := range h.Resources {
var found, onWaiting []string
var forOperator []inventory.Wait
for _, r := range resources {
if r.Module != module {
continue
}
// **Any build is excused while its wait for the operator checks out** (ADR 0283 decision 4): a secret not
// given is owed by every build alike, so it is no fault of this one, and the verdict carries it.
if r.State == link.StateWaiting {
forOperator = append(forOperator, r.Waits...)
continue
}
if waits && r.State == link.StateUnhealthy {
continue
}
@@ -541,8 +673,16 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
return healthNotYet, fmt.Sprintf("its %s %s on %s is still starting", r.Kind, r.Resource, machine)
case link.StateUnhealthy:
if on, held := f.heldOn[module+"@"+machine]; held {
// **Held under a provider that only waits for the operator** (novox/hq issue 405): a wait for a
// person, a pass carrying the wait (ADR 0254, ADR 0283 decision 4) — the walk never waits for the
// operator's secret, whichever module owes it.
if on.waits != "" {
onWaiting = append(onWaiting, fmt.Sprintf("its %s %s on %s waits on %s, which waits for you: %s",
r.Kind, r.Resource, machine, on.on, on.waits))
continue
}
return healthWaiting, fmt.Sprintf("its %s %s on %s waits on %s, which is unhealthy", r.Kind,
r.Resource, machine, on)
r.Resource, machine, on.on)
}
return healthNotYet, fmt.Sprintf("its %s %s on %s %s", r.Kind, r.Resource, machine, reasonWords(r))
default:
@@ -550,11 +690,14 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
reasonAfter(r.Reason))
}
}
if waits || len(found) > 0 {
var said []string
if waits || len(found) > 0 || len(forOperator) > 0 || len(onWaiting) > 0 {
said := onWaiting
if waits {
said = append(said, wait)
}
if len(forOperator) > 0 {
said = append(said, operatorWaitSaid(module, machine, forOperator))
}
if len(found) > 0 {
said = append(said, fmt.Sprintf("on %s, %s uses %s as found and waits for the operator to hand it over "+
"(`nox node hand-over %s <directory>` on the control-node)", machine, module, strings.Join(found, ", "), machine))
@@ -593,7 +736,7 @@ func healthLines(h inventory.NodeHealth, had bool, now time.Time) []string {
line += " — " + r.Reason
}
if r.Restarts > 0 {
line += fmt.Sprintf(", %d restart(s) counted", r.Restarts)
line += ", " + restartsWords(r.Restarts)
}
if r.State == link.StateUnhealthy && h.Streaks[r.Module] < moduleUnhealthyAfter {
line += " (unconfirmed: said once)"
+45 -21
View File
@@ -237,7 +237,15 @@ func moduleCommand(ctx context.Context, args []string) error {
says = append(says, "provides "+describeOffers(m.Provides))
}
if len(m.Requires) > 0 {
says = append(says, "requires "+strings.Join(m.Requires, ", "))
// An optional requirement says so (novox/hq ADR 0330): it is not refused when absent.
required := make([]string, 0, len(m.Requires))
for _, r := range m.Requires {
if m.IsOptional(r) {
r += " (optional)"
}
required = append(required, r)
}
says = append(says, "requires "+strings.Join(required, ", "))
}
for _, c := range m.Claims {
says = append(says, "claims "+c.At()+"/"+c.Name)
@@ -563,18 +571,14 @@ func settingsCommand(ctx context.Context, args []string) error {
return err
}
m, ok := known[positionals[0]]
if !ok || len(m.Settings) == 0 {
if !ok || len(catalogue.PreferencesOf(m)) == 0 {
return nil
}
var layers []catalogue.Layer
if *node != "" {
if mesh, has, err := inv.Layer(ctx, "", positionals[0]); err != nil {
// Every source on the node: mesh config, node config, then the module's own (novox/hq issue 630).
if layers, err = inv.SettingsFor(ctx, *node, positionals[0]); err != nil {
return err
} else if has {
layers = append(layers, catalogue.Layer{From: catalogue.MeshWideLayer, Values: mesh})
}
if has {
layers = append(layers, catalogue.Layer{From: *node, Values: values})
}
} else if has {
layers = append(layers, catalogue.Layer{From: catalogue.MeshWideLayer, Values: values})
@@ -604,7 +608,7 @@ func settingsCommand(ctx context.Context, args []string) error {
var listed []preferencesOf
for _, e := range entries {
m := e.Manifest
if len(m.Settings) == 0 || (only != "" && m.Module != only) {
if len(catalogue.PreferencesOf(m)) == 0 || (only != "" && m.Module != only) {
continue
}
if *node != "" && !containsString(e.On, *node) {
@@ -674,7 +678,11 @@ func describeEffective(module, where string, values []catalogue.SettingSource) s
fmt.Fprintf(&b, "%s on %s, every value and where it comes from:\n", module, where)
for _, v := range values {
value, _ := json.Marshal(v.Value)
fmt.Fprintf(&b, " %s = %s (%s", v.Key, value, v.From)
from := v.From
if v.From == where && where != "the whole mesh" {
from = valueSource(module, where, v)
}
fmt.Fprintf(&b, " %s = %s (%s", v.Key, value, from)
if v.HasDefault && !v.FromDefault {
d, _ := json.Marshal(v.Default)
fmt.Fprintf(&b, "; the default is %s", d)
@@ -715,28 +723,27 @@ func describePreferences(modules []preferencesOf) string {
on = "on " + strings.Join(p.Nodes, ", ")
}
fmt.Fprintf(&b, "%s (%s)\n", p.Manifest.Module, on)
keys := make([]string, 0, len(p.Manifest.Settings))
for k := range p.Manifest.Settings {
declared := catalogue.PreferencesOf(p.Manifest)
keys := make([]string, 0, len(declared))
for k := range declared {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
d := p.Manifest.Settings[k]
d := declared[k]
def, _ := json.Marshal(d.Default)
fmt.Fprintf(&b, " %s, default %s: %s\n", k, def, d.Why)
falls := ""
if d.Predefined != "" {
falls = ", falls back to the predefined key " + d.Predefined
}
fmt.Fprintf(&b, " %s, default %s%s: %s\n", k, def, falls, d.Why)
for _, n := range p.Nodes {
for _, s := range p.On[n] {
if s.Key != k {
continue
}
v, _ := json.Marshal(s.Value)
from := s.From
if s.FromDefault {
from = catalogue.DefaultLayer
} else if s.From != catalogue.MeshWideLayer {
from = "the node"
}
fmt.Fprintf(&b, " %s: %s (%s)\n", n, v, from)
fmt.Fprintf(&b, " %s: %s (%s)\n", n, v, valueSource(p.Manifest.Module, n, s))
}
}
}
@@ -744,6 +751,18 @@ func describePreferences(modules []preferencesOf) string {
return b.String()
}
// valueSource is where a value on a node comes from, as the four sources are said (novox/hq issue 630):
// "default", "mesh config", "node config (<node>)", or "<module> on <node>" for the module's own value.
func valueSource(module, node string, s catalogue.SettingSource) string {
switch {
case s.FromDefault:
return catalogue.DefaultLayer
case s.From == catalogue.MeshConfigLayer, s.From == catalogue.NodeConfigLayer(node), s.From == catalogue.MeshWideLayer:
return s.From
}
return module + " on " + node
}
// nodeFlag is ` --node <node>` for a machine's layer, nothing for the whole mesh's.
func nodeFlag(node string) string {
if node == "" {
@@ -1051,6 +1070,11 @@ func namesNoInstallation(m catalogue.Manifest) error {
if problems := catalogue.TrustProblems(m); len(problems) > 0 {
return fmt.Errorf("%s", strings.Join(problems, "; "))
}
// And a module that declares a setting names the keys of its mergeable files (novox/hq ADR 0326
// §3, issue 574): judged where a definition is written, never where a stored build is read again.
if problems := catalogue.WrittenSettingsProblems(m); len(problems) > 0 {
return fmt.Errorf("%s", strings.Join(problems, "; "))
}
named := catalogue.InstallationProblems(m)
if len(named) == 0 {
return nil
+5 -4
View File
@@ -3,10 +3,12 @@ package main
import (
"context"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/beside"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
@@ -239,13 +241,12 @@ func TestATakeoverIsNotComposedForAHubPlacedOffItsTunnel(t *testing.T) {
}
}
// theResolver is the catalogue's dnsmasq module as it is, or the test is skipped where the
// catalogue is not beside this checkout.
// theResolver is the catalogue's dnsmasq module as it is (internal/beside).
func theResolver(t *testing.T) catalogue.Manifest {
t.Helper()
raw, err := os.ReadFile("../../../mesh-catalog/modules/dnsmasq/module.json")
raw, err := os.ReadFile(filepath.Join(beside.Catalogue(t), "dnsmasq", "module.json"))
if err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err)
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
+117 -4
View File
@@ -19,6 +19,7 @@ import (
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/overlay"
"github.com/novox/mesh-controller/internal/token"
)
@@ -44,7 +45,7 @@ func nodeCommand(ctx context.Context, args []string) error {
if len(args) != 2 {
return errors.New("node show <name>")
}
return showNode(ctx, inv, args[1])
return showNode(ctx, open, args[1])
case "add":
return addNode(ctx, inv, args[1:])
@@ -127,9 +128,14 @@ func nodeCommand(ctx context.Context, args []string) error {
// and whoever may call a verb includes agents.
return nodeSetuidSearch(ctx, open, args[1:])
case "judge-directories":
// Which of a node's declared directories the walk that follows no link another account could make would
// refuse (novox/hq issue 358). Read-only on the node and here, so a verb may run it too.
return nodeJudgeDirectories(ctx, open, args[1:])
default:
return fmt.Errorf("node has no %q; it has add, list, show, public-domain, account, agent-account, hand-over "+
"and setuid-search", args[0])
return fmt.Errorf("node has no %q; it has add, list, show, public-domain, account, agent-account, hand-over, "+
"setuid-search and judge-directories", args[0])
}
}
@@ -787,7 +793,8 @@ func roughly(d time.Duration) string {
//
// It is also where "what should it be configured as" is read. The same line that gates an
// assignment carries `card1-DP-1`, and a person composing settings for that machine needs it.
func showNode(ctx context.Context, inv *inventory.Inventory, name string) error {
func showNode(ctx context.Context, stored *stores, name string) error {
inv := stored.inventory
node, err := inv.NodeByName(ctx, name)
if err != nil {
return err
@@ -803,6 +810,19 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
if err := showMode(ctx, inv, node); err != nil {
return err
}
// What it was last sent, by whom, under which epoch, from which generation, naming which modules, and what
// it answered (novox/hq issue 485): the record of sends is read through this verb, never the store's reader.
// Unreadable is said, not taken for none.
switch last, found, err := inv.LastSend(ctx, name); {
case err != nil:
fmt.Printf(" what it was last sent could NOT be read: %v\n", err)
case !found:
fmt.Printf(" last sent no send recorded since the mesh began keeping them\n")
default:
for _, line := range lastSendLines(last) {
fmt.Println(line)
}
}
// Whom agents run as here, and whether that account can become root without a person (ADR 0266).
for _, line := range agentAccountLines(ctx, inv, node) {
fmt.Println(line)
@@ -856,6 +876,27 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
}
}
// What it is made of, in one block with its age (novox/hq issue 160): collected and shown in the same
// breath, so no fact the machine says is kept and read by nothing.
if account, had, err := inv.MachineOf(ctx, name); err != nil {
fmt.Printf("\n what it is made of could NOT be read: %v\n", err)
} else {
fmt.Println()
for _, line := range machineLines(account, had, time.Now()) {
fmt.Println(line)
}
// And which of the networks it is attached to count as its local network, and why the others
// do not (novox/hq ADR 0340 §8).
outward, err := inv.OutwardLinksOf(ctx, name)
if err != nil {
fmt.Printf(" its local network could NOT be judged: %v\n", err)
} else {
for _, line := range localNetworkLines(account, had, judgedNetworks(account, had, outward, overlay.Interface)) {
fmt.Println(line)
}
}
}
held, err := inv.Profile(ctx, name)
if err != nil {
return err
@@ -889,6 +930,24 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
if len(assigned) > 0 {
fmt.Printf("\n assigned: %s\n", strings.Join(assigned, ", "))
}
// And which of them a push leaves out, and why (novox/hq issue 380): judged as the push judges it, the same
// reason its condition says. Not computable is said, never read as "all applied".
plan, settings, err := planFor(ctx, stored, name)
switch {
case err != nil:
fmt.Printf("\n whether a send leaves any assigned module out is NOT known: %s\n", oneLine(err.Error()))
default:
for _, line := range notAppliedLines(name, leftOutOf(plan, settings, node.Adopted)) {
fmt.Println(line)
}
// And which optional requirements of its modules nothing assigned answers (novox/hq ADR 0330 §7).
if lines := unboundLines(plan); len(lines) > 0 {
fmt.Println()
for _, line := range lines {
fmt.Println(line)
}
}
}
return nil
}
@@ -912,3 +971,57 @@ func printJSONTo(w io.Writer, v any) error {
fmt.Fprintln(w, string(body))
return nil
}
const judgeDirectoriesUsage = "node judge-directories <node> — read-only: which of the directories <node> is " +
"declared the node-engine's walk that follows no link another account could make would refuse, one line each " +
"(novox/hq issue 358). Nothing is made, chmodded or chowned"
// nodeJudgeDirectories asks the node's engine to judge its declared directories, signed with the mesh's key as a
// hand-over is. Read-only: the engine changes nothing and keeps nothing.
func nodeJudgeDirectories(ctx context.Context, open *stores, args []string) error {
known := func(node string) error {
_, err := open.inventory.NodeByName(ctx, node)
return err
}
ask := func(node, by string) (link.HandOverAnswer, error) {
ident, err := open.Identity(ctx)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("the mesh's signing key cannot be read, so nothing was asked of %s: %w",
node, err)
}
address, err := broker.BusAddress()
if err != nil {
return link.HandOverAnswer{}, err
}
js, err := broker.Dial(address)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("cannot reach the bus, so nothing was asked of %s: %w", node, err)
}
defer js.Close()
return link.AskJudgeDirectories(ctx, js.Conn(), ident, node, by, link.HandOverWithin)
}
return judgeDirectoriesAsked(args, known, ask, os.Stdout)
}
// judgeDirectoriesAsked is the line with its two acts given: whether the mesh knows the node, and the ask. The
// engine's refusal is this command's failure.
func judgeDirectoriesAsked(args []string, known func(node string) error,
ask func(node, by string) (link.HandOverAnswer, error), out io.Writer) error {
if len(args) != 1 || args[0] == "" || strings.HasPrefix(args[0], "-") {
return errors.New(judgeDirectoriesUsage)
}
node := args[0]
if err := known(node); err != nil {
return fmt.Errorf("nothing was asked: %w", err)
}
answer, err := ask(node, handOverBy())
if err != nil {
return err
}
if answer.Refused != "" {
return fmt.Errorf("%s refused: %s", node, answer.Refused)
}
fmt.Fprintf(out, "%s (read-only: nothing on %s was changed)\n", node, node)
fmt.Fprintln(out, answer.Said)
return nil
}
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"bytes"
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq issue 237: every line of an act's answer comes from one resolution of the node taken after
// its write, so the answer agrees with `plan` for the node right after.
func aProviderAndItsConsumer(t *testing.T, open *stores) {
t.Helper()
register(t, open, catalogue.Manifest{Module: "db", Version: "1",
Provides: []catalogue.Offer{{Name: "rows", Scope: catalogue.ScopeNode}}})
register(t, open, catalogue.Manifest{Module: "app", Version: "1", Requires: []string{"rows"}})
}
// twoRivalsAndANote is a node that does not resolve — two modules claim one node seat there — and a
// module beside them that changes nothing about it.
func twoRivalsAndANote(t *testing.T, open *stores) {
t.Helper()
one, two := rivals()
register(t, open, one)
register(t, open, two)
register(t, open, catalogue.Manifest{Module: "notes", Version: "1"})
}
func TestAnUnassignThatLeavesItsNodeUnresolvedSaysSoOnceInItsAnswer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
twoRivalsAndANote(t, open)
// Straight into the store, as a node that came to this before the act.
for _, m := range []string{"rival-one", "rival-two", "notes"} {
if _, err := open.inventory.Assign(ctx, "laptop", m); err != nil {
t.Fatal(err)
}
}
said := answered(t, func() error { return assignCommand(ctx, "unassign", []string{"laptop", "notes"}) })
if _, _, err := planFor(ctx, open, "laptop"); err == nil || !unresolvable(err) {
t.Fatalf("laptop resolves with two claimants of one seat: %v", err)
}
if n := strings.Count(said, "laptop does not resolve now"); n != 1 {
t.Errorf("the answer says laptop does not resolve %d times, want once:\n%s", n, said)
}
if strings.Contains(said, "is left out of the rest of the mesh") {
t.Errorf("the answer carries a line from a read that is not its resolution:\n%s", said)
}
}
func TestAnUnassignThatLeavesItsNodeResolvedSaysNothingOfIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
aProviderAndItsConsumer(t, open)
if _, err := assign(ctx, open, "laptop", "db", "app"); err != nil {
t.Fatal(err)
}
said, err := unassign(ctx, open, "laptop", "app")
if err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
if strings.Contains(said, "does not resolve") || strings.Contains(said, "could not be read") {
t.Errorf("a node that resolves is said not to:\n%s", said)
}
}
func TestAnAssignThatLeavesItsNodeUnresolvedSaysOnlyThePlansRefusal(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
twoRivalsAndANote(t, open)
if _, err := assign(ctx, open, "laptop", "rival-one"); err != nil {
t.Fatal(err)
}
// Kept and not resolvable: a second claimant of the seat, which assignment keeps.
said, err := assign(ctx, open, "laptop", "rival-two")
if err == nil {
t.Fatalf("two claimants of one node seat resolved:\n%s", said)
}
_, _, planErr := planFor(ctx, open, "laptop")
if planErr == nil || planErr.Error() != err.Error() {
t.Errorf("the act's refusal is not the plan's right after:\nact: %v\nplan: %v", err, planErr)
}
if strings.Contains(said, "now has") {
t.Errorf("the answer says a seat is now held on a node that does not resolve:\n%s", said)
}
}
// The other readers of the whole mesh still say a machine they leave out, on standard error (novox/hq
// issue 188); an act sends it elsewhere.
func TestAMachineLeftOutIsSaidWhereTheContextSays(t *testing.T) {
var buf bytes.Buffer
sayLeftOut(leftOutSaidBy(context.Background(), &buf), "laptop", errors.New("why"))
if buf.String() != "laptop is left out of the rest of the mesh: it does not resolve: why\n" {
t.Errorf("said %q", buf.String())
}
got := answered(t, func() error {
sayLeftOut(context.Background(), "laptop", errors.New("why"))
return nil
})
if !strings.Contains(got, "laptop is left out") {
t.Errorf("without an act's writer the line did not reach standard error: %q", got)
}
}
+386
View File
@@ -0,0 +1,386 @@
package main
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A part that waits for the operator's secret or setting (novox/hq ADR 0283, issue 386).
//
// **A module's tool check may say it waits**: nothing of it is wrong but a part that cannot work until the operator
// gives one of its own secrets or one of its settings. The node-engine states such a resource `waiting`, with what
// it waits for. A module saying so is an assertion, so **the controller checks each wait before it excuses it**:
//
// - a wait for a secret names an own secret the module's manifest declares — by its name, or as a member of a
// secret family — said `"issued-by": "outside"`, and the store holds no value a person gave for it on that
// machine;
// - a wait for a setting names a setting the manifest declares (checked by name only: the controller cannot tell
// whether a free-form value covers a part, and the needs-operator condition is where a false one shows).
//
// An excused wait is read by the first-node gate as *waits for a person* (ADR 0254), a pass carried in the verdict,
// for any build of the module — a secret not given is owed by every build alike. It is said to the operator as
// `module.<module>.<machine>.needs-operator`, naming the act. A wait that fails the check is judged unhealthy, saying
// why, and raises the module's `unhealthy` condition.
// kindNeedsOperator is a module's condition while a part of it waits for the operator's secret or setting.
const kindNeedsOperator = "needs-operator"
// needsOperatorKey is a module's needs-operator condition on a machine.
func needsOperatorKey(module, node string) string {
return conditions.Key(conditions.ScopeModule, module+"."+node, kindNeedsOperator)
}
// operatorWaitFacts is what the controller holds to check a module's waits: the manifest judged per module, and per
// "<module>@<machine>" the own secrets a person gave there, with when. A module or a machine absent is not known,
// and no wait of it is excused.
type operatorWaitFacts struct {
manifests map[string]catalogue.Manifest
given map[string]map[string]time.Time
// outside is, per provision some module offers with a credential issued outside the mesh, that provider module
// and the machines it is assigned to (novox/hq ADR 0337); pairsGiven, per "<module>@<machine>", the provisions
// it holds a value a person gave for, sealed to the keys both ends have now. Absent is not known: no wait for a
// provision of it is excused.
outside map[string]outsideProvider
pairsGiven map[string]map[string]bool
}
// outsideProvider is the module offering a provision whose credential only a person gives, and where it runs.
type outsideProvider struct {
Module string
Machines []string
}
// acceptLine is the one line that gives an outside provision's credential for every consumer of its provider: on
// the provider's machine when it runs on one, named as <machine> when on several or none.
func (p outsideProvider) acceptLine(provision string) string {
machine := "<its machine>"
if len(p.Machines) == 1 {
machine = p.Machines[0]
}
return inventory.AcceptLine(machine, p.Module, provision)
}
// checkWait is nil when a wait is excused, and otherwise why not, in words. Pure.
func checkWait(module, machine string, w inventory.Wait, f operatorWaitFacts) error {
m, known := f.manifests[module]
if !known {
return fmt.Errorf("says it waits for %s, and the mesh holds no manifest of %s to check it against", waitNames(w), module)
}
named := 0
for _, n := range []string{w.Secret, w.Setting, w.Provision} {
if n != "" {
named++
}
}
switch {
case named != 1:
return fmt.Errorf("says it waits, naming %s, where a wait names one secret, one setting or one provision",
waitNames(w))
case w.Provision != "":
return checkProvisionWait(module, machine, w, m, f)
case w.Setting != "":
if !catalogue.DeclaresPreference(m, w.Setting) {
return fmt.Errorf("says it waits for the setting %s, which %s does not declare", w.Setting, module)
}
return nil
}
own, _, declared := m.OwnSecrets.Lookup(w.Secret)
if !declared {
return fmt.Errorf("says it waits for the secret %s, which %s does not declare", w.Secret, module)
}
if own.IssuedBy != catalogue.IssuedOutside {
return fmt.Errorf("says it waits for the secret %s, which the mesh makes itself: only a secret issued outside "+
"the mesh waits for the operator", w.Secret)
}
given, readable := f.given[module+"@"+machine]
if !readable {
return fmt.Errorf("says it waits for the secret %s, and what was given on %s could not be read", w.Secret, machine)
}
if at, was := given[w.Secret]; was {
return fmt.Errorf("says it waits for the secret %s, which was given at %s", w.Secret,
at.UTC().Format("2006-01-02 15:04 MST"))
}
return nil
}
// checkProvisionWait excuses a wait for a provision's credential only when the store bears it out (novox/hq ADR 0337
// §6): the module requires the provision, a provider offers it with a credential issued outside the mesh, and the
// module on this machine holds no value a person gave for it under the keys both ends have now.
func checkProvisionWait(module, machine string, w inventory.Wait, m catalogue.Manifest, f operatorWaitFacts) error {
if !slices.Contains(m.Requires, w.Provision) {
return fmt.Errorf("says it waits for the provision %s, which %s does not require", w.Provision, module)
}
if _, outside := f.outside[w.Provision]; !outside {
return fmt.Errorf("says it waits for the provision %s, whose credential the mesh gives itself: only a "+
"credential issued outside the mesh waits for the operator", w.Provision)
}
given, readable := f.pairsGiven[module+"@"+machine]
if !readable {
return fmt.Errorf("says it waits for the provision %s, and what was given on %s could not be read", w.Provision, machine)
}
if given[w.Provision] {
return fmt.Errorf("says it waits for the provision %s, whose credential was given", w.Provision)
}
return nil
}
// waitRefusedPrefix opens every reason checkWait gives, so the words of a refused wait are told from a check's own.
const waitRefusedPrefix = "says it waits"
// waitNames is what a wait names, as "the secret x" or "the setting y".
func waitNames(w inventory.Wait) string {
var names []string
if w.Secret != "" {
names = append(names, "the secret "+w.Secret)
}
if w.Setting != "" {
names = append(names, "the setting "+w.Setting)
}
if w.Provision != "" {
names = append(names, "the provision "+w.Provision)
}
if len(names) == 0 {
return "nothing"
}
return strings.Join(names, " and ")
}
// checkWaiting reads one machine's resources against the facts: every waiting resource whose waits all check out is
// kept as said; one with a wait that does not, or with no wait at all, is answered as unhealthy with why. Pure; the
// statement as kept is not changed.
func checkWaiting(machine string, rs []inventory.ResourceHealth, f operatorWaitFacts) []inventory.ResourceHealth {
out := make([]inventory.ResourceHealth, 0, len(rs))
for _, r := range rs {
if r.State == link.StateWaiting {
var why error
if len(r.Waits) == 0 {
why = fmt.Errorf("says it waits, and names nothing it waits for")
}
waits := make([]inventory.Wait, 0, len(r.Waits))
for _, w := range r.Waits {
if why == nil {
why = checkWait(r.Module, machine, w, f)
}
if p, outside := f.outside[w.Provision]; outside && w.Provision != "" {
w.Act = p.acceptLine(w.Provision)
}
waits = append(waits, w)
}
r.Waits = waits
if why != nil {
r.State, r.Reason = link.StateUnhealthy, why.Error()
}
}
out = append(out, r)
}
return out
}
// operatorWait is whether everything not healthy of a module on a machine is waiting with its waits checked
// (checkWaiting already applied), and those waits. A module with anything unhealthy, starting or unknown beside it
// does not wait: it is judged as before.
func operatorWait(module string, rs []inventory.ResourceHealth) ([]inventory.Wait, bool) {
var waits []inventory.Wait
for _, r := range rs {
if r.Module != module {
continue
}
switch r.State {
case link.StateHealthy:
case link.StateWaiting:
waits = append(waits, r.Waits...)
default:
return nil, false
}
}
return waits, len(waits) > 0
}
// operatorWaitSaid is a module's wait for the operator in one sentence, for the gate's verdict and the condition's
// summary: what the operator gives and what it names, and for a secret the line that opens the desk prompt.
func operatorWaitSaid(module, machine string, waits []inventory.Wait) string {
var parts []string
for _, w := range waits {
part := fmt.Sprintf("%s (%s", w.What, waitNames(w))
if w.Secret != "" {
part += fmt.Sprintf(", given with `nox secret ask %s %s %s`", machine, module, w.Secret)
}
if w.Provision != "" && w.Act != "" {
// One line for the provider, which seals it to every consumer of it (ADR 0337 §3, §7).
part += fmt.Sprintf(", given for every consumer with `%s`", w.Act)
}
parts = append(parts, part+")")
}
return fmt.Sprintf("%s on %s waits for the operator: %s", module, machine, strings.Join(parts, "; "))
}
// needsOperatorObservation is a module whose only parts not healthy wait for the operator (ADR 0283): the operator's,
// a warning however long it stands, its plain words naming the act and never saying there is nothing to do.
func needsOperatorObservation(module, node string, waits []inventory.Wait, rs []inventory.ResourceHealth) conditions.Observation {
o := moduleUnhealthyObservation(module, node, rs)
o.Token, o.Kind, o.Resolver, o.Severity = kindNeedsOperator, kindNeedsOperator, conditions.ResolverOperator, conditions.Warning
o.Summary = operatorWaitSaid(module, node, waits)
w := needsOperatorWords(module, node, waits)
o.Headline, o.Explanation, o.Needs, o.Resolved, o.Actions = w.Headline, w.Explanation, w.Needs, w.Resolved, nil
return o
}
// needsOperatorWords is what the operator reads of a module waiting for them (ADR 0253, ADR 0283): the act, for a
// secret typed at the machine's desk prompt and for a setting approved when an agent proposes it. The secret's and
// the setting's names, and the line, are in the summary for whoever looks closer.
func needsOperatorWords(module, node string, waits []inventory.Wait) words {
var acts []string
seen := map[string]bool{}
secret, provision := false, false
for _, w := range waits {
var act string
switch {
case w.Secret != "":
act, secret = fmt.Sprintf("type %s at %s's desk prompt", w.What, node), true
case w.Setting != "":
act = fmt.Sprintf("approve %s of %s on %s when it is proposed to you", w.Setting, module, node)
case w.Provision != "":
// ADR 0337 §7: "give <what> of <provider> on <machine>"; the line is in the summary.
act, provision = "give "+w.What, true
}
if act != "" && !seen[act] {
seen[act] = true
acts = append(acts, act)
}
}
needs := strings.Join(acts, "; and ") + "."
// Plain words hold one sentence of at most conditions.NeedsMax characters: several acts are named in the
// summary instead.
if len(acts) == 0 || len(needs) > conditions.NeedsMax {
needs = fmt.Sprintf("give what %s waits for on %s; the details name each secret and setting.", module, node)
}
explanation := fmt.Sprintf("Part of %s on %s cannot work until you give what it waits for.", module, node)
if secret {
explanation += " A hidden prompt opens at the desk when the secret is asked for, and what you type there " +
"is sealed to the machine."
}
if provision {
explanation += " A key only you can give is given once for the app that issued it, and reaches every " +
"module that uses it."
}
explanation += " Its update is in place and nothing was undone; it carries on by itself once it is given."
return words{
Headline: fmt.Sprintf("%s waits for you on %s", module, node),
Needs: needs,
Explanation: explanation,
Resolved: fmt.Sprintf("%s on %s no longer waits for you", module, node),
}
}
// readWaitFacts reads what the controller holds to check the waits of the modules named on one machine: the
// manifests (the catalogue's, or those given) and the secrets given there. A read that fails leaves that module
// unknown, so none of its waits is excused.
func readWaitFacts(ctx context.Context, inv *inventory.Inventory, machine string, modules []string,
manifests map[string]catalogue.Manifest, f *operatorWaitFacts) {
if f.manifests == nil {
f.manifests = map[string]catalogue.Manifest{}
}
if f.given == nil {
f.given = map[string]map[string]time.Time{}
}
if f.outside == nil {
f.outside = map[string]outsideProvider{}
}
if f.pairsGiven == nil {
f.pairsGiven = map[string]map[string]bool{}
}
if inv == nil {
return
}
var shelf map[string]catalogue.Manifest
sort.Strings(modules)
for _, module := range modules {
if _, has := f.manifests[module]; !has {
if m, given := manifests[module]; given {
f.manifests[module] = m
} else {
if shelf == nil {
var err error
if shelf, err = inv.Catalogue(ctx); err != nil {
shelf = map[string]catalogue.Manifest{}
}
}
if m, known := shelf[module]; known {
f.manifests[module] = m
}
}
}
if _, read := f.given[module+"@"+machine]; !read {
if given, err := inv.GivenOwnSecrets(ctx, machine, module); err == nil {
f.given[module+"@"+machine] = given
}
}
if _, read := f.pairsGiven[module+"@"+machine]; !read {
if given, err := inv.GivenPairsOf(ctx, machine, module); err == nil {
f.pairsGiven[module+"@"+machine] = given
}
}
// The provisions it requires whose credential only a person gives, and where their provider runs (ADR 0337).
m, known := f.manifests[module]
if !known {
continue
}
for _, provision := range m.Requires {
if _, read := f.outside[provision]; read {
continue
}
if shelf == nil {
var err error
if shelf, err = inv.Catalogue(ctx); err != nil {
shelf = map[string]catalogue.Manifest{}
}
}
provider, outside := catalogue.OffersIssuedOutside(shelf, provision)
if !outside {
continue
}
f.outside[provision] = outsideProvider{Module: provider, Machines: machinesRunning(ctx, inv, provider)}
}
}
}
// machinesRunning is every machine a module is assigned to, by name; a read that fails is left out.
func machinesRunning(ctx context.Context, inv *inventory.Inventory, module string) []string {
nodes, err := inv.Nodes(ctx)
if err != nil {
// Only the summary's accept line reads it, which then names the provider's machine "<its machine>".
// empty-on-error: no wait is excused or refused by it
return nil
}
var out []string
for _, n := range nodes {
assigned, err := inv.Assigned(ctx, n.Name)
if err == nil && slices.Contains(assigned, module) {
out = append(out, n.Name)
}
}
sort.Strings(out)
return out
}
// waitingModules is every module with a waiting resource in a statement.
func waitingModules(rs []inventory.ResourceHealth) []string {
seen := map[string]bool{}
var out []string
for _, r := range rs {
if r.State == link.StateWaiting && r.Module != "" && !seen[r.Module] {
seen[r.Module] = true
out = append(out, r.Module)
}
}
return out
}
+264
View File
@@ -0,0 +1,264 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A part that waits for the operator's secret or setting (novox/hq ADR 0283, issue 386).
var mountsManifest = catalogue.Manifest{Module: "mounts", Version: "1",
Settings: map[string]catalogue.SettingDeclaration{"smb-users": {}, "sources": {}},
OwnSecrets: catalogue.OwnSecrets{
"smb-password-*": {Path: "/s/smb-password-*.secret", IssuedBy: catalogue.IssuedOutside},
"broker": {Path: "/s/broker"},
"made": {Path: "/s/made", Taken: catalogue.TakenAtStart},
"licence": {Path: "/s/licence", IssuedBy: catalogue.IssuedOutside},
}}
var passwordWait = inventory.Wait{Part: "the source games", Secret: "smb-password-games",
What: "the password of the source games"}
var usernameWait = inventory.Wait{Part: "the source games", Setting: "smb-users", What: "the username of the source games"}
func waitingResource(waits ...inventory.Wait) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: "mounts", Resource: "mounts.watch", Kind: "process",
Target: "mesh-mounts-watch.service", State: link.StateWaiting, Check: "tool",
Reason: "the source games waits for its password", Waits: waits}
}
func factsGiven(given map[string]time.Time) operatorWaitFacts {
return operatorWaitFacts{manifests: map[string]catalogue.Manifest{"mounts": mountsManifest},
given: map[string]map[string]time.Time{"mounts@workstation": given}}
}
// Rule 3: a wait is excused only when what it names is the module's, issued outside the mesh, and not given there.
func TestAWaitIsExcusedOnlyWhenItChecksOut(t *testing.T) {
at := time.Date(2026, 10, 10, 15, 8, 0, 0, time.UTC)
for _, c := range []struct {
name string
w inventory.Wait
f operatorWaitFacts
says string // "" when excused
}{
{"a member of an outside family, not given", passwordWait, factsGiven(nil), ""},
{"an outside secret by name, not given", inventory.Wait{Part: "p", Secret: "licence", What: "w"}, factsGiven(nil), ""},
{"a declared setting", usernameWait, factsGiven(nil), ""},
{"a member given", passwordWait, factsGiven(map[string]time.Time{"smb-password-games": at}), "was given at 2026-10-10 15:08"},
{"a secret not declared", inventory.Wait{Part: "p", Secret: "smb-credentials", What: "w"}, factsGiven(nil), "does not declare"},
{"a secret the mesh makes", inventory.Wait{Part: "p", Secret: "made", What: "w"}, factsGiven(nil), "mesh makes itself"},
{"the bus account", inventory.Wait{Part: "p", Secret: "broker", What: "w"}, factsGiven(nil), "mesh makes itself"},
{"a setting not declared", inventory.Wait{Part: "p", Setting: "logins", What: "w"}, factsGiven(nil), "does not declare"},
{"both", inventory.Wait{Part: "p", Secret: "licence", Setting: "smb-users", What: "w"}, factsGiven(nil), "one secret, one setting or one provision"},
{"neither", inventory.Wait{Part: "p", What: "w"}, factsGiven(nil), "one secret, one setting or one provision"},
{"no manifest known", passwordWait, operatorWaitFacts{}, "no manifest"},
{"what was given cannot be read", passwordWait,
operatorWaitFacts{manifests: map[string]catalogue.Manifest{"mounts": mountsManifest}}, "could not be read"},
} {
err := checkWait("mounts", "workstation", c.w, c.f)
switch {
case c.says == "" && err != nil:
t.Errorf("%s: refused: %v", c.name, err)
case c.says != "" && (err == nil || !strings.Contains(err.Error(), c.says)):
t.Errorf("%s: %v; want a refusal saying %q", c.name, err, c.says)
}
}
}
// A waiting resource whose wait does not check out, or that names nothing, is judged unhealthy, saying why.
func TestAWaitThatFailsItsCheckIsUnhealthy(t *testing.T) {
given := factsGiven(map[string]time.Time{"smb-password-games": time.Now()})
got := checkWaiting("workstation", []inventory.ResourceHealth{waitingResource(passwordWait)}, given)
if got[0].State != link.StateUnhealthy || !strings.Contains(got[0].Reason, "was given") {
t.Fatalf("a wait for a secret given: %+v", got[0])
}
got = checkWaiting("workstation", []inventory.ResourceHealth{waitingResource()}, factsGiven(nil))
if got[0].State != link.StateUnhealthy || !strings.Contains(got[0].Reason, "names nothing") {
t.Fatalf("a wait naming nothing: %+v", got[0])
}
got = checkWaiting("workstation", []inventory.ResourceHealth{waitingResource(passwordWait, usernameWait)}, factsGiven(nil))
if got[0].State != link.StateWaiting {
t.Fatalf("two waits that check out: %+v", got[0])
}
}
// Rule 4: the gate passes a module whose only parts not healthy wait for the operator, carrying the wait; anything
// else beside it is judged as before; and any build is excused, not only one that added something.
func TestTheGatePassesAWaitForTheOperatorCarriedAlong(t *testing.T) {
now := time.Now()
since := now.Add(-time.Minute)
healthy := inventory.ResourceHealth{Module: "mounts", Resource: "mounts.apply", Kind: "process",
Target: "mesh-mounts-apply.service", State: link.StateHealthy}
f := gateFacts{now: now, waits: factsGiven(nil), groupsAdded: map[string]bool{"mounts": false},
health: map[string]inventory.NodeHealth{"workstation": {Node: "workstation", HeardAt: now,
Resources: []inventory.ResourceHealth{healthy, waitingResource(passwordWait)}}}}
h, why := moduleHealthWord("mounts", "workstation", since, f)
if h != healthPerson || !strings.Contains(why, "waits for the operator: the password of the source games") ||
!strings.Contains(why, "nox secret ask workstation mounts smb-password-games") {
t.Fatalf("an excused wait reads %v %q; want a wait for a person naming the act", h, why)
}
// A second resource unhealthy beside it: not yet, as before.
down := healthy
down.State, down.Reason = link.StateUnhealthy, "down"
f.health["workstation"] = inventory.NodeHealth{Node: "workstation", HeardAt: now,
Resources: []inventory.ResourceHealth{down, waitingResource(passwordWait)}}
if h, why := moduleHealthWord("mounts", "workstation", since, f); h != healthNotYet {
t.Fatalf("a resource down beside the wait reads %v %q", h, why)
}
// The password given and the module still saying it waits: not excused.
f.waits = factsGiven(map[string]time.Time{"smb-password-games": now})
f.health["workstation"] = inventory.NodeHealth{Node: "workstation", HeardAt: now,
Resources: []inventory.ResourceHealth{healthy, waitingResource(passwordWait)}}
if h, why := moduleHealthWord("mounts", "workstation", since, f); h != healthNotYet || !strings.Contains(why, "was given") {
t.Fatalf("a wait for a secret given reads %v %q", h, why)
}
// Facts never read (no manifest): never excused.
f.waits = operatorWaitFacts{}
if h, _ := moduleHealthWord("mounts", "workstation", since, f); h != healthNotYet {
t.Fatalf("a wait nothing could check reads %v", h)
}
}
func needsOperatorOpen(t *testing.T, k *conditions.Keeper) (*conditions.Condition, []conditions.Condition) {
t.Helper()
open, err := k.Open(t.Context())
if err != nil {
t.Fatal(err)
}
for i, c := range open {
if c.Key == needsOperatorKey("mounts", "workstation") {
return &open[i], open
}
}
return nil, open
}
// Rule 5: two statements of an excused wait raise needs-operator, the operator's, a warning however long, naming the
// act; a statement without it clears it.
func TestTheNeedsOperatorConditionNamesTheAct(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
rs := map[string][]inventory.ResourceHealth{"mounts": {waitingResource(passwordWait)}}
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": 1}, time.Now()); err != nil {
t.Fatal(err)
}
if got, _ := needsOperatorOpen(t, k); got != nil {
t.Fatal("raised on one statement")
}
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": 2}, time.Now()); err != nil {
t.Fatal(err)
}
got, open := needsOperatorOpen(t, k)
if got == nil {
t.Fatalf("not raised on two statements: %+v", open)
}
for _, c := range open {
if c.Kind == kindModuleUnhealthy {
t.Fatalf("raised as a fault too: %+v", c)
}
}
if got.Kind != kindNeedsOperator || got.Resolver != conditions.ResolverOperator || got.Severity != conditions.Warning {
t.Fatalf("the condition: %+v", got)
}
if !strings.Contains(got.Needs, "type the password of the source games at workstation's desk prompt") ||
!strings.Contains(got.Explanation, "hidden prompt opens at the desk") {
t.Fatalf("its needs do not name the act: %q", got.Needs)
}
if strings.Contains(strings.ToLower(got.Explanation), "nothing for you") || !strings.Contains(got.Explanation, "nothing was undone") {
t.Fatalf("its explanation: %q", got.Explanation)
}
if !strings.Contains(got.Summary, "smb-password-games") || !strings.Contains(got.Summary, "nox secret ask workstation mounts smb-password-games") {
t.Fatalf("its summary does not name the secret and the line: %q", got.Summary)
}
// Long open is still a warning: only the operator can end it.
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": 3}, time.Now().Add(48*time.Hour)); err != nil {
t.Fatal(err)
}
if got, _ := needsOperatorOpen(t, k); got == nil || got.Severity == conditions.Urgent {
t.Fatalf("after two days: %+v", got)
}
// Given: the next statement does not say it, and it clears.
if err := judgeModuleHealth(ctx, nil, k, "workstation", map[string][]inventory.ResourceHealth{}, nil, time.Now()); err != nil {
t.Fatal(err)
}
if got, _ := needsOperatorOpen(t, k); got != nil {
t.Fatal("not cleared once given")
}
}
func TestASettingsWaitAsksForTheApproval(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
rs := map[string][]inventory.ResourceHealth{"mounts": {waitingResource(usernameWait)}}
for i := 1; i <= 2; i++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": i}, time.Now()); err != nil {
t.Fatal(err)
}
}
got, _ := needsOperatorOpen(t, k)
if got == nil || !strings.Contains(got.Needs, "approve smb-users of mounts on workstation when it is proposed to you") {
t.Fatalf("the condition: %+v", got)
}
}
// A wait that fails its check is the module's own fault: unhealthy, with why, and no needs-operator.
func TestAWaitThatFailsItsCheckRaisesUnhealthy(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
checked := checkWaiting("workstation", []inventory.ResourceHealth{waitingResource(passwordWait)},
factsGiven(map[string]time.Time{"smb-password-games": time.Now()}))
rs := map[string][]inventory.ResourceHealth{"mounts": checked}
for i := 1; i <= 2; i++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": i}, time.Now()); err != nil {
t.Fatal(err)
}
}
got, open := needsOperatorOpen(t, k)
if got != nil {
t.Fatalf("a wait for a secret given raised needs-operator: %+v", got)
}
unhealthy := false
for _, c := range open {
unhealthy = unhealthy || c.Key == moduleUnhealthyKey("mounts", "workstation")
}
if !unhealthy {
t.Fatalf("not raised as unhealthy: %+v", open)
}
}
// A module that waited and is then broken says so when the wait clears, and the other way round.
func TestANeedsOperatorThatBecameUnhealthySaysSo(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
waiting := map[string][]inventory.ResourceHealth{"mounts": {waitingResource(passwordWait)}}
for i := 1; i <= 2; i++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", waiting, map[string]int{"mounts": i}, time.Now()); err != nil {
t.Fatal(err)
}
}
broken := waitingResource()
broken.State, broken.Reason, broken.Waits = link.StateUnhealthy, "the source games refused its login", nil
for i := 3; i <= 4; i++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", map[string][]inventory.ResourceHealth{"mounts": {broken}},
map[string]int{"mounts": i}, time.Now()); err != nil {
t.Fatal(err)
}
}
got, open := needsOperatorOpen(t, k)
if got != nil {
t.Fatal("needs-operator still open after it became a fault")
}
found := false
for _, c := range open {
found = found || c.Key == moduleUnhealthyKey("mounts", "workstation")
}
if !found {
t.Fatalf("the fault is not raised: %+v", open)
}
}
@@ -0,0 +1,83 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// One accept for a provider reaches every consumer that binds the provision from it, and each consumer's send
// places it; the per-pair form naming a consumer is refused for such a provision, naming the provider's form; a
// value the mesh made before the rule is not placed (novox/hq ADR 0337 §2, §3, §4, issue 164, issue 165). Needs the
// store (MESH_TEST_POSTGRES), and is named in testdata/store-tests.
func TestOneAcceptForAProviderIsPlacedForEveryConsumer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
// Before the rule: tv's offer said nothing of its credential, and the send made one per pair.
register(t, open, catalogue.Manifest{Module: "tv", Version: "1",
Provides: []catalogue.Offer{{Name: "tv-api", Scope: catalogue.ScopeMesh}}})
for _, m := range []string{"requests", "indexer"} {
register(t, open, catalogue.Manifest{Module: m, Version: "1", Requires: []string{"tv-api"},
Secrets: map[string]string{"tv-api": "/var/lib/mesh/" + m + "/tv-api.secret"}})
}
for _, a := range [][2]string{{"anchor", "tv"}, {"laptop", "requests"}, {"laptop", "indexer"}} {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatal(err)
}
}
placed := func(module string) bool {
t.Helper()
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatalf("laptop was refused: %v", err)
}
declared, err := declarationFor(ctx, open, "laptop", plan, settings)
if err != nil {
t.Fatalf("laptop did not compose: %v", err)
}
for _, r := range declared.Resources {
if r["path"] == "/var/lib/mesh/"+module+"/tv-api.secret" && r["sealed"] != nil {
return true
}
}
return false
}
if !placed("requests") || !placed("indexer") {
t.Fatal("before the rule, the send made and placed a credential per pair")
}
// The offer now says only a person gives it: the values the mesh made are not placed.
register(t, open, catalogue.Manifest{Module: "tv", Version: "2",
Provides: []catalogue.Offer{{Name: "tv-api", Scope: catalogue.ScopeMesh,
Credential: &catalogue.OfferCredential{IssuedBy: catalogue.IssuedOutside}}}})
if placed("requests") || placed("indexer") {
t.Fatal("a credential nobody gave was placed")
}
// The per-pair form is refused, naming the provider's form, and reads nothing.
file := filepath.Join(t.TempDir(), "key")
if err := os.WriteFile(file, []byte("the app's key\n"), 0o600); err != nil {
t.Fatal(err)
}
err := secretCommand(ctx, []string{"accept", "laptop", "requests", "tv-api", "--provider", "anchor", "--from", file})
if err == nil || !strings.Contains(err.Error(), "secret accept anchor tv tv-api") {
t.Fatalf("the per-pair form was not refused naming the provider's: %v", err)
}
consumers, unresolved, err := consumersFrom(ctx, open, "anchor", "tv", "tv-api")
if err != nil || len(unresolved) != 0 {
t.Fatalf("the consumers of tv on anchor could not be found: %v %v", err, unresolved)
}
if len(consumers) != 2 {
t.Fatalf("the consumers of tv on anchor are %+v; want requests and indexer on laptop", consumers)
}
if _, err := open.inventory.AcceptSecretForProvision(ctx, "tv-api", "anchor", "tv", "the app's key", consumers); err != nil {
t.Fatal(err)
}
if !placed("requests") || !placed("indexer") {
t.Fatal("the value given once for the provider was not placed for every consumer")
}
}
@@ -0,0 +1,120 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// An own secret issued outside the mesh that nobody gave is never made (novox/hq issue 401, ADR 0228). Seen on
// 2026-10-10: the send made a random value for the mounts module's smb-credentials and placed it, and the module read
// it as a credential given and wrong. The send now makes none, places none, and places a value only once a person
// gave one — and a value the mesh made for it before is not placed either.
// anAssistant is a module with a secret the mesh makes and one only an outside party issues.
func anAssistant() catalogue.Manifest {
return catalogue.Manifest{Module: "assistant", Version: "1",
OwnSecrets: catalogue.OwnSecrets{
"password": {Path: "/var/lib/mesh/assistant/password", Taken: catalogue.TakenAtStart},
"vendor-key": {Path: "/var/lib/mesh/assistant/vendor-key", Taken: catalogue.TakenAtStart, IssuedBy: catalogue.IssuedOutside},
},
Resources: []map[string]any{
{"id": "state", "type": "directory", "path": "/var/lib/mesh/assistant", "mode": "0700"},
}}
}
// sentTo composes what a push sends to a machine, on the send path.
func sentTo(t *testing.T, open *stores, node string) []map[string]any {
t.Helper()
plan, settings, err := planFor(t.Context(), open, node)
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(t.Context(), open, node, plan, settings)
if err != nil {
t.Fatalf("the send to %s did not compose: %v", node, err)
}
return declared.Resources
}
func placedAt(resources []map[string]any, path string) map[string]any {
for _, r := range resources {
if r["path"] == path && r["sealed"] != nil {
return r
}
}
return nil
}
func TestTheSendMakesNoValueForAnOutsideSecretNobodyGave(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, anAssistant())
if _, err := assign(ctx, open, "laptop", "assistant"); err != nil {
t.Fatal(err)
}
sent := sentTo(t, open, "laptop")
if placedAt(sent, "/var/lib/mesh/assistant/password") == nil {
t.Fatal("the secret the mesh makes was not placed")
}
if got := placedAt(sent, "/var/lib/mesh/assistant/vendor-key"); got != nil {
t.Fatalf("a secret nobody gave was placed: %v", got)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "assistant", "vendor-key"); err != nil || held {
t.Fatalf("the send made a value for a secret issued outside the mesh (held %v, %v)", held, err)
}
// A value the mesh made before this rule is not one a person gave: it is not placed.
if _, err := open.inventory.SecretForModule(ctx, "laptop", "assistant", "vendor-key"); err != nil {
t.Fatal(err)
}
if got := placedAt(sentTo(t, open, "laptop"), "/var/lib/mesh/assistant/vendor-key"); got != nil {
t.Fatalf("a value the mesh made for a secret issued outside was placed: %v", got)
}
// Given, it is placed.
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "assistant", "vendor-key", "the vendor's key"); err != nil {
t.Fatal(err)
}
given, ok, err := open.inventory.GivenOwnSecret(ctx, "laptop", "assistant", "vendor-key")
if err != nil || !ok || !given.Current {
t.Fatalf("the value given is not read as given: %+v %v %v", given, ok, err)
}
placed := placedAt(sentTo(t, open, "laptop"), "/var/lib/mesh/assistant/vendor-key")
if placed == nil || placed["sealed"] != given.Sealed {
t.Fatalf("the value given is placed as %v", placed)
}
}
// The read paths answer as the send does: a question about the machine composes without the secret nobody gave, and
// asking writes nothing.
func TestAQuestionComposesWithoutAnOutsideSecretNobodyGave(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, anAssistant())
if _, err := assign(ctx, open, "laptop", "assistant"); err != nil {
t.Fatal(err)
}
pushedBy(t, open, "laptop") // the mesh's own secret is made by the send
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
for _, choosing := range []Choosing{Reading, Foreseeing} {
got, err := declarationWith(ctx, open, "laptop", plan, settings, gens, choosing)
if err != nil {
t.Fatalf("%v: %v", choosing, err)
}
if placedAt(got.Resources, "/var/lib/mesh/assistant/vendor-key") != nil || len(got.foreseen) != 0 {
t.Fatalf("%v: a secret nobody gave is composed (foreseen %v)", choosing, got.foreseen)
}
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "assistant", "vendor-key"); err != nil || held {
t.Fatalf("a question made a value (held %v, %v)", held, err)
}
}
+113
View File
@@ -0,0 +1,113 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A consumer may wait for a provision's credential only a person gives (novox/hq ADR 0337 §5–§7, issue 164).
var requestsManifest = catalogue.Manifest{Module: "requests", Version: "1", Requires: []string{"tv-api", "db"},
Secrets: map[string]string{"tv-api": "/s/tv-api", "db": "/s/db"}}
var tvKeyWait = inventory.Wait{Part: "the tv library", Provision: "tv-api", What: "the API key of tv"}
func outsideFacts(given map[string]bool) operatorWaitFacts {
return operatorWaitFacts{manifests: map[string]catalogue.Manifest{"requests": requestsManifest},
outside: map[string]outsideProvider{"tv-api": {Module: "tv", Machines: []string{"media"}}},
pairsGiven: map[string]map[string]bool{"requests@laptop": given}}
}
// §6: excused only when the module requires the provision, its provider says it is issued outside, and no value a
// person gave is held for the pair under the keys both ends have now.
func TestAWaitForAProvisionIsExcusedOnlyWhenTheStoreBearsItOut(t *testing.T) {
for _, c := range []struct {
name string
w inventory.Wait
f operatorWaitFacts
says string
}{
{"required, issued outside, not given", tvKeyWait, outsideFacts(nil), ""},
{"given", tvKeyWait, outsideFacts(map[string]bool{"tv-api": true}), "was given"},
{"not required", inventory.Wait{Part: "p", Provision: "music-api", What: "w"}, outsideFacts(nil), "does not require"},
{"a credential the mesh gives itself", inventory.Wait{Part: "p", Provision: "db", What: "w"}, outsideFacts(nil),
"mesh gives itself"},
{"what was given cannot be read", tvKeyWait, operatorWaitFacts{
manifests: map[string]catalogue.Manifest{"requests": requestsManifest},
outside: map[string]outsideProvider{"tv-api": {Module: "tv"}}}, "could not be read"},
{"a provision and a secret", inventory.Wait{Part: "p", Provision: "tv-api", Secret: "x", What: "w"},
outsideFacts(nil), "one secret, one setting or one provision"},
} {
err := checkWait("requests", "laptop", c.w, c.f)
switch {
case c.says == "" && err != nil:
t.Errorf("%s: refused: %v", c.name, err)
case c.says != "" && (err == nil || !strings.Contains(err.Error(), c.says)):
t.Errorf("%s: %v; want a refusal saying %q", c.name, err, c.says)
}
}
}
// §7: an excused wait is the consumer's needs-operator condition, in plain words, its summary naming the one line
// that gives the credential for every consumer of the provider; the wait as kept is not changed.
func TestAWaitForAProvisionNamesTheOneAcceptForItsProvider(t *testing.T) {
r := inventory.ResourceHealth{Module: "requests", Resource: "requests.app", Kind: "container", Target: "requests",
State: link.StateWaiting, Check: "tool", Waits: []inventory.Wait{tvKeyWait}}
kept := []inventory.ResourceHealth{r}
got := checkWaiting("laptop", kept, outsideFacts(nil))
if got[0].State != link.StateWaiting {
t.Fatalf("an excused wait for a provision was judged: %+v", got[0])
}
if kept[0].Waits[0].Act != "" {
t.Fatal("checking the wait changed the statement as kept")
}
waits, waiting := operatorWait("requests", got)
if !waiting {
t.Fatal("the module does not read as waiting for the operator")
}
o := needsOperatorObservation("requests", "laptop", waits, got)
if !strings.Contains(o.Summary, "secret accept media tv tv-api") {
t.Fatalf("the summary does not name the accept for the provider: %q", o.Summary)
}
if !strings.HasPrefix(o.Needs, "give the API key of tv") {
t.Fatalf("the needs do not name the act: %q", o.Needs)
}
if why, ok := conditions.PlainWords(conditions.Words{Headline: o.Headline, Explanation: o.Explanation,
Resolved: o.Resolved, Needs: o.Needs, Actions: o.Actions}, "laptop"); !ok {
t.Fatalf("the condition is not in plain words: %s", why)
}
// A provider on several machines, or none: the line names its machine as a place to fill.
if line := (outsideProvider{Module: "tv", Machines: []string{"a", "b"}}).acceptLine("tv-api"); line !=
"secret accept <its machine> tv tv-api" {
t.Fatalf("a provider on two machines reads %q", line)
}
}
// §3: the consumers one accept reaches are those whose resolution answers the provision from that provider.
func TestTheConsumersOfAProviderAreThoseItAnswers(t *testing.T) {
plan := catalogue.Resolution{Needs: []catalogue.Needed{
{Name: "tv-api", From: "media", Module: "tv", For: "requests"},
{Name: "tv-api", From: "media", Module: "tv", For: "indexer", Local: "main"},
{Name: "tv-api", From: "other", Module: "tv", For: "elsewhere"},
{Name: "tv-api", From: "media", Module: "another-tv", For: "not-this-module"},
{Name: "db", From: "media", Module: "tv", For: "requests"},
{Name: "tv-api", ByRecord: true, For: "by-record"},
}}
got := bindsFrom(plan, "laptop", "media", "tv", "tv-api")
if len(got) != 2 || got[0].ConsumerModule != "indexer" || got[0].Local != "main" || got[1].ConsumerModule != "requests" {
t.Fatalf("the consumers from tv on media are %+v", got)
}
said := acceptedSaid("media", "tv", "tv-api", inventory.ProvisionAccepted{
Sealed: []inventory.PairEnd{{Consumer: "laptop", ConsumerModule: "requests"}},
Unsealable: []inventory.PairEnd{{Consumer: "new", ConsumerModule: "indexer"}}}, []string{"down: refused"})
for _, want := range []string{"sealed to requests on laptop", "no sealing key", "down: refused"} {
if !strings.Contains(said, want) {
t.Errorf("what the accept says lacks %q:\n%s", want, said)
}
}
}
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A delivery over its budget is loud (novox/hq ADR 0282 decision 7, issue 382): the self-check reads
// mesh-delivery's `times` and raises the warning `delivery.<class>.over-budget` when the newest delivery of a
// class whose delivery time is known took longer than its class's budget — three minutes for a leaf module (ADR
// 0304 decision 4), ten for a core module — naming the delivery and its longest phase. It clears when the next delivery of that class
// lands within its budget. Measurement only: the condition says, it never holds or acts on a delivery.
// probeBudgetsID is the probe that reads the delivery times.
const probeBudgetsID = "D16"
// kindOverBudget is what a delivery over its class's budget raises.
const kindOverBudget = "over-budget"
// deliveryBudgets are the budgets by class (ADR 0282 decision 1, the leaf's as ADR 0304 decision 4 changed it);
// the probe raises nothing for another class.
var deliveryBudgets = map[string]time.Duration{inventory.ClassLeaf: 3 * time.Minute, inventory.ClassCore: 10 * time.Minute}
// timesAnswer is what mesh-delivery's `times` answers, as far as the probe reads it.
type timesAnswer struct {
Classes []timesClass `json:"classes"`
}
// timesClass is one class's line of `times`.
type timesClass struct {
Class string `json:"class"`
Latest *timesLatest `json:"latest,omitempty"`
}
// timesLatest is the newest delivery of a class whose delivery time is known.
type timesLatest struct {
ID string `json:"id"`
TookMS int64 `json:"took_ms"`
Longest string `json:"longest,omitempty"`
Landed string `json:"landed,omitempty"`
}
// deliveryTimes is what the delivery's owner says of its delivery times; nothing when no holder is on record or
// none answers (D3 says that one).
func deliveryTimes(ctx context.Context, conn *nats.Conn, held bool) (*timesAnswer, error) {
if !held {
return nil, nil
}
raw, err := askDeliveryOwner(ctx, conn, "times", map[string]any{})
if errors.Is(err, link.ErrNothingServes) {
return nil, nil
}
if err != nil {
return nil, err
}
var a timesAnswer
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("%s.times answered something unreadable: %w", catalogue.DeliverySeat, err)
}
return &a, nil
}
// overBudgetObservations are the conditions of the classes whose newest delivery took longer than its budget:
// strictly longer, so a delivery of exactly its budget is within it.
func overBudgetObservations(a *timesAnswer) []conditions.Observation {
if a == nil {
return nil
}
var out []conditions.Observation
for _, c := range a.Classes {
budget, ok := deliveryBudgets[c.Class]
if !ok || c.Latest == nil {
continue
}
took := time.Duration(c.Latest.TookMS) * time.Millisecond
if took <= budget {
continue
}
longest := c.Latest.Longest
if longest == "" {
longest = "not known"
}
out = append(out, conditions.Observation{Scope: conditions.ScopeDelivery, ID: c.Class, Kind: kindOverBudget,
Severity: conditions.Warning,
Summary: fmt.Sprintf("the %s delivery %s took %s from its merge to running everywhere, over its budget of %s; "+
"its longest phase: %s — `mesh-delivery.times`", c.Class, c.Latest.ID, humanDuration(took),
humanDuration(budget), longest),
Said: fmt.Sprintf("%s took %s (budget %s), longest phase %s", c.Latest.ID, took.Round(time.Second), budget, longest),
Headline: fmt.Sprintf("A %s delivery took %s, over its %s budget", c.Class, humanDuration(took),
humanDuration(budget)),
Explanation: fmt.Sprintf("The delivery %s took %s from its merge until every machine ran it; a %s module is "+
"held to %s (ADR 0282). Most of the time went to %s. Nothing was held or changed because of this: it is "+
"a measurement.", c.Latest.ID, humanDuration(took), c.Class, humanDuration(budget), longest),
Resolved: fmt.Sprintf("the next %s delivery lands within %s", c.Class, humanDuration(budget)),
})
}
return out
}
// probeBudgets is D16: the newest delivery of each class lands within its class's budget.
func probeBudgets(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
entries, err := d.open.inventory.Catalogued(ctx)
if err != nil {
return nil, err
}
var conn *nats.Conn
if d.js != nil {
conn = d.js.Conn()
}
a, err := deliveryTimes(ctx, conn, deliverySeatHeld(entries))
if err != nil {
return nil, err
}
return overBudgetObservations(a), nil
}
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A leaf delivery of 3 minutes 10 seconds raises delivery.leaf.over-budget naming its longest phase; one of
// exactly three minutes, a core one of nine and a class without a budget raise nothing (ADR 0282 decision 7,
// the leaf's budget as ADR 0304 decision 4 changed it).
func TestADeliveryOverItsBudgetIsLoud(t *testing.T) {
a := &timesAnswer{Classes: []timesClass{
{Class: inventory.ClassLeaf, Latest: &timesLatest{ID: "novox/mesh-catalog@abc", TookMS: (3*time.Minute + 10*time.Second).Milliseconds(),
Longest: "judgement 2m40s"}},
{Class: inventory.ClassCore, Latest: &timesLatest{ID: "novox/mesh-controller@def", TookMS: (9 * time.Minute).Milliseconds(),
Longest: "build 1m"}},
{Class: "unclassed", Latest: &timesLatest{ID: "x@y", TookMS: time.Hour.Milliseconds()}},
}}
obs := overBudgetObservations(a)
if len(obs) != 1 {
t.Fatalf("one class over its budget, got %d: %+v", len(obs), obs)
}
o := obs[0]
if o.Key() != "delivery.leaf.over-budget" || !strings.Contains(o.Summary, "judgement 2m40s") ||
!strings.Contains(o.Summary, "novox/mesh-catalog@abc") {
t.Fatalf("the condition names its delivery and longest phase: %s — %s", o.Key(), o.Summary)
}
// The next leaf delivery within its budget clears it: the probe raises nothing for the class.
a.Classes[0].Latest = &timesLatest{ID: "novox/mesh-catalog@ghi", TookMS: (3 * time.Minute).Milliseconds()}
if obs := overBudgetObservations(a); len(obs) != 0 {
t.Fatalf("a delivery of exactly its budget is within it: %+v", obs)
}
a.Classes[1].Latest.TookMS = (10*time.Minute + time.Second).Milliseconds()
if obs := overBudgetObservations(a); len(obs) != 1 || obs[0].Key() != "delivery.core.over-budget" {
t.Fatalf("a core delivery over ten minutes: %+v", obs)
}
if obs := overBudgetObservations(nil); obs != nil {
t.Fatal("no answer raises nothing")
}
// No delivery of a class with a known time yet: nothing said of it.
if obs := overBudgetObservations(&timesAnswer{Classes: []timesClass{{Class: inventory.ClassLeaf}}}); len(obs) != 0 {
t.Fatalf("a class with no delivery: %+v", obs)
}
}
// The probe's question is one the controller's grant names: a question the bus refuses checks nothing.
func TestTheControllerMayAskForTheDeliveryTimes(t *testing.T) {
found := false
for _, v := range broker.VerbsTheControllerAsksTheDeliveryOwner {
found = found || (v.Seat == catalogue.DeliverySeat && v.Verb == "times")
}
if !found {
t.Fatal("the grant does not name mesh-delivery.times")
}
}
// A walk's class is core when it walks a module of the controller's own path or one holding the mesh's resolver,
// leaf otherwise; its window closed at its batch's window, or its maximum, never after its cut.
func TestAWalksClassAndWindowAreReadAtItsCut(t *testing.T) {
entries := []inventory.Entry{
{Manifest: catalogue.Manifest{Module: "dnsmasq", Claims: []catalogue.Claim{{Name: resolverSeat}}}},
{Manifest: catalogue.Manifest{Module: "gitea"}},
}
walk := func(modules ...string) inventory.Plan {
p := inventory.Plan{Modules: map[string]*inventory.PlanModule{}}
for _, m := range modules {
p.Modules[m] = &inventory.PlanModule{}
}
return p
}
for want, w := range map[string]inventory.Plan{
inventory.ClassLeaf: walk("gitea"), inventory.ClassCore: walk("gitea", "mesh-controller"),
} {
if got := classOf(w, entries); got != want {
t.Errorf("%v: %s, want %s", w.Modules, got, want)
}
}
if got := classOf(walk("dnsmasq"), entries); got != inventory.ClassCore {
t.Errorf("the resolver's holder is core: %s", got)
}
now := time.Date(2026, 10, 10, 18, 0, 0, 0, time.UTC)
batch := inventory.Plan{Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{
ClosesAt: now.Add(-20 * time.Second), AtMost: now.Add(5 * time.Minute)}}}
times := walkTimesAtCut(batch, walk("gitea"), entries, now)
if times.Cut == nil || !times.Cut.Equal(now) || times.WindowClosed == nil || !times.WindowClosed.Equal(now.Add(-20*time.Second)) ||
times.Class != inventory.ClassLeaf {
t.Fatalf("times at the cut: %+v", times)
}
batch.Delivery.Batch.AtMost = now.Add(-time.Minute)
if times := walkTimesAtCut(batch, walk("gitea"), entries, now); !times.WindowClosed.Equal(now.Add(-time.Minute)) {
t.Fatalf("a window closed at its maximum: %v", times.WindowClosed)
}
if times := walkTimesAtCut(inventory.Plan{Delivery: &inventory.PlanDelivery{Alone: true}}, walk("gitea"), entries, now); times.WindowClosed != nil {
t.Fatalf("a merge walked alone had no window: %v", times.WindowClosed)
}
}
// What each machine of the rest was sent is kept only for the machines the send reached.
func TestTheRestIsKeptForTheMachinesTheSendReached(t *testing.T) {
got := restOf([]string{"ace", "g14"}, []string{"ace", "shanks"}, map[string]inventory.SentDeclaration{"ace": {Digest: "d1"}})
if len(got) != 1 || got["ace"].Digest != "d1" {
t.Fatalf("rest: %+v", got)
}
if restOf([]string{"g14"}, []string{"ace"}, nil) != nil {
t.Fatal("no machine reached: nothing kept")
}
}
+138
View File
@@ -81,6 +81,13 @@ var plainWordings = map[string]func(conditions.Observation) words{
"It usually catches up by itself.", m, m),
Resolved: m + " applied the change"}
}),
kindApplyFailing: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: m + " is not applying its update",
Explanation: fmt.Sprintf("%s has refused or only partly applied what the mesh sent it, again and again. "+
"The details say what fails.", conditions.Capital(m)),
Resolved: m + " applies what it is sent again"}
}),
"tools-silent": worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "Tools on " + m + " are not answering",
@@ -175,6 +182,20 @@ var plainWordings = map[string]func(conditions.Observation) words{
"register it, so it was not made.", module, m),
Resolved: "Resolved: " + module + " on " + m + " is settled"}
}),
// The controller or the node-engine kept at the build a machine was last sent, over an older record (novox/hq
// issue 201).
kindCoreDowngrade: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
component := idPart(o, 0)
if o.Scope != conditions.ScopeCore || !guarded(component) {
component = "core component"
}
return words{Headline: m + " keeps a newer " + component + " than the mesh holds",
Needs: fmt.Sprintf("push %s to take the older build, or a newer merge or rebuild moves the record forward.", m),
Explanation: fmt.Sprintf("The mesh's record of its own %s is older than the build %s was last sent, so "+
"the mesh's sends keep the newer one there rather than put it back.", component, m),
Resolved: "Resolved: " + m + " runs the " + component + " the mesh holds"}
}),
kindBindingKept: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "A module's data source is held on " + m,
@@ -221,6 +242,24 @@ var plainWordings = map[string]func(conditions.Observation) words{
w := usedAsFoundObservation(orModule(module), machineOr(o, "a machine"), o.Summary, nil)
return words{Headline: w.Headline, Explanation: w.Explanation, Needs: w.Needs, Resolved: w.Resolved}
}),
kindNeedsOperator: worded(func(o conditions.Observation) words {
// The observation carries the act itself (ADR 0283); these are its words when only the kind is known.
module := ""
if o.Scope == conditions.ScopeModule && o.Machine != "" {
module = strings.TrimSuffix(o.ID, "."+o.Machine)
}
node := machineOr(o, "a machine")
w := needsOperatorWords(orModule(module), node, nil)
return w
}),
kindLeftOut: worded(func(o conditions.Observation) words {
// The observation carries its own words (novox/hq issue 380); these are its kind's alone.
module := ""
if o.Scope == conditions.ScopeModule && o.Machine != "" {
module = strings.TrimSuffix(o.ID, "."+o.Machine)
}
return leftOutWords(orModule(module), machineOr(o, "a machine"), "")
}),
kindProviderFailing: worded(func(o conditions.Observation) words {
thing, consumer := conditions.ThingWords(o), idPart(o, 2)
if consumer == "" {
@@ -596,6 +635,19 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "The bus refused messages from a part of the mesh, so what they carried did not happen.",
Resolved: "Resolved: the bus takes the messages again"}
}),
kindBucketOrLogFilling: worded(func(o conditions.Observation) words {
return words{Headline: "A module's bucket or log on the bus is filling up",
Needs: "decide whether to raise its cap or have the module keep less.",
Explanation: "A bucket or log a module keeps on the bus holds three quarters of its cap or more. When it " +
"is full, the bus refuses what the module writes there.",
Resolved: "It has room again"}
}),
kindBucketOrLogUnread: worded(func(o conditions.Observation) words {
return words{Headline: "A module's bucket or log could not be read",
Explanation: "The controller could not read how full a bucket or log a module keeps on the bus is, so it " +
"cannot say whether it is filling up. It asks again at its next check.",
Resolved: "It can be read again"}
}),
"stream-wrong": worded(func(o conditions.Observation) words {
return words{Headline: "Part of the bus's storage is wrong",
Needs: "check the machine the bus runs on; the details say what is missing.",
@@ -818,6 +870,9 @@ func stalledWords(l stalledLine, o conditions.Observation) (headline, explanatio
"to check it: the forge never announced it. It cannot merge until it is checked.", pull, long),
fmt.Sprintf("%s has a merge check now, or is closed", pull), needs, nil
}
if l.State == "unanswered" {
return unansweredWords(l, o)
}
held := l.State
if held == "" {
held = "held"
@@ -847,6 +902,89 @@ func stalledWords(l stalledLine, o conditions.Observation) (headline, explanatio
fmt.Sprintf("Delivery of %s is no longer %s", name, held), needs, actions
}
// unansweredWords are the plain words of a pull request's head whose merge check was asked and has not answered
// within its bound (novox/hq issue 438): mesh-delivery holds no delivery of it, so there is nothing to stop, release
// or close, and no action is offered. The operator's acts are a new commit, which asks the check again, or a look
// at the build queue, where a check that never ran may still wait.
//
// A head whose delivery is held in ready or rejected is said the same way (novox/hq issue 441), and mesh-delivery
// names that state as data (Held). Its check answered once, so the build queue is the wrong place to look: the step
// is a recheck of the delivery, named by its id, which asks the check again (novox/hq issue 451). It is advised, not
// offered as an action, since the controller is not granted the recheck.
func unansweredWords(l stalledLine, o conditions.Observation) (headline, explanation, resolved, needs string,
actions []conditions.Action) {
repository, _, _ := strings.Cut(l.ID, "@")
pull := "A pull request of " + repoName(repository)
if l.Number > 0 {
pull = fmt.Sprintf("Pull request %s #%d", repoName(repository), l.Number)
}
long, limit := "for too long", ""
if d, err := time.ParseDuration(l.For); err == nil {
long = "for " + humanDuration(d)
}
if d, err := time.ParseDuration(l.Bound); err == nil {
limit = ", past its limit of " + humanDuration(d)
}
check := "merge check"
if said := uncheckedWaitWords(l); len(said) > 0 {
check = "merge check (" + strings.Join(said, ", ") + ")"
}
which := "Its " + check
switch l.Held {
case "ready", "rejected":
which = "Its delivery is " + l.Held + ", but on the forge its " + check
if o.Resolver == conditions.ResolverOperator {
needs = "recheck its delivery, which asks its check again — mesh-delivery.recheck with id " + l.ID +
" and why — or push a new commit to its branch, " + FromMeshMCPServer
}
default:
if o.Resolver == conditions.ResolverOperator {
needs = "push a new commit to its branch, which asks the check again, or see whether its check still " +
"waits in the build queue: mesh-controller.queue."
}
}
return fmt.Sprintf("%s's merge check has not answered %s", pull, long),
fmt.Sprintf("%s is open on a branch that requires the merge check. %s was asked and has not answered %s%s. "+
"It cannot merge until its check answers.", pull, which, long, limit),
fmt.Sprintf("%s has an answer from its merge check, or is closed", pull), needs, nil
}
// uncheckedWaitWords are the merge checks an unanswered line waits on, as the operator reads them: from the checks
// given as data, each time in the controller's local zone, as every other time in a message — "mesh/merge-gate
// pending since 02:11" — never the UTC time inside mesh-delivery's finished words, which cannot be said again in
// another zone (novox/hq issue 443). A line from a mesh-delivery that gives no such data is said by its words.
func uncheckedWaitWords(l stalledLine) []string {
if len(l.Checks) == 0 {
return l.Waiting
}
out := make([]string, 0, len(l.Checks))
for _, c := range l.Checks {
switch c.State {
case "never-set":
out = append(out, c.Context+" never set")
case "pending":
out = append(out, c.Context+" pending"+sinceWords(c.Since))
default:
out = append(out, c.Context+" not answered")
}
}
return out
}
// sinceWords is " since 02:11" in the controller's local zone, with its day when that is not today ("since 23:05 on
// 9 Oct"), so the time stays absolute; a time not given, or not readable, is said so and never made up.
func sinceWords(since string) string {
at, err := time.Parse(time.RFC3339, since)
if err != nil {
return ", since a time the forge did not say"
}
local, today := at.In(wordsZone()), wordsNow().In(wordsZone())
if local.YearDay() == today.YearDay() && local.Year() == today.Year() {
return " since " + local.Format("15:04")
}
return " since " + local.Format("15:04 on 2 Jan")
}
// causeWords is a hand-act's cause as a person says it.
func causeWords(cause string) string {
return strings.NewReplacer(".", " ", "_", " ").Replace(cause)
+151
View File
@@ -1,6 +1,7 @@
package main
import (
"encoding/json"
"regexp"
"strings"
"testing"
@@ -347,3 +348,153 @@ func TestAnUnannouncedPullRequestSaysToPushANewCommit(t *testing.T) {
t.Fatalf("it reads %q / %q", o.Headline, o.Explanation)
}
}
// **A pull request whose merge check never answered says which check, since when, and what to do** (novox/hq issue
// 438): mesh-delivery says one as a stalled line in state `unanswered`, the line below exactly as its test makes it.
// No delivery of the head is held, so there is nothing to stop, release or close: no action is offered, and the
// operator's acts are a new commit, which asks the check again, or a look at the build queue.
func TestAnUnansweredMergeCheckSaysWhichCheckAndWhatToDo(t *testing.T) {
var l stalledLine
if err := json.Unmarshal([]byte(`{"id":"novox/mesh-controller@c11222026a3b","number":212,"state":"unanswered",`+
`"waiting":["mesh/merge-gate pending since 2026-10-11T00:11:39Z","mesh/repo-check never set"],"for":"1h1m0s",`+
`"bound":"1h0m0s","h2":"none: no delivery of it is held here, so there is none to close — the operator's",`+
`"says":"novox/mesh-controller#212 is open on main, which requires the merge check, and its head's check `+
`started and never answered: mesh/merge-gate pending since 2026-10-11T00:11:39Z, mesh/repo-check never set. `+
`No delivery of it is held here, so nothing asks it again. A new commit on its branch announces it and asks `+
`its check"}`), &l); err != nil {
t.Fatal(err)
}
obs := stalledObservations([]stalledLine{l})
if len(obs) != 1 || obs[0].Resolver != conditions.ResolverOperator {
t.Fatalf("an unanswered merge check is not the operator's: %+v", obs)
}
o := obs[0]
t.Logf("headline: %s\nexplanation: %s\nneeds: %s\nresolved: %s", o.Headline, o.Explanation, o.Needs, o.Resolved)
if len(o.Actions) != 0 {
t.Fatalf("it offers an action a head with no delivery cannot take: %+v", o.Actions)
}
if strings.Contains(o.Needs, "stop") || strings.Contains(o.Needs, "release") || !strings.Contains(o.Needs, "commit") ||
!strings.Contains(o.Needs, "in the build queue: mesh-controller.queue") || strings.Contains(o.Needs, "build seat") {
t.Fatalf("it says to %q", o.Needs)
}
if o.Headline != "Pull request mesh-controller #212's merge check has not answered for 61 minutes" {
t.Fatalf("its headline reads %q", o.Headline)
}
for _, want := range []string{"mesh/merge-gate pending since 2026-10-11T00:11:39Z", "mesh/repo-check never set",
"past its limit of 60 minutes"} {
if !strings.Contains(o.Explanation, want) {
t.Fatalf("its explanation does not say %q: %q", want, o.Explanation)
}
}
if strings.Contains(o.Explanation, "never asked") || strings.Contains(o.Explanation, "never announced") {
t.Fatalf("it says the mesh was never asked, of a head whose check started: %q", o.Explanation)
}
// A line from a mesh-delivery that names no checks still says the check did not answer, and offers nothing.
var bare stalledLine
if err := json.Unmarshal([]byte(`{"id":"novox/mesh-controller@c11222026a3b","number":212,"state":"unanswered",`+
`"for":"1h1m0s","bound":"1h0m0s","h2":"none: no delivery of it is held here, so there is none to close — the operator's"}`),
&bare); err != nil {
t.Fatal(err)
}
o = stalledObservations([]stalledLine{bare})[0]
if len(o.Actions) != 0 || !strings.Contains(o.Explanation, "has not answered") || strings.Contains(o.Headline, "Delivery of") {
t.Fatalf("a line naming no checks reads %q / %q, actions %+v", o.Headline, o.Explanation, o.Actions)
}
}
// **An unanswered merge check's time reaches the operator in their own time, never as raw UTC** (novox/hq issue
// 443): mesh-delivery says each check waited on as data — its context, its state and since when, in RFC 3339 — beside
// the finished words it gave before, and the controller words it in its local zone, as every other time in a message.
// The line is the one mesh-delivery says, as in the report of issue 443, with the checks it now carries.
func TestAnUnansweredMergeChecksTimeIsSaidInLocalTime(t *testing.T) {
// The operator's zone given through the seam, never by writing time.Local: other tests' goroutines read it, and
// the build seat runs the suite under the race detector.
wasZone, wasNow := wordsZone, wordsNow
wordsZone = func() *time.Location { return time.FixedZone("CEST", 2*60*60) }
wordsNow = func() time.Time { return time.Date(2026, 10, 11, 1, 12, 39, 0, time.UTC) }
t.Cleanup(func() { wordsZone, wordsNow = wasZone, wasNow })
var l stalledLine
if err := json.Unmarshal([]byte(`{"id":"novox/mesh-controller@c11222026a3b","number":212,"state":"unanswered",`+
`"waiting":["mesh/merge-gate pending since 2026-10-11T00:11:39Z","mesh/repo-check never set"],`+
`"checks":[{"context":"mesh/merge-gate","state":"pending","since":"2026-10-11T00:11:39Z"},`+
`{"context":"mesh/repo-check","state":"never-set"}],"for":"1h1m0s",`+
`"bound":"1h0m0s","h2":"none: no delivery of it is held here, so there is none to close — the operator's",`+
`"says":"novox/mesh-controller#212 is open on main, which requires the merge check, and its head's check `+
`started and never answered: mesh/merge-gate pending since 2026-10-11T00:11:39Z, mesh/repo-check never set. `+
`No delivery of it is held here, so nothing asks it again. A new commit on its branch announces it and asks `+
`its check"}`), &l); err != nil {
t.Fatal(err)
}
o := stalledObservations([]stalledLine{l})[0]
t.Logf("explanation: %s", o.Explanation)
for _, raw := range []string{"2026-10-11T00:11:39Z", "00:11", "UTC"} {
if strings.Contains(o.Explanation, raw) || strings.Contains(o.Headline, raw) {
t.Fatalf("the operator reads %q: %q / %q", raw, o.Headline, o.Explanation)
}
}
for _, want := range []string{"mesh/merge-gate pending since 02:11", "mesh/repo-check never set"} {
if !strings.Contains(o.Explanation, want) {
t.Fatalf("its explanation does not say %q: %q", want, o.Explanation)
}
}
// A check pending since another day than today says which day, so the time stays absolute.
l.Checks[0].Since = "2026-10-09T21:05:00Z"
if o = stalledObservations([]stalledLine{l})[0]; !strings.Contains(o.Explanation, "mesh/merge-gate pending since 23:05 on 9 Oct") {
t.Fatalf("a check pending since an earlier day reads %q", o.Explanation)
}
// A pending check whose time the forge did not say is said so, with no time made up.
l.Checks[0].Since = ""
if o = stalledObservations([]stalledLine{l})[0]; !strings.Contains(o.Explanation, "mesh/merge-gate pending, since a time the forge did not say") {
t.Fatalf("a pending check without its time reads %q", o.Explanation)
}
}
// **An unanswered merge check on a ready or rejected delivery is told to recheck it** (novox/hq issue 451):
// mesh-delivery says such a line too (issue 441) and names the delivery's state as data, `held`. That check already
// answered once, so the build queue is the wrong place to look; a recheck of the delivery, named by its id, asks the
// check again. A line with no delivery held keeps the advice of issue 438, and so does a line from a mesh-delivery
// that does not say `held` yet: the controller never guesses the state from the words.
func TestAnUnansweredMergeCheckOnAReadyOrRejectedDeliveryOffersARecheck(t *testing.T) {
line := func(held string) stalledLine {
var l stalledLine
raw := `{"id":"novox/mesh-controller@c11222026a3b","number":212,"state":"unanswered",` +
`"waiting":["mesh/merge-gate pending since 2026-10-11T00:11:39Z"],"for":"1h1m0s","bound":"1h0m0s",` +
`"h2":"none: its delivery is ` + held + `, a state the operator moves — the operator's",` +
`"says":"novox/mesh-controller#212 is open on main, which requires the merge check, and its delivery is ` +
held + ` here, but on the forge its check has not answered: mesh/merge-gate pending since ` +
`2026-10-11T00:11:39Z. The pull request waits on that status and nothing asks it again. A recheck of the ` +
`delivery asks its check again, as does a new commit on its branch"`
if held != "" {
raw += `,"held":"` + held + `"`
}
if err := json.Unmarshal([]byte(raw+`}`), &l); err != nil {
t.Fatal(err)
}
return l
}
for _, held := range []string{"ready", "rejected"} {
o := stalledObservations([]stalledLine{line(held)})[0]
t.Logf("%s: %s\nneeds: %s", held, o.Explanation, o.Needs)
if !strings.Contains(o.Needs, "mesh-delivery.recheck") || !strings.Contains(o.Needs, "novox/mesh-controller@c11222026a3b") {
t.Fatalf("a %s delivery's unanswered check is not offered its recheck: %q", held, o.Needs)
}
if strings.Contains(o.Needs, "build queue") || strings.Contains(o.Needs, "mesh-controller.queue") {
t.Fatalf("a %s delivery's unanswered check, which answered once, is sent to the build queue: %q", held, o.Needs)
}
if !strings.Contains(o.Explanation, "Its delivery is "+held) {
t.Fatalf("a %s delivery's explanation does not say its state: %q", held, o.Explanation)
}
if len(o.Actions) != 0 {
t.Fatalf("it offers an action the controller is not granted: %+v", o.Actions)
}
}
// No delivery held, or a mesh-delivery that does not say: the advice of issue 438, and no recheck.
o := stalledObservations([]stalledLine{line("")})[0]
if !strings.Contains(o.Needs, "in the build queue: mesh-controller.queue") || strings.Contains(o.Needs, "recheck") {
t.Fatalf("a line with no delivery held says to %q", o.Needs)
}
}
+200 -31
View File
@@ -7,6 +7,7 @@ import (
"errors"
"flag"
"fmt"
"io"
"os"
"slices"
"sort"
@@ -181,12 +182,21 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
}
var secret inventory.Secret
var err error
if n.SharedOwn != "" {
if outsideModule, outside := issuedOutsideBy(shelf, n); outside {
// Only a person can give it (novox/hq ADR 0337, issue 164): read as given, never made.
secret, err = inv.GivenPairSecret(ctx, n.Name, nodeName, n.For, n.From, outsideModule, n.Local)
} else if n.SharedOwn != "" {
// The provider's one credential, sealed to this consumer too (novox/hq ADR 0158).
secret, err = inv.SharedSecretFor(ctx, n.Name, nodeName, n.For, n.From, providerModuleOf(resolved, open, ctx, n), n.Local, n.SharedOwn)
} else {
secret, err = inv.SecretFor(ctx, n.Name, nodeName, n.For, n.From, n.Local)
}
if inventory.IsNotGiven(err) {
// **Not given, and the rest of the consumer runs** (ADR 0337 §4): no credential file, so the module can
// tell it was not given and says it waits for it; the machine is not refused for what only a person can
// give, and nothing the mesh made stands in for it.
continue
}
if err != nil {
// Said rather than skipped. A machine that resolves cleanly and receives no
// credential is one that will fail to authenticate at some later, less obvious
@@ -291,7 +301,7 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
if err != nil {
// Said, not skipped: a machine dropped here offers nothing and holds nothing as far
// as every other machine's plan can tell (novox/hq issue 188).
fmt.Fprintf(os.Stderr, "%s is left out of the rest of the mesh: it does not resolve: %v\n", o.node.Name, err)
sayLeftOut(ctx, o.node.Name, err)
continue
}
firstHeld = append(firstHeld, got.Claims...)
@@ -336,7 +346,7 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
// resolved without its offers, and one that consumes them cannot resolve here by
// design — that is not the machine being dropped, it is the view being partial.
if exclude == "" {
fmt.Fprintf(os.Stderr, "%s is left out of the rest of the mesh: it does not resolve: %v\n", o.node.Name, err)
sayLeftOut(ctx, o.node.Name, err)
}
continue
}
@@ -346,6 +356,35 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
return world, nil
}
// leftOutKey carries where theRestOfTheMesh says a machine it leaves out, for an act that answers with
// its own resolution instead (novox/hq issue 237).
type leftOutKey struct{}
// leftOutSaidBy is ctx with theRestOfTheMesh's "left out" lines written to w rather than to standard
// error.
//
// **An act's answer is one resolution taken after its write** (novox/hq issue 237, decided 2026-10-11).
// `assign` read the whole mesh once before its write — to record the seats held by derivation — and that
// read said, on standard error, that the node did not resolve for lack of the seat the act was about to
// give it. The controller seat's verb answers with both streams, so the stale line came last, after the
// act's own "now has it held", and named as its remedy the assignment just made. An act gives io.Discard
// here: what the node resolves to after the write is said by its own planFor, and every other machine
// that does not resolve by blockedElsewhere, both read after the write.
func leftOutSaidBy(ctx context.Context, w io.Writer) context.Context {
return context.WithValue(ctx, leftOutKey{}, w)
}
// sayLeftOut says that a machine is left out of the rest of the mesh, where ctx says to: standard
// error unless an act said otherwise. Said, not skipped: a machine dropped there offers nothing and
// holds nothing as far as every other machine's plan can tell (novox/hq issue 188).
func sayLeftOut(ctx context.Context, node string, err error) {
w, ok := ctx.Value(leftOutKey{}).(io.Writer)
if !ok {
w = os.Stderr
}
fmt.Fprintf(w, "%s is left out of the rest of the mesh: it does not resolve: %v\n", node, err)
}
// declarationFor is everything a node would be sent.
//
// One place, because there were three and one of them was written before credentials existed and
@@ -412,6 +451,12 @@ func declarationWith(ctx context.Context, open *stores, node string,
if err != nil {
return sendable{}, err
}
// Which ports the filter admits from the local network, and from which networks, for plan to say
// (novox/hq ADR 0340 §8). The same rules the composition rendered, never sent.
rules, err := plan.Rules(with)
if err != nil {
return sendable{}, err
}
// And what was taken on it, said in every declaration it is sent from this one place.
adoption, err := adoptionOf(ctx, open.inventory, record, plan, composed)
if err != nil {
@@ -420,7 +465,9 @@ func declarationWith(ctx context.Context, open *stores, node string,
out := sendable{Resources: composed.Resources, Adoption: adoption,
Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut, withheld: with.Withheld,
unbound: with.Unbound, foreseen: composed.Foreseen, unplaced: composed.Unplaced}
unbound: with.Unbound, foreseen: composed.Foreseen, unplaced: composed.Unplaced,
modules: namedModules(plan, composed.LeftOut),
localNetwork: catalogue.LocalNetworkPorts(rules, with.LocalNetworks)}
// And which build of each module it carries, for the send to record (novox/hq issue 259, ADR
// 0221). Read only on the send path: a question about what would be sent records nothing.
if choosing == Allocating {
@@ -559,6 +606,13 @@ func busCredentialIssued(ctx context.Context, inv *inventory.Inventory, node, mo
return nil
}
// givenUnderAnOldKey is the refusal for a value a person gave — a family's member, or an own secret issued outside
// the mesh — sealed to a key the machine no longer holds: the mesh cannot make another, so it is given again.
func givenUnderAnOldKey(module, node, name string) error {
return fmt.Errorf("%s on %s holds %q, which was given to the mesh rather than made by it, and %s has "+
"since generated a new sealing key. The mesh cannot make another; give it again", module, node, name, node)
}
// renderingFor is everything a node's declaration is composed with, and the node's record.
func renderingFor(ctx context.Context, open *stores, node string,
plan catalogue.Resolution, settings catalogue.SettingsBy,
@@ -664,7 +718,54 @@ func renderingFor(ctx context.Context, open *stores, node string,
needed := map[string]map[string]string{}
foreseen := map[string]map[string]bool{}
for _, m := range plan.Modules {
for name := range m.OwnSecrets {
// **A secret family is never made** (novox/hq ADR 0283): each member a person gave on this machine is
// placed, and one not given is nothing — the module says it waits for it.
for _, family := range m.OwnSecrets.Families() {
members, err := inv.GivenMembers(ctx, node, m.Module, family)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
for _, g := range members {
if _, fam, ok := m.OwnSecrets.Lookup(g.Name); !ok || fam != family {
continue // a longer family's member, or a name no longer of this family
}
if !g.Current {
if choosing == Allocating {
return catalogue.Rendering{}, inventory.Node{}, givenUnderAnOldKey(m.Module, node, g.Name)
}
continue
}
if needed[m.Module] == nil {
needed[m.Module] = map[string]string{}
}
needed[m.Module][g.Name] = g.Sealed
}
}
for name, own := range m.OwnSecrets.Plain() {
// **An own secret issued outside the mesh is never made** (novox/hq issue 401, ADR 0228; ADR 0283 for
// a family): only a value a person gave is placed. One not given is nothing — no file, and an empty
// value where a file of the module names it — so the module can tell it was not given, where a value
// the mesh made looked given and wrong. A value the mesh made for it before is not one either.
if own.IssuedBy == catalogue.IssuedOutside {
g, given, err := inv.GivenOwnSecret(ctx, node, m.Module, name)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
if !given {
continue
}
if !g.Current {
if choosing == Allocating {
return catalogue.Rendering{}, inventory.Node{}, givenUnderAnOldKey(m.Module, node, name)
}
continue
}
if needed[m.Module] == nil {
needed[m.Module] = map[string]string{}
}
needed[m.Module][name] = g.Sealed
continue
}
// Minted on the send path and only read on every other. Making one is an insert, and
// a question that writes is a question that can block against the machine it is about.
var sealed string
@@ -875,6 +976,12 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// And which of the networks it is attached to count as its local network (novox/hq ADR 0340):
// private ranges only, on a link facing outside it. Reported by the machine, never set.
local, err := localNetworksOf(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// Where this machine reaches each mesh seat's holder, for ${seat:<seat>:reach} (novox/hq ADR
// 0222): the artifact store as this network reaches it, which the container runtime is told to
// trust (ADR 0082). The same address composed into every reference the mesh built.
@@ -899,7 +1006,8 @@ func renderingFor(ctx context.Context, open *stores, node string,
Machines: machines, Zones: zones, Holders: replicas,
Suffix: overlay.Suffix(), MeshRange: meshRange, TunnelInterface: overlay.Interface, Accounts: accounts, Foundation: foundation,
Kept: kept, Adopted: record.Adopted, OutwardLinks: outwardLinks,
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, SeatReach: reach, Built: built,
LocalNetworks: catalogue.CountedLocalNetworks(local),
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, SeatReach: reach, Built: built,
BusUsers: busUsers, Withheld: withheld, Unbound: unbound,
}, record, nil
}
@@ -1230,6 +1338,14 @@ func planCommand(ctx context.Context, args []string) error {
fmt.Printf("%s is assigned nothing\n", args[0])
return nil
}
// Which optional requirements are unbound (novox/hq ADR 0330 §7): beside the document on the
// error stream where the answer is the declaration itself, which carries no such field — the
// node-engine refuses an envelope key it does not know.
if *asDiff || *asJSON {
for _, line := range unboundLines(plan) {
fmt.Fprintf(os.Stderr, "%s:%s\n", args[0], line)
}
}
if *asDiff {
declared, err := declarationFor(ctx, open, args[0], plan, settings)
if err != nil {
@@ -1264,6 +1380,11 @@ func planCommand(ctx context.Context, args []string) error {
return nil
}
// What the machine was last told, by whom and from which assignment generation (novox/hq issue 234),
// beside what it would be told now. A record that cannot be read is said, not taken for none.
if err := writeLastSend(ctx, os.Stdout, open.inventory, args[0]); err != nil {
fmt.Printf("what %s was last sent could not be read: %v\n", args[0], err)
}
// Which modules a push would leave out, and why — said before the plan, since the plan is of
// what the machine would be told (novox/hq ADR 0163, rule 6). Judged, never composed: `plan`
// without --json allocates nothing.
@@ -1306,6 +1427,9 @@ func planCommand(ctx context.Context, args []string) error {
for _, n := range plan.Needs {
fmt.Printf(" needs %s from %s, for %s\n", n.Name, n.From, n.For)
}
for _, line := range unboundLines(plan) {
fmt.Println(line)
}
declared, err := declarationFor(ctx, open, args[0], plan, settings)
if err != nil {
return err
@@ -1316,6 +1440,11 @@ func planCommand(ctx context.Context, args []string) error {
fmt.Printf(" %-20s settings from %s\n", module, layer.From)
}
}
// What each port reached from the local network resolved to on this machine (novox/hq ADR 0340 §8),
// or that it resolved to none.
for _, p := range declared.localNetwork {
fmt.Println(" " + p.Said())
}
fmt.Printf("\n%d resource(s)\n", len(resources))
if *show {
@@ -1641,6 +1770,17 @@ func busUserList(ctx context.Context, inv *inventory.Inventory,
// providerModuleOf is which module answers a need on the providing node: the one in this node's
// own set when the provider is here, else the one the catalogue says offers it.
// issuedOutsideBy is whether a need's provision is offered by its provider with a credential issued outside the
// mesh (novox/hq ADR 0337), and that provider module: the one answering it when known, else any that offers it so.
func issuedOutsideBy(shelf map[string]catalogue.Manifest, n catalogue.Needed) (string, bool) {
if n.Module != "" {
if m, known := shelf[n.Module]; known {
return n.Module, m.IssuedOutsideOf(n.Name)
}
}
return catalogue.OffersIssuedOutside(shelf, n.Name)
}
func providerModuleOf(resolved catalogue.Resolution, open *stores, ctx context.Context, n catalogue.Needed) string {
for _, m := range resolved.Modules {
if _, shared := m.SharedCredentialOf(n.Name); shared {
@@ -1659,17 +1799,27 @@ func providerModuleOf(resolved catalogue.Resolution, open *stores, ctx context.C
return ""
}
// unheldLogged is what was last logged about each node's unmet seat dependencies, so the log says
// each change once (novox/hq ADR 0207), on stderr so `status --json` stays a document.
// unheldLogger remembers what was last logged about each node's unmet seat dependencies, so the
// log says each change once (novox/hq ADR 0207), on stderr so `status --json` stays a document.
// A test gives its own writer and memory, and no longer shares the process's (novox/hq issue 486).
//
// **The serving controller's log only.** planFor runs for every node on every push, assignment and
// status — `blockedElsewhere` alone resolves the whole mesh — and a one-shot command starts with an
// empty memory, so every node's report was "a change" and a push printed the whole mesh's list,
// burying the line about the node it acted on. A command says what concerns its own act instead
// (unheldChange, reportUnheldPushed); the full list is `status`'s.
type unheldLogger struct {
mu sync.Mutex
logged map[string]string
out io.Writer
}
func newUnheldLogger(out io.Writer) *unheldLogger {
return &unheldLogger{logged: map[string]string{}, out: out}
}
var (
unheldLogged = map[string]string{}
unheldLoggedMu sync.Mutex
unheldLog = newUnheldLogger(os.Stderr)
logUnheldChanges bool
)
@@ -1679,23 +1829,27 @@ func logUnheld(node string, unheld []catalogue.Unheld) {
if !logUnheldChanges {
return
}
unheldLog.log(node, unheld)
}
func (l *unheldLogger) log(node string, unheld []catalogue.Unheld) {
lines := make([]string, 0, len(unheld))
for _, u := range unheld {
lines = append(lines, u.String())
}
now := strings.Join(lines, "\n")
unheldLoggedMu.Lock()
before, seen := unheldLogged[node]
unheldLogged[node] = now
unheldLoggedMu.Unlock()
l.mu.Lock()
defer l.mu.Unlock()
before, seen := l.logged[node]
l.logged[node] = now
if (seen && before == now) || (!seen && now == "") {
return
}
if now == "" {
fmt.Fprintf(os.Stderr, "%s: every seat its modules depend on is held (novox/hq ADR 0207)\n", node)
fmt.Fprintf(l.out, "%s: every seat its modules depend on is held (novox/hq ADR 0207)\n", node)
return
}
fmt.Fprintf(os.Stderr, "%s: %d unmet seat dependenc(ies), reported and not refused (novox/hq ADR 0207):\n %s\n",
fmt.Fprintf(l.out, "%s: %d unmet seat dependenc(ies), reported and not refused (novox/hq ADR 0207):\n %s\n",
node, len(lines), strings.Join(lines, "\n "))
}
@@ -1704,28 +1858,43 @@ func logUnheld(node string, unheld []catalogue.Unheld) {
// which includes every one of a module just assigned — and each now met that was not. Nothing about
// any other node, and nothing that was already true before the act.
func unheldChange(shelf map[string]catalogue.Manifest, node string, before, after []string) []string {
judge := func(names []string) map[string]catalogue.Unheld {
var set []catalogue.Manifest
for _, n := range names {
if m, known := shelf[n]; known {
set = append(set, m)
}
return unheldChangeTo(shelf, node, before, after, unheldOver(shelf, node, after))
}
// unheldOver is the node's unmet seat dependencies judged over a set of module names alone.
func unheldOver(shelf map[string]catalogue.Manifest, node string, names []string) []catalogue.Unheld {
var set []catalogue.Manifest
for _, n := range names {
if m, known := shelf[n]; known {
set = append(set, m)
}
out := map[string]catalogue.Unheld{}
for _, u := range catalogue.UnheldDependencies(shelf, node, set, nil) {
out[u.Module+" "+u.Seat] = u
}
return out
}
was, now := judge(before), judge(after)
return catalogue.UnheldDependencies(shelf, node, set, nil)
}
// unheldChangeTo is unheldChange with the node's dependencies as they now stand given: `now` is what
// the act's one resolution after its write says is unmet, and `after` the modules that resolution has
// on the node (novox/hq issue 237). A line about how things stand after the act is never judged from
// anything else, so it cannot disagree with `plan` for the node.
func unheldChangeTo(shelf map[string]catalogue.Manifest, node string, before, after []string,
now []catalogue.Unheld) []string {
key := func(u catalogue.Unheld) string { return u.Module + " " + u.Seat }
was := map[string]catalogue.Unheld{}
for _, u := range unheldOver(shelf, node, before) {
was[key(u)] = u
}
is := map[string]catalogue.Unheld{}
for _, u := range now {
is[key(u)] = u
}
var lines []string
for _, k := range sortedNames(now) {
for _, k := range sortedNames(is) {
if _, already := was[k]; !already {
lines = append(lines, "but "+now[k].String())
lines = append(lines, "but "+is[k].String())
}
}
for _, k := range sortedNames(was) {
if _, still := now[k]; still {
if _, still := is[k]; still {
continue
}
u := was[k]
+26
View File
@@ -0,0 +1,26 @@
package main
import (
"fmt"
"regexp"
)
// planID is every id the mesh gives a plan: a merge's walk or batch (plan-<nanoseconds>) or a release
// (release-<nanoseconds>).
var planID = regexp.MustCompile(`^(plan|release)-[0-9]+$`)
// planIDArguments are the arguments of plans that name a plan. wait is among them for when it is declared
// (novox/hq issue 229); an argument not declared is refused before this is asked.
var planIDArguments = []string{"id", "wait", "stop", "close", "retry", "go"}
// plainPlanIDs refuses a plans call naming a plan by anything but a plan's id (novox/hq issue 561): checked
// once, where the call becomes a command line, so nothing else a caller sent reaches a command or the store.
func plainPlanIDs(given map[string]string) error {
for _, arg := range planIDArguments {
if id, named := given[arg]; named && !planID.MatchString(id) {
return fmt.Errorf("plans %s: %q is not a plan's id (plan-<digits> or release-<digits>, as `plans` "+
"lists them); nothing was done", arg, id)
}
}
return nil
}
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"strings"
"testing"
)
// **A plan's id is checked once, where a call becomes a command line** (novox/hq issue 561): every argument of
// plans that names a plan is refused unless it is an id the mesh gives a plan, so a quote never reaches a
// command, nor the store behind it.
func TestAPlanIdThatIsNoPlansIdIsRefusedBeforeACommandStarts(t *testing.T) {
for _, arg := range []string{"id", "stop", "close", "retry", "go"} {
for _, bad := range []string{"x' or '1'='1", "plan-1'", "plan-", "plan-1 2", "batch-1", "PLAN-1"} {
args := map[string]any{arg: bad}
if arg == "stop" || arg == "close" || arg == "go" {
args["why"] = "a test"
}
argv, err := argvFor("plans", args)
if err == nil {
t.Errorf("plans %s %q was composed: %v", arg, bad, argv)
continue
}
if !strings.Contains(err.Error(), "is not a plan's id") {
t.Errorf("plans %s %q was refused for another reason: %v", arg, bad, err)
}
}
for _, good := range []string{"plan-1791813626034291775", "release-1791811938267404408"} {
args := map[string]any{arg: good}
if arg == "stop" || arg == "close" || arg == "go" {
args["why"] = "a test"
}
if _, err := argvFor("plans", args); err != nil {
t.Errorf("plans %s %q was refused: %v", arg, good, err)
}
}
}
}
+45 -3
View File
@@ -251,6 +251,16 @@ func resumed(p *inventory.Plan, why string) {
if p.State == inventory.PlanFailed && len(failedIn(*p)) == 0 {
p.State = inventory.PlanBuilding
p.Note = why
goesOnAgain(p)
}
}
// goesOnAgain clears a stop from a walk that walks again: a stop said why the walk ended, and a walk resumed has
// not ended. Kept, it would end a retried walk that fails again as stopped, not failed (novox/hq ADR 0308
// decision 1), and keep the search for the merge that brought the failure from running (searchAfterFailures).
func goesOnAgain(p *inventory.Plan) {
if p.Delivery != nil {
p.Delivery.Stopped, p.Delivery.StoppedWhy = "", ""
}
}
@@ -278,6 +288,13 @@ func retryPlan(ctx context.Context, open *stores, id string) (string, error) {
plans = append(plans, recent...)
// **A failed walk whose earlier merges are walked alone is not retried** (novox/hq ADR 0276): the search for
// the merge that brought the failure answers them now, and a retried walk would name them twice.
// **A walk stopped without some merges is not retried** (novox/hq ADR 0299): the merges it left out were stopped,
// and the others are another walk's now.
if p.Delivery != nil && (p.Delivery.WalkedAgainBy != "" || len(p.Delivery.LeftOut) > 0) {
return "", fmt.Errorf("%s was stopped without %s: those merges were stopped, and %s walks the others; a "+
"newer merge, or `rebuild <module>`, builds again", p.ID, mergesLeftOutWords(p.Delivery.LeftOut),
orWords(p.Delivery.WalkedAgainBy, "nothing"))
}
if p.Delivery != nil && len(p.Delivery.Merges) > 0 {
kept, err := inv.MergesOf(ctx, p.ID)
if err != nil {
@@ -288,13 +305,24 @@ func retryPlan(ctx context.Context, open *stores, id string) (string, error) {
"those walks answer them; a newer merge, or `rebuild <module>`, builds again", p.ID)
}
}
// Settled from the build records before anything is judged (novox/hq issue 457): a build of the tier
// that failed after the plan did is as failed as the one that failed it. What toRetry says is the
// failed set every step below works from.
var failed []string
if p.Tier < len(p.Tiers) {
recorded, byID, err := recordsOfAsked(ctx, inv, &p, p.Tiers[p.Tier])
if err != nil {
return "", err
}
failed = toRetry(&p, recorded, byID)
}
if err := retryRefusal(p, plans); err != nil {
return "", err
}
if again := unjudgedAtGate(p); len(failedIn(p)) == 0 && len(again) > 0 {
if again := unjudgedAtGate(p); len(failed) == 0 && len(again) > 0 {
return retryTierWhole(ctx, open, &p, again)
}
if len(failedIn(p)) == 0 {
if len(failed) == 0 {
return retryRollouts(ctx, open, &p)
}
entries, err := inv.Catalogued(ctx)
@@ -305,7 +333,6 @@ func retryPlan(ctx context.Context, open *stores, id string) (string, error) {
for _, e := range entries {
byName[e.Manifest.Module] = e
}
failed := failedIn(p)
var asked []string
for _, m := range failed {
askModule(ctx, &p, m, byName)
@@ -443,6 +470,7 @@ func retryRollouts(ctx context.Context, open *stores, p *inventory.Plan) (string
said = append(said, m+" to "+strings.Join(again, ", "))
}
p.State = inventory.PlanRolling
goesOnAgain(p)
p.Note = fmt.Sprintf("tier %d retried by hand; sent %s first again", p.Tier, strings.Join(said, "; "))
if err := open.inventory.SavePlan(ctx, p); err != nil {
return "", err
@@ -508,6 +536,7 @@ func retryTierWhole(ctx context.Context, open *stores, p *inventory.Plan, again
}
if p.State == inventory.PlanFailed && len(failedIn(*p)) == 0 {
p.State = inventory.PlanBuilding
goesOnAgain(p)
p.Note = fmt.Sprintf("tier %d retried by hand: %s asked again; %s sent again with the tier", p.Tier,
strings.Join(again, ", "), orNone(strings.Join(resent, ", ")))
}
@@ -525,3 +554,16 @@ func retryTierWhole(ctx context.Context, open *stores, p *inventory.Plan, again
func sendAgain(s *inventory.PlanModule) {
s.First, s.FirstAt, s.Gate, s.GatedBy, s.Previous, s.Why = nil, nil, nil, "", "", ""
}
// toRetry is the modules a retry asks again: every one of the tier that failed, the plan's state
// settled from the build records first (novox/hq issue 457). A plan fails on the first failure in its
// tier, and an outcome arriving after that finds no open plan to answer — it is kept only in the build
// records. Read from the plan alone, a retry asked only the build that failed first, and the records
// then failed the plan again on the next: each failed build of a tier took a retry of its own. What
// still runs is left asked, and its outcome is the plan's once the retry sets it building.
func toRetry(p *inventory.Plan, recorded map[string][]inventory.Build, byID map[string]inventory.Build) []string {
if p.Tier < len(p.Tiers) {
settleFromRecords(p, p.Tiers[p.Tier], recorded, byID)
}
return failedIn(*p)
}
@@ -0,0 +1,57 @@
package main
import (
"reflect"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A retry asks every failed build of the tier, not one (novox/hq issue 457). Seen 2026-10-11: gitea
// and plex both failed in tier 0 while the registry was held still; gitea's failure failed the plan,
// and plex's, arriving after, found no open plan and was kept only in the build records. The first
// retry asked gitea alone, the records then failed the plan again on plex, and a second retry asked
// plex.
func TestARetryAsksEveryFailedBuildOfTheTier(t *testing.T) {
asked := time.Date(2026, 10, 11, 1, 29, 0, 0, time.UTC)
failedAt := asked.Add(2 * time.Minute)
p := inventory.Plan{ID: "plan-457", State: inventory.PlanFailed, Tier: 0,
Tiers: [][]string{{"gitea", "plex"}}, Note: "gitea failed to build in tier 0",
Modules: map[string]*inventory.PlanModule{
"gitea": {State: "failed", AskedAt: &asked, Build: "build-gitea", Why: "cannot reach the registry"},
"plex": {State: "asked", AskedAt: &asked, Build: "build-plex"},
}}
byID := map[string]inventory.Build{
"build-plex": {ID: "build-plex", Module: "plex", At: failedAt, Failed: "cannot reach the registry"},
}
if got := toRetry(&p, nil, byID); !reflect.DeepEqual(got, []string{"gitea", "plex"}) {
t.Fatalf("a retry of a tier where gitea and plex failed asks %v", got)
}
}
// A build of the tier still running when the plan failed is not asked again: its outcome is the plan's
// once the retry sets it building, and one that is recorded built is taken as built.
func TestARetryLeavesABuildThatRunsOrWorked(t *testing.T) {
asked := time.Date(2026, 10, 11, 1, 29, 0, 0, time.UTC)
builtAt := asked.Add(3 * time.Minute)
p := inventory.Plan{ID: "plan-457", State: inventory.PlanFailed, Tier: 0,
Tiers: [][]string{{"a", "b", "c"}},
Modules: map[string]*inventory.PlanModule{
"a": {State: "failed", AskedAt: &asked, Build: "build-a", Why: "broken"},
"b": {State: "asked", AskedAt: &asked, Build: "build-b"},
"c": {State: "asked", AskedAt: &asked, Build: "build-c"},
}}
byID := map[string]inventory.Build{"build-c": {ID: "build-c", Module: "c", At: builtAt, Commit: "c0ffee"}}
if got := toRetry(&p, nil, byID); !reflect.DeepEqual(got, []string{"a"}) {
t.Fatalf("asks %v, want a alone", got)
}
if s := p.Modules["c"]; s.State != "built" || s.Commit != "c0ffee" {
t.Errorf("c, recorded built, is %+v", s)
}
if s := p.Modules["b"]; s.State != "asked" {
t.Errorf("b, still building, is %+v", s)
}
}
+159
View File
@@ -0,0 +1,159 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"strconv"
"strings"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Following a plan (novox/hq issue 229): `plans wait <id>` holds its answer until the plan moves, reaches its
// next tier or ends, or until its bound passes, and says which. Before it, an agent following a rollout had
// `plans` answer once, in prose, and wrote a loop of its own round the mesh's tools to be told the plan moved.
//
// **No wait is longer than a call is answered within** (link.LongestAnswer, inside the console's thirty
// seconds): a longer one would answer "still running" and leave its caller to ask `calls` — the loop again.
// So a caller following a plan to its end asks again; each call answers the moment the plan moves.
// maxPlanWait is the longest bound a wait takes, and the bound when none is given: what a call is answered
// within, less the time a verb's command takes to start and read the store.
var maxPlanWait = link.LongestAnswer - planWaitStart
// planWaitStart is what the verb's window allows beyond the bound: starting the command and opening the stores.
// A store slow enough to push the read and the answer past it degrades as any long call does: the caller is
// answered that the call is still running, and `calls` with its id says what came of it.
const planWaitStart = 5 * time.Second
// planWaitEvery is how often a wait reads the plan again. A variable so a test reads it more often
// (novox/hq issue 591).
var planWaitEvery = time.Second
// The things a wait waits until.
var planWaitUntil = []string{"moved", "tier", "done"}
// planWaitAsked checks what a wait is asked for: until (moved when empty) and a bound (a duration, or a number
// of seconds; maxPlanWait when empty).
func planWaitAsked(until, bound string) (string, time.Duration, error) {
if until == "" {
until = "moved"
}
known := false
for _, u := range planWaitUntil {
known = known || u == until
}
if !known {
return "", 0, fmt.Errorf("a wait is until %s, not %q", strings.Join(planWaitUntil, ", "), until)
}
d := maxPlanWait
if bound = strings.TrimSpace(bound); bound != "" {
var err error
if secs, isNumber := strconv.Atoi(bound); isNumber == nil {
d = time.Duration(secs) * time.Second
} else if d, err = time.ParseDuration(bound); err != nil {
return "", 0, fmt.Errorf("the bound %q is not a duration (such as 20s) or a number of seconds", bound)
}
if d <= 0 || d > maxPlanWait {
return "", 0, fmt.Errorf("the bound is at most %s, the longest a call is answered within; %s was asked "+
"— to follow a plan longer, ask again: each call answers the moment the plan moves", maxPlanWait, d)
}
}
return until, d, nil
}
// planEnded is a plan that will not move again: done, failed or superseded. A batch being assembled is not one.
func planEnded(p inventory.Plan) bool { return !p.Open() && !p.Batch() }
// waitOnPlan reads the plan until what it waits for happens or the bound passes, and answers the plan as last
// read and what ended the wait: moved (any save of it), tier (its tier changed), done (it ended — which ends
// every wait, since nothing comes after it), or bound. Moved and tier are judged against the plan as first read.
func waitOnPlan(ctx context.Context, read func(context.Context) (inventory.Plan, error), until string,
bound, every time.Duration) (inventory.Plan, string, error) {
first, err := read(ctx)
if err != nil {
return inventory.Plan{}, "", err
}
if planEnded(first) {
return first, "done", nil
}
deadline := time.Now().Add(bound)
tick := time.NewTicker(every)
defer tick.Stop()
last := first
for {
select {
case <-ctx.Done():
return last, "", ctx.Err()
case <-tick.C:
}
if !time.Now().Before(deadline) {
return last, "bound", nil
}
p, err := read(ctx)
if err != nil {
return last, "", err
}
last = p
switch {
case planEnded(p):
return p, "done", nil
case until == "moved" && p.Revision != first.Revision:
return p, "moved", nil
case until == "tier" && p.Tier != first.Tier:
return p, "tier", nil
}
}
}
// planWaited is what `plans wait` answers: the plan, what ended the wait, and the line a person reads (`text`,
// which the verb answers as its output and not again in the data).
type planWaited struct {
// Came is what ended the wait: moved, tier, done, or bound when nothing did.
Came string `json:"came"`
Until string `json:"until"`
Waited string `json:"waited"`
Plan planStatus `json:"plan"`
// Revision is the plan's as read: a caller asking again with the same plan sees whether it moved since.
Revision int64 `json:"revision"`
Text string `json:"text"`
}
func planWaitedDoc(p inventory.Plan, st planStatus, until, came string, waited time.Duration, line string) planWaited {
said := map[string]string{
"moved": "moved", "tier": "reached another tier", "done": "ended",
"bound": "did not " + map[string]string{"moved": "move", "tier": "reach another tier", "done": "end"}[until],
}[came]
text := fmt.Sprintf("%s %s in %s (waited until %s)\n%s — %s\n", p.ID, said, waited.Round(time.Second), until,
p.ID, line)
return planWaited{Came: came, Until: until, Waited: waited.Round(time.Millisecond).String(), Plan: st,
Revision: p.Revision, Text: text}
}
// planWaitCommand is `plans wait <id>`: the plan followed until it moves, or the bound passes.
func planWaitCommand(ctx context.Context, inv *inventory.Inventory, id, untilAsked, boundAsked string) error {
until, bound, err := planWaitAsked(untilAsked, boundAsked)
if err != nil {
return err
}
began := time.Now()
p, came, err := waitOnPlan(ctx, func(ctx context.Context) (inventory.Plan, error) {
return inv.PlanByID(ctx, id)
}, until, bound, planWaitEvery)
if err != nil {
return err
}
// Said as `plans <id>` says its first line, and as `status` says the plan.
now, one := time.Now(), []inventory.Plan{p}
bounds, pause := readTierBounds(ctx, inv, now), buildSeatPause(ctx, inv, one)
line := planLineSaid(p, now, pause, bounds.of(p.Repository), func(module string) []string {
on, _ := inv.Running(ctx, module)
return on
}, waitsOf(ctx, inv, one))
doc := planWaitedDoc(p, planStatuses(one, now, pause, bounds)[0], until, came, now.Sub(began), line)
return json.NewEncoder(os.Stdout).Encode(doc)
}

Some files were not shown because too many files have changed in this diff Show More