Compare commits

...
Author SHA1 Message Date
jochen f0fd7fe886 Leave a kept core move out of what a gate judges (issue 201, review 91)
mesh/merge-gate out of date: judged against trunk ba685b17f5e1, the trunk is now d8f36a8fb706; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk ba685b17f5e1, the trunk is now d8f36a8fb706; judged again on a recheck or when it is merged
mesh/delivery checking
gatedSend took its judged moves from the record, so on a machine kept at
a newer controller the gate judged the controller moving back to the
record, read its own send's kept controller as a later build there, and
was superseded with nothing judged. The kept moves are now left out
there as in ungatedIn (withoutKept). The doc says the bus step keeps
too, and the three store tests are listed.
2026-10-12 19:16:26 +02:00
jochen 41a9ff15dd Merge origin/main into fix/201-a-push-never-downgrades-the-core 2026-10-12 19:07:20 +02:00
mesh-admin ba685b17f5 Merge pull request 'Let plans wait for a plan to move and answer it as data (issue 229)' (#266) from fix/229-plans-wait into main 2026-10-12 17:06:50 +00:00
jochen d170491256 Keep the core over an older record instead of refusing the send (issue 201, review 90)
mesh/merge-gate out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Refusing the whole send wedged a machine whose gate put back the
controller and whose put-back send was refused: every later automatic
send there was refused, retried each tick, and held up every other walk.

A send now composes the controller and the node-engine at the build the
machine was last sent when the mesh's record is older, and carries the
rest; nothing is refused or retried, and a plan whose own build is the
older one ends with its gate superseded. A sent build that failed its
gate is never kept, so the next send completes a put-back. The condition
names the two ways out and clears on any send that carries the record,
a person's push included.
2026-10-12 18:51:17 +02:00
jochen 6bb1c9b768 Merge remote-tracking branch 'origin/main' into fix/201-a-push-never-downgrades-the-core 2026-10-12 18:40:12 +02:00
mesh-admin f8daec01d3 Merge pull request 'Read a plan by its id as a parameter, never as SQL (issue 561)' (#268) from fix/561-plan-id-parameter into main 2026-10-12 16:37:07 +00:00
jochen 9038d4b0d3 Refuse a send that puts the controller or node-engine behind its machine's build (issue 201)
mesh/merge-gate out of date: judged against trunk f10aedac5880, the trunk is now f8daec01d3f0; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f10aedac5880, the trunk is now f8daec01d3f0; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
On 2026-10-02 a plan sent the control node a controller older than the one
running there, whose seat row the newer build had widened; the older binary
refused the row and the mesh went quiet for ten minutes. ADR 0185 took the
outage out of it, and the race stayed open.

Every send but a person's push now refuses a move of mesh-controller or
mesh-host to a build asked before the one its machine was last sent (the
builds' registration order, issue 219), whatever composed it — gated or not.
A gate's own put-back is the one exception, for the modules it puts back,
and says so in its send's scope. The refusal is the send's error, which the
plan keeps as its note, and a warning condition on the machine, cleared by
the next send there that carries no such move.
2026-10-12 18:21:17 +02:00
mesh-admin f10aedac58 Merge pull request 'A merge check judges the head merged into its trunk, and names the trunk commit (hq ADR 0323, issue 341)' (#271) from feat/341-a-merge-check-judges-the-merge into main 2026-10-12 16:11:27 +00:00
jschoubben 875ebc093e Format PullUpdated, and keep the announced base head on a rechecked member
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/341-a-merge-check-judges-the-merge delivered: every member is delivered
A head checked again through mesh-delivery reached the direct verdict with no
base head, so a head no build judges was answered with no trunk commit and the
merge tool would refuse it (novox/hq ADR 0323).
2026-10-12 17:52:12 +02:00
mesh-admin 1e28342ede Merge pull request 'Make no value for an own secret issued outside the mesh that nobody gave (issue 401)' (#262) from fix/401-an-outside-secret-not-given-is-not-made into main 2026-10-12 15:48:12 +00:00
jschoubben bc623bc1f9 Name the announced trunk commit on a verdict no build judges
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: not gofmt'd by the toolchain's gofmt: internal/link/events.go
mesh/delivery-group group feat/341-a-merge-check-judges-the-merge rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
A change touching no module, in a repository with no merge-check.sh, was
answered by the controller with no trunk commit, so the merge tool refused it,
asked again and refused it for ever (novox/mesh-tools-go; novox/hq ADR 0323).
2026-10-12 17:44:18 +02:00
jschoubben ce9dc8e157 Run no hook and no signature setting when a check makes its merge
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
A build machine's or a clone's git configuration must not run code or change
the merge a check judges (novox/hq ADR 0323).
2026-10-12 17:39:59 +02:00
mesh-admin c672b3df6d Merge pull request 'Stop composing ADR 0312's group grant, so deliveries to the media machine pass again (hq issue 567)' (#270) from fix/567-revert-the-group-grant into main 2026-10-12 15:36:12 +00:00
jschoubben 25deb499bf Say why a merge's changed files read as none on an error
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 17:31:36 +02:00
jschoubben 83a9cf6e3a Judge a pull request's head merged into its trunk, and name the trunk commit
A check judged the head alone, so two pull requests that each passed could
break the trunk together (node-engine #66 on 2026-10-09), and a recheck after
the trunk moved judged the same head again. The build seat now merges the head
into the trunk's head in its own clone, never pushed, fails a head that does not
merge cleanly naming the files, and the verdict names the head, the trunk
commit and the commits read beside it, for the forge and mesh-delivery to weigh
against the trunk's next move (novox/hq ADR 0323, issue 341).
2026-10-12 17:27:01 +02:00
jschoubben 7b050abdcb Stop composing ADR 0312's group grant, so deliveries to the media machine pass again
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Since #256 delivered, the node-engine judges every read-write access by the path's group alone and
states six media modules unhealthy on ace, though they write their folders as the owner; every
delivery there fails its gate (hq issue 567). ADR 0325 replaces ADR 0312; until it is built, the
controller sends no grant, as before #256.

This reverts commit 3cfc121670.
2026-10-12 17:25:46 +02:00
jochen c3b4f187e1 Name issue 401's two store tests, so a skip without the store fails the check (issue 459's rule)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
2026-10-12 17:15:18 +02:00
jochen edd33f21d2 Make no value for an own secret issued outside the mesh that nobody gave (issue 401)
The send minted a random value for a plain own secret marked issued-by outside and placed it, so a
module could not tell "not given" from "given and wrong" (ADR 0228). It now places only a value a
person gave: not given is no file, an empty value where a module's file names it, and nothing to
restart on, as ADR 0283 decided for a family's member. A value the mesh made for one earlier is not
read as given.
2026-10-12 17:15:12 +02:00
jochen 5ad36127ee Hold the plans schema's longest bound to the value, and say what a slow store gives (review 75)
mesh/merge-gate head 5ad36127ee8b merged into trunk f8daec01d3f0 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 5ad36127ee8b merged into trunk f8daec01d3f0 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 16:50:59 +02:00
jochen a1a6f6493d Refuse a plans argument that is not a plan's id (issue 561)
mesh/merge-gate head a1a6f6493d50 merged into trunk f10aedac5880 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head a1a6f6493d50 merged into trunk f10aedac5880 — pass: its merge-check.sh passed
mesh/delivery delivered
Every argument of plans that names a plan (id, wait, stop, close, retry,
go) is checked once where the call becomes a command line, against the
ids the mesh gives plans (plan-<digits>, release-<digits>), so a quote
from a caller never reaches a command or the store (review 75 on #266).
2026-10-12 16:46:29 +02:00
jochen e9f060e0be Read a plan by its id as a parameter, never as SQL (issue 561)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
PlanByID pasted its id into the query, and the id comes from a caller of
the plans verb: an id holding a quote changed the query (x' or '1'='1
answered another plan). The id is now $1, and RecentPlans's limit too, so
no plan query is built from a value.
2026-10-12 16:33:23 +02:00
mesh-admin 3cfc121670 Merge pull request 'Compose each read-write access's grant for the node-engine (hq ADR 0312, issue 537)' (#256) from feat/537-a-read-write-access-grants-the-paths-group into main 2026-10-12 14:20:27 +00:00
jochen d20b52cb19 Let plans wait for a plan to move and answer it as data (issue 229)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
An agent following a rollout had plans answer once, in prose, and polled the
console with a curl loop of its own to be told a plan moved. plans now takes
wait (a plan's id), until (moved, tier or done) and bound, and answers when the
plan moves or the bound passes, saying which, with the plan as data and its
line as the output.

A call's one answer has to reach the console inside its thirty seconds, so
link.AnswersWithin lets a handler that waits on purpose hold its answer up to
LongestAnswer (25s) instead of AnswerWithin; the bound is at most 20s, and a
caller following a plan further asks again.
2026-10-12 16:18:54 +02:00
mesh-admin db7e4869ae Merge pull request 'Raise a machine that refuses or only partly applies its declarations past one apply interval (issue 233)' (#263) from fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 into main 2026-10-12 14:11:10 +00:00
mesh-admin 9eddf36503 Merge pull request 'A directory's owner may follow the account its module runs as (issue 279)' (#260) from fix/279-owner-follows-a-setting into main 2026-10-12 14:00:23 +00:00
jochen d4de4543cc Judge S21 by time since the first failing report while the machine is heard, not by how many reports said it (issue 233 review)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 delivered: every member is delivered
2026-10-12 15:39:08 +02:00
jochen a371de37fe Keep a directory owner's settings at the controller's terminal (issue 279)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Root chowns a directory to the number its owner's setting names, so a key
an owner asks for must not change through the settings verb: any caller
could have set puid to 0 or to its own uid. TerminalKeys now takes every
${setting:…} in a directory's owner, whatever else asks for it, and the
owner is written as the parsed number so "0001" never reaches the
node-engine as text (review 60).
2026-10-12 15:37:29 +02:00
jochen 77acb40092 Let a directory's owner follow the account its module runs as (issue 279)
A media manager runs as the puid/pgid its assignment sets per node, but a
declared directory's owner was literal, so a folder the app must write was
refused on a machine whose identity was not the image's default. A
directory's owner now takes ${setting:<key>} (user, or user:group), filled
at composition and refused unless the value is a whole number, and the
module check refuses any other shape or a key the module asks for nowhere
else.
2026-10-12 15:37:23 +02:00
mesh-admin 5facd8b08e Merge pull request 'An act's answer is said from one resolution after its write (hq issue 237)' (#253) from fix/237-one-resolution-for-assign into main 2026-10-12 13:31:03 +00:00
mesh-admin 72edb458cc Merge pull request 'The node-reads seat: its verbs, its holder's own account, and the node's secret places (hq ADR 0310, issue 418)' (#257) from fix/418-node-reads into main 2026-10-12 13:20:23 +00:00
mesh-admin 2843834813 Merge pull request 'The build queue answers as data too (issue 455)' (#259) from fix/455-the-queue-answers-as-data into main 2026-10-12 13:04:26 +00:00
jochen 3e10ad1535 Raise a machine that refuses or only partly applies its declarations past one apply interval, naming the package manager and its words, so the mesh says it rather than only status (issue 233)
mesh/delivery-group group fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 ready: every member ready, and composed together they pass
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 14:49:33 +02:00
mesh-admin 27b13e4586 Merge pull request 'Ask a node's engine which of its directories the walk would refuse, read-only (issue 358)' (#258) from fix/358-judge-directories-read-only into main 2026-10-12 12:48:56 +00:00
mesh-admin 142a5ad71d Merge pull request 'An archive may be an upstream release pinned by URL and sha256, mirrored by the build seat (hq issue 345, ADR 0316)' (#255) from fix/345-builder-archive-pins into main 2026-10-12 12:33:58 +00:00
jochen 63a99d4323 Name the vault's placed directories among a node's secret places, fail loudly on one that cannot be resolved, and keep the reads account out of every group (review 54, issue 418)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/418-node-reads delivered: every member is delivered
The real mesh-vault manifest declares its directories without a path, so the
list left its state and ledger out, and a place still holding a placeholder
vanished from it without a word. The provider's directories are now resolved
where the node places them, an unresolved place fails the composition naming
the module, and a node-reads holder whose account is put in a group is refused.
2026-10-12 14:30:00 +02:00
jschoubben 0cd8bc0bfb Answer the build queue as data too, so a check can say how far back it waits
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The queue verb answered only a table, so nothing outside the controller
could say an ask's place in the queue. It now asks its command for JSON:
each ask's state, kind, repository, commit, machine, since and position
among the waiting, with the table carried beside it and answered as the
verb's output, both written from one read. A repository's password, if
one were ever written into it, is left out of what is shown.

novox/hq issue 455
2026-10-12 14:22:22 +02:00
jschoubben 7385d9c092 Ask a node's engine which of its directories the walk would refuse, read-only (issue 358)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/358-judge-directories-read-only delivered: every member is delivered
Before the node-engine gives a mesh directory its mode and owner only through
a walk that follows no link another account could make, the operator is to
see which directories that walk would refuse on every node, the control-node
included, where no tool reads a path. `node judge-directories <node>` signs
the ask as a hand-over is, under its own context, and prints the engine's
one line per directory; it changes nothing, so it is a seat verb an agent may
call. The node is granted the subject, the controller alone its publish.
2026-10-12 14:17:01 +02:00
mesh-admin db9e9c5b87 Merge pull request 'Advise a recheck for an unanswered check on a ready or rejected delivery (issue 451)' (#254) from fix/451-a-recheck-is-the-step into main 2026-10-12 12:14:43 +00:00
jochen 4ead5a3df7 Define the node-reads seat, refuse a holder without an account of its own, and tell it where its node keeps a secret (hq ADR 0310, issue 418)
mesh/delivery-group group fix/418-node-reads rejected: a member's own check failed
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Reading a file, searching one, listing a directory, the owner and mode of each
component of a path and looking up an account had no mesh tool, so they went
through the login shell's execute as the operator's account. node-reads is a
node seat with six verbs, each saying the commands it replaces. Its holder must
run as an account of its own (ADR 0310 §3), and its own runtime is given
MESH_SECRET_PLACES: the directory of every module's own secret, every credential
file and grant directory, the vault's directories and the sealed-secret keep,
so it refuses them by name before any read (§5).
2026-10-12 13:57:24 +02:00
jschoubben a23b190747 Merge remote-tracking branch 'origin/main' into fix/451-a-recheck-is-the-step
mesh/merge-gate pass: builds mesh-controller, route-proxy → ace, novox; no bus step; every machine composes with the change as it did without (4 of 4 compo…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/451-a-recheck-is-the-step delivered: every member is delivered
2026-10-12 13:56:52 +02:00
jochen 3be6e86f77 Say an act's answer from one resolution after its write (issue 237)
mesh/merge-gate pass: builds mesh-controller, route-proxy → ace, novox; no bus step; every machine composes with the change as it did without (4 of 4 compo…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
assign read the whole mesh before its write, to record the seats held by
derivation, and that read printed on standard error that the node did not
resolve for lack of the seat the act was about to give it. The seat verb
answers with both streams, so the stale line came last and named the
assignment just made as its remedy.

The reads inside an act no longer print 'left out' lines; the node's
unmet seat dependencies and whether it resolves come from one planFor
taken after the write, and every other machine from blockedElsewhere.
unassign now says when its node does not resolve, from the same
resolution.
2026-10-12 13:53:19 +02:00
jochen b0c4b849a7 Judge an upstream pin's host like any name, and harden the upstream unpack (review 35, hq issue 345)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The url was skipped by the installation check, so an installation's host or
an address passed; it is judged whole now, the vendor's host declared in the
artifact's names-on-purpose, and an address or a reserved name as the host
is refused. Only the version's dotted-quad reading is skipped. The unpack
caps its entries, drains xz before waiting for it, reads a pax global
header and a nested within's parents, and a redirect off https is not
followed.
2026-10-12 13:51:35 +02:00
jochen b849af53dd Let an archive be an upstream release pinned by URL and sha256, mirrored by the build seat (hq issue 345, ADR 0316)
A mail client's release is about 80 MB per version, too large to keep in a
repository at every upgrade, and an add-on whose licence forbids copying it
to third parties may not be in a repository at all. An archive artifact may
now name url, sha256 and version (and file or within) instead of from: the
build seat fetches it once, refuses it when its digest is not the pin's and
names both, keeps it in its workspace under the digest (a vendor that serves
only its newest version cannot break a later build of the same pin), and
packs it like any archive, so a node gets the mesh's own archive by digest
and never asks the vendor. The build seat's image carries xz, which Go's
standard library does not read. The installation check takes the vendor's
address and version of such a pin for the world's, as a registry is an
image's.
2026-10-12 13:51:35 +02:00
jschoubben 74bd90df7e Pin the node-engine at its merge of the access grant, and capture it beside (hq ADR 0312, issue 537)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
2026-10-12 13:47:28 +02:00
jschoubben 30599c1858 Pin the node-engine at its reviewed grant, and hold the mount match to the path itself (hq ADR 0312, issue 537 review)
A path beginning with the same letters is another path, and a read-only mount
of a read-write access is still granted: the grant follows what mounts it.
2026-10-12 13:47:17 +02:00
jschoubben 10d4712f82 Name who a read-write access grants its path's group to, so modules write shared data through the group the operator gave it (hq ADR 0312, issue 537)
Each read-write access is composed with a grant: the module's containers that
mount its path and the account its tools run as (the operator's, or the module's
own under ADR 0259 §8); that runtime restarts on the access, so it holds a group
the engine put its account in. A read access grants nothing. Sent only to an
engine whose statement says GrantContract (4), since an older one refuses the
field. Pins the node-engine at the branch that reads it.
2026-10-12 13:47:17 +02:00
jschoubben e737e13bbb Advise a recheck for an unanswered check on a ready or rejected delivery (issue 451)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.66s)
mesh/delivery-group group fix/451-a-recheck-is-the-step rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
That check answered once, so the build queue is the wrong place to look.
mesh-delivery names the delivery's state as data (held); the controller
reads it and never the words. A line without it keeps the advice of 438.
2026-10-12 13:47:11 +02:00
mesh-admin 4b1dee6b79 Merge pull request 'A merge that shares nothing with the assembling batch, and whose group waits for no one, is cut at once (hq issue 525, ADR 0304 §1, ADR 0306 §2-3)' (#250) from feat/525-a-lone-merge-skips-the-window into main 2026-10-12 11:46:59 +00:00
mesh-admin 4ab441ce16 Merge pull request 'Ask only once the bus enforces the controller's own grant, judged for real and failing closed (review of hq issue 353)' (#188) from fix/353-the-asker-judges-the-controllers-own-grant into main 2026-10-12 11:34:14 +00:00
jochen f996cf7b6b Take an older bus module's allowed as held, and name only remedies that work (review 31 of PR 188, hq issue 353)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
After mesh-catalog #166 merges and before the bus upgrade, the bus's
machine cannot be pushed, so any change to the composed list made the
list check read behind for good: every ask was withheld and the condition
named a push that is refused. An older module's allowed now counts as
held, the unconfirmed reload said once; the worst case is one ask refused
in the seconds before a reload, which is asked again (issue 369).

The list check stays for a module that says not allowed or cannot answer,
and its remedy is the plan's own reason when the list cannot be composed,
the bus upgrade while sends there wait on it, and a push otherwise. An
in_force false is a wait only when the module gives both times; otherwise
what is not known is named.
2026-10-12 13:21:32 +02:00
jochen bcfcfaaaf3 Keep asking on an older bus module and name the real remedy for each grant not held (review of hq issue 353)
A bus module older than its in_force answer read as not held, so a merge
before mesh-catalog #166 is live would silence every ask until a person's
bus upgrade. Such an answer now falls back to the check main makes (is the
list sent to the bus's machine the one composed now), says the reload is
not confirmed, and each case names its own remedy: push <node> for a list
behind or not allowing it, a ~5 s wait for a reload, the bus upgrade when
an older module's list cannot be judged.
2026-10-12 13:21:32 +02:00
jschoubben a3181e356b Ask only once the bus enforces the controller's own grant, judged for real and failing closed (review of hq issue 353)
change held asking back, a list sent was taken for a list the bus enforces, a plan that could not be
worked out read as not behind, and no test held the real judgement. Now the asker asks only when the
composition grants the controller itself to publish its ask, and the bus's own module says that grant
is in force on the running server (nats_user_can's in_force, mesh-catalog); each fact not known is not
held, said with why. brokerBehind takes a list nobody could compose as behind.
2026-10-12 13:21:32 +02:00
mesh-admin dd2fd28c92 Merge pull request 'closedNames waits for the verb's own connection by name, not the first closed one (issue 543)' (#252) from fix/543-wait-for-the-verbs-own-connection-by-name into main 2026-10-12 11:21:18 +00:00
mesh-admin 9d165c7500 Merge pull request 'plan-moved says what a walk touches, its bound and its outcome when it starts and ends (ADR 0308, issue 532)' (#247) from adr-0308-walk-touches into main 2026-10-12 11:08:53 +00:00
jochen 7630072e63 Wait for the verb's own connection by name, not the first closed one (issue 543)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
closedNames returned as soon as the bus's closed-connections list held
anything, which could be the test's own setup connection closing before
the verb subprocess's connection did. That raced TestAVerbsOwnProcessNamesItsConnection
red about once in 80 runs. It now waits until the connection it is named
for appears, or the deadline passes.
2026-10-12 13:05:15 +02:00
jochen 1e9dcf0667 Say every walk a verb keeps, and clear a stop from a walk that walks again
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
cancel, clear and rebuild kept a walk's end or restart without saying it, as plans stop, retry and
release-backlog did before them. The controller's run now says every save a verb makes on the bus, so the
next verb cannot lose a mark the same way. A stop is cleared when a walk is resumed, so one that fails again
ends failed and the search for its merge runs (review 21, ADR 0308, issue 532).
2026-10-12 12:54:20 +02:00
jochen cd6610cdf7 Say a walk's start or end when a person stops, closes, retries or releases it, and a hand stop as stopped
plans stop|close, plans retry and upgrade release-backlog kept a walk's start or end mark without saying it,
so the router would hold what the walk touched until its bound; each now runs on the bus as plans go does.
A hand stop keeps who stopped it, so its outcome is stopped, not failed (review 15, ADR 0308, issue 532).
2026-10-12 12:54:20 +02:00
jochen facc551ff2 Say what a walk touches, its bound and its outcome when it starts and ends (ADR 0308, issue 532)
The router holds what an open walk touches and tells its result once; it can do so
only if plan-moved names the walk's start and end, the modules and machines it moves,
what a moved tool runner or node-engine carries there, and how long to hold.
2026-10-12 12:54:20 +02:00
jschoubben 71b3aeaf2c Keep a group member of the other kind's batch from being overtaken (hq issue 525)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A merge's group member counted as merged when its merge waited in any batch,
but kinds never share a batch: the later member got a batch of its own, cut
at once, and walked before the earlier member's batch was cut, breaking the
group's order (ADR 0276 §5). Only a member kept in a batch of the merge's own
kind counts as merged now; one kept in the other kind's keeps the merge
waiting, and its batch names that member, so the two are cut in the order
their windows close (review of #250).
2026-10-12 12:54:11 +02:00
mesh-admin fafa3fd9ef Merge pull request 'Require the service manager's reset-failed, wanted-by and unlink-dangling verbs (issue 332)' (#248) from fix/332-require-service-manager-verbs into main 2026-10-12 10:53:20 +00:00
jschoubben 6e5a1df575 Judge the installer's user list against mesh-host with the groups grant (hq issue 525)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.78s)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
The controller now asks mesh-delivery.groups, and mesh-host a76baa4b carries
that grant in the installer's first user list (mesh-host #85). The captured
copy is moved to that commit so the genesis test judges the list that ships.
2026-10-12 12:31:50 +02:00
jochen 61d00a234a Give the store tests' service manager holders the three verbs it now must serve (issue 332)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
They named only the eight verbs from before issue 332 and were refused once the seat required
reset-failed, wanted-by and unlink-dangling.
2026-10-12 12:25:05 +02:00
jschoubben 4d203b5a5d Ask mesh-delivery whether a merge's group still assembles (hq issue 525)
A merge from any head waited for the window because the controller could not
see a group's members, so the window skip changed nothing in production and
unrelated merges walked one after another. The controller now asks
mesh-delivery's groups, within 3 s and before the plans are held: a merge
waits only when its head's group has another member whose pull request is
open and whose merge it does not keep yet (ADR 0306 §2). No answer means it
waits, and its batch says why. A headed merge in no waiting group and related
to no batch is cut at once, as one from no head is (ADR 0304 §1, ADR 0306 §3).

The controller's grant names mesh-delivery.groups. The installer's first user
list in mesh-host must carry it too before the genesis test can pass.
2026-10-12 12:14:46 +02:00
jschoubben 6196a4815e Let a merge that shares nothing skip the merge window (hq issue 525)
A merge waited the whole window even when nothing assembling shared anything
with it. It now joins only a batch it is related to (ADR 0304 §1, read by ADR
0306 §3: repository, a module the cut would move, a graph edge, a group head),
folding the newer into the oldest when it relates to two. A merge related to
none starts a batch whose window is closed at once (PlanBatch.CutAtOnce),
unless its delivery group may still have a member to merge (ADR 0306 §2).

The controller cannot see a group's unmerged members (that is mesh-delivery's
record), so a merge from any head still waits, in a batch of its own: the safe
error is to wait. Walks stay one at a time; a batch cut at once queues behind
an open walk. A stop's merges walked again join a related batch, or their own.
2026-10-12 12:14:46 +02:00
jochen 0a0f5b4a5a Require the service manager's reset-failed, wanted-by and unlink-dangling verbs now every holder serves them (issue 332)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestAVerbsOwnProcessNamesItsConnection (4.62s); and TestASeatsVerbGainsTheArgumentsTheBinaryNames
mesh/delivery superseded: a newer head of the same pull request
ADR 0246 step 3: the systemd module, the seat's only holder, serves all three on every node,
so a holder that stops serving one is refused rather than leaving a node without them.
2026-10-12 12:03:37 +02:00
105 changed files with 7912 additions and 310 deletions
+2
View File
@@ -6,3 +6,5 @@
/mesh-controller
/postgres-provisioner
/redis-provisioner
# And where `go build` in cmd/mesh-controller leaves it (novox/hq issue 233).
/cmd/mesh-controller/mesh-controller
+3 -2
View File
@@ -16,7 +16,8 @@ RUN CGO_ENABLED=0 go build -trimpath -ldflags '-s -w' -o /mesh-builder ./cmd/mes
FROM ${ALPINE_BASE}
# git to clone what it is asked to build, and the docker client to build and push it. The daemon
# is the machine's, reached through its socket — a build machine shares the runtime it was given
# rather than running one inside itself.
RUN apk add --no-cache git docker-cli
# rather than running one inside itself. xz to read an upstream release an archive is pinned to
# (novox/hq ADR 0316): Go's standard library reads gzip and bzip2 and not xz.
RUN apk add --no-cache git docker-cli xz
COPY --from=build /mesh-builder /usr/local/bin/mesh-builder
ENTRYPOINT ["/usr/local/bin/mesh-builder"]
+8 -2
View File
@@ -256,7 +256,13 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
v, err = builder.Check(building, builder.Command, checkSpecOf(request), workspace, registry, forgeFrom(), say)
if err == nil {
result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report,
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo)}
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo),
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside}
// What each of a group's heads was judged as (novox/hq ADR 0323 §6).
for _, m := range v.Members {
result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo,
Commit: m.Head, Trunk: m.Trunk})
}
}
} else if err == nil {
// The package-registry credential is a build input, so it is resolved before the clone: a
@@ -354,7 +360,7 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
}
for _, m := range c.Members {
spec.Group = append(spec.Group, builder.GroupHead{Owner: m.Owner, Repo: m.Repo, Repository: m.Repository,
Ref: m.Ref, Paths: m.Paths})
Ref: m.Ref, Paths: m.Paths, Base: m.Base})
}
return spec
}
+67 -15
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/broker"
"io"
"slices"
"sort"
"strings"
@@ -54,6 +55,9 @@ func assignWith(ctx context.Context, open *stores, node string, opts assignOptio
if err := refusedMovingTheController(modules); err != nil {
return "", err
}
// Every line of the answer comes from the one resolution after the write (novox/hq issue 237):
// what the reads before it would say about a machine left out is not the answer's.
ctx = leftOutSaidBy(ctx, io.Discard)
// Held while it is recorded, so it cannot land between a converge's preview and its flip and
// be taken without ever having been previewed (novox/hq ADR 0100).
ctx, release, err := holdNodes(ctx, open, []string{node})
@@ -125,22 +129,28 @@ func assignWith(ctx context.Context, open *stores, node string, opts assignOptio
for _, line := range settled {
answer += "\n " + line
}
// What this act changed about this node's unmet seat dependencies, and nothing else (novox/hq
// ADR 0207): a dependency of a module just assigned, or one this assignment met. The rest of the
// node's list, and every other node's, is `status`'s.
for _, line := range unheldChange(shelf, node, before, append(append([]string(nil), before...), added...)) {
answer += "\n " + line
}
// Its bus credential, in the same act (novox/hq issue 203): an assignment pushed before its
// credential exists delivers a process that cannot authenticate and crash-loops until somebody
// runs a second verb and a second push. Issued here when the module speaks on the bus and has
// no credential yet; kept when it has one, so re-assigning rotates nothing.
var issued []string
for _, module := range added {
if line := issueOnAssign(ctx, open, node, module); line != "" {
answer += "\n " + line
issued = append(issued, line)
}
}
plan, _, err := planFor(ctx, open, node)
// **The one resolution the rest of the answer is said from** (novox/hq issue 237), taken after
// the write: what this act changed about this node's unmet seat dependencies, and nothing else
// (novox/hq ADR 0207) — a dependency of a module just assigned, or one this assignment met; the
// rest of the node's list, and every other node's, is `status`'s — and whether the node resolves.
after := resolvedAfter(ctx, open, shelf, node)
for _, line := range unheldChangeTo(shelf, node, before, after.modules, after.unheld) {
answer += "\n " + line
}
for _, line := range issued {
answer += "\n " + line
}
plan, err := after.plan, after.err
if err != nil {
// Kept, and still refused. Both halves are the answer, and the rest of the mesh is still
// worth reporting: this machine's refusal is rarely the only consequence.
@@ -218,6 +228,8 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
if len(modules) == 0 {
return "", fmt.Errorf("unassign %s names no module", node)
}
// One resolution after the write, as assign's answer (novox/hq issue 237).
ctx = leftOutSaidBy(ctx, io.Discard)
ctx, release, err := holdNodes(ctx, open, []string{node})
if err != nil {
return "", err
@@ -262,15 +274,22 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
}
answer := fmt.Sprintf("%s no longer runs %s — run `push %s` to make it so",
node, strings.Join(modules, ", "), node)
var left []string
for _, a := range assigned {
if !slices.Contains(modules, a) {
left = append(left, a)
}
}
for _, line := range unheldChange(shelf, node, assigned, left) {
after := resolvedAfter(ctx, open, shelf, node)
for _, line := range unheldChangeTo(shelf, node, assigned, after.modules, after.unheld) {
answer += "\n " + line
}
// Whether the node itself still resolves, from the same resolution: before, only the rest of the
// mesh's reads said it, on standard error and once per other machine.
if after.err != nil {
if unresolvable(after.err) {
answer += fmt.Sprintf("\n but %s does not resolve now, so nothing will be sent to it:", node)
} else {
answer += fmt.Sprintf("\n but whether %s still resolves could not be read:", node)
}
for _, line := range strings.Split(strings.TrimRight(after.err.Error(), "\n"), "\n") {
answer += "\n " + strings.TrimSpace(line)
}
}
// What it leaves behind that is irreplaceable is kept and retired, never removed (novox/hq ADR 0233).
for _, line := range keptOnUnassign(ctx, open.inventory, node, modules) {
answer += "\n " + line
@@ -278,6 +297,39 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
return answer + blockedElsewhere(ctx, open, node), nil
}
// afterAct is the one resolution of a node an act's answer is said from, taken after its write
// (novox/hq issue 237): the plan or why there is none, the modules it has on the node, and its unmet
// seat dependencies.
type afterAct struct {
plan catalogue.Resolution
err error
modules []string
unheld []catalogue.Unheld
}
// resolvedAfter resolves the node as `plan` does, once, after the act's write. When the node resolves,
// its modules and unmet seat dependencies are the resolution's — the closure, so a holder pulled in by
// a requirement counts as `plan` counts it. When it does not, they are judged over the assignments read
// after the write, which can only find more unmet than the closure would: a seat said held there is
// held in the closure too.
func resolvedAfter(ctx context.Context, open *stores, shelf map[string]catalogue.Manifest, node string) afterAct {
plan, _, err := planFor(ctx, open, node)
if err == nil {
out := afterAct{plan: plan, unheld: plan.Unheld}
for _, m := range plan.Modules {
out.modules = append(out.modules, m.Module)
}
return out
}
out := afterAct{err: err}
assigned, readErr := open.inventory.Assigned(ctx, node)
if readErr != nil {
return out
}
out.modules, out.unheld = assigned, unheldOver(shelf, node, assigned)
return out
}
// splitModules is a surface's one `module` field as the modules it names: several, comma-separated,
// are one act (novox/hq ADR 0207), so the holders that depend on each other go on together from the
// command API and the controller seat's verbs as they do from the command line.
+118
View File
@@ -0,0 +1,118 @@
package main
import (
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// kindApplyFailing is S21's kind: a machine whose declarations have been refused, or applied only in part,
// for longer than one apply interval (novox/hq issue 233).
const kindApplyFailing = "apply-failing"
// applyEvery is how often a node-engine applies what it was last told when nothing new arrives: its
// ReconcileEvery. A failure still there after one more of them is not one the next apply clears.
const applyEvery = 5 * time.Minute
// packageManagerKey is how a node-engine names a package manager that does not answer among what failed
// (mesh-host's probe.go): "the package manager (pacman)".
const packageManagerKey = "the package manager ("
// watchApplyFailing is S21: **a machine that does not apply what it was sent is said by the mesh, not only
// to someone who asks for `status`** (novox/hq issue 233, issue 187's family). A workstation refused every
// declaration for an hour and a half because its package manager did not answer, and the mesh's only word
// on it was a line in `status`. Raised once the machine's first report of this failure is older than one apply
// interval **while the machine is heard** — judged by time, not by how many reports said it: a node-engine
// says a re-apply only when how it went changed, so the same failure is one report however long it lasts.
// Names the refusal's or the failure's words; cleared by the clean apply the node-engine then says. Not while
// the machine is silent, asleep or not known: S1 says silent, and asleep is not lost.
func watchApplyFailing(f *signalFacts) []conditions.Observation {
heard := map[string]bool{}
for _, m := range f.machines {
if !m.asleep() && !m.lastHeard.IsZero() && f.now.Sub(m.lastHeard) <= heartbeatBound(m.every) {
heard[m.name] = true
}
}
var out []conditions.Observation
for _, d := range f.doing {
if !d.Wrong() || d.Since == nil || f.now.Sub(*d.Since) <= applyEvery || !heard[d.Node] {
continue
}
out = append(out, applyFailing(d, f.now))
}
return out
}
// applyFailing is one machine's S21 condition.
func applyFailing(d inventory.Doing, now time.Time) conditions.Observation {
m := d.Node
since := d.Since.UTC().Format("2006-01-02 15:04 MST")
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: m, Kind: kindApplyFailing, Machine: m,
Severity: conditions.Warning, Said: doingWords(d),
Resolved: m + " applies what it is sent again"}
how := "refused its declaration whole"
if d.Outcome != inventory.OutcomeRefused {
how = "applied its declaration only in part"
}
o.Summary = fmt.Sprintf("%s has %s since %s (bound %s): %s", m, how, since, ago(applyEvery), failedNames(d))
o.Headline = m + " is not applying its update"
o.Explanation = fmt.Sprintf("%s has %s for %s, and trying again has not helped. The details say what fails.",
m, plainHow(d), humanDuration(now.Sub(*d.Since)))
if tool := unansweringTool(d); tool != "" {
o.Summary = fmt.Sprintf("%s: %s does not answer, so no package reaches it; everything else it was sent "+
"is applied, since %s (bound %s)", m, tool, since, ago(applyEvery))
o.Headline = fmt.Sprintf("%s cannot install packages", m)
o.Explanation = fmt.Sprintf("The package manager on %s (%s) has not answered for %s, so no package reaches "+
"it. Everything else is still applied, so an update that repairs it can land; if none does, it needs a "+
"look on the machine.", m, tool, humanDuration(now.Sub(*d.Since)))
}
return o
}
// plainHow is how a machine failed, as the operator reads it.
func plainHow(d inventory.Doing) string {
if d.Outcome == inventory.OutcomeRefused {
return "refused every update"
}
return "applied its updates only in part"
}
// unansweringTool is the package manager a machine said does not answer, or "".
func unansweringTool(d inventory.Doing) string {
for _, r := range d.Failed {
if rest, ok := strings.CutPrefix(r.ID, packageManagerKey); ok {
return strings.TrimSuffix(rest, ")")
}
}
return ""
}
// failedNames is what failed, by name: the refusal, or the failed resources' ids.
func failedNames(d inventory.Doing) string {
if d.Outcome == inventory.OutcomeRefused {
return "refused"
}
ids := make([]string, 0, len(d.Failed))
for _, r := range d.Failed {
ids = append(ids, r.ID)
}
sort.Strings(ids)
return "failed " + strings.Join(ids, ", ")
}
// doingWords is the machine's own words on what failed: the evidence, kept inside the mesh.
func doingWords(d inventory.Doing) string {
if d.Outcome == inventory.OutcomeRefused {
return "refused: " + oneLine(d.Refused)
}
var parts []string
for _, r := range d.Failed {
parts = append(parts, r.ID+": "+oneLine(r.Error))
}
sort.Strings(parts)
return strings.Join(parts, "; ")
}
+123
View File
@@ -0,0 +1,123 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// unansweringPackageManager is anchor's last report as the workstation's read on 2026-10-04, once its
// node-engine applies the rest (novox/hq issue 233): the package failed and the package manager did not answer,
// the same failure on times reports since for before now.
func unansweringPackageManager(now time.Time, forHowLong time.Duration, times int) inventory.Doing {
since := now.Add(-forHowLong)
return inventory.Doing{Node: "anchor", Outcome: inventory.OutcomeFailed, At: now.Add(-time.Second),
Since: &since, Times: times, Applied: 41, Failed: []inventory.FailedResource{
{ID: "apply", Error: `applying "pacman.base": nothing can be said about "base": this is the arch host and ` +
"pacman does not answer here. Either this machine is not Arch, or its package database is broken: " +
"pacman exited 1: error: config file /etc/pacman.conf could not be read: No such file or directory"},
{ID: "the package manager (pacman)", Error: "pacman does not answer here, so every package in this " +
"declaration fails and the rest of it is applied (novox/hq issue 233): pacman exited 1: error: config " +
"file /etc/pacman.conf could not be read: No such file or directory"}}}
}
// **A machine whose package manager does not answer is raised by the mesh itself, naming the tool and the
// probe's words** (novox/hq issue 233): the words in the evidence, the path kept out of the summary.
func TestAPackageManagerThatDoesNotAnswerIsRaisedNamingItsWords(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
f := calm(now)
f.doing = []inventory.Doing{unansweringPackageManager(now, 90*time.Minute, 18)}
got := watchApplyFailing(f)
if len(got) != 1 {
t.Fatalf("raised %d conditions, want one: %+v", len(got), got)
}
o := got[0]
if o.Key() != "machine.anchor.apply-failing" || o.Machine != "anchor" {
t.Errorf("the condition is not the machine's: %s", o.Key())
}
if !strings.Contains(o.Summary, "pacman does not answer") || !strings.Contains(o.Summary, "since 2026-10-04 12:30") {
t.Errorf("the summary does not name the tool and since when: %s", o.Summary)
}
if strings.Contains(o.Summary, "/etc/") {
t.Errorf("the summary carries a path, which belongs in the evidence: %s", o.Summary)
}
if !strings.Contains(o.Said, "could not be read") {
t.Errorf("the evidence does not carry the probe's words: %s", o.Said)
}
if !strings.Contains(o.Headline, "cannot install packages") || o.Explanation == "" || o.Resolved == "" {
t.Errorf("the plain words do not say it: %+v", o)
}
}
// **A whole refusal is raised too**, with the node-engine's words: what the workstation's report said before the
// node-engine applied the rest.
func TestARefusalThatGoesOnIsRaised(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
since := now.Add(-time.Hour)
f := calm(now)
f.doing = []inventory.Doing{{Node: "anchor", Outcome: inventory.OutcomeRefused, Since: &since, Times: 12,
Refused: "this is the arch host and pacman does not answer here", At: now.Add(-time.Minute)}}
got := watchApplyFailing(f)
if len(got) != 1 || !strings.Contains(got[0].Summary, "refused its declaration whole") ||
!strings.Contains(got[0].Said, "pacman does not answer") {
t.Fatalf("a refusal that went on for an hour was not raised with its words: %+v", got)
}
}
// **Judged by time while the machine is heard, not by how many reports said it** (review of novox/hq issue 233):
// a node-engine says the same failure once, so one report an apply interval old is a fault that goes on; one
// younger is not yet; and a silent or sleeping machine is S1's.
func TestOneReportOlderThanAnApplyIsRaisedAndAQuietMachineIsNot(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
f := calm(now)
f.doing = []inventory.Doing{unansweringPackageManager(now, 6*time.Minute, 1)}
if got := watchApplyFailing(f); len(got) != 1 {
t.Errorf("one report of a failure six minutes old was not raised: %+v", got)
}
f.doing = []inventory.Doing{unansweringPackageManager(now, 4*time.Minute, 1)}
if got := watchApplyFailing(f); len(got) != 0 {
t.Errorf("a failure four minutes old was raised before one apply could clear it: %+v", got)
}
f.doing = []inventory.Doing{unansweringPackageManager(now, time.Hour, 1)}
f.machines[0].lastHeard = now.Add(-time.Hour)
if got := watchApplyFailing(f); len(got) != 0 {
t.Errorf("a silent machine was raised for its last report: %+v", got)
}
f = calm(now)
f.doing = []inventory.Doing{unansweringPackageManager(now, time.Hour, 1)}
f.doing[0].Node = "a-machine-the-mesh-does-not-know"
if got := watchApplyFailing(f); len(got) != 0 {
t.Errorf("a machine not heard of was raised: %+v", got)
}
}
// **A failed re-apply followed by a clean one clears S21** (review of novox/hq issue 233): the node-engine says
// the clean re-apply, the store keeps it as applied with no failure since, and the condition the failure raised
// is cleared through the keeper on the next tick.
func TestACleanReapplyAfterAFailedOneClearsTheCondition(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
defer k.Close(context.Background())
w := &watchdogs{keeper: k, started: now.Add(-time.Hour)}
failing := calm(now)
failing.doing = []inventory.Doing{unansweringPackageManager(now, 20*time.Minute, 1)}
w.see(t.Context(), failing)
open, err := k.Open(t.Context())
if err != nil || len(open) != 1 || open[0].Kind != kindApplyFailing {
t.Fatalf("the failed re-apply raised %+v (%v), want apply-failing", open, err)
}
clean := calm(now.Add(5 * time.Minute))
clean.doing = nil // the clean re-apply recorded: NotDoingWhatTheyWereTold no longer lists the machine
w.see(t.Context(), clean)
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("the condition stayed open after the clean re-apply: %+v", open)
}
}
+9
View File
@@ -213,6 +213,8 @@ type asker struct {
saidNoRouter bool
saidNoGrant bool
// saidGrantCaveat is whether a grant held with a caveat was said, so it is said once.
saidGrantCaveat bool
mu sync.Mutex
nudged chan struct{}
@@ -317,6 +319,13 @@ func (a *asker) reconcile(ctx context.Context) error {
}
return a.sayUnasked(ctx, unasked, "the bus does not hold the controller's grant to ask yet: "+why)
}
if why != "" && !a.saidGrantCaveat {
// Held, with a caveat worth one line (an older bus module cannot confirm the reload).
a.logf("the controller asks: %s", why)
a.saidGrantCaveat = true
} else if why == "" {
a.saidGrantCaveat = false
}
a.saidNoGrant = false
}
channels := ""
+205
View File
@@ -12,6 +12,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
@@ -808,3 +809,207 @@ func TestARefusalIsAskedAgainAndTheUndeliveredConditionClearsOnceTaken(t *testin
t.Errorf("the explanation says its verdict twice: %q", e)
}
}
// hq issue 353, the review: the bus holds the controller's grant to ask only when both facts hold — the mesh
// composes the controller's own grant, and the bus's module says it is in force on the running server — and
// each fact not known is not held, said with why.
func TestTheGrantIsHeldOnlyWhenComposedAndInForceOnTheBus(t *testing.T) {
yes := func(context.Context) (bool, string, error) { return true, "", nil }
no := func(why string) grantJudge {
return func(context.Context) (bool, string, error) { return false, why, nil }
}
broken := func(context.Context) (bool, string, error) { return false, "", errors.New("the store is away") }
for _, c := range []struct {
name string
composed, inForce grantJudge
held bool
says string
}{
{"composed and in force", yes, yes, true, ""},
{"not composed", no("no router is assigned"), yes, false, "no router is assigned"},
{"composed, written, not reloaded", yes, no("the server has not reloaded"), false, "has not reloaded"},
{"the composition unknown", broken, yes, false, "could not be worked out: the store is away"},
{"the bus unknown", yes, broken, false, "could not be worked out: the store is away"},
{"nothing judges the bus", yes, nil, false, "not judged"},
} {
held, why, err := grantHeldBy(c.composed, c.inForce, time.Now)(context.Background())
if err != nil || held != c.held || !strings.Contains(why, c.says) {
t.Errorf("%s: held %v (%q, %v)", c.name, held, why, err)
}
}
// Remembered for grantLookEvery, then judged again.
now := time.Date(2026, 10, 9, 18, 0, 0, 0, time.UTC)
inForce := false
judge := grantHeldBy(yes, func(context.Context) (bool, string, error) { return inForce, "not yet", nil },
func() time.Time { return now })
if held, _, _ := judge(context.Background()); held {
t.Fatal("held before the bus enforces it")
}
inForce = true
if held, _, _ := judge(context.Background()); held {
t.Error("judged again inside grantLookEvery")
}
now = now.Add(grantLookEvery)
if held, _, _ := judge(context.Background()); !held {
t.Error("not judged again after grantLookEvery")
}
}
// The controller's own grant, as composed: present while a router that takes asks is assigned, absent
// otherwise — whatever else in the user list changes.
func TestTheControllersOwnGrantToAskIsReadFromTheComposition(t *testing.T) {
router := broker.Declared{Module: "messenger", Holds: []broker.Seat{{Name: broker.AsksSeat, Scope: "mesh",
Accepts: []string{"ask", "cancel"}, Emits: []string{"decided"}, ByCaller: []string{"ask", "cancel", "decided"}}}}
other := broker.Declared{Module: "dunst"}
with := broker.Records{Nodes: []string{"anchor"}, Assigned: map[string][]broker.Declared{"anchor": {router, other}}}
without := broker.Records{Nodes: []string{"anchor"}, Assigned: map[string][]broker.Declared{"anchor": {other}}}
if may, err := controllerMayAsk(with); err != nil || !may {
t.Errorf("with a router assigned: %v %v", may, err)
}
if may, err := controllerMayAsk(without); err != nil || may {
t.Errorf("with no router assigned: %v %v", may, err)
}
}
// The bus module's answer, with the record's word on the list sent as the check only where the answer does
// not settle it (reviews 22 and 31 of PR 188); each way it is not held names its own remedy.
func TestTheBusModulesAnswerNamesTheRemedyForEachCase(t *testing.T) {
const (
reloaded = `{"allowed":true,"in_force":true,"file_written":"2026-10-09T15:55:56Z","server_loaded":"2026-10-09T15:56:05Z"}`
notLoaded = `{"allowed":true,"in_force":false,"why":"the user list on the machine allows it, and the server has not reloaded since it was written","file_written":"2026-10-09T15:55:56Z","server_loaded":"2026-10-09T15:50:00Z"}`
noVarz = `{"allowed":true,"in_force":false,"why":"the server's last configuration load could not be read: refused"}`
noStat = `{"allowed":true,"in_force":false,"why":"when the bus's user list was written could not be read inside mesh-broker-nats: exit 1"}`
notAllowed = `{"allowed":false,"in_force":false,"why":"the user list on the machine does not allow it"}`
)
current := busListState{Machine: "anchor"}
behind := busListState{Machine: "anchor", Behind: true}
stepped := busListState{Machine: "anchor", Behind: true, BusStep: "sending anchor would replace the bus; `bus upgrade --why …`"}
uncomposed := busListState{Machine: "anchor", Behind: true, Uncomposed: errors.New("module foo requires a provision nobody provides")}
for _, c := range []struct {
name, answer string
askErr error
list busListState
listErr error
held bool
says, not string
}{
{"in force", reloaded, nil, behind, nil, true, "", "push"},
{"written, not reloaded", notLoaded, nil, current, nil, false, "within about 5 s", "push"},
{"in force not known: varz", noVarz, nil, current, nil, false, "is not known: the server's last configuration load", "wait"},
{"in force not known: stat", noStat, nil, current, nil, false, "is not known: when the bus's user list was written", "wait"},
{"not allowed, list behind", notAllowed, nil, behind, nil, false, "`push anchor` carries it", "5 s"},
{"not allowed, bus step pending", notAllowed, nil, stepped, nil, false, "`bus upgrade", "`push anchor`"},
{"not allowed, list uncomposed", notAllowed, nil, uncomposed, nil, false, "`plan anchor`", "`push anchor`"},
{"older module, allowed, list current", `{"allowed":true}`, nil, current, nil, true, "not confirmed", "push"},
{"older module, allowed, list behind", `{"allowed":true}`, nil, behind, nil, true, "not confirmed", "push"},
{"older module, allowed, list unknown", `{"allowed":true}`, nil, current, errors.New("the store is away"), true, "`bus upgrade`", "push"},
{"older module, not allowed, list behind", `{"allowed":false}`, nil, behind, nil, false, "`push anchor` carries it", "5 s"},
{"module cannot answer, list current", "", errors.New("no responders"), current, nil, true, "could not answer", "push"},
{"module cannot answer, list behind", "", errors.New("no responders"), behind, nil, false, "`push anchor` carries it", ""},
{"module cannot answer, list unknown", "", errors.New("no responders"), current, errors.New("the store is away"), false, "could not be worked out", ""},
{"answer unreadable, list behind", `not json`, nil, behind, nil, false, "could not be read", ""},
} {
judge := busGrantBy(
func(context.Context) (string, json.RawMessage, error) {
return "anchor", json.RawMessage(c.answer), c.askErr
},
func(context.Context) (busListState, error) { return c.list, c.listErr })
held, why, _ := judgeGrantOnce(judge)
if held != c.held || !strings.Contains(why, c.says) || (c.not != "" && strings.Contains(why, c.not)) {
t.Errorf("%s: held %v (%q)", c.name, held, why)
}
}
none := busGrantBy(func(context.Context) (string, json.RawMessage, error) { return "", nil, nil }, nil)
if held, why, _ := judgeGrantOnce(none); held || !strings.Contains(why, "no module holds the bus") {
t.Errorf("no bus: held %v (%q)", held, why)
}
}
// judgeGrantOnce is the whole judgement with the composed grant held, as the asker makes it.
func judgeGrantOnce(bus grantJudge) (bool, string, error) {
yes := func(context.Context) (bool, string, error) { return true, "", nil }
return grantHeldBy(yes, bus, time.Now)(context.Background())
}
// The orders reviews 22 and 31 of PR 188 found: this controller live before the bus module that answers
// in_force, with the bus's machine behind on its list — and, after mesh-catalog #166 merged, unpushable until a
// person's bus upgrade. An older module that allows the ask asks (once, with the caveat said); a new module that
// has not reloaded names the wait, not a push; one that cannot say whether it reloaded names what is not known.
func TestAnOlderBusModuleNeitherSilencesTheAsksNorNamesTheWrongRemedy(t *testing.T) {
for _, c := range []struct {
name, answer string
list busListState
asks bool
says, not string
}{
{"older module, list current", `{"allowed":true}`, busListState{Machine: "anchor"}, true, "not confirmed", ""},
{"older module, list behind, bus step pending", `{"allowed":true}`,
busListState{Machine: "anchor", Behind: true, BusStep: "a planned step"}, true, "not confirmed", ""},
{"new module, not reloaded",
`{"allowed":true,"in_force":false,"why":"not reloaded","file_written":"2026-10-09T15:55:56Z","server_loaded":"2026-10-09T15:50:00Z"}`,
busListState{Machine: "anchor"}, false, "within about 5 s", "push"},
{"new module, reload not known", `{"allowed":true,"in_force":false,"why":"the server's last configuration load could not be read: refused"}`,
busListState{Machine: "anchor"}, false, "is not known", "wait"},
} {
t.Run(c.name, func(t *testing.T) {
r := newAskerRig(t)
var said []string
r.a.logf = func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }
var raised [][]conditions.Observation
r.a.raise = func(_ context.Context, obs []conditions.Observation) error {
raised = append(raised, obs)
return nil
}
yes := func(context.Context) (bool, string, error) { return true, "", nil }
r.a.grantHeld = grantHeldBy(yes, busGrantBy(
func(context.Context) (string, json.RawMessage, error) {
return "anchor", json.RawMessage(c.answer), nil
},
func(context.Context) (busListState, error) { return c.list, nil }),
func() time.Time { return r.now })
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
_ = r.a.reconcile(context.Background())
sent := r.asksSent(t)
if c.asks {
if len(sent) != 1 || sent[0].About != heldCondition().Key {
t.Fatalf("not asked: %+v", sent)
}
n := 0
for _, s := range said {
if strings.Contains(s, c.says) {
n++
}
}
if n != 1 {
t.Errorf("the caveat said %d times: %q", n, said)
}
return
}
if len(sent) != 0 {
t.Fatalf("asked while the bus does not hold the grant: %+v", sent)
}
if len(raised) == 0 || len(raised[len(raised)-1]) != 1 ||
raised[len(raised)-1][0].Kind != "asks-undelivered" {
t.Fatalf("not said as undelivered: %+v", raised)
}
summary := raised[len(raised)-1][0].Summary
if !strings.Contains(summary, c.says) || (c.not != "" && strings.Contains(summary, c.not)) {
t.Errorf("the condition names the wrong remedy: %q", summary)
}
})
}
}
// The serving asker is wired to the real judgement: with records and a bus it cannot read, the grant is not
// held, and so nothing is asked (a stub saying yes, or no judgement at all, would ask).
func TestTheServingAskerJudgesTheGrantForReal(t *testing.T) {
a := servingAsker(&stores{}, nil, nil, nil)
if a.grantHeld == nil {
t.Fatal("the serving asker does not judge the bus's grant")
}
held, why, err := a.grantHeld(context.Background())
if err != nil || held || !strings.Contains(why, "could not be worked out") {
t.Errorf("held %v (%q, %v) with nothing to read", held, why, err)
}
}
+271 -21
View File
@@ -235,12 +235,27 @@ func routerHereIn(inv *inventory.Inventory) func(ctx context.Context) (bool, err
}
}
// grantHeldIn says whether the bus holds the controller's grant to ask (novox/hq issue 353): the user list the
// machine holding the bus was last sent is the one the mesh composes now (brokerBehind, the same judgement a
// push makes to send that machine first). While it is behind, the controller's ask is refused by the bus,
// whatever the record says of the router, so nothing is asked and the operator is told to push that machine.
// Judged at most every grantLookEvery: composing the list resolves the bus's machine whole.
func grantHeldIn(open *stores) func(ctx context.Context) (bool, string, error) {
// grantHeldIn says whether the bus holds the controller's grant to ask (novox/hq issue 353), judged on the two
// facts that make it so, and failing closed on each:
//
// 1. **composed**: the user list the mesh composes now grants the controller itself to publish its ask
// (controllerMayAsk) — the controller's own grant, not whether any user of the list changed;
// 2. **in force**: the bus's own module, on the machine holding the bus, says the user list there allows it
// and the server reloaded after that list was written (the nats module's `nats_user_can`, `in_force`) — a
// list sent is not a list the bus enforces until it reloads.
//
// Anything that cannot be worked out is not held, with why, so the conditions that need the operator are said
// as undelivered rather than asked into a refusal.
func grantHeldIn(open *stores, conn *nats.Conn) func(ctx context.Context) (bool, string, error) {
return grantHeldBy(composedGrantIn(open), busGrantIn(open, conn), time.Now)
}
// grantJudge is one of the two facts: whether it holds, and why not.
type grantJudge func(ctx context.Context) (bool, string, error)
// grantHeldBy is the judgement over the two facts, remembered for grantLookEvery: composing the list resolves
// the bus's machine whole, and asking the bus's module is a call.
func grantHeldBy(composed, inForce grantJudge, now func() time.Time) func(ctx context.Context) (bool, string, error) {
var mu sync.Mutex
var at time.Time
var held bool
@@ -248,22 +263,251 @@ func grantHeldIn(open *stores) func(ctx context.Context) (bool, string, error) {
return func(ctx context.Context) (bool, string, error) {
mu.Lock()
defer mu.Unlock()
if !at.IsZero() && time.Since(at) < grantLookEvery {
if !at.IsZero() && now().Sub(at) < grantLookEvery {
return held, why, nil
}
machine, behind, err := brokerBehind(ctx, open, nil)
held, why = judgeGrant(ctx, composed, inForce)
at = now()
return held, why, nil
}
}
// judgeGrant is held only when both facts hold; an error is a fact not known, never a yes. A fact that holds
// with a caveat (a reload not confirmed) gives that caveat as the why of a yes.
func judgeGrant(ctx context.Context, composed, inForce grantJudge) (bool, string) {
var caveat []string
for _, fact := range []struct {
judge grantJudge
what string
}{{composed, "whether the mesh composes the controller a grant to ask"}, {inForce, "whether the bus enforces it"}} {
if fact.judge == nil {
return false, fact.what + " is not judged here"
}
ok, why, err := fact.judge(ctx)
if err != nil {
return false, fact.what + " could not be worked out: " + err.Error()
}
if !ok {
return false, why
}
if why != "" {
caveat = append(caveat, why)
}
}
return true, strings.Join(caveat, "; ")
}
// controllerMayAsk says whether the controller's own grant, as composed from these records, lets it publish its
// ask (ADR 0259 §3: composed while a router that takes asks under its asker's name is assigned).
func controllerMayAsk(records broker.Records) (bool, error) {
users, err := broker.Users(records)
if err != nil {
return false, err
}
for _, u := range users {
if u.Kind != broker.KindController {
continue
}
perms, err := broker.PermissionsFor(u)
if err != nil {
return false, err
}
return broker.MayPublish(perms, asks.AskSubject(askerName)), nil
}
return false, errors.New("the composition holds no controller")
}
// composedGrantIn is the first fact, from the mesh's records.
func composedGrantIn(open *stores) grantJudge {
return func(ctx context.Context) (bool, string, error) {
if open == nil || open.inventory == nil {
return false, "", errors.New("the mesh's records are not open")
}
records, err := open.inventory.BusRecords(ctx)
if err != nil {
return false, "", err
}
at, held, why = time.Now(), !behind, ""
if behind {
why = fmt.Sprintf("the bus's user list on %s is behind what the mesh composes, so the bus has not been "+
"given the controller's grant to ask; `push %s` carries it", machine, machine)
may, err := controllerMayAsk(records)
if err != nil || may {
return may, "", err
}
return held, why, nil
return false, "the mesh composes the controller no grant to ask: no router that takes asks under its " +
"asker's name is assigned", nil
}
}
// busGrantIn is the second fact, asked of the bus's own module on the machine holding the bus, with the check
// main made before it (novox/hq issue 353, PR 186: is the user list sent to the bus's machine the one composed
// now) kept for the cases the module's word does not settle.
func busGrantIn(open *stores, conn *nats.Conn) grantJudge {
ask := func(ctx context.Context) (string, json.RawMessage, error) {
if open == nil || open.inventory == nil || conn == nil {
return "", nil, errors.New("the mesh's records or the bus are not open")
}
holders, err := seatHolders(ctx, open.inventory)
if err != nil {
return "", nil, err
}
h, held := holders[theBrokerSeat]
if !held || h.Node == "" || h.Module == "" {
return "", nil, nil
}
a, err := link.AskModuleToolOn(ctx, conn, h.Module, busUserCanTool, h.Node, map[string]any{
"user": broker.ControllerName, "action": "publish", "subject": asks.AskSubject(askerName)}, 15*time.Second)
if err != nil {
return h.Node, nil, err
}
if a.Error != "" {
return h.Node, nil, errors.New(a.Error)
}
return h.Node, a.Result, nil
}
list := func(ctx context.Context) (busListState, error) {
if open == nil || open.inventory == nil {
return busListState{}, errors.New("the mesh's records are not open")
}
s, err := brokerListState(ctx, open, nil)
if err != nil || s.Machine == "" || !s.Behind {
return s, err
}
step, err := busHeld(ctx, open.inventory, []string{s.Machine})
if err != nil {
return busListState{}, err
}
s.BusStep = step[s.Machine]
return s, nil
}
return busGrantBy(ask, list)
}
// busListState is the user list on the bus's machine as the controller's record knows it: whether the list
// sent there is behind the one composed now, why it could not be composed, and the bus step a send there waits
// on (that machine cannot be pushed until a person's `bus upgrade`).
type busListState struct {
Machine string
Behind bool
Uncomposed error
BusStep string
}
// remedy is the one thing that sends the bus's machine the list composed now: the plan's own reason when it
// cannot be composed, the bus upgrade while sends there wait on it, and otherwise a push of that machine.
func (s busListState) remedy(node string) string {
if s.Machine != "" {
node = s.Machine
}
switch {
case s.Uncomposed != nil:
return fmt.Sprintf("the user list for %s cannot be composed (%v); `plan %s` says what to fix first",
node, s.Uncomposed, node)
case s.BusStep != "":
return fmt.Sprintf("sends to %s wait on the bus's planned step, so a push will not carry it: %s", node, s.BusStep)
}
return fmt.Sprintf("`push %s` carries it", node)
}
// busGrantBy judges the bus's word, and names for each way it is not held the one thing that makes it held:
//
// - **in force**: held;
// - **allowed, from a module older than `in_force`**: held, with the caveat that the reload is not confirmed.
// The file allows the publish, and the bus reloads within about 5 s of a change to it, so the worst case is
// one ask refused in those seconds, which is asked again (issue 369). The list check is not consulted:
// while a bus step is pending the bus's machine cannot be pushed, so any change to the composed list would
// read as behind for good and silence every ask until a person's bus upgrade (the review of PR 188);
// - **allowed, written, not reloaded** (the module gives both times): wait about 5 s;
// - **allowed, in force not known** (the module could not read a time): what is not known, said;
// - **not allowed, or the module cannot answer**: the list check decides — the remedy that sends the list
// when it is behind (busListState.remedy), and, for a module that cannot answer, held while the list sent
// is current, as main judged it.
//
// ask answers the machine holding the bus and its module's answer ("" when nothing holds the bus); list is the
// record's word on the list sent to that machine.
func busGrantBy(ask func(ctx context.Context) (string, json.RawMessage, error),
list func(ctx context.Context) (busListState, error)) grantJudge {
return func(ctx context.Context) (bool, string, error) {
node, raw, askErr := ask(ctx)
if askErr == nil && node == "" {
return false, "no module holds the bus, so nothing says what it enforces", nil
}
var answer busAnswer
if askErr == nil {
if answer, askErr = readBusAnswer(raw); askErr != nil {
askErr = fmt.Errorf("its answer could not be read: %w", askErr)
}
}
if askErr != nil {
if node == "" {
node = "the bus's machine"
}
s, err := list(ctx)
if err != nil {
return false, "", fmt.Errorf("the bus module on %s could not answer (%v), and whether the list sent "+
"there is the one composed now could not be worked out: %w", node, askErr, err)
}
if s.Behind {
return false, fmt.Sprintf("the bus module on %s could not answer (%v), and the user list sent there is "+
"behind what the mesh composes; %s", node, askErr, s.remedy(node)), nil
}
return true, fmt.Sprintf("the bus module on %s could not answer (%v); judged by the user list sent there, "+
"which is the one composed now", node, askErr), nil
}
notAllowed := func(older string) (bool, string, error) {
s, err := list(ctx)
if err != nil {
return false, "", fmt.Errorf("the user list on %s does not allow the controller to ask%s, and whether "+
"the list sent there is the one composed now could not be worked out: %w", node, older, err)
}
if s.Behind {
return false, fmt.Sprintf("the user list on %s does not allow the controller to ask%s: it is behind "+
"what the mesh composes; %s", node, older, s.remedy(node)), nil
}
return false, fmt.Sprintf("the user list on %s does not allow the controller to ask%s, though the list "+
"sent there is the one composed now; %s again", node, older, s.remedy(node)), nil
}
switch {
case answer.InForce == nil && answer.Allowed:
return true, fmt.Sprintf("the user list on %s allows the controller to ask, but its reload is not "+
"confirmed: the bus module there is older than the answer whether a grant is in force, and the bus "+
"upgrade (`bus upgrade`) carries one that confirms it", node), nil
case answer.InForce == nil:
return notAllowed(" (the bus module there is older than the answer whether a grant is in force)")
case !answer.Allowed:
return notAllowed("")
case *answer.InForce:
return true, "", nil
case answer.FileWritten != "" && answer.ServerLoaded != "":
return false, fmt.Sprintf("the bus on %s has the controller's grant to ask written (%s) but has not "+
"reloaded since (last load %s); it reloads in place within about 5 s, so nothing is to be done but wait",
node, answer.FileWritten, answer.ServerLoaded), nil
}
why := answer.Why
if why == "" {
why = "it gives no reason"
}
return false, fmt.Sprintf("the user list on %s allows the controller to ask, but whether the bus enforces it "+
"is not known: %s", node, why), nil
}
}
// busUserCanTool is the bus module's answer to whether a user may do something, and whether it is in force.
const busUserCanTool = "nats_user_can"
// busAnswer is the bus module's answer to busUserCanTool; InForce is nil from a module older than that field,
// and FileWritten and ServerLoaded are given only when it could read both times (mesh-catalog's nats InForce).
type busAnswer struct {
Allowed bool `json:"allowed"`
InForce *bool `json:"in_force"`
Why string `json:"why"`
FileWritten string `json:"file_written"`
ServerLoaded string `json:"server_loaded"`
}
func readBusAnswer(raw json.RawMessage) (busAnswer, error) {
var answer busAnswer
err := json.Unmarshal(raw, &answer)
return answer, err
}
// grantLookEvery is how often the bus's user list is judged against the one its machine was last sent.
const grantLookEvery = 30 * time.Second
@@ -302,7 +546,19 @@ func startAsking(ctx context.Context, open *stores, server *link.Server, conn *n
fmt.Printf("the operator cannot be asked: %v\n", err)
return
}
a := &asker{
a := servingAsker(open, conn, js, keeper)
if err := server.Decides(a); err != nil {
fmt.Printf("the operator's answers cannot be heard, so nothing is asked: %v\n", err)
return
}
askerFrom = a
go a.keep(ctx)
}
// servingAsker is the serving controller's asker, wired to the mesh: what it reads, publishes, performs and
// records, and the judgements it asks by — the bus's grant to ask among them (novox/hq issue 353).
func servingAsker(open *stores, conn *nats.Conn, js jetstream.JetStream, keeper *conditions.Keeper) *asker {
return &asker{
open: keeper.Open,
silence: func(ctx context.Context, key string, d time.Duration, by, why string) error {
_, err := keeper.Silence(ctx, key, d, by, why)
@@ -322,7 +578,7 @@ func startAsking(ctx context.Context, open *stores, server *link.Server, conn *n
},
routerRecord: routerRecordOf(conn, open.inventory),
routerHere: routerHereIn(open.inventory),
grantHeld: grantHeldIn(open),
grantHeld: grantHeldIn(open, conn),
channels: channelsIn(open.inventory),
raise: func(ctx context.Context, obs []conditions.Observation) error {
return keeper.Reconcile(ctx, sourceAsker, obs)
@@ -330,10 +586,4 @@ func startAsking(ctx context.Context, open *stores, server *link.Server, conn *n
now: time.Now,
logf: func(format string, args ...any) { fmt.Printf(format+"\n", args...) },
}
if err := server.Decides(a); err != nil {
fmt.Printf("the operator's answers cannot be heard, so nothing is asked: %v\n", err)
return
}
askerFrom = a
go a.keep(ctx)
}
+232 -17
View File
@@ -37,6 +37,11 @@ import (
// own commit, the newest first, one walk each, until one is delivered; the older ones are then answered
// by that one.
//
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, ADR 0306 decisions
// 2 and 3, issue 525): it joins a batch it is related to, and one that is related to none and whose delivery
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Walks stay one at
// a time: such a batch is cut when no walk is open, and queued behind the one that is otherwise.
//
// A merge on the controller's own path (a module whose walk waits for nobody's word) never shares a batch with
// one that waits for mesh-delivery's: each kind has a batch of its own, an own-path batch is cut first and folds
// no waiting catalogue walk, and a catalogue walk let go while another walk runs starts once it ended — so no
@@ -173,6 +178,12 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
fmt.Printf("%s merged into %s (%.8s); nothing the mesh holds reads it\n", repository, m.Base, m.Commit)
return nil
}
// **Its delivery group's word, before the plans are held** (ADR 0306 decision 2): a slow answer holds up no walk.
own := ownPath(touches)
waits, err := groupWait(ctx, inv, m, deliverySeatHeld(entries), own)
if err != nil {
return notNow(err)
}
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return notNow(err)
@@ -198,7 +209,6 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
// **A merge on the controller's own path never shares a batch with one that waits for the delivery's
// word** (decided during the build, 2026-10-10): batched together, the catalogue's deliveries would start
// with the controller's and skip their turn. Each kind has a batch of its own.
own := ownPath(touches)
if p, ok, err := answeredByALaterMerge(ctx, inv, heard, touches, own, deliverySeatHeld(entries)); err != nil {
return notNow(err)
} else if ok {
@@ -218,7 +228,7 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
return cutBatchesHeld(ctx, f.open, now)
}
batch, err := theOpenBatch(ctx, inv, heard, now, own)
batch, why, err := batchToJoin(ctx, inv, heard, entries, read, now, own, waits)
if err != nil {
return notNow(err)
}
@@ -229,7 +239,7 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
if err := keepBatch(ctx, inv, &batch, now, ""); err != nil {
return notNow(err)
}
fmt.Printf("%s merged into %s (%.8s); it joins %s — %s\n", repository, m.Base, m.Commit, batch.ID,
fmt.Printf("%s merged into %s (%.8s); %s, it joins %s — %s\n", repository, m.Base, m.Commit, why, batch.ID,
batchWords(batch, now))
return cutBatchesHeld(ctx, f.open, now)
}
@@ -320,22 +330,211 @@ func ownPath(touches []inventory.Entry) bool {
return false
}
// theOpenBatch is the batch a merge heard now joins: the one of its kind not yet cut, or a new one.
func theOpenBatch(ctx context.Context, inv *inventory.Inventory, first inventory.BatchedMerge, now time.Time, own bool) (inventory.Plan, error) {
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, as ADR 0306
// decisions 2 and 3 read it). batchToJoin is the batch a merge heard now joins, and why in words:
//
// - an assembling or queued batch of its kind it is related to (related.go): it joins it and waits for the
// window, which its coming restarts. Related to two or more, the newer are folded into the oldest, which
// answers all their merges: a merge never waits in two windows, and what it shares with each walks in one.
// - none, and its delivery group has a member still to merge, or mesh-delivery could not say (groupWait, waits
// names why): a new batch, which waits for the window, beside the ones it shares nothing with; its words say
// why.
// - none, and no member of its group is still to merge: a new batch whose window is closed at once. Cut at
// once means only that: walks stay one at a time (ADR 0276 decision 4), so with a walk open the batch is
// queued behind it as any closed batch is.
func batchToJoin(ctx context.Context, inv *inventory.Inventory, heard inventory.BatchedMerge, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, now time.Time, own bool, waits string) (inventory.Plan, string, error) {
edges, err := inv.Dependencies(ctx)
if err != nil {
return inventory.Plan{}, "", err
}
joins, whys, _, err := relatedBatches(ctx, inv, []inventory.BatchedMerge{heard}, own, "", entries, read, edges)
if err != nil {
return inventory.Plan{}, "", err
}
if len(joins) > 0 {
into := joins[0]
why := "related to " + into.ID + " (" + whys[0] + ")"
if len(joins) > 1 {
plans, moves, err := foldInto(ctx, inv, &into, joins[1:])
if err != nil {
return inventory.Plan{}, "", err
}
if err := inv.SavePlansMoving(ctx, plans, moves); err != nil {
return inventory.Plan{}, "", err
}
var folded []string
for i, b := range joins[1:] {
folded = append(folded, b.ID+" ("+whys[i+1]+")")
}
why += ", and to " + strings.Join(folded, ", ") + ", folded into it"
}
into.Delivery.Batch.CutAtOnce = false
return into, why, nil
}
if waits != "" {
// Its own window, beside the batches it shares nothing with: waiting is for its group's members, which
// are related to it by their head and join it (ADR 0306 decision 2).
b, err := newBatch(ctx, inv, heard, now, own, false, waits)
return b, waits + ", it waits for the window in a batch of its own", err
}
b, err := newBatch(ctx, inv, heard, now, own, true, "")
return b, "it shares nothing with a batch assembling and no member of a group is still to merge: it is cut at once", err
}
// relatedBatches are the batches not yet cut of a kind, oldest first, parted into those related to merges (with
// the first reason of each) and the others. Each batch is judged by what its cut would move (ADR 0306 decision
// 3): its merges' moves as they were kept and as they read now — the catalogue may have moved since a merge was
// heard — widened to their dependents, and, for a catalogue batch, every walk waiting for its word, which the
// cut folds in (cutBatch's carry). A merge kept with no moves leaves its batch unknown, related to everything.
// except is a walk not to count as folded: the one being stopped.
func relatedBatches(ctx context.Context, inv *inventory.Inventory, merges []inventory.BatchedMerge, own bool,
except string, entries []inventory.Entry, read map[string][]inventory.ReadRepository,
edges []inventory.Edge) (joins []inventory.Plan, whys []string, others []inventory.Plan, err error) {
batches, err := inv.Batches(ctx)
if err != nil {
return inventory.Plan{}, err
return nil, nil, nil, err
}
var kind []inventory.Plan
for _, b := range batches {
if b.OwnPath() == own {
return b, nil
kind = append(kind, b)
}
}
if len(kind) == 0 {
return nil, nil, nil, nil
}
carry := newFootprint()
if !own {
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch may fold every one.
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, nil, nil, err
}
for _, p := range plans {
if p.Release != nil || !p.Waiting() || p.ID == except {
continue
}
kept, err := inv.MergesOf(ctx, p.ID)
if err != nil {
return nil, nil, nil, err
}
carry = unionFootprint(carry, footprintOfWalk(p, kept))
}
}
mine := footprintAsCut(merges, entries, read, edges)
for _, b := range kind {
held, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return nil, nil, nil, err
}
if b.Delivery == nil {
b.Delivery = &inventory.PlanDelivery{}
}
if b.Delivery.Batch == nil {
b.Delivery.Batch = &inventory.PlanBatch{Own: own}
}
if ok, why := related(mine, unionFootprint(footprintAsCut(held, entries, read, edges), carry), edges); ok {
joins, whys = append(joins, b), append(whys, why)
} else {
others = append(others, b)
}
}
return joins, whys, others, nil
}
// footprintAsCut is a batch's footprint as a cut now would read it: footprintOfBatch, widened by the modules each
// merge moves as the catalogue reads now (planBatch judges each merge again at the cut, so what it moves may
// have changed since it was heard). Both are kept: when in doubt, related.
func footprintAsCut(merges []inventory.BatchedMerge, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, edges []inventory.Edge) footprint {
f := footprintOfBatch(merges, edges)
var moved []string
for _, m := range merges {
s := announcedOf(m).SourceMoved
if s.Owner == "" {
continue
}
s.MergedAt = "" // news, whatever a cut marked seen since (as planBatch reads it)
for _, e := range wouldMove(s, entries, read) {
moved = append(moved, e.Manifest.Module)
}
}
for _, name := range reachableFrom(moved, edges) {
f.modules[name] = true
}
return f
}
// unionFootprint is what two changes reach together.
func unionFootprint(a, b footprint) footprint {
f := newFootprint()
for _, from := range []footprint{a, b} {
for k := range from.repos {
f.repos[k] = true
}
for k := range from.modules {
f.modules[k] = true
}
for k := range from.heads {
f.heads[k] = true
}
}
f.unknown = a.unknown
if f.unknown == "" {
f.unknown = b.unknown
}
return f
}
// foldInto folds batches into the oldest one related to a merge: the plans to write (into first, so no merge
// names a plan the store does not hold) and the merges to move, for one SavePlansMoving. The folded batches name
// the batch that took them over; a stop's closed window and its withheld modules (ADR 0299) go with their merges.
func foldInto(ctx context.Context, inv *inventory.Inventory, into *inventory.Plan, from []inventory.Plan) ([]*inventory.Plan,
[]inventory.MergeMove, error) {
if into.Delivery == nil {
into.Delivery = &inventory.PlanDelivery{}
}
if into.Delivery.Batch == nil {
into.Delivery.Batch = &inventory.PlanBatch{Own: into.OwnPath()}
}
plans := []*inventory.Plan{into}
var moves []inventory.MergeMove
for i := range from {
f := from[i]
merges, err := inv.MergesOf(ctx, f.ID)
if err != nil {
return nil, nil, err
}
for _, m := range merges {
moves = append(moves, inventory.MergeMove{Repository: m.Repository, Commit: m.Commit, Plan: into.ID, Alone: m.Alone})
}
if f.Delivery == nil {
f.Delivery = &inventory.PlanDelivery{}
}
if w := f.Delivery.Batch; w != nil {
into.Delivery.Batch.Closed = into.Delivery.Batch.Closed || w.Closed
into.Delivery.Batch.Withheld = union(into.Delivery.Batch.Withheld, w.Withheld)
sort.Strings(into.Delivery.Batch.Withheld)
}
f.State = inventory.PlanSuperseded
f.Delivery.Batch = nil
f.Delivery.TakenOverBy = into.ID
f.Note = fmt.Sprintf("folded into %s before it was cut: a merge related to both joined it, and %s answers "+
"its merges", into.ID, into.ID)
plans = append(plans, &f)
}
return plans, moves, nil
}
// newBatch is a new batch of a kind, kept before its first merge names it: a merge never names a plan the store
// does not hold. cutAtOnce closes its window at once; waits says why it waits for its group, when it does.
func newBatch(ctx context.Context, inv *inventory.Inventory, first inventory.BatchedMerge, now time.Time, own,
cutAtOnce bool, waits string) (inventory.Plan, error) {
b := inventory.Plan{ID: fmt.Sprintf("plan-%d", now.UnixNano()), Repository: first.Repository, Branch: first.Branch,
Commit: first.Commit, Merged: first.Merged, Created: now, State: inventory.PlanAssembling,
Tiers: [][]string{}, Modules: map[string]*inventory.PlanModule{},
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own}}}
// Kept before its first merge names it: a merge never names a plan the store does not hold.
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own, CutAtOnce: cutAtOnce, Waits: waits}}}
if err := inv.SavePlan(ctx, &b); err != nil {
return inventory.Plan{}, err
}
@@ -367,13 +566,16 @@ func keepBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan,
}
closes, latest := windowOf(merges, b.Created, window, atMost)
b.Delivery.Merges = named
var closed bool
var closed, cutAtOnce bool
var withheld []string
var waits string
if was := b.Delivery.Batch; was != nil {
closed, withheld = was.Closed, was.Withheld // a stop's, which no look takes back (novox/hq ADR 0299)
cutAtOnce = was.CutAtOnce // the merge's, until another joins it (ADR 0304 decision 1)
waits = was.Waits // its group's, as said when it was made (ADR 0306 decision 2)
}
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, Own: b.OwnPath(),
Closed: closed, Withheld: withheld}
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits}
b.State = inventory.PlanAssembling
if behind != "" && windowClosed(*b, now) {
b.State = inventory.PlanQueued
@@ -402,13 +604,14 @@ func windowOf(merges []inventory.BatchedMerge, created time.Time, window, atMost
return last.Add(window), created.Add(atMost)
}
// windowClosed says a batch's window is closed: no merge for the window's length, or its maximum reached.
// windowClosed says a batch's window is closed: closed by a stop, closed at once for a merge that shares
// nothing (ADR 0304 decision 1), no merge for the window's length, or its maximum reached.
func windowClosed(b inventory.Plan, now time.Time) bool {
if b.Delivery == nil || b.Delivery.Batch == nil {
return true
}
w := b.Delivery.Batch
return w.Closed || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
return w.Closed || w.CutAtOnce || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
}
// carriedOf is a set of merges as a walk carries them: the latest merge of each repository's branch, and
@@ -617,8 +820,9 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
return err
}
// The oldest batch whose window closed is cut, an own-path one before a catalogue one (it goes first, and
// the catalogue batch queues behind it rather than being cut and overtaken); one of each kind at most is
// open (theOpenBatch).
// the catalogue batch queues behind it rather than being cut and overtaken). Several of a kind may be open
// since a merge that shares nothing is cut at once (ADR 0304 decision 1): walks stay one at a time, so one
// is cut per look, and once its walk started the others are queued behind it.
sort.SliceStable(batches, func(i, j int) bool { return batches[i].OwnPath() && !batches[j].OwnPath() })
for i := range batches {
b := &batches[i]
@@ -886,6 +1090,9 @@ func walkTimesAtCut(batch, walk inventory.Plan, entries []inventory.Entry, now t
if !w.AtMost.IsZero() && w.AtMost.Before(closed) {
closed = w.AtMost
}
if w.CutAtOnce && !batch.Created.IsZero() {
closed = batch.Created // closed when it was made (ADR 0304 decision 1)
}
if !closed.IsZero() {
if closed.After(now) {
closed = now
@@ -1142,6 +1349,10 @@ func batchWords(b inventory.Plan, now time.Time) string {
if b.State == inventory.PlanQueued {
return fmt.Sprintf("queued behind %s; grouped: %s; plan not yet calculated", w.Behind, grouped)
}
if w.CutAtOnce {
return fmt.Sprintf("cut at once, sharing nothing with a batch assembling; grouped: %s; plan not yet calculated",
grouped)
}
left := w.ClosesAt.Sub(now)
if w.AtMost.Before(w.ClosesAt) {
left = w.AtMost.Sub(now)
@@ -1149,8 +1360,12 @@ func batchWords(b inventory.Plan, now time.Time) string {
if left < 0 {
left = 0
}
return fmt.Sprintf("assembling: %s left (at the latest %s); grouped: %s; plan not yet calculated",
secondsWords(left), w.AtMost.Local().Format("15:04:05"), grouped)
waits := ""
if w.Waits != "" {
waits = " — " + w.Waits
}
return fmt.Sprintf("assembling: %s left (at the latest %s)%s; grouped: %s; plan not yet calculated",
secondsWords(left), w.AtMost.Local().Format("15:04:05"), waits, grouped)
}
// groupedWords is a batch's commits, each with the earlier merges it answers.
+77 -13
View File
@@ -34,6 +34,7 @@ func windowed(t *testing.T) *stores {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
noGroups(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
@@ -58,19 +59,70 @@ func windowed(t *testing.T) *stores {
// t0 is the moment a test's clock starts: merges are dated by it, and heard after it.
var t0 = time.Now().UTC().Truncate(time.Second)
// catalogueMerge is a merge of the catalogue changing one module's directory, made at a moment.
// catalogueMerge is a merge of the catalogue changing one module's directory, made at a moment, from a pull
// request's head branch of its own, as the forge announces every merge. Whether its head names a delivery group
// with a member still to merge is mesh-delivery's word, which windowed stubs as no group at all (noGroups); a test
// whose merges wait for the window as a group's says so (groupsWaiting).
func catalogueMerge(commit, module string, made time.Time) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit,
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Head: "pr/" + commit, Commit: commit,
MergedAt: made.Format(time.RFC3339Nano), Paths: []string{"modules/" + module + "/module.json"},
ModuleDirs: []string{"modules/" + module}, ModuleDirsSaid: true}
}
// repoMerge is a merge of a repository of one module's own.
// repoMerge is a merge of a repository of one module's own, from a head branch of its own (catalogueMerge).
func repoMerge(repo, commit string, made time.Time) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Commit: commit,
return link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Head: "pr/" + commit, Commit: commit,
MergedAt: made.Format(time.RFC3339Nano), Paths: []string{"main.go"}}
}
// noGroups stubs mesh-delivery's word on groups for a test: no head names a group.
func noGroups(t *testing.T) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) { return nil, false, nil }
t.Cleanup(func() { deliveryGroupOf = was })
}
// groupsWaiting stubs mesh-delivery's word on groups for a test: each head named is a group with a member of another
// repository whose pull request is open (novox/elsewhere, ready), so a merge from it waits for the window (ADR 0306
// decision 2); every other head names no group.
func groupsWaiting(t *testing.T, heads ...string) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(_ context.Context, _ bool, head string) ([]groupMember, bool, error) {
if slices.Contains(heads, head) {
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
return nil, false, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
}
// everyHeadWaits stubs mesh-delivery's word on groups for a test whose merges each wait for the window as they did
// before novox/hq ADR 0304: every head is a group with a member of another repository still to merge.
func everyHeadWaits(t *testing.T) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
}
// fromHead is a merge from a head branch a test names: merges from one head are members of one delivery group,
// related by it (novox/hq ADR 0304 decision 1), so they share a batch whatever else they share.
func fromHead(m link.SourceMoved, head string) link.SourceMoved {
m.Head = head
return m
}
// lone is a merge that names no head: known to be no delivery group's, it waits for no window unless it is
// related to a batch assembling (ADR 0304 decision 1).
func lone(m link.SourceMoved) link.SourceMoved {
m.Head = ""
return m
}
// hear is a merge heard at a moment.
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
@@ -108,6 +160,7 @@ func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
// later commit, which names both merges — the earlier carried by the later.
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
@@ -152,9 +205,10 @@ func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
// Merges of three repositories in one window are one walk, one commit for each.
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/three")
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)),
hear(t, open, fromHead(repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)), "feat/three"),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
@@ -187,9 +241,10 @@ func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
// closes at most ten minutes after its first merge however busy.
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
hear(t, open, repoMerge("two", "c2", t0.Add(80*time.Second)), t0.Add(80*time.Second))
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(80*time.Second)), "feat/both"), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
@@ -200,12 +255,13 @@ func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
}
busy := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, repoMerge(repo, "c"+string(rune('a'+i)), last), last)
hear(t, busy, fromHead(repoMerge(repo, "c"+string(rune('a'+i)), last), "feat/busy"), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
@@ -223,10 +279,11 @@ func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
// what is grouped, and that the plan is not yet calculated. It says so on the bus as plan-moved, too.
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/553b7191claude")
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
inventory.PlanSaved = func(p inventory.Plan) { said = append(said, p) }
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { said = append(said, p) }
t.Cleanup(func() { inventory.PlanSaved = was })
now := time.Now().UTC()
claude := catalogueMerge("553b7191claude", "app", now.Add(-20*time.Second))
@@ -235,7 +292,7 @@ func TestTheAssemblingBatchIsShown(t *testing.T) {
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, repoMerge("one", "a6bc0931one", now), now)
hear(t, open, fromHead(repoMerge("one", "a6bc0931one", now), dunst.Head), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
@@ -360,6 +417,7 @@ func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
// moves — named at once on a walk done — and joins the next batch when it does not.
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
@@ -391,6 +449,7 @@ func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
// first; the first delivered answers the older ones, and the search stops.
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
@@ -459,6 +518,7 @@ func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
@@ -496,9 +556,10 @@ func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
// store, a merge handed over again is not doubled, and the batch is cut when its window closes.
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both")
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
hear(t, open, repoMerge("two", "c2", t0.Add(30*time.Second)), t0.Add(30*time.Second))
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(30*time.Second)), "feat/both"), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
@@ -506,7 +567,7 @@ func TestABatchSurvivesARestart(t *testing.T) {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, repoMerge("one", "c1", t0), t0.Add(60*time.Second)) // the bus hands it over again
hear(t, again, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
@@ -563,6 +624,7 @@ func TestAWaitingWalkNamesTheMergesItAnswers(t *testing.T) {
// nothing, and the merge joins the next batch.
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
@@ -716,6 +778,7 @@ func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
// still waits for the word, so no catalogue delivery skips its turn.
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
@@ -860,6 +923,7 @@ func sameMerge(a, b inventory.PlanMerge) bool {
// its delivery would skip its turn. It joins the next catalogue batch.
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
+4
View File
@@ -48,6 +48,10 @@ func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time)
if f.atMost.Before(f.closed) {
f.closed = f.atMost
}
if w.Batch.CutAtOnce && !b.Created.IsZero() {
// Its window closed when it was made, which is its latest too (ADR 0304 decision 1).
f.atMost, f.closed = b.Created, b.Created
}
}
if f.behind != "" {
if walk, err := inv.PlanByID(ctx, f.behind); err == nil {
+50 -7
View File
@@ -161,9 +161,7 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
return u, err == nil
})
fmt.Print(planText(plan))
direct := link.Checked{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Commit: p.Commit,
ID: link.NewBuildID(time.Now()), Verdict: "pass", Summary: noModuleTouched,
Gate: &link.CheckLayer{Verdict: "pass", Summary: noModuleTouched}, Plan: &plan}
direct := directVerdict(p, plan)
switch {
case !scope.gated() && !scope.Mesh:
fmt.Printf("%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
@@ -201,6 +199,24 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
return nil
}
// directVerdict is the verdict of a change no build judges — it touches no module, and either the repository is not
// the mesh's or the head holds no merge-check.sh. **It names the trunk commit it holds for** (novox/hq ADR 0323 §3):
// the base branch's head the forge announced with the head, so the merge tool can weigh it against the trunk's head
// and does not refuse it for ever (a verdict naming no trunk is never the trunk's head's). Nothing was built to
// judge: the trunk commit is the one the forge said the change was announced against.
func directVerdict(p link.PullUpdated, plan link.ChangePlan) link.Checked {
trunk := p.BaseSHA
if len(trunk) > 12 {
trunk = trunk[:12]
}
if trunk != "" {
plan.Trunk = trunk
}
return link.Checked{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Commit: p.Commit,
ID: link.NewBuildID(time.Now()), Verdict: "pass", Summary: noModuleTouched,
Gate: &link.CheckLayer{Verdict: "pass", Summary: noModuleTouched}, Plan: &plan, Trunk: trunk}
}
// checkRequestFor is the ask for one pull request's head: the repository as the mesh clones it, the head,
// and what is read beside it.
func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scope checkScope,
@@ -363,6 +379,9 @@ func checkedOf(result link.BuildResult) link.Checked {
case result.Check != nil:
c.Verdict, c.Summary, c.Report = result.Check.Verdict, result.Check.Summary, result.Check.Report
c.Gate, c.RepoCheck = result.Check.Gate, result.Check.RepoCheck
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and the commits of the
// repositories read beside it.
c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took
if c.Gate == nil {
// A build seat from before the layers: its verdict is the gate's.
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
@@ -388,14 +407,20 @@ func checkedOf(result link.BuildResult) link.Checked {
}
if result.Checked != nil {
c.Plan = result.Checked.Plan
if c.Plan != nil && c.Trunk != "" {
// The kept delivery plan records the trunk commit with its other inputs (ADR 0323 §3).
plan := *c.Plan
plan.Trunk = c.Trunk
c.Plan = &plan
}
// A delivery group's composed check (novox/hq ADR 0239): every head it judged, this one first.
if g := result.Checked; g.Group != "" {
c.Group = g.Group
c.Members = append(c.Members, link.CheckedMember{Owner: g.Owner, Repo: g.Repo, Number: g.Number,
Commit: result.Ref})
Commit: result.Ref, Trunk: c.Trunk})
for _, m := range g.Members {
c.Members = append(c.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo, Number: m.Number,
Commit: m.Ref})
Commit: m.Ref, Trunk: memberTrunk(result.Check, m)})
}
}
}
@@ -410,6 +435,20 @@ func checkedOf(result link.BuildResult) link.Checked {
return c
}
// memberTrunk is the trunk commit the build seat merged one of a group's heads into (novox/hq ADR 0323 §6), empty
// when it said none.
func memberTrunk(outcome *link.CheckOutcome, m link.GroupMember) string {
if outcome == nil {
return ""
}
for _, j := range outcome.Members {
if strings.EqualFold(j.Owner, m.Owner) && strings.EqualFold(j.Repo, m.Repo) && strings.HasPrefix(m.Ref, j.Commit) {
return j.Trunk
}
}
return ""
}
func prefixedAll(prefix string, items []string) []string {
out := make([]string, 0, len(items))
for _, i := range items {
@@ -424,8 +463,12 @@ func sayChecked(ctx context.Context, c link.Checked) {
if c.RepoCheck != nil {
repo = strings.ToUpper(c.RepoCheck.Verdict) + " — " + c.RepoCheck.Summary
}
fmt.Printf("%s: %s/%s#%d at %.8s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
c.Commit, orSomewhere(c.On), strings.ToUpper(c.Verdict), c.Summary, repo)
judged := ""
if c.Trunk != "" {
judged = " merged into trunk " + c.Trunk
}
fmt.Printf("%s: %s/%s#%d at %.8s%s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
c.Commit, judged, orSomewhere(c.On), strings.ToUpper(c.Verdict), c.Summary, repo)
if checkEvents == nil {
return
}
+78
View File
@@ -210,3 +210,81 @@ func TestABuildOffTheTrunkIsNeverPublished(t *testing.T) {
t.Errorf("a build seat that said nothing was refused: %v", err)
}
}
// **A verdict names the trunk commit it judged** (novox/hq ADR 0323 §3, issue 341): the controller's `checked`
// carries the head and the trunk commit the build seat merged it into, the commits of the repositories read beside
// it, how long it took, and the kept delivery plan records the trunk commit with its other inputs. A group's
// composed check names, per member, the trunk commit that head was merged into (§6).
func TestAVerdictNamesTheTrunkCommitItJudged(t *testing.T) {
plan := &link.ChangePlan{Repository: "novox/mesh-host", Base: "main", Head: "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
Summary: "builds mesh-host"}
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 66, Base: "main", Plan: plan}
c := checkedOf(link.BuildResult{ID: "b", Ref: "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", Checked: asked,
Check: &link.CheckOutcome{Verdict: "pass", Took: "4m12s", Head: "dfb1415aaaaa", Trunk: "d28ca95bbbbb",
Gate: &link.CheckLayer{Verdict: "pass"}, RepoCheck: &link.CheckLayer{Verdict: "pass"},
Beside: map[string]string{"mesh-catalog": "0123456789ab", "mesh-lab": "ba9876543210"}}})
if c.Trunk != "d28ca95bbbbb" || c.Commit != "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" {
t.Fatalf("the verdict names head %s and trunk %q", c.Commit, c.Trunk)
}
if c.Beside["mesh-catalog"] != "0123456789ab" || c.Beside["mesh-lab"] != "ba9876543210" {
t.Fatalf("the verdict names the repositories read beside it as %v", c.Beside)
}
if c.Took != "4m12s" {
t.Fatalf("the verdict says it took %q", c.Took)
}
if c.Plan == nil || c.Plan.Trunk != "d28ca95bbbbb" || c.Plan.Head != plan.Head {
t.Fatalf("the kept plan records %+v", c.Plan)
}
if plan.Trunk != "" {
t.Fatal("the plan the controller asked with was changed: the verdict's is a copy")
}
group := &link.CheckRequest{Owner: "novox", Repo: "mesh-controller", Number: 7, Base: "main", Group: "feat/x",
Members: []link.GroupMember{{Owner: "novox", Repo: "mesh-host", Number: 8, Ref: "cccccccccccc0000", Base: "main"}}}
c = checkedOf(link.BuildResult{ID: "g", Ref: "aaaaaaaaaaaa0000", Checked: group, Check: &link.CheckOutcome{
Verdict: "pass", Head: "aaaaaaaaaaaa", Trunk: "111111111111", Gate: &link.CheckLayer{Verdict: "pass"},
Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-host", Commit: "cccccccccccc", Trunk: "222222222222"}}}})
if len(c.Members) != 2 || c.Members[0].Trunk != "111111111111" || c.Members[1].Trunk != "222222222222" ||
c.Members[1].Commit != "cccccccccccc0000" {
t.Fatalf("the composed check names its members %+v", c.Members)
}
}
// **The merge made for a check is never registered** (novox/hq ADR 0323 §2): a build of it — a commit on no branch
// of the forge — is refused, naming the trunk. The rule of ADR 0238 §1 stands unchanged; this holds it for the
// merge commit a check makes.
func TestABuildOfACheckMergeIsRefusedNamingTheTrunk(t *testing.T) {
merge := link.BuildResult{ID: "b", Commit: "4b5c6d7e8f90aabbccddeeff0011223344556677", Trunk: "main"}
err := publishable(merge, "")
if !errors.Is(err, errOffTheTrunk) || !strings.Contains(err.Error(), "not on main") {
t.Fatalf("a build of a check's merge was answered %v", err)
}
}
// **A verdict no build judges names its trunk commit too** (novox/hq ADR 0323 §3): a change touching no module of a
// repository with no merge-check.sh (novox/mesh-tools-go) is answered by the controller itself. Naming no trunk, the
// merge tool refused it, asked again, got the same verdict, and refused for ever.
func TestADirectVerdictNamesTheTrunkTheForgeAnnounced(t *testing.T) {
p := link.PullUpdated{Owner: "novox", Repo: "mesh-tools-go", Number: 3, Base: "main", Commit: "1a2b3c4d5e6f7a8b9c0d",
BaseSHA: "7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b"}
c := directVerdict(p, link.ChangePlan{Repository: "novox/mesh-tools-go", Base: "main", Head: p.Commit})
if c.Trunk != "7a8b9c0d1e2f" || c.Plan == nil || c.Plan.Trunk != "7a8b9c0d1e2f" {
t.Fatalf("the direct verdict names trunk %q, its plan %+v", c.Trunk, c.Plan)
}
if c.Verdict != "pass" || c.Commit != p.Commit {
t.Fatalf("the direct verdict is %+v", c)
}
if c := directVerdict(link.PullUpdated{Owner: "novox", Repo: "x", Commit: "abc"}, link.ChangePlan{}); c.Trunk != "" {
t.Fatalf("a forge that said no base head gave the trunk %q", c.Trunk)
}
}
// A head checked again through mesh-delivery (`delivery check --members`, a recheck) reaches the same direct verdict:
// it names the base head the forge last announced, kept on the member, or the merge tool would refuse it for ever.
func TestARecheckedHeadsDirectVerdictNamesItsTrunk(t *testing.T) {
m := orderMember{ID: "novox/mesh-tools-go@1a2b3c4d5e6f", Repository: "novox/mesh-tools-go", Base: "main",
Head: "1a2b3c4d5e6f7a8b9c0d", Number: 3, BaseSHA: "7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b"}
if c := directVerdict(m.pull(), link.ChangePlan{}); c.Trunk != "7a8b9c0d1e2f" {
t.Fatalf("a rechecked head's direct verdict names trunk %q", c.Trunk)
}
}
+215
View File
@@ -0,0 +1,215 @@
package main
import (
"context"
"fmt"
"sort"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
)
// **No send but a person's `push <node>` and a gate's own put-back moves the mesh's own core to a build older
// than the one its machine was last sent** (novox/hq issue 201). Every send made through sendToEach keeps it — a
// plan's, a release's, a healer's, a rotation's, and the bus step too, which is a person's act but composes under
// sendKeeps like the rest. On 2026-10-02 two merges a minute apart made two plans;
// the controller's plan rolled the new controller, which widened its seat's row, and the node-engine's plan then
// sent the control node the controller build before it. The older binary read the row it could not serve and the
// mesh lost its voice for ten minutes — the thing that would have corrected it was the thing that was down. ADR
// 0185 took the outage out of it (a controller behind its row serves what it can); the race is closed here,
// without needing its trigger: whatever made the mesh's record older, the send keeps the build.
//
// **Kept, not refused.** The send composes the controller or the node-engine at the build the machine was last
// sent, as a recorded module is kept (ADR 0245), and carries everything else: no other module, secret, healer or
// walk waits on it, and nothing is retried. The kept move is no move for a gate either: ungatedIn and gatedSend
// leave it out (withoutKept), so a gate judges the other moves of its send. A plan whose own build is the older one finds its gate superseded by
// the build kept there (issue 352) and ends. The mesh's record stays older, and a warning condition says so until
// a send carries the record — a person's `push <node>` accepts the older build — or the record moves forward: a
// newer merge, or `rebuild mesh-controller`.
//
// Older is by the builds' registration order — when each was asked (novox/hq issue 219) — never by the commits'
// dates. So a retry or a rebuild of an old commit, asked anew, counts as newer: it is registered as the module's
// current build by the same order, and the guard does not stop that move. A build that failed its gate is never
// ahead: the machine was sent it and the record was put back before it (gateFailed), and if the put-back's own
// send was refused, the next send completes it rather than keeping the failed build there. A gate's own put-back
// carries its modules back whatever the order (sendScope.puttingBack); the controller's is held to the store's
// schema already (issue 352).
// kindCoreDowngrade is a core module kept at the build its machine was last sent, over an older record.
const kindCoreDowngrade = "core-downgrade-kept"
// coreDowngradeSource is the condition's source: the sends, which raise and clear it.
const coreDowngradeSource = "sends"
// downgradeGuarded is the core components a send may not move backwards: the controller, which reads the
// store and its seat's row, and the node-engine, which reads what the controller declares.
var downgradeGuarded = []string{lease.ComponentController, lease.ComponentEngine}
// askedBuild is what the order needs of one build of a module: when it was asked, and whether it failed its gate.
type askedBuild struct {
asked time.Time
failedGate bool
}
// buildAsked is the newest build of a module from a commit that worked — when it was asked (or recorded, when
// the request time is not known), and whether it failed its gate — and false when the records hold none. A
// variable, so a test reads its own order.
var buildAsked = func(ctx context.Context, inv *inventory.Inventory, module, commit string) (askedBuild, bool, error) {
builds, err := inv.Builds(ctx, module, askedLookBack)
if err != nil {
return askedBuild{}, false, err
}
for _, b := range builds {
if b.Worked() && sameCommit(b.Commit, commit) {
failed, err := inv.GateFailed(ctx, b.ID)
if err != nil {
return askedBuild{}, false, err
}
return askedBuild{asked: b.AskedOrAt(), failedGate: failed}, true, nil
}
}
return askedBuild{}, false, nil
}
// askedLookBack is how many of a module's newest builds are read for when a build was asked.
const askedLookBack = 50
// coreDowngrade is one core module a machine was last sent a build of that is newer than the mesh's record.
type coreDowngrade struct {
module, node, component string
// sent is the build the machine was last sent, record the one the mesh holds.
sent, record string
sentAsked, recordAsked time.Time
}
func (d coreDowngrade) String() string {
return fmt.Sprintf("%s was last sent %s %s, and the mesh holds the older %s (asked %s, before %s): a send there "+
"keeps %s; `push %s` sends the older build, and a newer merge or `rebuild %s` moves the record forward "+
"(novox/hq issue 201)", d.node, d.module, short(d.sent), short(d.record),
d.recordAsked.UTC().Format(time.RFC3339), d.sentAsked.UTC().Format(time.RFC3339), short(d.sent), d.node,
d.module)
}
// coreDowngrades is, for one machine, each guarded core module whose record is older than the build the machine
// was last sent, leaving out what a gate's own put-back puts back and a sent build that failed its gate. A move
// whose either build the records do not hold is not proved older and is left to the gate that judges it. Pure
// but for asked.
func coreDowngrades(node string, sent map[string]string, record map[string]string, scope sendScope,
asked func(module, commit string) (askedBuild, bool, error)) ([]coreDowngrade, error) {
var out []coreDowngrade
for module, was := range sent {
component := coreComponent(module)
now := record[module]
if !guarded(component) || was == "" || now == "" || sameCommit(was, now) {
continue
}
if scope.puttingBack && scope.modules[module] {
continue
}
from, known, err := asked(module, was)
if err != nil {
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(was), err)
}
if !known || from.failedGate {
continue
}
to, known, err := asked(module, now)
if err != nil {
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(now), err)
}
if known && to.asked.Before(from.asked) {
out = append(out, coreDowngrade{module: module, node: node, component: component, sent: was, record: now,
sentAsked: from.asked, recordAsked: to.asked})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].module < out[j].module })
return out, nil
}
func guarded(component string) bool {
for _, c := range downgradeGuarded {
if c == component {
return true
}
}
return false
}
// coreKept is, for a send that keeps (sendKeeps), each guarded core module of a machine to keep at the build it
// was last sent, module → commit, each said and raised as a condition.
func coreKept(ctx context.Context, inv *inventory.Inventory, node string, sent map[string]string,
current map[string]inventory.CurrentBuild) (map[string]string, error) {
record := map[string]string{}
for m, c := range current {
record[m] = c.Commit
}
found, err := coreDowngrades(node, sent, record, scopeOf(ctx), func(module, commit string) (askedBuild, bool, error) {
return buildAsked(ctx, inv, module, commit)
})
if err != nil || len(found) == 0 {
return nil, err
}
out := map[string]string{}
for _, d := range found {
out[d.module] = d.sent
sayCoreDowngrade(ctx, d)
}
return out, nil
}
// coreDowngradeObservation is a kept core build as a condition: `core.<component>.<machine>.<kind>`.
func coreDowngradeObservation(component, node, summary string) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeCore, ID: component + "." + node, Kind: kindCoreDowngrade,
Token: kindCoreDowngrade, Machine: node, Severity: conditions.Warning, Source: coreDowngradeSource,
Summary: summary}
}
func sayCoreDowngrade(ctx context.Context, d coreDowngrade) {
fmt.Printf("KEPT: %s\n", d)
keeper := conditionsFrom
if keeper == nil {
return
}
if _, err := keeper.Observe(ctx, coreDowngradeObservation(d.component, d.node, d.String())); err != nil {
fmt.Printf("the condition for %s kept on %s could not be raised: %v\n", d.module, d.node, err)
}
}
// settleCoreDowngrades clears, after a send to a machine is away, the condition of each guarded core module the
// send carried at the mesh's record — a person's push that accepted the older build, or a send after the record
// moved forward — or did not carry at all. A send that kept the newer build leaves it open.
func settleCoreDowngrades(ctx context.Context, inv *inventory.Inventory, node string, carried map[string]string) {
keeper := conditionsFrom
if keeper == nil || carried == nil {
return
}
var open []string
for _, c := range downgradeGuarded {
if _, found, err := keeper.Get(ctx, coreDowngradeObservation(c, node, "").Key()); err == nil && found {
open = append(open, c)
}
}
if len(open) == 0 {
return
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
fmt.Printf("whether %s still keeps a core build over the mesh's record cannot be read: %v\n", node, err)
return
}
for module, now := range current {
component := coreComponent(module)
if !guarded(component) {
continue
}
if was, has := carried[module]; has && !sameCommit(was, now.Commit) {
continue
}
key := coreDowngradeObservation(component, node, "").Key()
if _, err := keeper.Clear(ctx, key, node+" was sent the "+component+" the mesh holds"); err != nil {
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
}
}
}
+110 -31
View File
@@ -10,6 +10,7 @@ import (
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
@@ -240,7 +241,7 @@ func stopWalkWithout(ctx context.Context, inv *inventory.Inventory, id, by, why
}
return p, said, inv.SavePlan(ctx, &p)
}
target, err := batchToWalkAgain(ctx, inv, again, now)
target, folded, foldMoves, err := batchToWalkAgain(ctx, inv, p.ID, again, now)
if err != nil {
return p, said, fmt.Errorf("%s is not stopped: %w", p.ID, err)
}
@@ -261,8 +262,10 @@ func stopWalkWithout(ctx context.Context, inv *inventory.Inventory, id, by, why
if len(said.Kept) > 0 {
p.Note += "; " + keptWords(said.Kept)
}
// The batch first, so no merge names a plan the store does not hold; all of it, or nothing (decision 7).
if err := inv.SavePlansMoving(ctx, []*inventory.Plan{&target, &p}, moves); err != nil {
// The batch first, so no merge names a plan the store does not hold; all of it, or nothing (decision 7) — the
// batches folded into it too.
if err := inv.SavePlansMoving(ctx, append([]*inventory.Plan{&target, &p}, folded...),
append(moves, foldMoves...)); err != nil {
return p, said, fmt.Errorf("%s is not stopped, and none of its merges moved: %w", p.ID, err)
}
// What the batch holds and names now: written again by the cutter's next look should this fail.
@@ -317,36 +320,42 @@ func stopInBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Pla
return nil
}
// batchToWalkAgain is the batch a stopped walk's merges join: the open batch of their kind, or a new one, not yet
// written. The kind is read from what the merges moved.
func batchToWalkAgain(ctx context.Context, inv *inventory.Inventory, again []inventory.BatchedMerge,
now time.Time) (inventory.Plan, error) {
// batchToWalkAgain is the batch a stopped walk's merges join (novox/hq ADR 0299): a batch of their kind not yet cut
// that they are related to (ADR 0304 decision 1, ADR 0306 decision 3), or a new one, not yet written. Related to
// two or more, the newer are folded into the oldest: the plans and merge moves that fold them, to be written in
// the stop's one act. The kind is read from what the merges moved; the stopped walk itself is no walk a cut folds.
func batchToWalkAgain(ctx context.Context, inv *inventory.Inventory, stopping string, again []inventory.BatchedMerge,
now time.Time) (inventory.Plan, []*inventory.Plan, []inventory.MergeMove, error) {
own := false
for _, n := range movesOf(again) {
if _, on := onTheControllersPath[n]; on {
own = true
}
}
batches, err := inv.Batches(ctx)
entries, read, edges, err := theGraph(ctx, inv)
if err != nil {
return inventory.Plan{}, err
return inventory.Plan{}, nil, nil, err
}
for _, b := range batches {
if b.OwnPath() == own {
if b.Delivery == nil {
b.Delivery = &inventory.PlanDelivery{}
}
if b.Delivery.Batch == nil {
b.Delivery.Batch = &inventory.PlanBatch{Own: own}
}
return b, nil
joins, _, _, err := relatedBatches(ctx, inv, again, own, stopping, entries, read, edges)
if err != nil {
return inventory.Plan{}, nil, nil, err
}
if len(joins) > 0 {
into := joins[0]
if len(joins) == 1 {
return into, nil, nil, nil
}
plans, moves, err := foldInto(ctx, inv, &into, joins[1:])
if err != nil {
return inventory.Plan{}, nil, nil, err
}
return into, plans[1:], moves, nil
}
first := again[0]
return inventory.Plan{ID: fmt.Sprintf("plan-%d", now.UnixNano()), Repository: first.Repository, Branch: first.Branch,
Commit: first.Commit, Merged: first.Merged, Created: now, State: inventory.PlanAssembling,
Tiers: [][]string{}, Modules: map[string]*inventory.PlanModule{},
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own}}}, nil
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own}}}, nil, nil, nil
}
// leaveOut splits a walk's merges by the merges named to leave out, each `<repository>@<commit>` (a commit's
@@ -571,6 +580,9 @@ type orderMember struct {
CloneURL string `json:"clone_url,omitempty"`
// After are the repositories its pull request says it goes after (`after: <repository>`).
After []string `json:"after,omitempty"`
// BaseSHA is the base branch's head the forge last announced the head with (novox/hq ADR 0323): the trunk commit
// a verdict no build judges names when the head is checked again through mesh-delivery.
BaseSHA string `json:"base_sha,omitempty"`
}
// pull is the member as the forge announced it.
@@ -580,7 +592,7 @@ func (m orderMember) pull() link.PullUpdated {
if base == "" {
base = "main"
}
return link.PullUpdated{Owner: owner, Repo: repo, Number: m.Number, Base: base, Commit: m.Head,
return link.PullUpdated{Owner: owner, Repo: repo, Number: m.Number, Base: base, Commit: m.Head, BaseSHA: m.BaseSHA,
CloneURL: m.CloneURL, Paths: m.Paths, PathsTruncated: m.PathsTruncated, Removed: m.Removed,
ModuleDirs: m.ModuleDirs, ModuleDirsSaid: m.ModuleDirsSaid}
}
@@ -940,7 +952,7 @@ func deliveryCommand(ctx context.Context, args []string) error {
if *group == "" && len(members) == 1 {
// One head, checked again as the forge's announcement would have it (a recheck): the controller's
// own path for a pull request, run because the delivery's owner asked.
if err := sayingOnTheBus(ctx, func() error { return (following{open}).PullUpdated(ctx, members[0].pull()) }); err != nil {
if err := sayingOnTheBus(ctx, inv, func() error { return (following{open}).PullUpdated(ctx, members[0].pull()) }); err != nil {
return err
}
return answer(map[string]any{"rechecked": members[0].ID})
@@ -958,7 +970,7 @@ func deliveryCommand(ctx context.Context, args []string) error {
return errors.New("delivery go <plan> [--by <who>] [--why <text>]")
}
var p inventory.Plan
if err := sayingOnTheBus(ctx, func() (err error) {
if err := sayingOnTheBus(ctx, inv, func() (err error) {
p, err = letGo(ctx, inv, positionals[0], *by, strings.TrimSpace(*why))
return err
}); err != nil {
@@ -973,7 +985,7 @@ func deliveryCommand(ctx context.Context, args []string) error {
}
var p inventory.Plan
var said stopAnswer
if err := sayingOnTheBus(ctx, func() (err error) {
if err := sayingOnTheBus(ctx, inv, func() (err error) {
p, said, err = stopWalkWithout(ctx, inv, positionals[0], *by, strings.TrimSpace(*why), splitList(*without),
time.Now().UTC())
return err
@@ -1176,7 +1188,7 @@ func askGroupCheck(ctx context.Context, open *stores, group string, members []or
return "", fmt.Errorf("%s cannot be cloned for the group's check: %w", m.ID, err)
}
request.Check.Members = append(request.Check.Members, link.GroupMember{Owner: mp.Owner, Repo: mp.Repo,
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths})
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths, Base: mp.Base})
}
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
@@ -1215,15 +1227,19 @@ func shelfOf(entries []inventory.Entry) map[string]catalogue.Manifest {
// delivery it walks reads its steps from. Never in the way of the walk: said beside it, and a save that could
// not be said is logged — the delivery's owner, which also asks for the walks it follows, finds it by
// comparison. Records arriving out of order are told apart by the plan's revision.
func sayPlanMoved(ctx context.Context, bus link.Bus, p inventory.Plan) {
go publishPlanMoved(ctx, bus, p)
//
// On the save that starts the walk and the one that ends it, the walk also says what it touches, its bound and,
// at the end, its outcome (novox/hq ADR 0308 decision 1, walk_said.go), read from facts; nil facts say only what
// the walk recorded.
func sayPlanMoved(ctx context.Context, bus link.Bus, facts walkFacts, p inventory.Plan) {
go publishPlanMoved(ctx, bus, facts, p)
}
func publishPlanMoved(ctx context.Context, bus link.Bus, p inventory.Plan) {
func publishPlanMoved(ctx context.Context, bus link.Bus, facts walkFacts, p inventory.Plan) {
// Its phases so far (novox/hq ADR 0282 decision 6): the rest's reports come after the walk ends, and are
// read by whoever asks for the walk (`delivery walks`).
p.Phases = p.WalkPhases(time.Now(), nil)
body, err := json.Marshal(p)
body, err := json.Marshal(planMovedBody{Plan: p, walkSaid: sayWalk(ctx, facts, p)})
if err != nil {
return
}
@@ -1238,7 +1254,7 @@ func publishPlanMoved(ctx context.Context, bus link.Bus, p inventory.Plan) {
// controller's, or a connection of the command's own — a verb runs as a command of its own, and what it kept
// would otherwise be said by nobody until the delivery's owner read the walks back. Without a bus the command
// still runs; what it did is found by comparison.
func sayingOnTheBus(ctx context.Context, f func() error) error {
func sayingOnTheBus(ctx context.Context, facts walkFacts, f func() error) error {
if checkEvents != nil {
return f()
}
@@ -1249,9 +1265,10 @@ func sayingOnTheBus(ctx context.Context, f func() error) error {
return err
}
bus := link.OverNATS{Conn: conn, JS: js}
was := inventory.PlanSaved
checkEvents = bus
inventory.PlanSaved = func(p inventory.Plan) { publishPlanMoved(ctx, bus, p) }
defer func() { checkEvents, inventory.PlanSaved = nil, nil }()
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { publishPlanMoved(ctx, bus, facts, p) }
defer func() { checkEvents, inventory.PlanSaved = nil, was }()
ran = true
return f()
})
@@ -1261,3 +1278,65 @@ func sayingOnTheBus(ctx context.Context, f func() error) error {
}
return err
}
// sayingWhatAVerbKept is PlanSaved for a process that is not the serving controller: each walk the verb keeps is
// said as `plan-moved` before the save returns, with the facts of the inventory that kept it, on the bus the verb
// reaches. It reaches the bus on the first save and never for a verb that keeps no walk. When the bus cannot be
// reached, a save that started or ended a walk says so on the verb's error output: its mark is kept and no later
// save says it, so the router holds what the walk touched until its bound, or not at all for a start. done lets
// the bus go and takes the hook back.
func sayingWhatAVerbKept(ctx context.Context) (said func(*inventory.Inventory, inventory.Plan), done func()) {
var (
mu sync.Mutex
reached bool
bus link.Bus
closing func()
failed error
)
said = func(inv *inventory.Inventory, p inventory.Plan) {
mu.Lock()
defer mu.Unlock()
if !reached {
reached = true
if handActConn != nil {
bus, failed = overNATS(handActConn)
} else if js, err := aBus(); err != nil {
failed = err
} else if bus, failed = overNATS(js.Conn()); failed == nil {
closing = js.Close
} else {
js.Close()
}
}
if bus == nil {
switch p.WalkSaid {
case inventory.WalkStarted:
fmt.Fprintf(os.Stderr, "%s: its start is kept and could NOT be said as plan-moved (%v): the router "+
"does not hold what it touches\n", p.ID, failed)
case inventory.WalkEnded:
fmt.Fprintf(os.Stderr, "%s: its end is kept and could NOT be said as plan-moved (%v): the router holds "+
"what it touched until its bound\n", p.ID, failed)
}
return
}
publishPlanMoved(ctx, bus, inv, p)
}
done = func() {
mu.Lock()
defer mu.Unlock()
inventory.PlanSaved = nil
if closing != nil {
closing()
}
}
return said, done
}
// overNATS is the bus a connection carries, with its JetStream.
func overNATS(conn *nats.Conn) (link.Bus, error) {
js, err := conn.JetStream()
if err != nil {
return nil, err
}
return link.OverNATS{Conn: conn, JS: js}, nil
}
+8 -4
View File
@@ -47,10 +47,14 @@ type stalledLine struct {
// issue 443): a time inside a finished sentence cannot be said again in another zone. A mesh-delivery from before
// says none, and Waiting is said as it reads.
Checks []waitingCheck `json:"checks,omitempty"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
// Held is the state of the delivery of a line of the state `unanswered`, ready or rejected, when one is held
// (novox/hq issues 441, 451): its check answered once, so a recheck is the step. Empty for a head with no
// delivery, and from a mesh-delivery that does not say it yet; the words are never read for it.
Held string `json:"held,omitempty"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
}
// waitingCheck is one merge check a stalled line waits on, as mesh-delivery gives it: its context, its state —
@@ -119,6 +119,13 @@ func TestTheBusMachineIsBehindByItsUserListAlone(t *testing.T) {
if userListBehind("", "") {
t.Fatal("a machine sent no list reads as behind")
}
// A list that could not be composed is behind: not known is never "not behind" (the review of hq issue 353).
if !listBehind("", errors.New("the plan cannot be worked out"), digestOf([]byte(list))) {
t.Fatal("a list nobody could compose reads as current")
}
if listBehind(list, nil, digestOf([]byte(list))) || !listBehind(list, nil, "") {
t.Fatal("listBehind does not read the digest as userListBehind does")
}
}
// The machine holding the bus goes first: its declaration carries the user list the new grants are
+2 -2
View File
@@ -1039,7 +1039,7 @@ func gateFailed(ctx context.Context, open *stores, p *inventory.Plan, module str
}
return
}
sent, err := sendRollout(withScope(ctx, sendScope{modules: map[string]bool{module: true}}), open, g.Machines)
sent, err := sendRollout(withScope(ctx, sendScope{modules: map[string]bool{module: true}, puttingBack: true}), open, g.Machines)
if err != nil {
notBack(fmt.Sprintf("its registered build is back at %s, and sending it to %s was refused: %v — `push %s` "+
"sends it", short(previous.Commit), strings.Join(g.Machines, ", "), err, g.Machines[0]))
@@ -1132,7 +1132,7 @@ func sendRollbacks(ctx context.Context, open *stores, p *inventory.Plan, b *roll
}
inv := open.inventory
sort.Strings(b.machines)
sent, err := sendRollout(withScope(ctx, sendScope{modules: b.modules}), open, b.machines)
sent, err := sendRollout(withScope(ctx, sendScope{modules: b.modules, puttingBack: true}), open, b.machines)
var back []string
for _, r := range b.pending {
if err != nil {
+158
View File
@@ -0,0 +1,158 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group still assembles** (novox/hq ADR 0306 decision 2): a merge whose delivery group has a member
// not yet merged waits for the merge window, whether or not a batch is assembling, so the group's members walk in
// one batch, in their order (ADR 0276 decision 5). Which groups exist and where each member stands is
// mesh-delivery's record (ADR 0239): a group is the pull requests of two repositories or more sharing a head
// branch name, which is its id. The controller asks it, through its `groups` verb, when a merge from a head is
// heard; a member whose pull request is open (proposed, checked, ready or rejected) is still to merge. When
// mesh-delivery does not answer within groupAskWithin, or answers something unreadable, the merge waits: the safe
// error is to wait, and its batch says why.
// groupAskWithin is how long mesh-delivery is given to say a merge's group, while the merge is being heard.
var groupAskWithin = 3 * time.Second
// openPullStates are the states of a delivery whose pull request is open and not merged (mesh-delivery's table):
// a member in one of them is still to merge. Closed unmerged is stopped, final, and waits for nothing.
var openPullStates = map[string]bool{"proposed": true, "checked": true, "ready": true, "rejected": true}
// groupMember is one member of a delivery group as mesh-delivery's `groups` says it: `<owner/repo>@<commit> (state)`.
type groupMember struct {
Repository, State string
}
// deliveryGroupOf is what mesh-delivery says of the group a head names: its members, and whether the head names a
// group at all; an error when it cannot be asked or did not answer. A seam a test replaces.
var deliveryGroupOf = func(ctx context.Context, held bool, head string) ([]groupMember, bool, error) {
if !held {
return nil, false, fmt.Errorf("no holder of %s is on record to say it", catalogue.DeliverySeat)
}
serving := servingBus.Load()
if serving == nil {
return nil, false, errors.New("this controller is not on the bus")
}
asking, cancel := context.WithTimeout(ctx, groupAskWithin)
defer cancel()
raw, err := askDeliveryOwner(asking, serving.Conn(), "groups", map[string]any{})
if err != nil {
if errors.Is(err, link.ErrNothingServes) {
return nil, false, fmt.Errorf("nothing serves %s.groups", catalogue.DeliverySeat)
}
return nil, false, err
}
var lines []struct {
ID string `json:"id"`
Members []string `json:"members"`
}
if err := json.Unmarshal(raw, &lines); err != nil {
return nil, false, fmt.Errorf("%s.groups answered something unreadable: %w", catalogue.DeliverySeat, err)
}
for _, l := range lines {
if l.ID != head {
continue
}
var members []groupMember
for _, s := range l.Members {
members = append(members, memberOf(s))
}
return members, true, nil
}
return nil, false, nil
}
// memberOf reads one member as `groups` says it: `novox/mesh-sdk@d41d255f28ad (held)`.
func memberOf(s string) groupMember {
id, state := strings.TrimSpace(s), ""
if i := strings.LastIndex(id, " ("); i >= 0 && strings.HasSuffix(id, ")") {
id, state = id[:i], id[i+2:len(id)-1]
}
repository, _, _ := strings.Cut(id, "@")
return groupMember{Repository: repository, State: state}
}
// groupWait says whether a merge heard now waits for the window for its delivery group, and why in words; nothing
// when it names no head, its head names no group, or every other member is merged or closed. own is the merge's
// kind (ownPath).
//
// A member of another repository whose merge this controller already keeps in a batch not yet cut of the same
// kind is merged, whatever mesh-delivery says yet (it marks a member merged once its walk opened): the merge joins
// that batch by their head. One kept in a batch of the other kind keeps the merge waiting, whatever mesh-delivery
// says (review of #250): kinds never share a batch, so the merge's batch is a new one, and cut at once it would
// walk before that member's batch is cut and break the group's order (ADR 0276 decision 5). Waiting, the two
// batches are cut in the order their windows close, an own-path one first at one look, as cutBatchesHeld cuts
// every pair of kinds.
//
// Asked before the plans are held: a slow answer holds up no walk.
func groupWait(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, held, own bool) (string, error) {
if m.Head == "" {
return "", nil
}
same, other, err := keptFromHead(ctx, inv, m.Head, own)
if err != nil {
return "", err
}
if len(other) > 0 {
return "group " + m.Head + " has a member waiting in a batch of the other kind: " + strings.Join(other, ", "), nil
}
members, isGroup, err := deliveryGroupOf(ctx, held, m.Head)
if err != nil {
return fmt.Sprintf("%s did not say whether group %s has a member still to merge (%v): it waits for the "+
"window, the safe side", catalogue.DeliverySeat, m.Head, err), nil
}
if !isGroup {
return "", nil
}
var open []string
for _, g := range members {
if strings.EqualFold(g.Repository, m.Owner+"/"+m.Repo) || same[strings.ToLower(g.Repository)] ||
!openPullStates[g.State] {
continue
}
open = append(open, g.Repository+" ("+g.State+")")
}
if len(open) == 0 {
return "", nil
}
return "group " + m.Head + " has a member still to merge: " + strings.Join(open, ", "), nil
}
// keptFromHead are the merges from a head that batches not yet cut hold: the repositories (lower case) of those in
// a batch of the kind own says, and, in words, each of the other kind with its batch.
func keptFromHead(ctx context.Context, inv *inventory.Inventory, head string, own bool) (map[string]bool, []string, error) {
batches, err := inv.Batches(ctx)
if err != nil {
return nil, nil, err
}
same := map[string]bool{}
var other []string
for _, b := range batches {
merges, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return nil, nil, err
}
for _, k := range merges {
if announcedOf(k).Head != head {
continue
}
if b.OwnPath() == own {
same[strings.ToLower(k.Repository)] = true
} else {
other = append(other, fmt.Sprintf("%s@%s in %s", k.Repository, short(k.Commit), b.ID))
}
}
}
return same, other, nil
}
+51
View File
@@ -167,3 +167,54 @@ func TestASetuidSearchAsksOnlyAKnownNodeAndFailsOnARefusal(t *testing.T) {
t.Fatalf("a start: %v, printed %q", err, out.String())
}
}
// The directory judgement's line asks only a known node, one name and nothing else, fails on a refusal, and says it
// changed nothing (novox/hq issue 358).
func TestADirectoryJudgementAsksOnlyAKnownNodeAndSaysItIsReadOnly(t *testing.T) {
t.Setenv(link.CallerVar, "jo through mesh-cli on anchor")
asked := 0
ask := func(answer link.HandOverAnswer) func(node, by string) (link.HandOverAnswer, error) {
return func(node, by string) (link.HandOverAnswer, error) {
asked++
if node != "novox" || by != "jo through mesh-cli on anchor" {
t.Fatalf("asked %q %q", node, by)
}
return answer, nil
}
}
known := func(string) error { return nil }
var out bytes.Buffer
for _, args := range [][]string{nil, {"novox", "extra"}, {"-x"}} {
if err := judgeDirectoriesAsked(args, known, ask(link.HandOverAnswer{Said: "x"}), &out); err == nil || asked != 0 {
t.Fatalf("%v was asked: %v", args, err)
}
}
if err := judgeDirectoriesAsked([]string{"novox"}, func(string) error { return errors.New("no node called novox") },
ask(link.HandOverAnswer{Said: "x"}), &out); err == nil || asked != 0 {
t.Fatalf("an unknown node: %v", err)
}
if err := judgeDirectoriesAsked([]string{"novox"}, known, ask(link.HandOverAnswer{Refused: "no; nothing was judged"}),
&out); err == nil || !strings.Contains(err.Error(), "novox refused") || out.Len() != 0 {
t.Fatalf("a refusal: %v, printed %q", err, out.String())
}
said := "2 directories declared: 1 placed, 1 would be refused\n/var/lib/a: placed\n/var/lib/b: would be refused: …"
if err := judgeDirectoriesAsked([]string{"novox"}, known, ask(link.HandOverAnswer{Said: said}), &out); err != nil ||
out.String() != "novox (read-only: nothing on novox was changed)\n"+said+"\n" {
t.Fatalf("a judgement: %v, printed %q", err, out.String())
}
}
// The judgement is a seat verb an agent may call — it reads, and changes nothing on the node — and runs exactly
// `node judge-directories <node>` (novox/hq issue 358).
func TestTheDirectoryJudgementIsAVerbThatOnlyReads(t *testing.T) {
argv, err := argvFor("judge-directories", map[string]any{"node": "novox"})
if err != nil || strings.Join(argv, " ") != "node judge-directories novox" {
t.Fatalf("the verb runs %v, %v", argv, err)
}
if _, err := argvFor("judge-directories", map[string]any{}); err == nil {
t.Fatal("the verb ran with no node")
}
if !readingVerbs["judge-directories"] {
t.Fatal("the judgement is not counted a reading verb")
}
}
+259
View File
@@ -0,0 +1,259 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
)
// novox/hq issue 201: on 2026-10-02 a plan sent the control node a controller older than the one running there,
// whose seat row the newer one had already widened; the older binary refused the row and the mesh went quiet for
// ten minutes. No send but a person's push and a gate's own put-back moves the controller or the node-engine back
// behind the build its machine was last sent: the send keeps it there, and carries the rest.
// askedIn answers a build from a fixed table: module@commit → when it was asked; failed names those that failed
// their gate.
func askedIn(table map[string]time.Time, failed ...string) func(module, commit string) (askedBuild, bool, error) {
return func(module, commit string) (askedBuild, bool, error) {
at, known := table[module+"@"+commit]
b := askedBuild{asked: at}
for _, f := range failed {
if f == module+"@"+commit {
b.failedGate = true
}
}
return b, known, nil
}
}
func TestACoreModuleIsKeptOverAnOlderRecord(t *testing.T) {
old, now := t0, t0.Add(time.Minute)
table := map[string]time.Time{
"mesh-controller@c1": old, "mesh-controller@c2": now,
"mesh-host@h1": old, "mesh-host@h2": now,
"app@a1": old, "app@a2": now,
}
sent := map[string]string{"mesh-controller": "c2", "mesh-host": "h2", "app": "a2"}
record := map[string]string{"mesh-controller": "c1", "mesh-host": "h1", "app": "a1"}
found, err := coreDowngrades("anchor", sent, record, sendScope{judged: map[string]bool{"anchor": true}}, askedIn(table))
if err != nil {
t.Fatal(err)
}
if len(found) != 2 || found[0].module != "mesh-controller" || found[0].component != lease.ComponentController ||
found[0].sent != "c2" || found[1].module != "mesh-host" || found[1].component != lease.ComponentEngine {
t.Fatalf("the controller and the node-engine over an older record are kept, and nothing else: %+v", found)
}
if said := found[0].String(); !strings.Contains(said, "anchor was last sent mesh-controller c2") ||
!strings.Contains(said, "`push anchor`") || !strings.Contains(said, "`rebuild mesh-controller`") {
t.Fatalf("it says the machine, the build kept and the two ways out: %s", said)
}
// The record forward, the same build, or a build the records do not hold: nothing kept.
for _, c := range [][2]string{{"c1", "c2"}, {"c2", "c2"}, {"c2", "c9"}, {"c9", "c1"}} {
found, err := coreDowngrades("anchor", map[string]string{"mesh-controller": c[0]},
map[string]string{"mesh-controller": c[1]}, sendScope{}, askedIn(table))
if err != nil || len(found) != 0 {
t.Fatalf("sent %s, record %s: no downgrade proved, nothing kept: %+v %v", c[0], c[1], found, err)
}
}
// A sent build that failed its gate is never ahead: the record put back before it is sent.
if found, _ := coreDowngrades("anchor", sent, record, sendScope{}, askedIn(table, "mesh-controller@c2")); len(found) != 1 ||
found[0].module != "mesh-host" {
t.Fatalf("a controller that failed its gate was kept over the build put back: %+v", found)
}
// A gate's own put-back carries its own modules back, and nothing else.
putBack := sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true}
if found, _ := coreDowngrades("anchor", sent, record, putBack, askedIn(table)); len(found) != 1 || found[0].module != "mesh-host" {
t.Fatalf("a put-back of the controller carries it back, and not the node-engine: %+v", found)
}
if found, _ := coreDowngrades("anchor", sent, record, sendScope{modules: map[string]bool{"mesh-controller": true}},
askedIn(table)); len(found) != 2 {
t.Fatalf("a send for the controller that is no put-back carries it back: %+v", found)
}
}
// core201 is a mesh whose control node, anchor, was sent the newer controller c2 while the mesh's record of the
// controller is the build before it, c1, which passed its gate when it rolled out.
func core201(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
old, now := time.Now().Add(-2*time.Hour), time.Now().Add(-time.Minute)
record := func(commit string, asked time.Time) inventory.Build {
manifest, _ := json.Marshal(catalogue.Manifest{Module: "mesh-controller", Version: "1"})
b := inventory.Build{ID: "build-mesh-controller-" + commit, Module: "mesh-controller", Commit: commit,
Repository: "novox/mesh-controller", Manifest: manifest, Asked: asked, At: asked,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:" + commit + "0000000000000000"}}}
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
return b
}
before := record("c1aaaaaaaaaa", old)
if err := inv.RecordGate(ctx, inventory.GateVerdict{Build: before.ID, Module: "mesh-controller", Commit: before.Commit,
Machines: []string{"anchor"}, Verdict: inventory.GatePassed}); err != nil {
t.Fatal(err)
}
record("c2bbbbbbbbbb", now)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"}, inventory.Source{
Repository: "novox/mesh-controller", BuiltFrom: "c2bbbbbbbbbb", Head: "c2bbbbbbbbbb", Asked: now}); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"mesh-controller": "c2bbbbbbbbbb"}); err != nil {
t.Fatal(err)
}
if err := inv.RestoreModule(ctx, before); err != nil {
t.Fatal(err)
}
wasMoves := machineMoves
machineMoves = func(ctx context.Context, open *stores, f moveFacts, node string, all bool) ([]inventory.CarriedMove, error) {
modules, err := open.inventory.Assigned(ctx, node)
if err != nil {
return nil, err
}
sent, known, err := open.inventory.SentBuilds(ctx, node)
if err != nil {
return nil, err
}
return f.moves(node, modules, sent, known, all), nil
}
t.Cleanup(func() { machineMoves = wasMoves })
return open
}
// TestReplay201 replays the race on a store: the control node runs the newer controller and the mesh's record is
// the build before it. A plan's send there — the node-engine's plan of 2026-10-02 — goes, keeps the newer
// controller, and says so as a condition; the gate's own put-back carries it back; a send that carries the record,
// as a person's push does, clears the condition.
func TestReplay201(t *testing.T) {
open := core201(t)
ctx := t.Context()
k, _ := withConditionsInMemory(t)
judged := withScope(ctx, sendScope{judged: map[string]bool{"anchor": true}})
if kept, err := ungatedIn(judged, open, []string{"anchor"}, ""); err != nil || len(kept) != 1 {
t.Fatalf("a plan's send to the machine is made, the rest of it with it: %v %v", kept, err)
}
keeps, err := recordedKept(keepingRecorded(judged), open, "anchor")
if err != nil || keeps["mesh-controller"] != "c2bbbbbbbbbb" {
t.Fatalf("the send keeps the newer controller there: %v %v", keeps, err)
}
shelf := map[string]catalogue.Manifest{"mesh-controller": {Module: "mesh-controller", Version: "old"}}
if _, err := keepRecorded(keepingRecorded(judged), open, "anchor", shelf); err != nil || shelf["mesh-controller"].Version != "1" {
t.Fatalf("the send composes the kept build's manifest: %+v %v", shelf, err)
}
key := coreDowngradeObservation(lease.ComponentController, "anchor", "").Key()
if c, found, err := k.Get(ctx, key); err != nil || !found || !strings.Contains(c.Summary, "c2bbbbbb") {
t.Fatalf("the condition %s is raised, saying the build kept: %+v %v %v", key, c, found, err)
}
putBack := keepingRecorded(withScope(ctx, sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true}))
if keeps, err := recordedKept(putBack, open, "anchor"); err != nil || len(keeps) != 0 {
t.Fatalf("the gate's own put-back of the controller carries it back: %v %v", keeps, err)
}
settleCoreDowngrades(ctx, open.inventory, "anchor", map[string]string{"mesh-controller": "c2bbbbbbbbbb"})
if _, found, _ := k.Get(ctx, key); !found {
t.Fatal("a send that kept the newer controller cleared the condition")
}
settleCoreDowngrades(ctx, open.inventory, "anchor", map[string]string{"mesh-controller": "c1aaaaaaaaaa"})
if _, found, _ := k.Get(ctx, key); found {
t.Fatal("a send carrying the mesh's record — a person's push — leaves the condition open")
}
}
// The review of #273: a gate failed the newer controller, put the record back, and the put-back's own send was
// refused. The next automatic send — a healer's, which judges nothing — completes the put-back: a build that
// failed its gate is never kept.
func TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend(t *testing.T) {
open := core201(t)
ctx := t.Context()
k, _ := withConditionsInMemory(t)
if err := open.inventory.RecordGate(ctx, inventory.GateVerdict{Build: "build-mesh-controller-c2bbbbbbbbbb",
Module: "mesh-controller", Commit: "c2bbbbbbbbbb", Previous: "c1aaaaaaaaaa", Machines: []string{"anchor"},
Verdict: inventory.GateFailed, Rollback: inventory.NotRolledBack}); err != nil {
t.Fatal(err)
}
healer := withScope(ctx, sendScope{})
if kept, err := ungatedIn(healer, open, []string{"anchor"}, ""); err != nil || len(kept) != 1 {
t.Fatalf("the healer's send is made: %v %v", kept, err)
}
if keeps, err := recordedKept(keepingRecorded(healer), open, "anchor"); err != nil || len(keeps) != 0 {
t.Fatalf("the failed controller was kept over the build put back: %v %v", keeps, err)
}
if open, _ := k.Open(ctx); len(open) != 0 {
t.Fatalf("a completed put-back raised a condition: %+v", open)
}
}
// Review 91 of #273: a gated send to a machine whose controller is kept over an older record leaves the kept
// controller out of what its gate judges. Judged as moving back to the record, the gate would read the send's own
// kept controller as another build sent there after it (issue 352) and be superseded, judging nothing.
func TestAGateJudgesTheOtherMovesBesideAKeptCore(t *testing.T) {
open := core201(t)
ctx := t.Context()
withConditionsInMemory(t)
inv := open.inventory
old, now := time.Now().Add(-2*time.Hour), time.Now().Add(-time.Minute)
for _, b := range []struct {
commit string
at time.Time
}{{"a1aaaaaaaaaa", old}, {"a2bbbbbbbbbb", now}} {
manifest, _ := json.Marshal(catalogue.Manifest{Module: "app", Version: "1"})
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-app-" + b.commit, Module: "app", Commit: b.commit,
Repository: "novox/mesh-catalog", Path: "modules/app", Manifest: manifest, Asked: b.at, At: b.at,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:app-" + b.commit}}}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "app", Version: "1"}, inventory.Source{
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/app", BuiltFrom: b.commit, Head: b.commit,
Asked: b.at}); err != nil {
t.Fatal(err)
}
}
if _, err := inv.Assign(ctx, "anchor", "app"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor-2",
map[string]string{"mesh-controller": "c2bbbbbbbbbb", "app": "a1aaaaaaaaaa"}); err != nil {
t.Fatal(err)
}
// The send as it is made: the controller kept at c2, the app moved to the record.
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
return names, open.inventory.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor-3",
map[string]string{"mesh-controller": "c2bbbbbbbbbb", "app": "a2bbbbbbbbbb"})
}
t.Cleanup(func() { sendRollout = wasSend })
moves, sent, _, err := gatedSend(ctx, open, "anchor", nil)
if err != nil || len(sent) != 1 {
t.Fatalf("the gated send is made: %v %v", sent, err)
}
if len(moves) != 1 || moves[0].Module != "app" || moves[0].To != "a2bbbbbbbbbb" {
t.Fatalf("the gate judges the app's move and not the kept controller: %+v", moves)
}
builds, _, err := inv.SentBuilds(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
g := &inventory.PlanGate{Carried: moves}
f := gateFacts{commits: judgedCommits(g, nil), sentBuilds: map[string]map[string]string{"anchor": builds}}
for _, m := range []string{"app", "mesh-controller"} {
if why, superseded := f.supersededOn(m, "anchor"); superseded {
t.Fatalf("the gate is superseded by its own send: %s", why)
}
}
}
+11 -5
View File
@@ -123,11 +123,12 @@ func TestDeliveryStopTakesTheMergesToLeaveOut(t *testing.T) {
// that builds the other repository's module alone.
func TestAStopWithoutAMergeWalksTheOthersAgain(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
@@ -181,12 +182,13 @@ func TestAStopWithoutAMergeWalksTheOthersAgain(t *testing.T) {
// ended. A merge the walk does not answer is refused, and nothing ends.
func TestAStopWithoutAMergeEndsTheMergesThatContainIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
first := catalogueMerge("cat1aaaaaaaa", "app", t0)
later := catalogueMerge("cat2cccccccc", "notes", t0.Add(10*time.Second))
one := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), first.Head)
hear(t, open, first, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
hear(t, open, later, t0.Add(11*time.Second))
@@ -238,11 +240,12 @@ func TestAStopWithoutAMergeEndsTheMergesThatContainIt(t *testing.T) {
// (review of #232): its window closes, and its walk is at the newer commit, answering both.
func TestAStopWalksTheOthersAgainInTheOpenBatch(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one1 := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one1 := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one1, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
@@ -290,11 +293,12 @@ func TestAStopWalksTheOthersAgainInTheOpenBatch(t *testing.T) {
// that walked its merges again is refused a retry, saying why.
func TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one1 := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one1 := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one1, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
@@ -335,11 +339,12 @@ func TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt(t *testing.T) {
// with the others and withholds the stopped merge's modules; with every merge left out, the batch is stopped.
func TestAStopWithoutAMergeLeavesItOutOfABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
_, bs := walks(t, open)
@@ -382,6 +387,7 @@ func TestAStopWithoutAMergeLeavesItOutOfABatch(t *testing.T) {
// else since it was read refuses all of it, and no merge moves, no plan is written.
func TestSavingPlansAndMovingMergesIsOneAct(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
endsItsLine(t)
ctx := t.Context()
hear(t, open, catalogueMerge("cat1aaaaaaaa", "app", t0), t0.Add(time.Second))
+398
View File
@@ -0,0 +1,398 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// The tests of novox/hq issue 525's window skip: a merge waits for the merge window only when a batch assembling
// holds a merge related to it, or its delivery group may have a member still to merge (ADR 0304 decision 1, ADR
// 0306 decisions 2 and 3). Walks stay one at a time.
// A merge from no head that shares nothing, with no walk open, is cut into its walk when it is heard: no window.
func TestALoneMergeSharingNothingIsCutAtOnce(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, lone(repoMerge("one", "c1", t0)), t0)
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 || !ws[0].Open() || ws[0].Commit != "c1" {
t.Fatalf("a lone merge sharing nothing: %d walk(s) %+v, %d batch(es)", len(ws), ws, len(bs))
}
if ws[0].Times == nil || ws[0].Times.WindowClosed == nil || !ws[0].Times.WindowClosed.Equal(t0) {
t.Fatalf("its window closed at %+v; want when it was heard, %s", ws[0].Times, t0)
}
}
// A merge related to the batch assembling joins it and waits for the window, which its coming restarts: by its
// repository, by a module in common (a dependent of what the batch moves), by an edge of the module graph between
// their modules (one that widens nothing), or by its delivery group's head.
func TestARelatedMergeJoinsTheBatchAndWaitsForTheWindow(t *testing.T) {
ctx := t.Context()
for _, c := range []string{"same repository", "a module in common", "an edge of the graph", "the same group head"} {
t.Run(c, func(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c1")
asksWithPaths(t)
inv := open.inventory
first := repoMerge("one", "c1", t0)
second := lone(repoMerge("two", "c2", t0.Add(10*time.Second)))
why := map[string]string{"same repository": "both change novox/one", "a module in common": "both move two",
"an edge of the graph": "two depends on builder (built-by)",
"the same group head": "both are members of group " + first.Head}[c]
switch c {
case "same repository":
second = lone(repoMerge("one", "c2", t0.Add(10*time.Second)))
case "a module in common":
// two is built on one: a merge of one moves two too.
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "two", Version: "1",
Build: &catalogue.Build{On: []catalogue.BuildsOn{{Arg: "BASE", Module: "one"}}}},
inventory.Source{Repository: "novox/two", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
case "an edge of the graph":
// two is built by the build machine: an edge that orders and widens nothing.
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "builder", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-build-machine", Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/builder", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
first = repoMerge("builder", "c1", t0)
case "the same group head":
second = fromHead(repoMerge("two", "c2", t0.Add(10*time.Second)), first.Head)
}
hear(t, open, first, t0)
said := captured(t, func() error {
return (following{open: open}).hearMerge(ctx, second, t0.Add(10*time.Second))
})
if !strings.Contains(said, "("+why+")") {
t.Fatalf("the merge was not said related by %q: %s", why, said)
}
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the related merge: %d walk(s), batches %+v", len(ws), bs)
}
if bs[0].Delivery.Batch.CutAtOnce {
t.Fatal("a batch a related merge joined is cut at once")
}
cutAt(t, open, t0.Add(99*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window the related merge restarted closed: %+v", ws)
}
cutAt(t, open, t0.Add(100*time.Second))
ws, bs = walks(t, open)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
})
}
}
// A merge from a head whose delivery group has a member still to merge waits for the window, even with no batch
// assembling, and says why; its sibling joins the same batch by their head (ADR 0306 decision 2). A merge from a
// head that names no group shares nothing and is cut at once, as one from no head is.
func TestAMergeOfAGroupWaitsEvenAlone(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
groupsWaiting(t, "feat/x")
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/x"), t0)
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 1 || bs[0].Delivery.Batch.CutAtOnce || windowClosed(bs[0], t0) {
t.Fatalf("a group's merge alone: %d walk(s), batches %+v", len(ws), bs)
}
if words := batchWords(bs[0], t0); !strings.HasPrefix(words, "assembling: 90 s left") ||
!strings.Contains(words, "group feat/x has a member still to merge: novox/elsewhere (ready)") {
t.Fatalf("it reads %q", words)
}
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(5*time.Second)), "feat/x"), t0.Add(5*time.Second))
if _, bs := walks(t, open); len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the group's second member did not join the first: %+v", bs)
}
cutAt(t, open, t0.Add(94*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before its window closed: %+v", ws)
}
cutAt(t, open, t0.Add(95*time.Second))
ws, bs = walks(t, open)
if len(ws) != 1 || len(bs) != 0 || ws[0].CommitOf("novox/one") != "c1" || ws[0].CommitOf("novox/two") != "c2" {
t.Fatalf("at its window: %d walk(s) %+v, %d batch(es)", len(ws), ws, len(bs))
}
}
// A merge from a head that names no group, sharing nothing, with no walk open, is cut at once.
func TestAHeadedMergeInNoGroupIsCutAtOnce(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "fix/alone"), t0)
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 || !ws[0].Open() {
t.Fatalf("a headed merge in no group: %d walk(s), %d batch(es)", len(ws), len(bs))
}
}
// When mesh-delivery cannot say a head's group, the merge waits for the window, the safe side, and its batch says
// why.
func TestAGroupNobodyCanSayWaits(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return nil, false, errors.New("nothing serves mesh-delivery.groups")
}
t.Cleanup(func() { deliveryGroupOf = was })
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/x"), t0)
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 1 || bs[0].Delivery.Batch.CutAtOnce {
t.Fatalf("a merge whose group nobody could say: %d walk(s), batches %+v", len(ws), bs)
}
if words := batchWords(bs[0], t0); !strings.Contains(words, "mesh-delivery did not say whether group feat/x has a "+
"member still to merge (nothing serves mesh-delivery.groups): it waits for the window, the safe side") {
t.Fatalf("it reads %q", words)
}
}
// A member counts as still to merge only while its pull request is open, and not once this controller keeps its
// merge: a group whose other members are merged or closed makes nobody wait.
func TestAGroupsMergedOrClosedMembersMakeNobodyWait(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
members := []groupMember{{Repository: "novox/one", State: "ready"}, {Repository: "novox/two", State: "published"},
{Repository: "novox/three", State: "stopped"}}
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) { return members, true, nil }
t.Cleanup(func() { deliveryGroupOf = was })
m := fromHead(repoMerge("one", "c1", t0), "feat/x")
if why, err := groupWait(ctx, open.inventory, m, true, false); err != nil || why != "" {
t.Fatalf("the group's other members are merged or closed, yet: %q %v", why, err)
}
members = append(members, groupMember{Repository: "novox/four", State: "checked"})
if why, _ := groupWait(ctx, open.inventory, m, true, false); !strings.Contains(why, "novox/four (checked)") {
t.Fatalf("an open member: %q", why)
}
// novox/four's merge is kept in a batch already: merged, whatever mesh-delivery says yet.
hear(t, open, fromHead(repoMerge("two", "c4", t0), "feat/x"), t0)
members[3].Repository = "novox/two"
if why, _ := groupWait(ctx, open.inventory, m, true, false); why != "" {
t.Fatalf("a member this controller keeps the merge of still waits: %q", why)
}
if got := memberOf("novox/mesh-sdk@d41d255f28ad (held)"); got != (groupMember{Repository: "novox/mesh-sdk", State: "held"}) {
t.Fatalf("a member reads %+v", got)
}
}
// A merge related to two batches assembling folds the newer into the older, which answers every merge of both
// and names nothing twice; the folded one names the batch that took it over.
func TestAMergeRelatedToTwoBatchesFoldsThem(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/a", "feat/b")
asksWithPaths(t)
ctx := t.Context()
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/a"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(5*time.Second)), "feat/b"), t0.Add(5*time.Second))
_, bs := walks(t, open)
if len(bs) != 2 {
t.Fatalf("two merges sharing nothing: %d batch(es)", len(bs))
}
older, newer := bs[1], bs[0] // newest first
// Of one's repository and from two's head: related to both.
hear(t, open, fromHead(repoMerge("one", "c3", t0.Add(10*time.Second)), "feat/b"), t0.Add(10*time.Second))
_, bs = walks(t, open)
if len(bs) != 1 || bs[0].ID != older.ID || len(bs[0].Delivery.Merges) != 3 {
t.Fatalf("after a merge related to both: batches %+v; want %s answering three merges", bs, older.ID)
}
folded, err := open.inventory.PlanByID(ctx, newer.ID)
if err != nil || folded.State != inventory.PlanSuperseded || folded.Delivery == nil ||
folded.Delivery.TakenOverBy != older.ID {
t.Fatalf("the newer batch reads %s %+v (%v)", folded.State, folded.Delivery, err)
}
if left, _ := open.inventory.MergesOf(ctx, newer.ID); len(left) != 0 {
t.Fatalf("the folded batch still answers %+v", left)
}
cutAt(t, open, t0.Add(100*time.Second))
ws, bs := walks(t, open)
var cut []inventory.Plan
for _, w := range ws {
if w.State != inventory.PlanSuperseded {
cut = append(cut, w)
}
}
if len(cut) != 1 || len(bs) != 0 || cut[0].ID != older.ID || cut[0].CommitOf("novox/one") != "c3" ||
cut[0].CommitOf("novox/two") != "c2" {
t.Fatalf("the folded batch's walk: %+v, %d batch(es)", cut, len(bs))
}
}
// With a walk open, a batch cut at once still waits for it, said queued behind it (walks stay one at a time, ADR
// 0276 decision 4); it is cut once that walk ended, with no window to wait for.
func TestACutAtOnceBatchQueuesBehindAnOpenWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c1")
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("no walk open: %+v", ws)
}
first := ws[0]
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("two", "c2", heard)), heard)
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID ||
!bs[0].Delivery.Batch.CutAtOnce {
t.Fatalf("a lone merge during a walk: %d walk(s), batches %+v", len(ws), bs)
}
if !strings.HasPrefix(batchWords(bs[0], heard), "queued behind "+first.ID) {
t.Fatalf("it reads %q", batchWords(bs[0], heard))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, heard.Add(time.Second))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut once the walk ended: %+v %+v", ws, bs)
}
if ws[0].Times == nil || ws[0].Times.WindowClosed == nil || !ws[0].Times.WindowClosed.Equal(heard) {
t.Fatalf("its window closed at %+v; want when its merge was heard, %s", ws[0].Times, heard)
}
}
// A stopped walk's other merges are walked again in a batch they are related to, and a batch of their own when
// they share nothing with the one assembling (ADR 0299, ADR 0304 decision 1): never in a batch of another
// change, whose window the stop would close.
func TestAStopWalksTheOthersAgainInABatchOfTheirOwnWhenTheyShareNothing(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
walk := ws[0]
hear(t, open, repoMerge("two", "two1cccccccc", t0.Add(3*time.Minute)), t0.Add(3*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 {
t.Fatalf("the merge heard while the walk ran is in %d batches", len(bs))
}
other := bs[0]
_, said, err := stopWalkWithout(ctx, open.inventory, walk.ID, "mesh-delivery for jochen", "its group stopped",
[]string{"novox/mesh-catalog@" + cat.Commit}, t0.Add(3*time.Minute+10*time.Second))
if err != nil {
t.Fatal(err)
}
if said.WalkedAgainBy == "" || said.WalkedAgainBy == other.ID {
t.Fatalf("walked again by %q; want a batch of its own, not %s", said.WalkedAgainBy, other.ID)
}
b, _ := open.inventory.PlanByID(ctx, other.ID)
if b.Delivery.Batch == nil || b.Delivery.Batch.Closed || len(b.Delivery.Merges) != 1 {
t.Fatalf("the other change's batch reads %+v; want it untouched by the stop", b.Delivery)
}
}
// A catalogue batch's cut folds in every walk waiting for its delivery's word (cutBatch's carry), so what that
// walk moves is the batch's too: a lone merge related to the waiting walk joins the batch assembling and waits for
// its window, rather than being cut at once as sharing nothing with what the batch's own merges move.
func TestALoneMergeRelatedToAWalkTheCutFoldsJoinsTheBatch(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c3")
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
hear(t, open, repoMerge("three", "c3", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
hear(t, open, lone(repoMerge("one", "c2", t0.Add(2*time.Minute+5*time.Second))), t0.Add(2*time.Minute+5*time.Second))
_, bs := walks(t, open)
if len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 || bs[0].Delivery.Batch.CutAtOnce {
t.Fatalf("a lone merge related to the waiting walk: batches %+v", bs)
}
}
// A delivery group with a member of each kind (review of #250): a merge on the controller's own path and a
// catalogue merge never share a batch, so the later member cannot join the earlier one's. Kept in a batch of the
// other kind, the earlier member keeps the later one waiting, whatever mesh-delivery says yet — never cut at once
// to walk before it. The two batches are then cut in the order their windows close (cutBatchesHeld: the oldest
// closed batch first, an own-path one first at one look, and the other queued behind a walk that started): the
// earlier member's window closes first, so it walks first, in both orders.
func TestAGroupAcrossBothKindsKeepsItsOrder(t *testing.T) {
for _, ownFirst := range []bool{true, false} {
name := "catalogue member first"
if ownFirst {
name = "controller member first"
}
t.Run(name, func(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
was := deliveryGroupOf
// mesh-delivery still says both members ready: it marks one merged only once its walk opened.
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return []groupMember{{Repository: "novox/mesh-controller", State: "ready"},
{Repository: "novox/mesh-catalog", State: "ready"}}, true, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
own := fromHead(repoMerge("mesh-controller", "k1", t0), "feat/both-kinds")
cat := fromHead(catalogueMerge("cat1aaaaaaaa", "app", t0), "feat/both-kinds")
first, second := cat, own
if ownFirst {
first, second = own, cat
}
second.MergedAt = t0.Add(10 * time.Second).Format(time.RFC3339Nano)
hear(t, open, first, t0)
hear(t, open, second, t0.Add(10*time.Second))
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 2 {
t.Fatalf("both members heard: %d walk(s), %d batch(es); want two batches, nothing cut", len(ws), len(bs))
}
later := bs[0] // newest first
if later.Delivery.Batch.CutAtOnce || !strings.Contains(later.Delivery.Batch.Waits,
"has a member waiting in a batch of the other kind: novox/"+first.Repo+"@") {
t.Fatalf("the later member's batch reads %+v; want it waiting for the earlier member", later.Delivery.Batch)
}
// The earlier member's window closes first: its batch alone is cut.
cutAt(t, open, t0.Add(90*time.Second))
ws, bs = walks(t, open)
if len(ws) != 1 || !ws[0].Open() || ws[0].CommitOf("novox/"+first.Repo) != first.Commit || len(bs) != 1 ||
bs[0].ID != later.ID {
t.Fatalf("at the earlier window: walks %+v, batches %+v; want the earlier member alone cut", ws, bs)
}
earlier := ws[0]
// The later member's window closes while the earlier walk runs: it queues behind it.
cutAt(t, open, t0.Add(100*time.Second))
if _, bs := walks(t, open); len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != earlier.ID {
t.Fatalf("the later member's batch does not queue behind the earlier walk: %+v", bs)
}
earlier.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &earlier); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(101*time.Second))
got, err := open.inventory.PlanByID(ctx, later.ID)
if err != nil || got.Batch() || got.CommitOf("novox/"+second.Repo) != second.Commit {
t.Fatalf("the later member was not walked after the earlier: %s %v", got.State, err)
}
})
}
}
+9
View File
@@ -63,6 +63,15 @@ func run() error {
// 45 §6), so the next controller takes it at once rather than after its age.
defer theLease.release()
// Every walk a verb keeps is said on the bus, whichever verb kept it (novox/hq ADR 0308 decision 1): the save
// that starts or ends a walk keeps its mark, and no later save says it again. The serving controller says its
// own saves.
if args[0] != "serve" {
said, done := sayingWhatAVerbKept(ctx)
inventory.PlanSaved = said
defer done()
}
switch args[0] {
case "build":
return buildCommand(ctx, args[1:])
+61 -2
View File
@@ -127,9 +127,14 @@ func nodeCommand(ctx context.Context, args []string) error {
// and whoever may call a verb includes agents.
return nodeSetuidSearch(ctx, open, args[1:])
case "judge-directories":
// Which of a node's declared directories the walk that follows no link another account could make would
// refuse (novox/hq issue 358). Read-only on the node and here, so a verb may run it too.
return nodeJudgeDirectories(ctx, open, args[1:])
default:
return fmt.Errorf("node has no %q; it has add, list, show, public-domain, account, agent-account, hand-over "+
"and setuid-search", args[0])
return fmt.Errorf("node has no %q; it has add, list, show, public-domain, account, agent-account, hand-over, "+
"setuid-search and judge-directories", args[0])
}
}
@@ -937,3 +942,57 @@ func printJSONTo(w io.Writer, v any) error {
fmt.Fprintln(w, string(body))
return nil
}
const judgeDirectoriesUsage = "node judge-directories <node> — read-only: which of the directories <node> is " +
"declared the node-engine's walk that follows no link another account could make would refuse, one line each " +
"(novox/hq issue 358). Nothing is made, chmodded or chowned"
// nodeJudgeDirectories asks the node's engine to judge its declared directories, signed with the mesh's key as a
// hand-over is. Read-only: the engine changes nothing and keeps nothing.
func nodeJudgeDirectories(ctx context.Context, open *stores, args []string) error {
known := func(node string) error {
_, err := open.inventory.NodeByName(ctx, node)
return err
}
ask := func(node, by string) (link.HandOverAnswer, error) {
ident, err := open.Identity(ctx)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("the mesh's signing key cannot be read, so nothing was asked of %s: %w",
node, err)
}
address, err := broker.BusAddress()
if err != nil {
return link.HandOverAnswer{}, err
}
js, err := broker.Dial(address)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("cannot reach the bus, so nothing was asked of %s: %w", node, err)
}
defer js.Close()
return link.AskJudgeDirectories(ctx, js.Conn(), ident, node, by, link.HandOverWithin)
}
return judgeDirectoriesAsked(args, known, ask, os.Stdout)
}
// judgeDirectoriesAsked is the line with its two acts given: whether the mesh knows the node, and the ask. The
// engine's refusal is this command's failure.
func judgeDirectoriesAsked(args []string, known func(node string) error,
ask func(node, by string) (link.HandOverAnswer, error), out io.Writer) error {
if len(args) != 1 || args[0] == "" || strings.HasPrefix(args[0], "-") {
return errors.New(judgeDirectoriesUsage)
}
node := args[0]
if err := known(node); err != nil {
return fmt.Errorf("nothing was asked: %w", err)
}
answer, err := ask(node, handOverBy())
if err != nil {
return err
}
if answer.Refused != "" {
return fmt.Errorf("%s refused: %s", node, answer.Refused)
}
fmt.Fprintf(out, "%s (read-only: nothing on %s was changed)\n", node, node)
fmt.Fprintln(out, answer.Said)
return nil
}
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"bytes"
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq issue 237: every line of an act's answer comes from one resolution of the node taken after
// its write, so the answer agrees with `plan` for the node right after.
func aProviderAndItsConsumer(t *testing.T, open *stores) {
t.Helper()
register(t, open, catalogue.Manifest{Module: "db", Version: "1",
Provides: []catalogue.Offer{{Name: "rows", Scope: catalogue.ScopeNode}}})
register(t, open, catalogue.Manifest{Module: "app", Version: "1", Requires: []string{"rows"}})
}
// twoRivalsAndANote is a node that does not resolve — two modules claim one node seat there — and a
// module beside them that changes nothing about it.
func twoRivalsAndANote(t *testing.T, open *stores) {
t.Helper()
one, two := rivals()
register(t, open, one)
register(t, open, two)
register(t, open, catalogue.Manifest{Module: "notes", Version: "1"})
}
func TestAnUnassignThatLeavesItsNodeUnresolvedSaysSoOnceInItsAnswer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
twoRivalsAndANote(t, open)
// Straight into the store, as a node that came to this before the act.
for _, m := range []string{"rival-one", "rival-two", "notes"} {
if _, err := open.inventory.Assign(ctx, "laptop", m); err != nil {
t.Fatal(err)
}
}
said := answered(t, func() error { return assignCommand(ctx, "unassign", []string{"laptop", "notes"}) })
if _, _, err := planFor(ctx, open, "laptop"); err == nil || !unresolvable(err) {
t.Fatalf("laptop resolves with two claimants of one seat: %v", err)
}
if n := strings.Count(said, "laptop does not resolve now"); n != 1 {
t.Errorf("the answer says laptop does not resolve %d times, want once:\n%s", n, said)
}
if strings.Contains(said, "is left out of the rest of the mesh") {
t.Errorf("the answer carries a line from a read that is not its resolution:\n%s", said)
}
}
func TestAnUnassignThatLeavesItsNodeResolvedSaysNothingOfIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
aProviderAndItsConsumer(t, open)
if _, err := assign(ctx, open, "laptop", "db", "app"); err != nil {
t.Fatal(err)
}
said, err := unassign(ctx, open, "laptop", "app")
if err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
if strings.Contains(said, "does not resolve") || strings.Contains(said, "could not be read") {
t.Errorf("a node that resolves is said not to:\n%s", said)
}
}
func TestAnAssignThatLeavesItsNodeUnresolvedSaysOnlyThePlansRefusal(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
twoRivalsAndANote(t, open)
if _, err := assign(ctx, open, "laptop", "rival-one"); err != nil {
t.Fatal(err)
}
// Kept and not resolvable: a second claimant of the seat, which assignment keeps.
said, err := assign(ctx, open, "laptop", "rival-two")
if err == nil {
t.Fatalf("two claimants of one node seat resolved:\n%s", said)
}
_, _, planErr := planFor(ctx, open, "laptop")
if planErr == nil || planErr.Error() != err.Error() {
t.Errorf("the act's refusal is not the plan's right after:\nact: %v\nplan: %v", err, planErr)
}
if strings.Contains(said, "now has") {
t.Errorf("the answer says a seat is now held on a node that does not resolve:\n%s", said)
}
}
// The other readers of the whole mesh still say a machine they leave out, on standard error (novox/hq
// issue 188); an act sends it elsewhere.
func TestAMachineLeftOutIsSaidWhereTheContextSays(t *testing.T) {
var buf bytes.Buffer
sayLeftOut(leftOutSaidBy(context.Background(), &buf), "laptop", errors.New("why"))
if buf.String() != "laptop is left out of the rest of the mesh: it does not resolve: why\n" {
t.Errorf("said %q", buf.String())
}
got := answered(t, func() error {
sayLeftOut(context.Background(), "laptop", errors.New("why"))
return nil
})
if !strings.Contains(got, "laptop is left out") {
t.Errorf("without an act's writer the line did not reach standard error: %q", got)
}
}
@@ -0,0 +1,120 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// An own secret issued outside the mesh that nobody gave is never made (novox/hq issue 401, ADR 0228). Seen on
// 2026-10-10: the send made a random value for the mounts module's smb-credentials and placed it, and the module read
// it as a credential given and wrong. The send now makes none, places none, and places a value only once a person
// gave one — and a value the mesh made for it before is not placed either.
// anAssistant is a module with a secret the mesh makes and one only an outside party issues.
func anAssistant() catalogue.Manifest {
return catalogue.Manifest{Module: "assistant", Version: "1",
OwnSecrets: catalogue.OwnSecrets{
"password": {Path: "/var/lib/mesh/assistant/password", Taken: catalogue.TakenAtStart},
"vendor-key": {Path: "/var/lib/mesh/assistant/vendor-key", Taken: catalogue.TakenAtStart, IssuedBy: catalogue.IssuedOutside},
},
Resources: []map[string]any{
{"id": "state", "type": "directory", "path": "/var/lib/mesh/assistant", "mode": "0700"},
}}
}
// sentTo composes what a push sends to a machine, on the send path.
func sentTo(t *testing.T, open *stores, node string) []map[string]any {
t.Helper()
plan, settings, err := planFor(t.Context(), open, node)
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(t.Context(), open, node, plan, settings)
if err != nil {
t.Fatalf("the send to %s did not compose: %v", node, err)
}
return declared.Resources
}
func placedAt(resources []map[string]any, path string) map[string]any {
for _, r := range resources {
if r["path"] == path && r["sealed"] != nil {
return r
}
}
return nil
}
func TestTheSendMakesNoValueForAnOutsideSecretNobodyGave(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, anAssistant())
if _, err := assign(ctx, open, "laptop", "assistant"); err != nil {
t.Fatal(err)
}
sent := sentTo(t, open, "laptop")
if placedAt(sent, "/var/lib/mesh/assistant/password") == nil {
t.Fatal("the secret the mesh makes was not placed")
}
if got := placedAt(sent, "/var/lib/mesh/assistant/vendor-key"); got != nil {
t.Fatalf("a secret nobody gave was placed: %v", got)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "assistant", "vendor-key"); err != nil || held {
t.Fatalf("the send made a value for a secret issued outside the mesh (held %v, %v)", held, err)
}
// A value the mesh made before this rule is not one a person gave: it is not placed.
if _, err := open.inventory.SecretForModule(ctx, "laptop", "assistant", "vendor-key"); err != nil {
t.Fatal(err)
}
if got := placedAt(sentTo(t, open, "laptop"), "/var/lib/mesh/assistant/vendor-key"); got != nil {
t.Fatalf("a value the mesh made for a secret issued outside was placed: %v", got)
}
// Given, it is placed.
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "assistant", "vendor-key", "the vendor's key"); err != nil {
t.Fatal(err)
}
given, ok, err := open.inventory.GivenOwnSecret(ctx, "laptop", "assistant", "vendor-key")
if err != nil || !ok || !given.Current {
t.Fatalf("the value given is not read as given: %+v %v %v", given, ok, err)
}
placed := placedAt(sentTo(t, open, "laptop"), "/var/lib/mesh/assistant/vendor-key")
if placed == nil || placed["sealed"] != given.Sealed {
t.Fatalf("the value given is placed as %v", placed)
}
}
// The read paths answer as the send does: a question about the machine composes without the secret nobody gave, and
// asking writes nothing.
func TestAQuestionComposesWithoutAnOutsideSecretNobodyGave(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, anAssistant())
if _, err := assign(ctx, open, "laptop", "assistant"); err != nil {
t.Fatal(err)
}
pushedBy(t, open, "laptop") // the mesh's own secret is made by the send
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
for _, choosing := range []Choosing{Reading, Foreseeing} {
got, err := declarationWith(ctx, open, "laptop", plan, settings, gens, choosing)
if err != nil {
t.Fatalf("%v: %v", choosing, err)
}
if placedAt(got.Resources, "/var/lib/mesh/assistant/vendor-key") != nil || len(got.foreseen) != 0 {
t.Fatalf("%v: a secret nobody gave is composed (foreseen %v)", choosing, got.foreseen)
}
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "assistant", "vendor-key"); err != nil || held {
t.Fatalf("a question made a value (held %v, %v)", held, err)
}
}
+41 -5
View File
@@ -81,6 +81,13 @@ var plainWordings = map[string]func(conditions.Observation) words{
"It usually catches up by itself.", m, m),
Resolved: m + " applied the change"}
}),
kindApplyFailing: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: m + " is not applying its update",
Explanation: fmt.Sprintf("%s has refused or only partly applied what the mesh sent it, again and again. "+
"The details say what fails.", conditions.Capital(m)),
Resolved: m + " applies what it is sent again"}
}),
"tools-silent": worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "Tools on " + m + " are not answering",
@@ -175,6 +182,20 @@ var plainWordings = map[string]func(conditions.Observation) words{
"register it, so it was not made.", module, m),
Resolved: "Resolved: " + module + " on " + m + " is settled"}
}),
// The controller or the node-engine kept at the build a machine was last sent, over an older record (novox/hq
// issue 201).
kindCoreDowngrade: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
component := idPart(o, 0)
if o.Scope != conditions.ScopeCore || !guarded(component) {
component = "core component"
}
return words{Headline: m + " keeps a newer " + component + " than the mesh holds",
Needs: fmt.Sprintf("push %s to take the older build, or a newer merge or rebuild moves the record forward.", m),
Explanation: fmt.Sprintf("The mesh's record of its own %s is older than the build %s was last sent, so "+
"the mesh's sends keep the newer one there rather than put it back.", component, m),
Resolved: "Resolved: " + m + " runs the " + component + " the mesh holds"}
}),
kindBindingKept: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "A module's data source is held on " + m,
@@ -885,6 +906,11 @@ func stalledWords(l stalledLine, o conditions.Observation) (headline, explanatio
// within its bound (novox/hq issue 438): mesh-delivery holds no delivery of it, so there is nothing to stop, release
// or close, and no action is offered. The operator's acts are a new commit, which asks the check again, or a look
// at the build queue, where a check that never ran may still wait.
//
// A head whose delivery is held in ready or rejected is said the same way (novox/hq issue 441), and mesh-delivery
// names that state as data (Held). Its check answered once, so the build queue is the wrong place to look: the step
// is a recheck of the delivery, named by its id, which asks the check again (novox/hq issue 451). It is advised, not
// offered as an action, since the controller is not granted the recheck.
func unansweredWords(l stalledLine, o conditions.Observation) (headline, explanation, resolved, needs string,
actions []conditions.Action) {
repository, _, _ := strings.Cut(l.ID, "@")
@@ -899,13 +925,23 @@ func unansweredWords(l stalledLine, o conditions.Observation) (headline, explana
if d, err := time.ParseDuration(l.Bound); err == nil {
limit = ", past its limit of " + humanDuration(d)
}
which := "Its merge check"
check := "merge check"
if said := uncheckedWaitWords(l); len(said) > 0 {
which = "Its merge check (" + strings.Join(said, ", ") + ")"
check = "merge check (" + strings.Join(said, ", ") + ")"
}
if o.Resolver == conditions.ResolverOperator {
needs = "push a new commit to its branch, which asks the check again, or see whether its check still waits " +
"in the build queue: mesh-controller.queue."
which := "Its " + check
switch l.Held {
case "ready", "rejected":
which = "Its delivery is " + l.Held + ", but on the forge its " + check
if o.Resolver == conditions.ResolverOperator {
needs = "recheck its delivery, which asks its check again — mesh-delivery.recheck with id " + l.ID +
" and why — or push a new commit to its branch, " + FromMeshMCPServer
}
default:
if o.Resolver == conditions.ResolverOperator {
needs = "push a new commit to its branch, which asks the check again, or see whether its check still " +
"waits in the build queue: mesh-controller.queue."
}
}
return fmt.Sprintf("%s's merge check has not answered %s", pull, long),
fmt.Sprintf("%s is open on a branch that requires the merge check. %s was asked and has not answered %s%s. "+
+46
View File
@@ -452,3 +452,49 @@ func TestAnUnansweredMergeChecksTimeIsSaidInLocalTime(t *testing.T) {
t.Fatalf("a pending check without its time reads %q", o.Explanation)
}
}
// **An unanswered merge check on a ready or rejected delivery is told to recheck it** (novox/hq issue 451):
// mesh-delivery says such a line too (issue 441) and names the delivery's state as data, `held`. That check already
// answered once, so the build queue is the wrong place to look; a recheck of the delivery, named by its id, asks the
// check again. A line with no delivery held keeps the advice of issue 438, and so does a line from a mesh-delivery
// that does not say `held` yet: the controller never guesses the state from the words.
func TestAnUnansweredMergeCheckOnAReadyOrRejectedDeliveryOffersARecheck(t *testing.T) {
line := func(held string) stalledLine {
var l stalledLine
raw := `{"id":"novox/mesh-controller@c11222026a3b","number":212,"state":"unanswered",` +
`"waiting":["mesh/merge-gate pending since 2026-10-11T00:11:39Z"],"for":"1h1m0s","bound":"1h0m0s",` +
`"h2":"none: its delivery is ` + held + `, a state the operator moves — the operator's",` +
`"says":"novox/mesh-controller#212 is open on main, which requires the merge check, and its delivery is ` +
held + ` here, but on the forge its check has not answered: mesh/merge-gate pending since ` +
`2026-10-11T00:11:39Z. The pull request waits on that status and nothing asks it again. A recheck of the ` +
`delivery asks its check again, as does a new commit on its branch"`
if held != "" {
raw += `,"held":"` + held + `"`
}
if err := json.Unmarshal([]byte(raw+`}`), &l); err != nil {
t.Fatal(err)
}
return l
}
for _, held := range []string{"ready", "rejected"} {
o := stalledObservations([]stalledLine{line(held)})[0]
t.Logf("%s: %s\nneeds: %s", held, o.Explanation, o.Needs)
if !strings.Contains(o.Needs, "mesh-delivery.recheck") || !strings.Contains(o.Needs, "novox/mesh-controller@c11222026a3b") {
t.Fatalf("a %s delivery's unanswered check is not offered its recheck: %q", held, o.Needs)
}
if strings.Contains(o.Needs, "build queue") || strings.Contains(o.Needs, "mesh-controller.queue") {
t.Fatalf("a %s delivery's unanswered check, which answered once, is sent to the build queue: %q", held, o.Needs)
}
if !strings.Contains(o.Explanation, "Its delivery is "+held) {
t.Fatalf("a %s delivery's explanation does not say its state: %q", held, o.Explanation)
}
if len(o.Actions) != 0 {
t.Fatalf("it offers an action the controller is not granted: %+v", o.Actions)
}
}
// No delivery held, or a mesh-delivery that does not say: the advice of issue 438, and no recheck.
o := stalledObservations([]stalledLine{line("")})[0]
if !strings.Contains(o.Needs, "in the build queue: mesh-controller.queue") || strings.Contains(o.Needs, "recheck") {
t.Fatalf("a line with no delivery held says to %q", o.Needs)
}
}
+94 -22
View File
@@ -292,7 +292,7 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
if err != nil {
// Said, not skipped: a machine dropped here offers nothing and holds nothing as far
// as every other machine's plan can tell (novox/hq issue 188).
fmt.Fprintf(os.Stderr, "%s is left out of the rest of the mesh: it does not resolve: %v\n", o.node.Name, err)
sayLeftOut(ctx, o.node.Name, err)
continue
}
firstHeld = append(firstHeld, got.Claims...)
@@ -337,7 +337,7 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
// resolved without its offers, and one that consumes them cannot resolve here by
// design — that is not the machine being dropped, it is the view being partial.
if exclude == "" {
fmt.Fprintf(os.Stderr, "%s is left out of the rest of the mesh: it does not resolve: %v\n", o.node.Name, err)
sayLeftOut(ctx, o.node.Name, err)
}
continue
}
@@ -347,6 +347,35 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
return world, nil
}
// leftOutKey carries where theRestOfTheMesh says a machine it leaves out, for an act that answers with
// its own resolution instead (novox/hq issue 237).
type leftOutKey struct{}
// leftOutSaidBy is ctx with theRestOfTheMesh's "left out" lines written to w rather than to standard
// error.
//
// **An act's answer is one resolution taken after its write** (novox/hq issue 237, decided 2026-10-11).
// `assign` read the whole mesh once before its write — to record the seats held by derivation — and that
// read said, on standard error, that the node did not resolve for lack of the seat the act was about to
// give it. The controller seat's verb answers with both streams, so the stale line came last, after the
// act's own "now has it held", and named as its remedy the assignment just made. An act gives io.Discard
// here: what the node resolves to after the write is said by its own planFor, and every other machine
// that does not resolve by blockedElsewhere, both read after the write.
func leftOutSaidBy(ctx context.Context, w io.Writer) context.Context {
return context.WithValue(ctx, leftOutKey{}, w)
}
// sayLeftOut says that a machine is left out of the rest of the mesh, where ctx says to: standard
// error unless an act said otherwise. Said, not skipped: a machine dropped there offers nothing and
// holds nothing as far as every other machine's plan can tell (novox/hq issue 188).
func sayLeftOut(ctx context.Context, node string, err error) {
w, ok := ctx.Value(leftOutKey{}).(io.Writer)
if !ok {
w = os.Stderr
}
fmt.Fprintf(w, "%s is left out of the rest of the mesh: it does not resolve: %v\n", node, err)
}
// declarationFor is everything a node would be sent.
//
// One place, because there were three and one of them was written before credentials existed and
@@ -561,6 +590,13 @@ func busCredentialIssued(ctx context.Context, inv *inventory.Inventory, node, mo
return nil
}
// givenUnderAnOldKey is the refusal for a value a person gave — a family's member, or an own secret issued outside
// the mesh — sealed to a key the machine no longer holds: the mesh cannot make another, so it is given again.
func givenUnderAnOldKey(module, node, name string) error {
return fmt.Errorf("%s on %s holds %q, which was given to the mesh rather than made by it, and %s has "+
"since generated a new sealing key. The mesh cannot make another; give it again", module, node, name, node)
}
// renderingFor is everything a node's declaration is composed with, and the node's record.
func renderingFor(ctx context.Context, open *stores, node string,
plan catalogue.Resolution, settings catalogue.SettingsBy,
@@ -679,10 +715,7 @@ func renderingFor(ctx context.Context, open *stores, node string,
}
if !g.Current {
if choosing == Allocating {
return catalogue.Rendering{}, inventory.Node{}, fmt.Errorf(
"%s on %s holds %q, which was given to the mesh rather than made by it, and %s has "+
"since generated a new sealing key. The mesh cannot make another; give it again",
m.Module, node, g.Name, node)
return catalogue.Rendering{}, inventory.Node{}, givenUnderAnOldKey(m.Module, node, g.Name)
}
continue
}
@@ -692,7 +725,31 @@ func renderingFor(ctx context.Context, open *stores, node string,
needed[m.Module][g.Name] = g.Sealed
}
}
for name := range m.OwnSecrets.Plain() {
for name, own := range m.OwnSecrets.Plain() {
// **An own secret issued outside the mesh is never made** (novox/hq issue 401, ADR 0228; ADR 0283 for
// a family): only a value a person gave is placed. One not given is nothing — no file, and an empty
// value where a file of the module names it — so the module can tell it was not given, where a value
// the mesh made looked given and wrong. A value the mesh made for it before is not one either.
if own.IssuedBy == catalogue.IssuedOutside {
g, given, err := inv.GivenOwnSecret(ctx, node, m.Module, name)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
if !given {
continue
}
if !g.Current {
if choosing == Allocating {
return catalogue.Rendering{}, inventory.Node{}, givenUnderAnOldKey(m.Module, node, name)
}
continue
}
if needed[m.Module] == nil {
needed[m.Module] = map[string]string{}
}
needed[m.Module][name] = g.Sealed
continue
}
// Minted on the send path and only read on every other. Making one is an insert, and
// a question that writes is a question that can block against the machine it is about.
var sealed string
@@ -1751,28 +1808,43 @@ func (l *unheldLogger) log(node string, unheld []catalogue.Unheld) {
// which includes every one of a module just assigned — and each now met that was not. Nothing about
// any other node, and nothing that was already true before the act.
func unheldChange(shelf map[string]catalogue.Manifest, node string, before, after []string) []string {
judge := func(names []string) map[string]catalogue.Unheld {
var set []catalogue.Manifest
for _, n := range names {
if m, known := shelf[n]; known {
set = append(set, m)
}
return unheldChangeTo(shelf, node, before, after, unheldOver(shelf, node, after))
}
// unheldOver is the node's unmet seat dependencies judged over a set of module names alone.
func unheldOver(shelf map[string]catalogue.Manifest, node string, names []string) []catalogue.Unheld {
var set []catalogue.Manifest
for _, n := range names {
if m, known := shelf[n]; known {
set = append(set, m)
}
out := map[string]catalogue.Unheld{}
for _, u := range catalogue.UnheldDependencies(shelf, node, set, nil) {
out[u.Module+" "+u.Seat] = u
}
return out
}
was, now := judge(before), judge(after)
return catalogue.UnheldDependencies(shelf, node, set, nil)
}
// unheldChangeTo is unheldChange with the node's dependencies as they now stand given: `now` is what
// the act's one resolution after its write says is unmet, and `after` the modules that resolution has
// on the node (novox/hq issue 237). A line about how things stand after the act is never judged from
// anything else, so it cannot disagree with `plan` for the node.
func unheldChangeTo(shelf map[string]catalogue.Manifest, node string, before, after []string,
now []catalogue.Unheld) []string {
key := func(u catalogue.Unheld) string { return u.Module + " " + u.Seat }
was := map[string]catalogue.Unheld{}
for _, u := range unheldOver(shelf, node, before) {
was[key(u)] = u
}
is := map[string]catalogue.Unheld{}
for _, u := range now {
is[key(u)] = u
}
var lines []string
for _, k := range sortedNames(now) {
for _, k := range sortedNames(is) {
if _, already := was[k]; !already {
lines = append(lines, "but "+now[k].String())
lines = append(lines, "but "+is[k].String())
}
}
for _, k := range sortedNames(was) {
if _, still := now[k]; still {
if _, still := is[k]; still {
continue
}
u := was[k]
+26
View File
@@ -0,0 +1,26 @@
package main
import (
"fmt"
"regexp"
)
// planID is every id the mesh gives a plan: a merge's walk or batch (plan-<nanoseconds>) or a release
// (release-<nanoseconds>).
var planID = regexp.MustCompile(`^(plan|release)-[0-9]+$`)
// planIDArguments are the arguments of plans that name a plan. wait is among them for when it is declared
// (novox/hq issue 229); an argument not declared is refused before this is asked.
var planIDArguments = []string{"id", "wait", "stop", "close", "retry", "go"}
// plainPlanIDs refuses a plans call naming a plan by anything but a plan's id (novox/hq issue 561): checked
// once, where the call becomes a command line, so nothing else a caller sent reaches a command or the store.
func plainPlanIDs(given map[string]string) error {
for _, arg := range planIDArguments {
if id, named := given[arg]; named && !planID.MatchString(id) {
return fmt.Errorf("plans %s: %q is not a plan's id (plan-<digits> or release-<digits>, as `plans` "+
"lists them); nothing was done", arg, id)
}
}
return nil
}
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"strings"
"testing"
)
// **A plan's id is checked once, where a call becomes a command line** (novox/hq issue 561): every argument of
// plans that names a plan is refused unless it is an id the mesh gives a plan, so a quote never reaches a
// command, nor the store behind it.
func TestAPlanIdThatIsNoPlansIdIsRefusedBeforeACommandStarts(t *testing.T) {
for _, arg := range []string{"id", "stop", "close", "retry", "go"} {
for _, bad := range []string{"x' or '1'='1", "plan-1'", "plan-", "plan-1 2", "batch-1", "PLAN-1"} {
args := map[string]any{arg: bad}
if arg == "stop" || arg == "close" || arg == "go" {
args["why"] = "a test"
}
argv, err := argvFor("plans", args)
if err == nil {
t.Errorf("plans %s %q was composed: %v", arg, bad, argv)
continue
}
if !strings.Contains(err.Error(), "is not a plan's id") {
t.Errorf("plans %s %q was refused for another reason: %v", arg, bad, err)
}
}
for _, good := range []string{"plan-1791813626034291775", "release-1791811938267404408"} {
args := map[string]any{arg: good}
if arg == "stop" || arg == "close" || arg == "go" {
args["why"] = "a test"
}
if _, err := argvFor("plans", args); err != nil {
t.Errorf("plans %s %q was refused: %v", arg, good, err)
}
}
}
}
+12
View File
@@ -251,6 +251,16 @@ func resumed(p *inventory.Plan, why string) {
if p.State == inventory.PlanFailed && len(failedIn(*p)) == 0 {
p.State = inventory.PlanBuilding
p.Note = why
goesOnAgain(p)
}
}
// goesOnAgain clears a stop from a walk that walks again: a stop said why the walk ended, and a walk resumed has
// not ended. Kept, it would end a retried walk that fails again as stopped, not failed (novox/hq ADR 0308
// decision 1), and keep the search for the merge that brought the failure from running (searchAfterFailures).
func goesOnAgain(p *inventory.Plan) {
if p.Delivery != nil {
p.Delivery.Stopped, p.Delivery.StoppedWhy = "", ""
}
}
@@ -460,6 +470,7 @@ func retryRollouts(ctx context.Context, open *stores, p *inventory.Plan) (string
said = append(said, m+" to "+strings.Join(again, ", "))
}
p.State = inventory.PlanRolling
goesOnAgain(p)
p.Note = fmt.Sprintf("tier %d retried by hand; sent %s first again", p.Tier, strings.Join(said, "; "))
if err := open.inventory.SavePlan(ctx, p); err != nil {
return "", err
@@ -525,6 +536,7 @@ func retryTierWhole(ctx context.Context, open *stores, p *inventory.Plan, again
}
if p.State == inventory.PlanFailed && len(failedIn(*p)) == 0 {
p.State = inventory.PlanBuilding
goesOnAgain(p)
p.Note = fmt.Sprintf("tier %d retried by hand: %s asked again; %s sent again with the tier", p.Tier,
strings.Join(again, ", "), orNone(strings.Join(resent, ", ")))
}
+158
View File
@@ -0,0 +1,158 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"strconv"
"strings"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Following a plan (novox/hq issue 229): `plans wait <id>` holds its answer until the plan moves, reaches its
// next tier or ends, or until its bound passes, and says which. Before it, an agent following a rollout had
// `plans` answer once, in prose, and wrote a loop of its own round the mesh's tools to be told the plan moved.
//
// **No wait is longer than a call is answered within** (link.LongestAnswer, inside the console's thirty
// seconds): a longer one would answer "still running" and leave its caller to ask `calls` — the loop again.
// So a caller following a plan to its end asks again; each call answers the moment the plan moves.
// maxPlanWait is the longest bound a wait takes, and the bound when none is given: what a call is answered
// within, less the time a verb's command takes to start and read the store.
var maxPlanWait = link.LongestAnswer - planWaitStart
// planWaitStart is what the verb's window allows beyond the bound: starting the command and opening the stores.
// A store slow enough to push the read and the answer past it degrades as any long call does: the caller is
// answered that the call is still running, and `calls` with its id says what came of it.
const planWaitStart = 5 * time.Second
// planWaitEvery is how often a wait reads the plan again.
const planWaitEvery = time.Second
// The things a wait waits until.
var planWaitUntil = []string{"moved", "tier", "done"}
// planWaitAsked checks what a wait is asked for: until (moved when empty) and a bound (a duration, or a number
// of seconds; maxPlanWait when empty).
func planWaitAsked(until, bound string) (string, time.Duration, error) {
if until == "" {
until = "moved"
}
known := false
for _, u := range planWaitUntil {
known = known || u == until
}
if !known {
return "", 0, fmt.Errorf("a wait is until %s, not %q", strings.Join(planWaitUntil, ", "), until)
}
d := maxPlanWait
if bound = strings.TrimSpace(bound); bound != "" {
var err error
if secs, isNumber := strconv.Atoi(bound); isNumber == nil {
d = time.Duration(secs) * time.Second
} else if d, err = time.ParseDuration(bound); err != nil {
return "", 0, fmt.Errorf("the bound %q is not a duration (such as 20s) or a number of seconds", bound)
}
if d <= 0 || d > maxPlanWait {
return "", 0, fmt.Errorf("the bound is at most %s, the longest a call is answered within; %s was asked "+
"— to follow a plan longer, ask again: each call answers the moment the plan moves", maxPlanWait, d)
}
}
return until, d, nil
}
// planEnded is a plan that will not move again: done, failed or superseded. A batch being assembled is not one.
func planEnded(p inventory.Plan) bool { return !p.Open() && !p.Batch() }
// waitOnPlan reads the plan until what it waits for happens or the bound passes, and answers the plan as last
// read and what ended the wait: moved (any save of it), tier (its tier changed), done (it ended — which ends
// every wait, since nothing comes after it), or bound. Moved and tier are judged against the plan as first read.
func waitOnPlan(ctx context.Context, read func(context.Context) (inventory.Plan, error), until string,
bound, every time.Duration) (inventory.Plan, string, error) {
first, err := read(ctx)
if err != nil {
return inventory.Plan{}, "", err
}
if planEnded(first) {
return first, "done", nil
}
deadline := time.Now().Add(bound)
tick := time.NewTicker(every)
defer tick.Stop()
last := first
for {
select {
case <-ctx.Done():
return last, "", ctx.Err()
case <-tick.C:
}
if !time.Now().Before(deadline) {
return last, "bound", nil
}
p, err := read(ctx)
if err != nil {
return last, "", err
}
last = p
switch {
case planEnded(p):
return p, "done", nil
case until == "moved" && p.Revision != first.Revision:
return p, "moved", nil
case until == "tier" && p.Tier != first.Tier:
return p, "tier", nil
}
}
}
// planWaited is what `plans wait` answers: the plan, what ended the wait, and the line a person reads (`text`,
// which the verb answers as its output and not again in the data).
type planWaited struct {
// Came is what ended the wait: moved, tier, done, or bound when nothing did.
Came string `json:"came"`
Until string `json:"until"`
Waited string `json:"waited"`
Plan planStatus `json:"plan"`
// Revision is the plan's as read: a caller asking again with the same plan sees whether it moved since.
Revision int64 `json:"revision"`
Text string `json:"text"`
}
func planWaitedDoc(p inventory.Plan, st planStatus, until, came string, waited time.Duration, line string) planWaited {
said := map[string]string{
"moved": "moved", "tier": "reached another tier", "done": "ended",
"bound": "did not " + map[string]string{"moved": "move", "tier": "reach another tier", "done": "end"}[until],
}[came]
text := fmt.Sprintf("%s %s in %s (waited until %s)\n%s — %s\n", p.ID, said, waited.Round(time.Second), until,
p.ID, line)
return planWaited{Came: came, Until: until, Waited: waited.Round(time.Millisecond).String(), Plan: st,
Revision: p.Revision, Text: text}
}
// planWaitCommand is `plans wait <id>`: the plan followed until it moves, or the bound passes.
func planWaitCommand(ctx context.Context, inv *inventory.Inventory, id, untilAsked, boundAsked string) error {
until, bound, err := planWaitAsked(untilAsked, boundAsked)
if err != nil {
return err
}
began := time.Now()
p, came, err := waitOnPlan(ctx, func(ctx context.Context) (inventory.Plan, error) {
return inv.PlanByID(ctx, id)
}, until, bound, planWaitEvery)
if err != nil {
return err
}
// Said as `plans <id>` says its first line, and as `status` says the plan.
now, one := time.Now(), []inventory.Plan{p}
bounds, pause := readTierBounds(ctx, inv, now), buildSeatPause(ctx, inv, one)
line := planLineSaid(p, now, pause, bounds.of(p.Repository), func(module string) []string {
on, _ := inv.Running(ctx, module)
return on
}, waitsOf(ctx, inv, one))
doc := planWaitedDoc(p, planStatuses(one, now, pause, bounds)[0], until, came, now.Sub(began), line)
return json.NewEncoder(os.Stdout).Encode(doc)
}
+196
View File
@@ -0,0 +1,196 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A plan as the store would answer it at each read: the reads are counted, and each answers the next.
type planReads struct {
plans []inventory.Plan
n int
}
func (r *planReads) read(context.Context) (inventory.Plan, error) {
p := r.plans[min(r.n, len(r.plans)-1)]
r.n++
return p, nil
}
func followedPlan(rev int64, state string, tier int) inventory.Plan {
return inventory.Plan{ID: "plan-1", Repository: "novox/x", Commit: "abc", State: state, Tier: tier,
Tiers: [][]string{{"a"}, {"b"}}, Revision: rev}
}
// **`plans` waits for a plan, and says what ended the wait** (novox/hq issue 229): an agent following a
// rollout is told when the plan moved, reached its next tier or ended, or that the bound passed — in one
// call, without a loop of its own.
func TestAWaitForAPlanEndsWhenWhatItWaitsForHappens(t *testing.T) {
cases := []struct {
name string
until string
reads []inventory.Plan
came string
times int
}{
{"moved: any save", "moved", []inventory.Plan{
followedPlan(1, inventory.PlanBuilding, 0), followedPlan(1, inventory.PlanBuilding, 0),
followedPlan(2, inventory.PlanBuilding, 0)}, "moved", 3},
{"tier: a save inside the tier is not enough", "tier", []inventory.Plan{
followedPlan(1, inventory.PlanBuilding, 0), followedPlan(2, inventory.PlanBuilding, 0),
followedPlan(3, inventory.PlanRolling, 1)}, "tier", 3},
{"tier: a plan that ended has no next tier to wait for", "tier", []inventory.Plan{
followedPlan(1, inventory.PlanBuilding, 0), followedPlan(2, inventory.PlanFailed, 0)}, "done", 2},
{"done: a new tier is not the end", "done", []inventory.Plan{
followedPlan(1, inventory.PlanBuilding, 0), followedPlan(2, inventory.PlanRolling, 1),
followedPlan(3, inventory.PlanDone, 1)}, "done", 3},
{"done: a plan already ended answers at once", "done", []inventory.Plan{
followedPlan(4, inventory.PlanSuperseded, 1)}, "done", 1},
{"the bound passes", "moved", []inventory.Plan{followedPlan(1, inventory.PlanBuilding, 0)}, "bound", -1},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
r := &planReads{plans: c.reads}
p, came, err := waitOnPlan(context.Background(), r.read, c.until, 50*time.Millisecond, time.Millisecond)
if err != nil {
t.Fatal(err)
}
if came != c.came {
t.Fatalf("the wait ended as %q, want %q", came, c.came)
}
if c.times > 0 && r.n != c.times {
t.Errorf("read the plan %d times, want %d: it answered before or after what it waited for", r.n, c.times)
}
if want := c.reads[min(r.n, len(c.reads))-1]; p.Revision != want.Revision {
t.Errorf("answered revision %d, not the plan as it was last read (%d)", p.Revision, want.Revision)
}
})
}
}
// What a wait is asked for is checked before anything is read: what it waits until, and a bound no longer than
// a call is answered within.
func TestAWaitIsRefusedWhatItCannotDo(t *testing.T) {
for _, bad := range []struct{ until, bound string }{{"landed", ""}, {"", "1h"}, {"", "-3s"}, {"", "soon"}} {
if _, _, err := planWaitAsked(bad.until, bad.bound); err == nil {
t.Errorf("until %q, bound %q was taken", bad.until, bad.bound)
}
}
until, bound, err := planWaitAsked("", "")
if err != nil || until != "moved" || bound != maxPlanWait {
t.Fatalf("the defaults are %q %s %v, want moved and %s", until, bound, err, maxPlanWait)
}
if _, bound, _ := planWaitAsked("tier", "12"); bound != 12*time.Second {
t.Errorf("a bare number is seconds; read %s", bound)
}
}
// **The answer is data, with its prose as a field beside it** (novox/hq issue 229): the verb's `answer` carries
// the plan and what ended the wait, and its output is the line a person reads — never JSON inside a string.
func TestAWaitForAPlanAnswersAsData(t *testing.T) {
p := followedPlan(7, inventory.PlanRolling, 1)
st := planStatuses([]inventory.Plan{p}, time.Now(), pauseView{}, tierBounds{})[0]
doc := planWaitedDoc(p, st, "tier", "tier", 3*time.Second, "rolling, tier 2 of 2")
stdout, _ := json.Marshal(doc)
a := answerOf([]string{"plans", "wait", "plan-1"}, stdout, "", true)
data, ok := a.Answer.(map[string]any)
if !ok {
t.Fatalf("the answer is not data: %v", a.Answer)
}
if data["came"] != "tier" || data["revision"] != float64(7) {
t.Errorf("the data does not say what ended the wait: %v", data)
}
if plan, _ := data["plan"].(map[string]any); plan["state"] != "rolling" || plan["tier"] != float64(1) {
t.Errorf("the data does not carry the plan: %v", data["plan"])
}
if _, text := data["text"]; text {
t.Errorf("the prose is in the data too: %v", data)
}
if strings.Contains(a.Output, "{") || !strings.Contains(a.Output, "plan-1") || !strings.Contains(a.Output, "tier") {
t.Errorf("the output is not the line a person reads: %q", a.Output)
}
}
// The verb composes the wait, and asks for a window as long as its bound.
func TestThePlansVerbComposesAWait(t *testing.T) {
argv, err := argvFor("plans", map[string]any{"wait": "plan-1", "until": "done", "bound": "15s"})
if err != nil {
t.Fatal(err)
}
if got := strings.Join(argv, " "); got != "plans wait plan-1 --until done --bound 15s" {
t.Fatalf("composed %q", got)
}
if _, err := argvFor("plans", map[string]any{"until": "done"}); err == nil {
t.Error("until without wait was taken")
}
if _, err := argvFor("plans", map[string]any{"wait": "plan-1", "bound": "1h"}); err == nil {
t.Error("a bound past what a call can wait was taken")
}
}
// Live, with the store: `plans wait` run as the verb runs it answers when the plan reaches its next tier, as
// data, and says the bound passed when nothing moved (novox/hq issue 229).
func TestAPlanFollowedFromTheStoreAnswersWhenItMoves(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
now := time.Now().UTC()
walk := inventory.Plan{ID: "plan-229", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c229c229",
Created: now, State: inventory.PlanBuilding, Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {}, "b": {}}}
if err := open.inventory.SavePlan(ctx, &walk); err != nil {
t.Fatal(err)
}
moved := make(chan error, 1)
go func() {
time.Sleep(1500 * time.Millisecond)
p, err := open.inventory.PlanByID(ctx, walk.ID)
if err == nil {
p.Tier, p.State = 1, inventory.PlanRolling
err = open.inventory.SavePlan(ctx, &p)
}
moved <- err
}()
began := time.Now()
var got planWaited
out := printed(t, func() error {
return plansCommand(ctx, []string{"wait", walk.ID, "--until", "tier", "--bound", "15s"})
})
if err := <-moved; err != nil {
t.Fatal(err)
}
if err := json.Unmarshal([]byte(out), &got); err != nil {
t.Fatalf("plans wait did not answer data: %v: %q", err, out)
}
if got.Came != "tier" || got.Plan.Tier != 1 || got.Plan.State != inventory.PlanRolling {
t.Fatalf("the wait answered %+v", got)
}
if waited := time.Since(began); waited > 10*time.Second {
t.Errorf("it answered after %s, not when the plan moved", waited)
}
out = printed(t, func() error {
return plansCommand(ctx, []string{"wait", walk.ID, "--bound", "2"})
})
if err := json.Unmarshal([]byte(out), &got); err != nil || got.Came != "bound" || got.Until != "moved" {
t.Fatalf("a plan that did not move answered %+v (%v)", got, err)
}
}
// The verb's schema says the longest bound in words, and the words are the value (review 75 on #266): the
// catalogue cannot read maxPlanWait, so this holds them together.
func TestThePlansSchemaSaysTheLongestBoundThereIs(t *testing.T) {
v, ok := controllerVerb("plans")
if !ok {
t.Fatal("no plans verb")
}
props, _ := v.Input["properties"].(map[string]any)
bound, _ := props["bound"].(map[string]any)
said, _ := bound["description"].(string)
if !strings.Contains(said, maxPlanWait.String()+", the default, is the longest") {
t.Fatalf("the schema says %q; the longest bound is %s", said, maxPlanWait)
}
}
+1 -1
View File
@@ -591,7 +591,7 @@ func proposeCommand(ctx context.Context, module, node, valuesArg string, clear,
return err
},
routerHere: routerHereIn(open.inventory),
grantHeld: grantHeldIn(open),
grantHeld: grantHeldIn(open, conn),
routerRecord: func(ctx context.Context, id string) (string, error) {
bucket, err := asksRecords(ctx, open.inventory)
if err != nil || bucket == "" {
+33 -14
View File
@@ -249,7 +249,7 @@ func serve(ctx context.Context) (err error) {
// (novox/hq to-be 45 §9).
checkEvents = bus
// And every walk this controller keeps in a new state, for the delivery it walks (novox/hq ADR 0239).
inventory.PlanSaved = func(p inventory.Plan) { sayPlanMoved(ctx, bus, p) }
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { sayPlanMoved(ctx, bus, open.inventory, p) }
if err := server.Checks(following{open}); err != nil {
return err
}
@@ -1027,6 +1027,9 @@ func (b overTheBus) declare(ctx context.Context, s readyNode, body []byte) (stri
if err != nil {
return "", err
}
// A send that carried the mesh's record of the controller and the node-engine — a person's push included —
// ends the condition of one kept over it (novox/hq issue 201).
settleCoreDowngrades(ctx, b.open.inventory, s.node, s.declared.Builds)
if len(s.declared.Bindings) > 0 {
// And where it bound each consumer of a provision that keeps its data (novox/hq ADR 0232):
// what the next resolution keeps it at. On the same outliving context as the send's record.
@@ -1084,42 +1087,58 @@ func brokerFirst(names []string, holder string, behind bool) []string {
// send anywhere, and a module running there always put it in its first wave. A digest of the list
// last sent is kept for this (ADR 0043: the list is composed on each push, never kept itself).
func brokerBehind(ctx context.Context, open *stores, names []string) (string, bool, error) {
s, err := brokerListState(ctx, open, names)
return s.Machine, s.Behind, err
}
// brokerListState is brokerBehind with why the list could not be composed, when it could not: the plan's own
// reason, which a remedy names rather than a push that would fail on it (the review of hq issue 353).
func brokerListState(ctx context.Context, open *stores, names []string) (busListState, error) {
inv := open.inventory
holders, err := seatHolders(ctx, inv)
if err != nil {
return "", false, err
return busListState{}, err
}
h, held := holders[theBrokerSeat]
if !held || h.Node == "" {
return "", false, nil
return busListState{}, nil
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
return "", false, err
return busListState{}, err
}
if m, known := shelf[h.Module]; !known || m.BusUsers == "" {
// A holder that is sent no user list carries no grant: nothing to send first.
return "", false, nil
return busListState{}, nil
}
for _, n := range names {
if n == h.Node {
return h.Node, false, nil
return busListState{Machine: h.Node}, nil
}
}
// **What cannot be worked out is behind** (the review of hq issue 353). The list it would be sent cannot be
// composed, so whether it carries the grants a send relies on is not known; it is taken as behind, so a push
// sends that machine first and says why it cannot (`plan` says it too), and a rollout that needs the list
// refuses rather than sending code whose grants the bus may not hold. It said "not behind" until then: a
// send went ahead as if the bus held a list nobody could compose.
list, composeErr := "", error(nil)
plan, _, err := planFor(ctx, open, h.Node)
if err != nil {
// It cannot be worked out: sending it would refuse the whole send, and `plan` says why.
return h.Node, false, nil
}
list, _, err := busUserList(ctx, inv, plan.Modules)
if err != nil {
return h.Node, false, nil
composeErr = err
} else if list, _, err = busUserList(ctx, inv, plan.Modules); err != nil {
composeErr = err
}
sent, err := inv.SentBusUsers(ctx, h.Node)
if err != nil {
return "", false, err
return busListState{}, err
}
return h.Node, userListBehind(list, sent), nil
return busListState{Machine: h.Node, Behind: listBehind(list, composeErr, sent), Uncomposed: composeErr}, nil
}
// listBehind is whether the bus's machine is behind: the list composed now is not the one last sent, or it
// could not be composed at all — not known is behind, never "not behind".
func listBehind(list string, composeErr error, sentDigest string) bool {
return composeErr != nil || userListBehind(list, sentDigest)
}
// userListBehind is whether the user list composed now is not the one last sent, by its digest. An
+81 -3
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"io"
"os"
"regexp"
"sort"
"strings"
"time"
@@ -67,18 +68,95 @@ func listQueue(ctx context.Context, args []string, w io.Writer) error {
if err != nil {
return err
}
now := time.Now()
if *asJSON {
body, err := json.MarshalIndent(q, "", " ")
body, err := json.MarshalIndent(queueAnswerOf(q, now), "", " ")
if err != nil {
return err
}
fmt.Fprintln(w, string(body))
return nil
}
fmt.Fprint(w, queueText(q, time.Now()))
fmt.Fprint(w, queueText(q, now))
return nil
}
// queueAnswer is the queue as data (novox/hq issue 455): what a caller reads to say how far back an ask
// waits — "waiting in the build queue behind 2" — which the table alone could not tell it. Text is the
// table, written from the same read: the verb answers it as its output and the rest as its answer, so the
// two never disagree.
type queueAnswer struct {
Seat string `json:"seat"`
Waiting int `json:"waiting"`
InFlight int `json:"in-flight"`
Dead int `json:"dead"`
MaxDeliver int `json:"max-deliver"`
Asks []queuedAnswer `json:"asks"`
Text string `json:"text"`
}
// queuedAnswer is one ask as data. Never what the request carries beyond its source — not `held`, every
// artifact the mesh has built, nor a check's checkouts beside it.
type queuedAnswer struct {
ID string `json:"id"`
State string `json:"state"`
// Kind is "check" for a pull request's merge check, "build" for anything else.
Kind string `json:"kind"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
// Commit is the ref when it names a commit; empty when the ask follows a branch or the default.
Commit string `json:"commit,omitempty"`
// Machine is the one building an ask in flight; WasOn the one that started it and handed it back.
Machine string `json:"machine,omitempty"`
WasOn string `json:"was-on,omitempty"`
AskedAt time.Time `json:"asked-at,omitzero"`
// Since is when the ask came to its state: its start on the machine for one in flight that a
// machine said it started, otherwise when it was asked.
Since time.Time `json:"since,omitzero"`
// Position is its place among the waiting, 1 being handed out next; 0 for an ask not waiting.
Position int `json:"position,omitempty"`
Seq uint64 `json:"seq"`
}
// aCommitRef is a ref that names a commit by its full hash (SHA-1 or SHA-256).
var aCommitRef = regexp.MustCompile(`^(?:[0-9a-f]{40}|[0-9a-f]{64})$`)
// queueAnswerOf is the queue as data, in the table's order: in flight, the waiting from the next one to
// be handed out, then the dead. A pure function of one read (novox/hq issue 455).
func queueAnswerOf(q link.Queue, now time.Time) queueAnswer {
waiting, running, dead := q.Of(link.AskWaiting), q.Of(link.AskInFlight), q.Of(link.AskDead)
out := queueAnswer{Seat: q.Seat, Waiting: len(waiting), InFlight: len(running), Dead: len(dead),
MaxDeliver: q.MaxDeliver, Asks: []queuedAnswer{}, Text: queueText(q, now)}
one := func(a link.QueuedAsk, position int) queuedAnswer {
kind := "build"
if a.Request.Check != nil {
kind = "check"
}
since := a.AskedAt
if a.State == link.AskInFlight && a.On != "" && !a.Started.IsZero() {
since = a.Started
}
commit := ""
if aCommitRef.MatchString(a.Ref) {
commit = a.Ref
}
return queuedAnswer{ID: a.ID, State: a.State, Kind: kind, Repository: link.WithoutPassword(a.Repository), Path: a.Path,
Ref: a.Ref, Commit: commit, Machine: a.On, WasOn: a.Was, AskedAt: a.AskedAt, Since: since,
Position: position, Seq: a.Seq}
}
for _, a := range running {
out.Asks = append(out.Asks, one(a, 0))
}
for i, a := range waiting {
out.Asks = append(out.Asks, one(a, i+1))
}
for _, a := range dead {
out.Asks = append(out.Asks, one(a, 0))
}
return out
}
// queueText is the queue as a person reads it: a summary line, then each kind in queue order.
// Never what an ask carries as `held` — that is every artifact the mesh has built.
func queueText(q link.Queue, now time.Time) string {
@@ -92,7 +170,7 @@ func queueText(q link.Queue, now time.Time) string {
return "asked " + now.Sub(t).Round(time.Second).String() + " ago"
}
what := func(a link.QueuedAsk) string {
s := a.Repository
s := link.WithoutPassword(a.Repository)
if a.Path != "" {
s += " at " + a.Path
}
+137
View File
@@ -0,0 +1,137 @@
package main
import (
"encoding/json"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// The queue answers as data as well as text (novox/hq issue 455): each ask's repository, commit, kind,
// machine, since and position among the waiting, read from the same queue the table is written from.
const aCommit = "0123456789abcdef0123456789abcdef01234567"
func aQueueOfEachKind(now time.Time) link.Queue {
asked := now.Add(-5 * time.Minute)
started := now.Add(-2 * time.Minute)
return link.Queue{Seat: link.TheBuildMachine, Delivered: 2, AckPending: 1, MaxDeliver: 5, Asks: []link.QueuedAsk{
{Seq: 1, ID: "build-dead", Repository: "ssh://forge/novox/x", AskedAt: asked, State: link.AskDead},
{Seq: 2, ID: "build-running", Repository: "ssh://forge/novox/mesh-host", Ref: aCommit, AskedAt: asked,
State: link.AskInFlight, On: "ace", Started: started,
Request: link.BuildRequest{ID: "build-running", Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host"}}},
{Seq: 3, ID: "build-first", Repository: "ssh://forge/novox/mesh-catalog", Ref: "main", AskedAt: asked,
State: link.AskWaiting},
{Seq: 4, ID: "build-second", Repository: "ssh://forge/novox/mesh-catalog", Path: "modules/gitea", Ref: aCommit,
AskedAt: now.Add(-time.Minute), State: link.AskWaiting,
Request: link.BuildRequest{ID: "build-second", Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog"}}},
}}
}
func TestTheQueueVerbAnswersEachAskAsData(t *testing.T) {
now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
q := aQueueOfEachKind(now)
got := queueAnswerOf(q, now)
if got.Waiting != 2 || got.InFlight != 1 || got.Dead != 1 || got.Seat != link.TheBuildMachine {
t.Fatalf("counts: %+v", got)
}
if got.Text != queueText(q, now) {
t.Fatalf("the data's text is not the table read from the same queue:\n%s", got.Text)
}
byID := map[string]queuedAnswer{}
for _, a := range got.Asks {
byID[a.ID] = a
}
running := byID["build-running"]
if running.Kind != "check" || running.Machine != "ace" || running.Commit != aCommit || running.Position != 0 ||
!running.Since.Equal(now.Add(-2*time.Minute)) || running.State != link.AskInFlight {
t.Errorf("in flight: %+v", running)
}
first, second := byID["build-first"], byID["build-second"]
if first.Position != 1 || first.Kind != "build" || first.Commit != "" || first.Ref != "main" || first.Machine != "" ||
!first.Since.Equal(now.Add(-5*time.Minute)) {
t.Errorf("first waiting: %+v", first)
}
if second.Position != 2 || second.Kind != "check" || second.Commit != aCommit || second.Path != "modules/gitea" ||
second.Repository != "ssh://forge/novox/mesh-catalog" {
t.Errorf("second waiting: %+v", second)
}
if dead := byID["build-dead"]; dead.Position != 0 || dead.State != link.AskDead {
t.Errorf("dead: %+v", dead)
}
// In the table's order: in flight, then the waiting by position, then the dead.
var order []string
for _, a := range got.Asks {
order = append(order, a.ID)
}
if want := []string{"build-running", "build-first", "build-second", "build-dead"}; !reflect.DeepEqual(order, want) {
t.Errorf("order %v, want %v", order, want)
}
}
// The verb asks its command for JSON, and its answer carries the data while its output stays the table a
// person reads, so a text caller reads what it always did.
func TestTheQueueVerbKeepsTheTableAndCarriesTheData(t *testing.T) {
argv, err := argvFor("queue", nil)
if err != nil || !reflect.DeepEqual(argv, []string{"queue", "--json"}) {
t.Fatalf("queue's command line: %v %v", argv, err)
}
now := time.Now()
q := aQueueOfEachKind(now)
body, err := json.Marshal(queueAnswerOf(q, now))
if err != nil {
t.Fatal(err)
}
answer := answerOf(argv, body, "", true)
if answer.Output != queueText(q, now) {
t.Fatalf("the output is not the table:\n%s", answer.Output)
}
data, ok := answer.Answer.(map[string]any)
if !ok {
t.Fatalf("no data: %#v", answer.Answer)
}
if _, doubled := data["text"]; doubled {
t.Error("the table is said twice: once as the output and again inside the data")
}
asks, _ := data["asks"].([]any)
if len(asks) != 4 || data["waiting"] != float64(2) {
t.Fatalf("data: %v", data)
}
second := asks[2].(map[string]any)
if second["position"] != float64(2) || second["kind"] != "check" || second["commit"] != aCommit {
t.Errorf("second waiting: %v", second)
}
for _, a := range asks {
if _, held := a.(map[string]any)["held"]; held {
t.Error("an ask carries what the mesh holds")
}
}
// A failure is said as text, as before.
failed := answerOf(argv, nil, "mesh-controller: the bus cannot be reached\n", false)
if failed.Answer != nil || !strings.Contains(failed.Output, "cannot be reached") {
t.Errorf("a failure: %+v", failed)
}
}
// A repository named with a password never shows it, in the table or the data.
func TestTheQueueNeverShowsARepositorysPassword(t *testing.T) {
if got := link.WithoutPassword("https://agent:s3cret@forge.example/novox/x.git"); strings.Contains(got, "s3cret") ||
!strings.Contains(got, "agent@forge.example/novox/x.git") {
t.Errorf("redacted: %s", got)
}
now := time.Now()
q := link.Queue{Seat: link.TheBuildMachine, Asks: []link.QueuedAsk{{Seq: 1, ID: "build-1",
Repository: "https://agent:s3cret@forge.example/novox/x.git", State: link.AskWaiting}}}
body, _ := json.Marshal(queueAnswerOf(q, now))
if strings.Contains(string(body), "s3cret") {
t.Errorf("the queue said a password: %s", body)
}
for _, kept := range []string{"ssh://git@forge:222/novox/x.git", "novox/x", "git@forge:novox/x.git", ""} {
if got := link.WithoutPassword(kept); got != kept {
t.Errorf("%q became %q", kept, got)
}
}
}
+1 -1
View File
@@ -326,7 +326,7 @@ func TestTheQueueVerbsRunTheirCommands(t *testing.T) {
args map[string]any
want []string
}{
{"queue", nil, []string{"queue"}},
{"queue", nil, []string{"queue", "--json"}},
{"cancel", map[string]any{"id": "build-1"}, []string{"cancel", "build-1"}},
{"clear", nil, []string{"clear"}},
{"clear", map[string]any{"dead": "true"}, []string{"clear", "--dead"}},
+1 -1
View File
@@ -80,7 +80,7 @@ func TestAVerbsOwnProcessNamesItsConnection(t *testing.T) {
// A silence acts, so it is still its own process; it dials, and finds no such condition.
_, _ = handlers["conditions"](context.Background(),
json.RawMessage(`{"silence":"bus.nothing.here","for":"1h","why":"a test"}`))
names := closedNames(t, bus)
names := closedNames(t, bus, "mesh-controller verb conditions")
found := false
for _, n := range names {
found = found || n == "mesh-controller verb conditions"
+17
View File
@@ -135,6 +135,17 @@ func recordedKept(ctx context.Context, open *stores, node string) (map[string]st
}
out[m] = was
}
// **And the controller and the node-engine at the build the machine was last sent, over an older record**
// (novox/hq issue 201, core_downgrade.go). Under keepingEveryBuild they are kept already.
if !every {
core, err := coreKept(ctx, inv, node, sent, current)
if err != nil {
return nil, err
}
for m, was := range core {
out[m] = was
}
}
return out, nil
}
@@ -159,6 +170,12 @@ func keepRecorded(ctx context.Context, open *stores, node string, shelf map[stri
return nil, fmt.Errorf("%s runs %s's build %s, which the build records no longer hold: the bus's user "+
"list cannot be sent there alone with it kept (novox/hq issue 490)", node, m, short(kept[m]))
}
if !found && guarded(coreComponent(m)) {
return nil, fmt.Errorf("%s was last sent %s's build %s, newer than the one the mesh holds, and the build records "+
"no longer hold it: this send cannot keep it and does not move it back — `push %s` sends the older build on "+
"a person's word, and a newer merge or `rebuild %s` moves the record forward (novox/hq issue 201)",
node, m, short(kept[m]), node, m)
}
if !found {
return nil, fmt.Errorf("%s records rather than rolls out, and %s runs its build %s, which the build "+
"records no longer hold: this send cannot keep it and does not move it — `push %s` sends the new "+
+48 -17
View File
@@ -51,6 +51,9 @@ type sendScope struct {
modules map[string]bool
// person is a person's act: it carries what it carries.
person bool
// puttingBack is a gate's own put-back of its modules: the one send that may carry a core module to a build
// older than the one its machine runs (novox/hq issue 201).
puttingBack bool
}
type sendScopeKey struct{}
@@ -237,6 +240,32 @@ var machineMoves = func(ctx context.Context, open *stores, f moveFacts, node str
return f.moves(node, modules, sent, known, all), nil
}
// withoutKept is a machine's moves less those its send keeps at the build the machine was last sent, and so does
// not make: the grants step keeps every module (novox/hq issue 490), and every send keeps the controller and the
// node-engine over an older record (issue 201). Read as the send composes it (sendKeeps, set after the checks that
// call this), so a move it could not keep is still counted.
func withoutKept(ctx context.Context, open *stores, node string, moves []inventory.CarriedMove) ([]inventory.CarriedMove, error) {
if len(moves) == 0 {
return moves, nil
}
keeping := ctx
if _, on := keptExcept(ctx); !on {
keeping = keepingRecorded(ctx)
}
keeps, err := recordedKept(keeping, open, node)
if err != nil || len(keeps) == 0 {
return moves, err
}
var out []inventory.CarriedMove
for _, mv := range moves {
if was, kept := keeps[mv.Module]; kept && sameCommit(was, mv.From) {
continue
}
out = append(out, mv)
}
return out, nil
}
// ungatedIn refuses a send whose machines would move a build no gate has seen, outside its scope: the
// machines named, and why; the machine holding the bus, added only for its user list, is left instead.
func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder string) ([]string, error) {
@@ -254,19 +283,11 @@ func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder st
if err != nil {
return nil, err
}
// The grants step composes every module at the build its machine was last sent (novox/hq issue 490):
// a move it keeps is not made. Read, not assumed, so a move it could not keep is still refused here.
var keeps map[string]string
if everyBuildKept(ctx, n) {
if keeps, err = recordedKept(ctx, open, n); err != nil {
return nil, err
}
if moves, err = withoutKept(ctx, open, n, moves); err != nil {
return nil, err
}
var waiting []string
for _, mv := range moves {
if was, kept := keeps[mv.Module]; kept && sameCommit(was, mv.From) {
continue
}
if !scope.judged[n] && !scope.modules[mv.Module] {
waiting = append(waiting, fmt.Sprintf("%s %s → %s", mv.Module, short(mv.From), short(mv.To)))
}
@@ -308,6 +329,11 @@ func gatedSend(ctx context.Context, open *stores, node string, owns []inventory.
if err != nil {
return nil, nil, nil, err
}
// A move the send keeps is not made, and is no move for the gate to judge (novox/hq issue 201): a core module
// kept over an older record judged as moving back to it would read the gate as superseded by its own send.
if moves, err = withoutKept(ctx, open, node, moves); err != nil {
return nil, nil, nil, err
}
own := func(module string) bool {
return slices.ContainsFunc(owns, func(o inventory.CarriedMove) bool { return o.Module == module })
}
@@ -812,14 +838,19 @@ func backlogCommand(ctx context.Context, sub string, args []string) error {
}
defer open.Close()
if sub == "release-backlog" {
release, err := open.inventory.HoldPlans(ctx, true)
if err != nil {
// Said on the bus (novox/hq ADR 0308 decision 1): the release plan starts walking here, and its start is
// said by the verb that opens it.
var p *inventory.Plan
if err := sayingOnTheBus(ctx, open.inventory, func() error {
release, err := open.inventory.HoldPlans(ctx, true)
if err != nil {
return err
}
defer release()
why.record(ctx, "upgrade release-backlog", nil)
p, err = releaseBacklog(ctx, open, link.Caller())
return err
}
defer release()
why.record(ctx, "upgrade release-backlog", nil)
p, err := releaseBacklog(ctx, open, link.Caller())
if err != nil {
}); err != nil {
return err
}
if p == nil {
+50 -23
View File
@@ -1463,6 +1463,10 @@ func plansCommand(ctx context.Context, args []string) error {
modules := set.String("modules", "", "or the modules it would change, comma-separated")
moduleDirs := set.String("module-dirs", "", "with --paths: the directories holding a module.json at the commit, "+
"comma-separated, as the forge's announcer says them (novox/hq issue 278)")
// Following a plan (novox/hq issue 229): `plans wait <id>`.
until := set.String("until", "", "with wait: what to wait for — moved (any change), tier (its next tier) or done (its end)")
bound := set.String("bound", "", "with wait: how long to wait at most, as 15s or seconds (default and at most the "+
"longest a call is answered within)")
// Ending a plan by hand is a repair, and says why (novox/hq to-be 45 §7).
why := addHandActFlags(set)
positionals, err := parseAround(set, args)
@@ -1482,6 +1486,9 @@ func plansCommand(ctx context.Context, args []string) error {
defer open.Close()
inv := open.inventory
now := time.Now()
if len(positionals) == 2 && positionals[0] == "wait" {
return planWaitCommand(ctx, inv, positionals[1], *until, *bound)
}
if len(positionals) == 1 {
p, err := inv.PlanByID(ctx, positionals[0])
if err != nil {
@@ -1559,7 +1566,7 @@ func plansCommand(ctx context.Context, args []string) error {
if len(positionals) == 2 && positionals[0] == "go" {
why.record(ctx, "plans go", positionals[1:])
var p inventory.Plan
if err := sayingOnTheBus(ctx, func() (err error) {
if err := sayingOnTheBus(ctx, inv, func() (err error) {
p, err = letGo(ctx, inv, positionals[1], "a person ("+whoAsked()+")", strings.TrimSpace(*why.why))
return err
}); err != nil {
@@ -1571,8 +1578,13 @@ func plansCommand(ctx context.Context, args []string) error {
}
// `retry` (novox/hq ADR 0219): a failed plan's failed builds asked again, and the plan goes on.
if len(positionals) == 2 && positionals[0] == "retry" {
said, err := retryPlan(ctx, open, positionals[1])
if err != nil {
// Said on the bus (novox/hq ADR 0308 decision 1): a retried walk starts again, and its start is said
// by the verb that starts it.
var said string
if err := sayingOnTheBus(ctx, inv, func() (err error) {
said, err = retryPlan(ctx, open, positionals[1])
return err
}); err != nil {
return err
}
fmt.Println(said)
@@ -1586,26 +1598,29 @@ func plansCommand(ctx context.Context, args []string) error {
if positionals[0] == "close" {
how = "closed"
}
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return err
}
defer release()
p, err := inv.PlanByID(ctx, positionals[1])
if err != nil {
return err
}
if !p.Open() {
return fmt.Errorf("%s is already %s", p.ID, p.State)
}
why.record(ctx, "plans "+positionals[0], positionals[1:])
p.State = inventory.PlanFailed
p.Note = how + " by hand at tier " + fmt.Sprint(p.Tier) + ": " + strings.TrimSpace(*why.why)
sayUnsent(&p, func(m string) bool {
u, err := inv.UpgradeOf(ctx, m)
return err == nil && u.RollOut
})
if err := inv.SavePlan(ctx, &p); err != nil {
// Said on the bus (novox/hq ADR 0308 decision 1): the walk's end, outcome stopped, is said by the verb
// that ends it, so what it touched is not held until its bound.
var p inventory.Plan
if err := sayingOnTheBus(ctx, inv, func() error {
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return err
}
defer release()
if p, err = inv.PlanByID(ctx, positionals[1]); err != nil {
return err
}
if !p.Open() {
return fmt.Errorf("%s is already %s", p.ID, p.State)
}
why.record(ctx, "plans "+positionals[0], positionals[1:])
stopByHand(&p, how, "a person ("+whoAsked()+")", strings.TrimSpace(*why.why))
sayUnsent(&p, func(m string) bool {
u, err := inv.UpgradeOf(ctx, m)
return err == nil && u.RollOut
})
return inv.SavePlan(ctx, &p)
}); err != nil {
return err
}
fmt.Printf("%s %s at tier %d of %d; what was asked still builds and registers, nothing further is asked\n",
@@ -1924,3 +1939,15 @@ func forgetDeleted(ctx context.Context, inv *inventory.Inventory, module string,
"`module forget %s`: %v\n", p.ID, module, module, firstLine(err.Error()))
}
}
// stopByHand ends a walk a person stopped or closed (`plans stop|close`): failed, with who stopped it and why
// kept as the delivery's owner keeps its own stop, so the walk ends with outcome stopped, not failed (novox/hq
// ADR 0308 decision 1).
func stopByHand(p *inventory.Plan, how, by, why string) {
p.State = inventory.PlanFailed
p.Note = how + " by hand at tier " + fmt.Sprint(p.Tier) + ": " + why
if p.Delivery == nil {
p.Delivery = &inventory.PlanDelivery{}
}
p.Delivery.Stopped, p.Delivery.StoppedWhy = by, why
}
+85
View File
@@ -0,0 +1,85 @@
package main
import (
"bytes"
"io"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// TestReplay237 replays novox/hq issue 237 (2026-10-04): a workstation already ran a module that
// depends on a node seat nothing there held, and the module holding that seat was assigned to it. The
// answer said the seat was now held, and then that the workstation was left out of the rest of the mesh
// for lack of it, naming as the remedy the assignment just made. `plan` for the node, right after,
// resolved. The outcome asserted: every line of the answer, as the controller seat's verb gives it
// (standard output, then standard error), agrees with `plan` for the node right after the act. Written
// with only what the controller had before its fix — register, assign's command, planFor — so it is laid
// over the commit before.
func TestReplay237(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aContainer("web"))
register(t, open, catalogue.Manifest{Module: "docker", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.ContainerRuntimeSeat, Scope: catalogue.ScopeNode}}})
// Already on the node, from before the rule refused it: the node does not resolve until the
// holder comes.
if _, err := open.inventory.Assign(ctx, "laptop", "web"); err != nil {
t.Fatal(err)
}
said := answered(t, func() error { return assignCommand(ctx, "assign", []string{"laptop", "docker"}) })
if _, _, err := planFor(ctx, open, "laptop"); err != nil {
t.Fatalf("laptop does not resolve right after the act: %v\nthe answer was:\n%s", err, said)
}
if !strings.Contains(said, "web on laptop now has "+catalogue.ContainerRuntimeSeat+" held") {
t.Errorf("the answer does not say the seat is now held:\n%s", said)
}
for _, not := range []string{"laptop is left out", "does not resolve", "which nothing on laptop holds"} {
if strings.Contains(said, not) {
t.Errorf("the answer says %q, which `plan` right after contradicts:\n%s", not, said)
}
}
}
// answered is what a command says as the controller seat's verb answers it: standard output, then
// standard error (answerOf).
func answered(t *testing.T, f func() error) string {
t.Helper()
read := func(w **os.File) (func() string, error) {
old := *w
r, pw, err := os.Pipe()
if err != nil {
return nil, err
}
*w = pw
got := make(chan string)
go func() {
var buf bytes.Buffer
_, _ = io.Copy(&buf, r)
got <- buf.String()
}()
return func() string {
_ = pw.Close()
*w = old
return <-got
}, nil
}
stdout, err := read(&os.Stdout)
if err != nil {
t.Fatal(err)
}
stderr, err := read(&os.Stderr)
if err != nil {
t.Fatal(err)
}
runErr := f()
said := stdout() + stderr()
if runErr != nil {
t.Fatalf("%v\n%s", runErr, said)
}
return said
}
+11 -3
View File
@@ -84,8 +84,12 @@ func asAProcess(t *testing.T, url string) {
t.Setenv(broker.CertificateVar, "")
}
// closedNames are the names of the connections the bus saw closed.
func closedNames(t *testing.T, bus *server.Server) []string {
// closedNames are the names of the connections the bus saw closed, waited for by want's own name: the
// bus can register another connection's close first (the test's own setup connection, say), so returning
// on the first non-empty list names whichever closed first, not necessarily the one being waited for
// (novox/hq issue 543). closedNames instead waits until want itself appears among the closed connections,
// or the deadline passes, and returns whatever names it last saw either way.
func closedNames(t *testing.T, bus *server.Server, want string) []string {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
var names []string
@@ -95,10 +99,14 @@ func closedNames(t *testing.T, bus *server.Server) []string {
t.Fatal(err)
}
names = names[:0]
found := false
for _, c := range connz.Conns {
names = append(names, c.Name)
if c.Name == want {
found = true
}
}
if len(names) > 0 {
if found {
return names
}
time.Sleep(50 * time.Millisecond)
@@ -15,7 +15,8 @@ import (
func serviceManagerHolder() catalogue.Manifest {
return catalogue.Manifest{Module: "systemd", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.ServiceManagerSeat, Scope: catalogue.ScopeNode,
Serves: []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal"}}},
Serves: []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal",
"reset-failed", "wanted-by", "unlink-dangling"}}},
Resources: []map[string]any{{"id": "systemd", "type": "package", "package": "systemd"}}}
}
+54 -3
View File
@@ -314,6 +314,11 @@ func (a *verbArguments) commandLine() ([]string, error) {
return nil, err
}
return []string{"node", "show", str("node")}, nil
case "judge-directories":
if err := need("node"); err != nil {
return nil, err
}
return []string{"node", "judge-directories", str("node")}, nil
case "modules":
return []string{"module", "list", "--json"}, nil
case "seats":
@@ -331,6 +336,9 @@ func (a *verbArguments) commandLine() ([]string, error) {
}
return argv, nil
case "plans":
if err := plainPlanIDs(a.given); err != nil {
return nil, err
}
if r := str("repository"); r != "" {
argv := []string{"plans", "--what-if", r}
if p := str("paths"); p != "" {
@@ -360,6 +368,21 @@ func (a *verbArguments) commandLine() ([]string, error) {
return argv, nil
}
}
if id := str("wait"); id != "" {
// Following a plan (novox/hq issue 229): checked here too, so a call asking what no wait can do is
// refused before a command is started.
if _, _, err := planWaitAsked(a.given["until"], a.given["bound"]); err != nil {
return nil, err
}
argv := []string{"plans", "wait", id}
if u := str("until"); u != "" {
argv = append(argv, "--until", u)
}
if b := str("bound"); b != "" {
argv = append(argv, "--bound", b)
}
return argv, nil
}
if id := str("id"); id != "" {
return []string{"plans", id}, nil
}
@@ -426,7 +449,8 @@ func (a *verbArguments) commandLine() ([]string, error) {
return argv, nil
// The build queue (novox/hq ADR 0219).
case "queue":
return []string{"queue"}, nil
// As data, the table its output (novox/hq issue 455).
return []string{"queue", "--json"}, nil
case "cancel", "kill":
if err := need("id"); err != nil {
return nil, err
@@ -977,7 +1001,16 @@ var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true, "co
// What the records say the registries may keep (novox/hq ADR 0251).
"artifacts": true, "collect": true, "images": true, "mirrors": true,
// The delivery's owner's verbs answer JSON where they read (plan, order, check, walks) — novox/hq ADR 0239.
"delivery": true}
"delivery": true,
// The build queue, each ask with its position among the waiting (novox/hq issue 455).
"queue": true,
// A plan followed (novox/hq issue 229): `plans wait` answers JSON; its other forms say text, and so answer no data.
"plans": true}
// tableBesideVerbs are the JSON verbs whose document carries the table a person reads as its `text`: the
// verb answers that table as its output, as it did before it answered data, and the rest as `answer` —
// both from the one read, so they never disagree (novox/hq issue 455).
var tableBesideVerbs = map[string]bool{"queue": true, "plans": true}
// overviewVerbs are the JSON verbs whose answer is said once, as data, and not again as the text the
// command printed: the overviews, which grow with the mesh (novox/hq issue 314). Every reader of theirs
@@ -1097,6 +1130,12 @@ func answerOf(argv []string, stdout []byte, stderr string, ok bool) verbAnswer {
var parsed any
if json.Unmarshal(bytes.TrimSpace(stdout), &parsed) == nil {
answer.Answer = parsed
if doc, isDoc := parsed.(map[string]any); isDoc && tableBesideVerbs[argv[0]] {
if table, said := doc["text"].(string); said {
answer.Output = table + stderr
delete(doc, "text")
}
}
if overviewVerbs[argv[0]] {
// Once, as data: the same document again as text doubled an answer that already
// outgrew one message of the bus (novox/hq issue 314).
@@ -1245,6 +1284,16 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
// Whatever it did, `status` is composed again once it has.
defer statusFrom.nudge()
}
if verb == "plans" && len(argv) > 1 && argv[1] == "wait" {
// A wait holds its answer up to its bound (novox/hq issue 229), and is answered in full inside it.
asked := ""
if i := slices.Index(argv, "--bound"); i > 0 && i+1 < len(argv) {
asked = argv[i+1]
}
if _, bound, err := planWaitAsked("", asked); err == nil {
link.AnswersWithin(ctx, bound+planWaitStart)
}
}
if answer, read := readHere(ctx, argv); read {
return answer, nil
}
@@ -1479,7 +1528,9 @@ func seatAnnouncement(handlers map[string]link.ToolHandler) micro.Info {
}
// nodeReads are the `node` subcommands a verb may run: the ones that only read.
var nodeReads = map[string]bool{"list": true, "show": true}
var nodeReads = map[string]bool{"list": true, "show": true,
// Read-only on the node too: the engine judges its directories and changes nothing (novox/hq issue 358).
"judge-directories": true}
// flagsOnly says a command line's rest names no subcommand: empty, or beginning with a flag. For a command
// with no subcommands every word is a flag, its value or a name it reads.
+1 -1
View File
@@ -256,7 +256,7 @@ func stringLit(e ast.Expr) (string, bool) {
var accountedFlags = map[string]map[string]string{
"status": {"json": "set by the verb: the answer is data"},
"seats": {"json": "set by the verb: the answer is data"},
"queue": {"json": "not set: the verb answers the table a person reads"},
"queue": {"json": "set by the verb: the answer is data, its output the table (novox/hq issue 455)"},
"plan": {"json": "set by the verb unless files is asked"},
"push": {
"wait": "set by the verb to 0: a tool call cannot hold a connection for a whole apply",
+10
View File
@@ -254,6 +254,16 @@ var signalsTable = []signalRow{
newest: func(f *signalFacts) time.Time {
return newestOf(f.bus.waits, func(w busWaitFacts) time.Time { return w.since })
}},
{Row: "S21", Signal: "a clean apply", Emitter: "node-engine", Trigger: "each report of a declaration refused or applied only in part (novox/hq issue 233)",
Bound: "one apply interval (5 min) past the first report of the same failure, judged by time while the machine " +
"is heard — the node-engine says a re-apply only when how it went changed; not while it is silent or " +
"asleep; naming the refusal, or the package manager that does not answer with its words; cleared by the " +
"clean apply it then says",
Kind: kindApplyFailing, Severity: conditions.Warning, Phase: 2,
needs: func(f *signalFacts) error { return f.doingErr }, watch: watchApplyFailing,
newest: func(f *signalFacts) time.Time {
return newestOf(f.doing, func(d inventory.Doing) time.Time { return d.At })
}},
}
// watchFacts is S14: the snapshot a merge check is fed is older than its bound, or none was kept since
+8
View File
@@ -186,6 +186,14 @@ var suppressions = map[string]suppression{
},
past: func(f *signalFacts) { f.refusedSends = []inventory.Send{refusedSend(f.now.Add(-time.Second))} },
},
// A machine whose package manager does not answer (novox/hq issue 233): said once its first report of the
// failure is older than one apply interval while it is heard. Inside: exactly five minutes old.
"S21": {
inside: func(f *signalFacts) { f.doing = []inventory.Doing{unansweringPackageManager(f.now, 5*time.Minute, 1)} },
past: func(f *signalFacts) {
f.doing = []inventory.Doing{unansweringPackageManager(f.now, 5*time.Minute+time.Second, 1)}
},
},
// Twice by hand within a fortnight is a healer wanted; once, or the first of two a day too old, is not.
"S15": {
inside: func(f *signalFacts) {
+1 -1
View File
@@ -172,7 +172,7 @@ func composeStatus(open *stores) func(context.Context) ([]byte, error) {
var readingVerbs = map[string]bool{
"tools": true, "calls": true, "status": true, "nodes": true, "node": true, "modules": true,
"seats": true, "builds": true, "plan": true, "queue": true, "durations": true, "hand-acts": true,
"doctor": true, "conditions": true, "healers": true,
"doctor": true, "conditions": true, "healers": true, "judge-directories": true,
}
// nudgingListener is the enrolment, nudging the summary when a machine said something new.
@@ -344,3 +344,58 @@ func TestAnyKeyOfAModuleWithATrustedMergeableFileIsRefusedThroughAVerb(t *testin
t.Fatalf("a file marked untrusted through the verb: %v", err)
}
}
// Who owns a module's directory is said at the terminal alone (novox/hq issue 279): a directory's owner may name
// the settings the module runs as, root chowns the folder to them at the next push, and through the settings verb
// any caller could have set puid to 0 or to its own uid. The key is refused through every verb route even where a
// file nothing trusts also asks for it, and taken at the terminal; the layer a refused call meets is unchanged.
func TestADirectoryOwnersKeyIsRefusedThroughAVerb(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "sonarr", Version: "1",
Resources: []map[string]any{
{"id": "recycle-bin", "type": "directory", "mode": "0755", "owner": "${setting:puid}:${setting:pgid}"},
{"id": "identity", "type": "file", "path": "/etc/sonarr.env", "mode": "0644", "trusted": false,
"content": "PUID=${setting:puid}\nPGID=${setting:pgid}\nX=${setting:x}\n"}}})
if err := atTheTerminal(t, "settings", "set", "sonarr", `{"puid":1001,"pgid":2000,"x":0}`,
"--node", "anchor"); err != nil {
t.Fatalf("the owner at the terminal: %v", err)
}
if _, err := assign(ctx, open, "anchor", "sonarr"); err != nil {
t.Fatal(err)
}
layer := func() string {
t.Helper()
values, _, err := open.inventory.Layer(ctx, "anchor", "sonarr")
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(values)
return string(raw)
}
kept := layer()
refused := func(what string, err error) {
t.Helper()
if err == nil || !strings.Contains(err.Error(), "controller's terminal") {
t.Fatalf("%s: %v", what, err)
}
}
refused("puid 0 through the verb", throughVerb(t, "settings", map[string]any{"module": "sonarr",
"node": "anchor", "values": `{"puid":0,"pgid":2000,"x":0}`}))
refused("pgid through the verb, mesh-wide", throughVerb(t, "settings", map[string]any{"module": "sonarr",
"values": `{"pgid":0}`}))
refused("cleared through the verb", throughVerb(t, "settings", map[string]any{"module": "sonarr",
"node": "anchor", "clear": "true"}))
if err := throughVerb(t, "command", map[string]any{
"command": `settings set sonarr '{"puid":0,"pgid":2000,"x":0}' --node anchor`}); err == nil {
t.Fatal("the command verb set the owner")
}
if got := layer(); got != kept {
t.Fatalf("a refused call changed the layer: %s, was %s", got, kept)
}
// Another key, the owner kept, is still the verb's.
if err := throughVerb(t, "settings", map[string]any{"module": "sonarr", "node": "anchor",
"values": `{"puid":1001,"pgid":2000,"x":1}`}); err != nil {
t.Fatalf("another key through the verb, the owner kept: %v", err)
}
}
+334
View File
@@ -0,0 +1,334 @@
package main
import (
"context"
"fmt"
"os"
"slices"
"sort"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// What a walk touches, its bound and its outcome (novox/hq ADR 0308 decision 1, decision 5): said in
// `plan-moved` on the save that starts the walk and the save that ends it, beside the walk whole, for the holder
// of the operator's channel to hold what the walk touches while it runs and tell its result once.
//
// **What a walk touches** is every module it moves on every machine it sends it to, and every module a moved
// tool runner (node-tools) or node-engine (mesh-host) carries on that machine: what restarts there. At the start
// that is what the walk will send — each module of its tiers whose policy rolls it out, on every machine running
// it; at the end it is what it reached — the machines each module was sent to, first and rest, and what a gated
// send carried with it.
//
// **Its bound** is twice the largest delivery budget among the walk's deliveries (deliveryBudgets: a core
// module's ten minutes, a leaf module's budget in force), and fifteen minutes, not doubled, for a walk none of
// whose deliveries has a budget: a release plan (builds waiting for a person's gate), a walk whose every module's
// policy records rather than rolls out (held for a person, ADR 0282 decision 1), or a walk kept without a class.
// walkNoBudgetBound is the bound of a walk with no budget (ADR 0308 decision 5): not doubled.
const walkNoBudgetBound = 15 * time.Minute
// budgetNone is the budget class of a walk none of whose deliveries has a budget.
const budgetNone = "none"
// walkSaid is what `plan-moved` adds on a walk's start and end, beside the walk whole.
type walkSaid struct {
Walk string `json:"walk"`
Phase string `json:"phase"`
// Touches is, per machine, the modules the walk moves or reaches there and what a moved tool runner or
// node-engine carries there; never null.
Touches []walkTouch `json:"touches"`
// Outcome is on the end only: delivered, failed or stopped.
Outcome string `json:"outcome,omitempty"`
// BoundSeconds and BudgetClass are said on both, read from the walk's class at the start.
BoundSeconds int `json:"bound_seconds"`
BudgetClass string `json:"budget_class"`
// StartedAt is when the walk was said started, on both; EndedAt when it was said ended, on the end: the
// bound runs from StartedAt, whenever the event is heard.
StartedAt *time.Time `json:"started_at,omitempty"`
EndedAt *time.Time `json:"ended_at,omitempty"`
}
// walkTouch is what a walk touches on one machine.
type walkTouch struct {
Machine string `json:"machine"`
Modules []string `json:"modules"`
// Carried names those of Modules that are there only because a moved tool runner or node-engine carries
// them: what restarts with it, not what the walk builds.
Carried []string `json:"carried,omitempty"`
}
// planMovedBody is `plan-moved`'s body: the walk whole, as before, and on its start and end what it touches.
type planMovedBody struct {
inventory.Plan
*walkSaid
}
// walkFacts is what the touches are read from: the controller's inventory.
type walkFacts interface {
Running(ctx context.Context, module string) ([]string, error)
UpgradeOf(ctx context.Context, module string) (inventory.Upgrade, error)
Assigned(ctx context.Context, node string) ([]string, error)
SentBuilds(ctx context.Context, node string) (map[string]string, bool, error)
Catalogue(ctx context.Context) (map[string]catalogue.Manifest, error)
}
// sayWalk is what `plan-moved` adds to the save that started or ended the walk; nil on every other save. A fact
// that cannot be read is said on the controller's log and left out, never in the way of the event: a walk said
// with fewer touches is held less, never longer.
func sayWalk(ctx context.Context, facts walkFacts, p inventory.Plan) *walkSaid {
if p.WalkSaid == "" {
return nil
}
w := &walkSaid{Walk: p.ID, Phase: p.WalkSaid}
if p.Times != nil {
w.StartedAt = p.Times.Started
if p.WalkSaid == inventory.WalkEnded {
w.EndedAt, w.Outcome = p.Times.Ended, p.Outcome()
}
}
read := readWalkFacts(ctx, facts, p)
w.BudgetClass = walkBudgetClass(p, read.rollsOut)
w.BoundSeconds = int(walkBound(w.BudgetClass) / time.Second)
var moved map[string][]string
if p.WalkSaid == inventory.WalkStarted {
moved = walkSends(p, read.running, read.rollsOut)
} else {
moved = walkReached(p, read.running)
}
var shelf map[string]catalogue.Manifest
if len(moved[broker.RuntimeModule]) > 0 {
// Which modules the tool runner serves is the catalogue's: read only when the walk moves it.
shelf = read.shelf()
}
w.Touches = touchesOf(moved, read.onMachine(moved), shelf)
return w
}
// walkBudgetClass is the largest budget class among the walk's deliveries: its class read at the cut when one of
// its modules rolls out, and none when it has no budget.
func walkBudgetClass(p inventory.Plan, rollsOut func(string) bool) string {
if p.Release != nil || p.Times == nil {
return budgetNone
}
if _, budgeted := deliveryBudgets[p.Times.Class]; !budgeted {
return budgetNone
}
for _, tier := range p.Tiers {
for _, m := range tier {
if rollsOut(m) {
return p.Times.Class
}
}
}
return budgetNone
}
// walkBound is a walk's bound by its budget class: twice its budget, or fifteen minutes with none.
func walkBound(class string) time.Duration {
if b, ok := deliveryBudgets[class]; ok {
return 2 * b
}
return walkNoBudgetBound
}
// walkSends is what a walk will send at its start, per module the machines it goes to: each module of its tiers
// whose policy rolls it out, on every machine running it (for a release plan, those of its order).
func walkSends(p inventory.Plan, running func(string) []string, rollsOut func(string) bool) map[string][]string {
out := map[string][]string{}
for _, tier := range p.Tiers {
for _, m := range tier {
if s := p.Modules[m]; s != nil && s.State == planDeleted {
continue
}
if p.Release == nil && !rollsOut(m) {
continue
}
for _, n := range running(m) {
if p.Release != nil && !slices.Contains(p.Release.Order, n) {
continue
}
out[m] = appendNew(out[m], n)
}
}
}
return out
}
// walkReached is what a walk sent by its end, per module the machines it reached: its first machines, its rest,
// and every module a gated send carried with it. A release plan reached the machines it walked: those done and
// the one being judged.
func walkReached(p inventory.Plan, running func(string) []string) map[string][]string {
out := map[string][]string{}
carried := func(g *inventory.PlanGate) {
if g == nil {
return
}
for _, c := range g.Carried {
out[c.Module] = appendNew(out[c.Module], c.Node)
}
}
if r := p.Release; r != nil {
reached := append([]string{}, r.Done...)
if r.Gate != nil {
reached = appendNew(reached, r.Gate.Machines...)
carried(r.Gate)
}
for _, tier := range p.Tiers {
for _, m := range tier {
for _, n := range running(m) {
if slices.Contains(reached, n) {
out[m] = appendNew(out[m], n)
}
}
}
}
return out
}
for m, s := range p.Modules {
if s == nil {
continue
}
out[m] = appendNew(out[m], s.First...)
for n := range s.Rest {
out[m] = appendNew(out[m], n)
}
carried(s.Gate)
if len(out[m]) == 0 {
delete(out, m)
}
}
return out
}
// touchesOf turns what a walk moves, per module, into what it touches, per machine: the modules it moves there,
// and, where it moves the tool runner, every module there whose tools it serves, and where it moves the
// node-engine, every module there. Sorted by machine and module.
func touchesOf(moved map[string][]string, on map[string][]string, shelf map[string]catalogue.Manifest) []walkTouch {
byMachine := map[string][]string{}
for m, machines := range moved {
for _, n := range machines {
byMachine[n] = appendNew(byMachine[n], m)
}
}
out := make([]walkTouch, 0, len(byMachine))
for n, modules := range byMachine {
t := walkTouch{Machine: n, Modules: modules}
engine, runner := slices.Contains(modules, hostModule), slices.Contains(modules, broker.RuntimeModule)
if engine || runner {
for _, m := range on[n] {
if slices.Contains(t.Modules, m) || (!engine && !servesTools(shelf[m])) {
continue
}
t.Modules = append(t.Modules, m)
t.Carried = append(t.Carried, m)
}
}
sort.Strings(t.Modules)
sort.Strings(t.Carried)
out = append(out, t)
}
sort.Slice(out, func(i, j int) bool { return out[i].Machine < out[j].Machine })
return out
}
// servesTools says the tool runner serves something of this module: a bundle it loads entrypoints from, or the
// module's own tools.
func servesTools(m catalogue.Manifest) bool {
if len(m.Tools) > 0 {
return true
}
for _, b := range m.Bundles {
if len(b.Loads) > 0 {
return true
}
}
return false
}
// walkRead is the facts a walk's touches are worked out from, read once per event.
type walkRead struct {
running func(string) []string
rollsOut func(string) bool
onMachine func(moved map[string][]string) map[string][]string
shelf func() map[string]catalogue.Manifest
}
// readWalkFacts reads what the walk's touches need. Without facts (a test, a command with no store), nothing runs
// anywhere and every module rolls out: the touches are then what the walk itself recorded.
func readWalkFacts(ctx context.Context, facts walkFacts, p inventory.Plan) walkRead {
cantRead := func(what string, err error) {
fmt.Fprintf(os.Stderr, "%s: %s cannot be read for what the walk touches: %v\n", p.ID, what, err)
}
r := walkRead{
running: func(string) []string { return nil },
rollsOut: func(string) bool { return true },
onMachine: func(map[string][]string) map[string][]string { return nil },
shelf: func() map[string]catalogue.Manifest { return nil },
}
if facts == nil {
return r
}
running := map[string][]string{}
r.running = func(m string) []string {
if on, read := running[m]; read {
return on
}
on, err := facts.Running(ctx, m)
if err != nil {
cantRead("where "+m+" runs", err)
}
running[m] = on
return on
}
rolls := map[string]bool{}
r.rollsOut = func(m string) bool {
if yes, read := rolls[m]; read {
return yes
}
u, err := facts.UpgradeOf(ctx, m)
if err != nil {
cantRead(m+"'s policy", err)
}
// A policy that cannot be read is taken to roll out: the walk is then held more, never less.
rolls[m] = err != nil || u.RollOut
return rolls[m]
}
r.onMachine = func(moved map[string][]string) map[string][]string {
out := map[string][]string{}
for _, core := range []string{hostModule, broker.RuntimeModule} {
for _, n := range moved[core] {
if _, read := out[n]; read {
continue
}
var there []string
if sent, known, err := facts.SentBuilds(ctx, n); err != nil {
cantRead("what "+n+" was sent", err)
} else if known {
for m := range sent {
there = append(there, m)
}
}
if assigned, err := facts.Assigned(ctx, n); err != nil {
cantRead("what "+n+" is assigned", err)
} else {
there = appendNew(there, assigned...)
}
sort.Strings(there)
out[n] = there
}
}
return out
}
r.shelf = func() map[string]catalogue.Manifest {
shelf, err := facts.Catalogue(ctx)
if err != nil {
cantRead("the catalogue", err)
}
return shelf
}
return r
}
+514
View File
@@ -0,0 +1,514 @@
package main
import (
"context"
"encoding/json"
"os"
"reflect"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// fakeWalkFacts is the inventory as far as a walk's touches read it.
type fakeWalkFacts struct {
running map[string][]string
records map[string]bool // modules whose policy records rather than rolls out
sent map[string]map[string]string
assigned map[string][]string
shelf map[string]catalogue.Manifest
}
func (f fakeWalkFacts) Running(_ context.Context, m string) ([]string, error) {
return f.running[m], nil
}
func (f fakeWalkFacts) UpgradeOf(_ context.Context, m string) (inventory.Upgrade, error) {
return inventory.Upgrade{RollOut: !f.records[m]}, nil
}
func (f fakeWalkFacts) Assigned(_ context.Context, n string) ([]string, error) {
return f.assigned[n], nil
}
func (f fakeWalkFacts) SentBuilds(_ context.Context, n string) (map[string]string, bool, error) {
s, ok := f.sent[n]
return s, ok, nil
}
func (f fakeWalkFacts) Catalogue(context.Context) (map[string]catalogue.Manifest, error) {
return f.shelf, nil
}
// theMesh is four machines: the tool runner on each, messenger and mesh-issues on novox, gitea on ace.
func theMesh() fakeWalkFacts {
return fakeWalkFacts{
running: map[string][]string{
broker.RuntimeModule: {"ace", "g14", "novox", "shanks"},
"messenger": {"novox"},
"gitea": {"ace"},
"plex": {"ace"},
},
sent: map[string]map[string]string{
"novox": {broker.RuntimeModule: "c0", "messenger": "c1", "mesh-issues": "c2", "postgres": "c3"},
"ace": {broker.RuntimeModule: "c0", "gitea": "c4"},
},
assigned: map[string][]string{"g14": {broker.RuntimeModule, "desktop"}, "shanks": {broker.RuntimeModule}},
shelf: map[string]catalogue.Manifest{
"messenger": {Module: "messenger", Tools: []string{"notify"}},
"mesh-issues": {Module: "mesh-issues", Bundles: []catalogue.Bundle{{Name: "tools", Loads: []string{"main"}}}},
"postgres": {Module: "postgres"},
"gitea": {Module: "gitea", Tools: []string{"gitea_api"}},
"desktop": {Module: "desktop", Tools: []string{"notify_send"}},
},
}
}
func walkOfTiers(class string, tiers ...[]string) inventory.Plan {
cut := time.Date(2026, 10, 12, 11, 0, 0, 0, time.UTC)
started := cut.Add(time.Second)
p := inventory.Plan{ID: "plan-308", State: inventory.PlanBuilding, Tiers: tiers,
Modules: map[string]*inventory.PlanModule{}, Times: &inventory.PlanTimes{Cut: &cut, Class: class, Started: &started}}
for _, tier := range tiers {
for _, m := range tier {
p.Modules[m] = &inventory.PlanModule{}
}
}
p.WalkSaid = inventory.WalkStarted
return p
}
// novox/hq ADR 0308 How it is checked §1: the start names the modules the walk moves and the machines it sends
// them to, and every module a moved tool runner carries there.
func TestAWalksStartNamesWhatItTouchesWithWhatTheToolRunnerCarries(t *testing.T) {
p := walkOfTiers(inventory.ClassCore, []string{broker.RuntimeModule}, []string{"messenger"})
w := sayWalk(t.Context(), theMesh(), p)
if w == nil || w.Walk != "plan-308" || w.Phase != inventory.WalkStarted || w.Outcome != "" {
t.Fatalf("the start is said: %+v", w)
}
want := []walkTouch{
{Machine: "ace", Modules: []string{"gitea", broker.RuntimeModule}, Carried: []string{"gitea"}},
{Machine: "g14", Modules: []string{"desktop", broker.RuntimeModule}, Carried: []string{"desktop"}},
// postgres serves no tools: the tool runner does not carry it.
{Machine: "novox", Modules: []string{"mesh-issues", "messenger", broker.RuntimeModule}, Carried: []string{"mesh-issues"}},
{Machine: "shanks", Modules: []string{broker.RuntimeModule}},
}
if !reflect.DeepEqual(w.Touches, want) {
t.Fatalf("touches:\n got %+v\nwant %+v", w.Touches, want)
}
if w.BudgetClass != inventory.ClassCore || w.BoundSeconds != 1200 || w.StartedAt == nil {
t.Fatalf("a core walk is bound at 20 minutes: %+v", w)
}
}
// A moved node-engine carries every module on its machine; a module whose policy records is not sent, and not
// touched.
func TestAMovedNodeEngineCarriesEverythingOnItsMachine(t *testing.T) {
f := theMesh()
f.running[hostModule] = []string{"novox"}
f.records = map[string]bool{"plex": true}
p := walkOfTiers(inventory.ClassCore, []string{hostModule, "plex"})
w := sayWalk(t.Context(), f, p)
want := []walkTouch{{Machine: "novox",
Modules: []string{"mesh-host", "mesh-issues", "messenger", "node-tools", "postgres"},
Carried: []string{"mesh-issues", "messenger", "node-tools", "postgres"}}}
if !reflect.DeepEqual(w.Touches, want) {
t.Fatalf("touches:\n got %+v\nwant %+v", w.Touches, want)
}
}
// The end names what the walk reached — its first machines, its rest, what a gated send carried — and its
// outcome; the modules a reached tool runner carries are touched there too.
func TestAWalksEndNamesWhatItReachedAndItsOutcome(t *testing.T) {
p := walkOfTiers(inventory.ClassCore, []string{broker.RuntimeModule}, []string{"messenger"})
ended := p.Times.Started.Add(4 * time.Minute)
p.Times.Ended = &ended
p.State, p.WalkSaid = inventory.PlanDone, inventory.WalkEnded
p.Modules[broker.RuntimeModule] = &inventory.PlanModule{First: []string{"shanks"},
Rest: map[string]inventory.SentDeclaration{"novox": {}},
Gate: &inventory.PlanGate{Carried: []inventory.CarriedMove{{Module: "gitea", Node: "shanks"}}}}
// messenger was built and found unchanged: no move, not reached.
p.Modules["messenger"] = &inventory.PlanModule{Why: "no move"}
w := sayWalk(t.Context(), theMesh(), p)
if w.Phase != inventory.WalkEnded || w.Outcome != inventory.WalkDelivered || w.EndedAt == nil {
t.Fatalf("the end is said with its outcome: %+v", w)
}
want := []walkTouch{
{Machine: "novox", Modules: []string{"mesh-issues", "messenger", broker.RuntimeModule},
Carried: []string{"mesh-issues", "messenger"}},
{Machine: "shanks", Modules: []string{"gitea", broker.RuntimeModule}},
}
if !reflect.DeepEqual(w.Touches, want) {
t.Fatalf("touches:\n got %+v\nwant %+v", w.Touches, want)
}
for outcome, end := range map[string]func(*inventory.Plan){
inventory.WalkFailed: func(p *inventory.Plan) { p.State = inventory.PlanFailed },
inventory.WalkStopped: func(p *inventory.Plan) {
p.State, p.Delivery = inventory.PlanFailed, &inventory.PlanDelivery{Stopped: "a person"}
},
} {
q := p
end(&q)
if got := sayWalk(t.Context(), theMesh(), q).Outcome; got != outcome {
t.Errorf("outcome %q, want %q", got, outcome)
}
}
}
// novox/hq ADR 0308 How it is checked §1 and decision 5: the bound is twice the largest budget in force among
// the walk's deliveries — 20 minutes for a core walk, twice the leaf budget for a leaf walk — and 15 minutes, not
// doubled, for a walk none of whose deliveries has a budget.
func TestAWalksBoundIsTwiceItsBudgetOrFifteenMinutesWithNone(t *testing.T) {
f := theMesh()
f.records = map[string]bool{"plex": true}
release := walkOfTiers("", []string{"gitea"})
release.Times, release.Release = &inventory.PlanTimes{}, &inventory.PlanRelease{Order: []string{"ace"}}
for name, c := range map[string]struct {
p inventory.Plan
class string
bound int
}{
"core": {walkOfTiers(inventory.ClassCore, []string{"messenger"}), inventory.ClassCore, 1200},
"leaf": {walkOfTiers(inventory.ClassLeaf, []string{"messenger"}), inventory.ClassLeaf, int(2 * deliveryBudgets[inventory.ClassLeaf] / time.Second)},
"held for a person": {walkOfTiers(inventory.ClassLeaf, []string{"plex"}), budgetNone, 900},
"a release": {release, budgetNone, 900},
"kept without a class": {walkOfTiers("", []string{"messenger"}), budgetNone, 900},
} {
w := sayWalk(t.Context(), f, c.p)
if w.BudgetClass != c.class || w.BoundSeconds != c.bound {
t.Errorf("%s: %s bound %ds, want %s %ds", name, w.BudgetClass, w.BoundSeconds, c.class, c.bound)
}
}
// The leaf budget in force is three minutes (ADR 0304 decision 4): a leaf walk is bound at six.
if deliveryBudgets[inventory.ClassLeaf] == 3*time.Minute {
if w := sayWalk(t.Context(), f, walkOfTiers(inventory.ClassLeaf, []string{"messenger"})); w.BoundSeconds != 360 {
t.Errorf("a leaf walk at a three-minute budget is bound at 360 s: %d", w.BoundSeconds)
}
}
}
// plan-moved's body: the walk whole as before, and on its start the walk's fields beside it; on any other save
// none of them.
func TestPlanMovedCarriesTheWalkWholeAndOnItsStartWhatItTouches(t *testing.T) {
p := walkOfTiers(inventory.ClassLeaf, []string{"messenger"})
body, err := json.Marshal(planMovedBody{Plan: p, walkSaid: sayWalk(t.Context(), theMesh(), p)})
if err != nil {
t.Fatal(err)
}
var said map[string]any
if err := json.Unmarshal(body, &said); err != nil {
t.Fatal(err)
}
for _, k := range []string{"id", "state", "tiers", "modules", "times", "walk", "phase", "touches", "bound_seconds", "budget_class", "started_at"} {
if _, ok := said[k]; !ok {
t.Errorf("plan-moved lacks %q: %s", k, body)
}
}
if _, ok := said["outcome"]; ok {
t.Errorf("a start says no outcome: %s", body)
}
if said["walk"] != "plan-308" || said["phase"] != "started" || said["budget_class"] != "leaf" {
t.Errorf("the walk's fields: %s", body)
}
p.WalkSaid = ""
body, _ = json.Marshal(planMovedBody{Plan: p, walkSaid: sayWalk(t.Context(), theMesh(), p)})
said = map[string]any{}
_ = json.Unmarshal(body, &said)
for _, k := range []string{"walk", "phase", "touches", "bound_seconds"} {
if _, ok := said[k]; ok {
t.Errorf("a save neither starting nor ending the walk says %q: %s", k, body)
}
}
// A walk that touches nothing says an empty list, never null.
q := walkOfTiers(inventory.ClassLeaf, []string{"nobody-runs-this"})
body, _ = json.Marshal(planMovedBody{Plan: q, walkSaid: sayWalk(t.Context(), theMesh(), q)})
said = map[string]any{}
_ = json.Unmarshal(body, &said)
if touches, ok := said["touches"].([]any); !ok || len(touches) != 0 {
t.Errorf("touches nothing: %s", body)
}
}
// Live, with the store and a bus: a walk let go by the delivery's verb is said started with what it touches and
// its bound, and stopped by the verb is said ended, stopped (novox/hq ADR 0308 How it is checked §1).
func TestAWalkLetGoAndStoppedByVerbsIsSaidStartedAndEnded(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
next := planMovedHeard(t)
register(t, open, catalogue.Manifest{Module: "app", Version: "1"})
if _, err := assign(ctx, open, "anchor", "app"); err != nil {
t.Fatal(err)
}
cut := time.Now().UTC()
waiting := inventory.Plan{ID: "plan-308-live", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c8c8c8c8",
Created: cut, State: inventory.PlanBuilding, Tiers: [][]string{{"app"}},
Modules: map[string]*inventory.PlanModule{"app": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat},
Times: &inventory.PlanTimes{Cut: &cut, Class: inventory.ClassLeaf}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if err := deliveryCommand(ctx, []string{"go", waiting.ID, "--why", "its turn"}); err != nil {
t.Fatal(err)
}
started := next(inventory.WalkStarted)
touches, _ := json.Marshal(started["touches"])
if started["walk"] != waiting.ID || string(touches) != `[{"machine":"anchor","modules":["app"]}]` ||
started["budget_class"] != inventory.ClassLeaf ||
started["bound_seconds"] != float64(2*deliveryBudgets[inventory.ClassLeaf]/time.Second) {
t.Fatalf("the start: %v", started)
}
if err := deliveryCommand(ctx, []string{"stop", waiting.ID, "--why", "a test", "--by", "a person"}); err != nil {
t.Fatal(err)
}
ended := next(inventory.WalkEnded)
touches, _ = json.Marshal(ended["touches"])
if ended["walk"] != waiting.ID || ended["outcome"] != inventory.WalkStopped || string(touches) != `[]` {
t.Fatalf("the end, which sent nothing: %v", ended)
}
}
// planMovedHeard raises a bus of the test's own, points the verbs' hand-act connection at it, and returns what
// waits for the next plan-moved said with the given phase, failing the test when none comes within 5 s.
func planMovedHeard(t *testing.T) func(phase string) map[string]any {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
before := handActConn
handActConn = js.Conn()
t.Cleanup(func() { handActConn = before })
heard := make(chan *nats.Msg, 64)
sub, err := js.Conn().ChanSubscribe(link.SeatEventSubject(link.MeshControllerSeat, link.KeyPlanMoved), heard)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
return func(phase string) map[string]any {
t.Helper()
deadline := time.After(5 * time.Second)
for {
select {
case m := <-heard:
var said map[string]any
if err := json.Unmarshal(m.Data, &said); err != nil {
t.Fatal(err)
}
if said["phase"] == phase {
return said
}
case <-deadline:
t.Fatalf("no walk was said %s", phase)
}
}
}
}
// Live, with the store and a bus: a walk a person stops with `plans stop` is said ended, outcome stopped, by the
// verb itself, and is kept stopped by that person; `plans close` ends one the same way (novox/hq ADR 0308
// decision 1, review 15 on mesh-controller#247).
func TestAWalkStoppedOrClosedByAPersonIsSaidEndedStopped(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
next := planMovedHeard(t)
register(t, open, catalogue.Manifest{Module: "app", Version: "1"})
if _, err := assign(ctx, open, "anchor", "app"); err != nil {
t.Fatal(err)
}
for _, verb := range []string{"stop", "close"} {
cut := time.Now().UTC()
walk := inventory.Plan{ID: "plan-308-" + verb, Repository: "novox/mesh-catalog", Branch: "main",
Commit: "c8c8c8c8", Created: cut, State: inventory.PlanBuilding, Tiers: [][]string{{"app"}},
Modules: map[string]*inventory.PlanModule{"app": {}},
Times: &inventory.PlanTimes{Cut: &cut, Started: &cut, Class: inventory.ClassLeaf}}
if err := open.inventory.SavePlan(ctx, &walk); err != nil {
t.Fatal(err)
}
if err := plansCommand(ctx, []string{verb, walk.ID, "--why", "a test"}); err != nil {
t.Fatal(err)
}
ended := next(inventory.WalkEnded)
if ended["walk"] != walk.ID || ended["outcome"] != inventory.WalkStopped {
t.Fatalf("plans %s: the end is %v", verb, ended)
}
p, err := open.inventory.PlanByID(ctx, walk.ID)
if err != nil {
t.Fatal(err)
}
if p.Outcome() != inventory.WalkStopped || p.Delivery == nil || p.Delivery.StoppedWhy != "a test" {
t.Fatalf("plans %s kept the walk %s, %+v", verb, p.Outcome(), p.Delivery)
}
}
}
// Live, with the store and a bus: a failed walk retried with `plans retry` starts again, and the verb says its
// start (novox/hq ADR 0308 decision 1, review 15 on mesh-controller#247).
func TestAWalkRetriedByAPersonIsSaidStartedAgain(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asksRecorded(t)
next := planMovedHeard(t)
twoTiers(t, open)
before := time.Now().UTC().Add(-time.Hour)
ended := before.Add(time.Minute)
failed := inventory.Plan{ID: "plan-308-retry", Repository: "novox/a", Branch: "main", Commit: "c0ffee",
Created: before, State: inventory.PlanFailed, Tier: 0, Tiers: [][]string{{"a"}, {"b"}},
Note: "a failed to build in tier 0",
Modules: map[string]*inventory.PlanModule{"a": {State: "failed", AskedAt: &before, Build: "build-1",
Why: link.KilledByHand}},
Times: &inventory.PlanTimes{Cut: &before, Started: &before, Ended: &ended, Class: inventory.ClassLeaf}}
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
if err := plansCommand(ctx, []string{"retry", failed.ID}); err != nil {
t.Fatal(err)
}
started := next(inventory.WalkStarted)
if started["walk"] != failed.ID || started["budget_class"] != inventory.ClassLeaf {
t.Fatalf("the retried start: %v", started)
}
}
// Live, with the store and a bus: the release plan `upgrade release-backlog` opens is said started by the verb
// (novox/hq ADR 0308 decision 1, review 15 on mesh-controller#247).
func TestABacklogReleasedByAPersonIsSaidStarted(t *testing.T) {
b := aBacklog(t)
next := planMovedHeard(t)
if err := backlogCommand(t.Context(), "release-backlog", []string{"--why", "a test"}); err != nil {
t.Fatal(err)
}
started := next(inventory.WalkStarted)
if p := b.release(t); started["walk"] != p.ID {
t.Fatalf("the release's start: %v", started)
}
}
// asTyped runs a verb as the controller's command line runs it, through run, as a person's terminal would.
func asTyped(t *testing.T, args ...string) error {
t.Helper()
was := os.Args
os.Args = append([]string{"mesh-controller"}, args...)
t.Cleanup(func() { os.Args = was })
err := run()
// run gives back the lease as the process would end; the test's process goes on, and takes it again.
theLease.mu.Lock()
theLease.held, theLease.kv, theLease.noBus = nil, nil, false
theLease.mu.Unlock()
return err
}
// aStartedWalkAsking keeps a live walk, said started, whose first tier asked a as build.
func aStartedWalkAsking(t *testing.T, open *stores, id, build string, asked time.Time) inventory.Plan {
t.Helper()
walk := inventory.Plan{ID: id, Repository: "novox/a", Branch: "main", Commit: "c0ffee", Created: asked,
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &asked, Build: build}},
Times: &inventory.PlanTimes{Cut: &asked, Started: &asked, Class: inventory.ClassLeaf}}
if err := open.inventory.SavePlan(t.Context(), &walk); err != nil {
t.Fatal(err)
}
return walk
}
// Live, with the store and a bus: `cancel <build>` and `clear` fail the walk that asked the build, and the verb
// says its end, failed (novox/hq ADR 0308 decision 1, review 21 on mesh-controller#247).
func TestAWalkEndedByCancelOrClearIsSaidEnded(t *testing.T) {
_, ask := aBuildQueue(t)
open := aMesh(t)
next := planMovedHeard(t)
twoTiers(t, open)
for _, verb := range [][]string{{"cancel"}, {"clear"}} {
id := link.NewBuildID(time.Now())
ask(id, "https://forge.example/novox/a.git")
asked, _ := link.BuildAskedAt(id)
walk := aStartedWalkAsking(t, open, "plan-308-"+verb[0], id, asked)
args := verb
if verb[0] == "cancel" {
args = append(args, id)
}
if err := asTyped(t, args...); err != nil {
t.Fatal(err)
}
ended := next(inventory.WalkEnded)
if ended["walk"] != walk.ID || ended["outcome"] != inventory.WalkFailed {
t.Fatalf("%s: the end is %v", verb[0], ended)
}
// Ended, and so the next walk may be cut on the same repository.
p, _ := open.inventory.PlanByID(t.Context(), walk.ID)
if p.State != inventory.PlanFailed {
t.Fatalf("%s left the walk %s", verb[0], p.State)
}
}
}
// Live, with the store and a bus: `rebuild <module>` joining a failed walk sets it building again, and the verb
// says its start (novox/hq ADR 0308 decision 1, review 21 on mesh-controller#247).
func TestAWalkRestartedByARebuildIsSaidStarted(t *testing.T) {
open := aMesh(t)
asksRecorded(t)
next := planMovedHeard(t)
twoTiers(t, open)
before := time.Now().UTC().Add(-time.Hour)
ended := before.Add(time.Minute)
failed := inventory.Plan{ID: "plan-308-rebuild", Repository: "novox/a", Branch: "main", Commit: "c0ffee",
Created: before, State: inventory.PlanFailed, Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "failed", AskedAt: &before, Build: "build-1",
Why: link.CancelledByHand}},
Times: &inventory.PlanTimes{Cut: &before, Started: &before, Ended: &ended, Class: inventory.ClassLeaf}}
if err := open.inventory.SavePlan(t.Context(), &failed); err != nil {
t.Fatal(err)
}
if err := asTyped(t, "rebuild", "a"); err != nil {
t.Fatal(err)
}
if started := next(inventory.WalkStarted); started["walk"] != failed.ID {
t.Fatalf("the rebuilt walk's start: %v", started)
}
}
// Live, with the store: a walk a person stopped and then retried, by `plans retry` or by `rebuild`, walks again
// without the stop, so when it fails again it ends failed, not stopped, and the search for the merge that
// brought the failure is not kept from it (novox/hq ADR 0308 decision 1, review 21 on mesh-controller#247).
func TestAWalkStoppedByHandAndRetriedEndsFailedWhenItFailsAgain(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asksRecorded(t)
twoTiers(t, open)
for _, again := range [][]string{{"plans", "retry"}, {"rebuild", "a"}} {
before := time.Now().UTC().Add(-time.Hour)
walk := aStartedWalkAsking(t, open, "plan-308-again-"+again[0], "build-1", before)
if err := asTyped(t, "plans", "stop", walk.ID, "--why", "a test"); err != nil {
t.Fatal(err)
}
// The build asked before the stop fails after it.
p, _ := open.inventory.PlanByID(ctx, walk.ID)
p.Modules["a"].State, p.Modules["a"].Why = "failed", "it broke"
if err := open.inventory.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
args := again
if again[0] == "plans" {
args = append(append([]string(nil), again...), walk.ID)
}
if err := asTyped(t, args...); err != nil {
t.Fatal(err)
}
p, _ = open.inventory.PlanByID(ctx, walk.ID)
if p.State != inventory.PlanBuilding || (p.Delivery != nil && p.Delivery.Stopped != "") {
t.Fatalf("%s: the walk is %s and keeps its stop: %+v", again[0], p.State, p.Delivery)
}
asking, _ := link.BuildAskedAt(p.Modules["a"].Build)
planBuilt(ctx, open, "a", "c0ffee", "it broke again", asking, p.Modules["a"].Build)
if p, _ = open.inventory.PlanByID(ctx, walk.ID); p.State != inventory.PlanFailed || p.Outcome() != inventory.WalkFailed {
t.Fatalf("%s: failing again, the walk is %s and ends %s", again[0], p.State, p.Outcome())
}
}
}
+5
View File
@@ -59,6 +59,10 @@ type signalFacts struct {
// from, each naming its sender (S20, novox/hq issue 234).
refusedSends []inventory.Send
refusedSendsErr error
// doing is every machine whose last report was not a clean apply, with since when it has said the same
// failure (S21, novox/hq issue 233).
doing []inventory.Doing
doingErr error
loop loopFacts
loopErr error
@@ -369,6 +373,7 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
}
f.handActs, f.handActsErr = w.gatherHandActs(ctx, now)
f.refusedSends, f.refusedSendsErr = inv.RefusedSendsSince(ctx, now.Add(-generationRefusalsSaid))
f.doing, f.doingErr = inv.NotDoingWhatTheyWereTold(ctx)
f.facts.taken, _, f.facts.began, f.facts.err = exportedFacts.last()
return f
}
+14 -2
View File
@@ -262,9 +262,12 @@ func MaySubscribe(perms Permissions, subject string) bool {
// delivery held past its bound, and calls them on the operator's warrant, with its why.
//
// And, since novox/hq ADR 0282, `times`: the self-check reads the delivery times to say a delivery over its budget.
//
// And, since novox/hq ADR 0306, `groups`: a merge heard from a head waits for the window while its delivery group
// has a member still to merge.
var VerbsTheControllerAsksTheDeliveryOwner = []SeatVerb{{Seat: "mesh-delivery", Verb: "stalled"},
{Seat: "mesh-delivery", Verb: "close"}, {Seat: "mesh-delivery", Verb: "release"}, {Seat: "mesh-delivery", Verb: "stop"},
{Seat: "mesh-delivery", Verb: "times"}}
{Seat: "mesh-delivery", Verb: "times"}, {Seat: "mesh-delivery", Verb: "groups"}}
// VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant
// chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the
@@ -333,6 +336,13 @@ func AskHandOverSubject(node string) string { return "mesh.node." + node + ".ask
// a hand-over is (link.SetuidSearchContext), for the same reason.
func AskSetuidSearchSubject(node string) string { return "mesh.node." + node + ".ask.setuid-search" }
// AskJudgeDirectoriesSubject is where the controller asks one machine's node-engine to judge, read-only, which of
// its declared directories the walk that follows no link another account could make would refuse (novox/hq issue
// 358): a request answered once, signed as a hand-over is (link.JudgeDirectoriesContext), for the same reason.
func AskJudgeDirectoriesSubject(node string) string {
return "mesh.node." + node + ".ask.judge-directories"
}
// inbox is a principal's own reply space. No user is ever granted a bare `_INBOX.>` (design 25
// §4): with one account, inbox privacy is the permission list or it is nothing, so each user's
// inbox is derived from its own identity and its permissions name that prefix and no other.
@@ -611,7 +621,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
// issue 356), which it answers on the request's reply: the one request a node is asked.
AskHandOverSubject(p.Node),
// And asking it for a fresh search for setuid programs (novox/hq issue 361), answered the same way.
AskSetuidSearchSubject(p.Node)}
AskSetuidSearchSubject(p.Node),
// And asking it to judge its declared directories, read-only (novox/hq issue 358), answered the same way.
AskJudgeDirectoriesSubject(p.Node)}
// The node-engine witnesses the core builds it places (novox/hq to-be 45 §8, ADR 0236; the
// contract is lease/witness.go): it asks its own machine's node tools PING, and, where the
// machine runs the controller, reads the lease's one key — read, never written.
+3 -1
View File
@@ -114,7 +114,9 @@ func TestOnlyWhatCanBeAskedMayAnswer(t *testing.T) {
node, _ := PermissionsFor(Principal{Kind: KindNode, Node: "one", PasswordHash: "x"})
if !node.AllowResponses || !slices.Contains(node.Subscribe, AskHandOverSubject("one")) ||
!slices.Contains(node.Subscribe, AskSetuidSearchSubject("one")) ||
slices.Contains(node.Subscribe, AskSetuidSearchSubject("two")) {
slices.Contains(node.Subscribe, AskSetuidSearchSubject("two")) ||
!slices.Contains(node.Subscribe, AskJudgeDirectoriesSubject("one")) ||
slices.Contains(node.Subscribe, AskJudgeDirectoriesSubject("two")) {
t.Fatalf("a node cannot answer the hand-over it is asked: %v %v", node.AllowResponses, node.Subscribe)
}
person, _ := PermissionsFor(Principal{Kind: KindPerson, Node: "one", Module: "jo", PasswordHash: "x"})
+2 -2
View File
@@ -24,7 +24,7 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
allow_responses: { max: 1, ttl: "1m" }
} }
@@ -34,7 +34,7 @@ accounts {
} }
{ user: "node.one", password: "$2a$11$nnnnnnnnnnnnnnnnnnnnnn", permissions: {
publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.INFO.NODES.one", "$SRV.PING.node-tools.one", "mesh.control.one.>"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.judge-directories", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "one.nats", password: "$2a$11$bbbbbbbbbbbbbbbbbbbbbb", permissions: {
+5
View File
@@ -96,6 +96,11 @@ var WritersTable = []WriterRow{
{State: "a fresh setuid search asked of a machine", Writer: "controller, at its terminal",
KeptIn: "the machine, which throws its last search away", Others: "the engine verifies the mesh's signature and starts it, or refuses",
Subjects: []string{"mesh.node.*.ask.setuid-search"}, Writes: isController},
// A read-only judgement of a machine's declared directories asked of its engine (novox/hq issue 358): signed as a
// hand-over is, under a signing context of its own. It changes nothing on the machine.
{State: "a judgement of a machine's directories asked of it", Writer: "controller",
KeptIn: "nowhere: the engine answers it and keeps nothing", Others: "the engine verifies the mesh's signature and answers, or refuses",
Subjects: []string{"mesh.node.*.ask.judge-directories"}, Writes: isController},
{State: "a machine's applied state and its report", Writer: "the node-engine's apply queue",
KeptIn: "the machine; the report on the bus", Others: "the reconcile and a delivery enqueue, never apply",
// And its health statement between reports (novox/hq ADR 0240): the same writer stating the same
+19
View File
@@ -17,6 +17,7 @@ var designRows = []string{
"a machine's declaration",
"a hand-over asked of a machine",
"a fresh setuid search asked of a machine",
"a judgement of a machine's directories asked of it",
"a machine's applied state and its report",
"the controller lease",
"plans and their tiers",
@@ -186,3 +187,21 @@ func TestASetuidSearchAskHasOnePublisher(t *testing.T) {
}
}
}
// **A judgement of a machine's directories asked of it has one publisher, the controller** (novox/hq issue 358), as a
// hand-over has.
func TestADirectoryJudgementAskHasOnePublisher(t *testing.T) {
for _, p := range []Principal{
{Kind: KindNode, Node: "laptop"},
{Kind: KindNodeTools, Node: "laptop", Module: RuntimeModule},
{Kind: KindModule, Node: "laptop", Module: "notes"},
} {
err := CheckWriters(p, []string{AskJudgeDirectoriesSubject("laptop")})
if err == nil || !strings.Contains(err.Error(), "a judgement of a machine's directories asked of it") {
t.Errorf("%s may ask a judgement of directories: %v", p.Username(), err)
}
}
if err := CheckWriters(Principal{Kind: KindController}, []string{AskJudgeDirectoriesSubject("laptop")}); err != nil {
t.Fatalf("the controller may not ask a judgement: %v", err)
}
}
+13 -2
View File
@@ -971,8 +971,19 @@ func one(ctx context.Context, run Runner, publish Publisher,
return catalogue.Built{Name: a.Name, Kind: a.Kind, Reference: reference}, nil
case catalogue.ArtifactArchive:
src.ownHas(cleanEntry(a.From) + "/**")
body, err := pack(filepath.Join(tree, a.From))
var body []byte
var err error
if a.URL != "" {
// An upstream file pinned by URL and sha256 (novox/hq ADR 0316): its build source is the pin, which
// is in module.json, always part of it.
body, err = upstreamArchive(ctx, workspace, a, say)
if err != nil {
return catalogue.Built{}, fmt.Errorf("%s: %s: %w", module, a.Name, err)
}
} else {
src.ownHas(cleanEntry(a.From) + "/**")
body, err = pack(filepath.Join(tree, a.From))
}
if err != nil {
return catalogue.Built{}, fmt.Errorf("%s: packing %s failed: %w", module, a.Name, err)
}
+77 -6
View File
@@ -49,8 +49,10 @@ import (
//
// One check:
//
// 1. clones the repository at the pull request's head, and beside it the repositories its check
// reads — the controller the mesh runs, the catalogue, the host, the lab — each at the ref asked;
// 1. clones the repository at the pull request's head and **makes the merge of that head into the trunk's
// head**, the check's trunk commit, in its own clone (novox/hq ADR 0323, issue 341; check_merge.go): what
// is judged is what merging makes, never pushed. Beside it the repositories its check reads — the
// controller the mesh runs, the catalogue, the host, the lab — each at the ref asked;
// 2. reads the facts snapshot the controller keeps, and with it **the versions the mesh runs**: the
// store a check's tests stand on is the store's own release, and the bus the bus's;
// 3. raises a throwaway PostgreSQL and a throwaway bus of those versions, labelled with the ask so a
@@ -79,7 +81,8 @@ type CheckSpec struct {
Modules []string
New []string
Manifests []string
// Base is the branch the pull request merges into: what the gate compares the change against.
// Base is the branch the pull request merges into: what the gate compares the change against, and the trunk
// whose head the change is merged into before it is judged (novox/hq ADR 0323). Empty judges the head alone.
Base string
// Judge is who judges the gate: "" the controller the mesh runs, "self" the change's own, "validator"
// the running one with the change's validator.
@@ -104,6 +107,9 @@ type GroupHead struct {
Repository string
Ref string
Paths []string
// Base is the branch it merges into: the head is merged into that trunk's head before it is composed
// (novox/hq ADR 0323 §6). Empty composes the head alone, as a controller from before it asks.
Base string
}
// Gated is whether the change touches the mesh's graph, and so whether the gate runs.
@@ -139,6 +145,16 @@ type CheckVerdict struct {
Took time.Duration
Gate *Layer
Repo *Layer
// Head and Trunk are what was judged (novox/hq ADR 0323 §3): the pull request's head merged into the trunk
// commit, each as twelve characters. Trunk is empty for a check asked with no trunk, which judged the head
// alone. Merge is the merge commit made for the check, whole: only checked, never pushed or registered.
Head, Trunk, Merge string
// Beside are the commits of the repositories the check read beside its own, by the directory each was
// cloned under, as twelve characters: named, not judged against (ADR 0323 §3, issue 568).
Beside map[string]string
// Members are a delivery group's other heads as the composed check judged them, each with the trunk
// commit it was merged into (ADR 0323 §6).
Members []JudgedHead
}
// Layer is one layer of a check, judged.
@@ -281,6 +297,34 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
return CheckVerdict{}, err
}
tree := filepath.Join(root, name)
// **What merging makes is judged, not the head** (novox/hq ADR 0323, issue 341): the head merged into the
// trunk's head as this clone found it, in this clone alone. A head that does not merge cleanly fails here.
judged := CheckVerdict{Head: twelve(spec.Ref)}
paths := spec.Paths
if spec.Base != "" {
m, err := mergeIntoTrunk(ctx, run, tree, spec.Base, spec.Ref)
if err != nil {
return CheckVerdict{}, err
}
judged.Head, judged.Trunk = twelve(m.Head), twelve(m.Trunk)
if len(m.Conflicts) > 0 {
said := conflictWords(m)
say("check", "%s", said)
return CheckVerdict{Verdict: "fail", Summary: said, Report: said, Took: time.Since(began),
Gate: &Layer{Verdict: "fail", Summary: said}, Repo: &Layer{Verdict: "fail", Summary: said},
Head: judged.Head, Trunk: judged.Trunk}, nil
}
judged.Merge = m.Merge
if changed := changedByMerge(ctx, run, tree, m); len(changed) > 0 {
paths = changed
}
say("check", "head %s merged into trunk %s (%s): the merge is judged, and never pushed", judged.Head,
judged.Trunk, spec.Base)
}
named := func(v CheckVerdict) CheckVerdict {
v.Head, v.Trunk, v.Merge, v.Beside, v.Members = judged.Head, judged.Trunk, judged.Merge, judged.Beside, judged.Members
return v
}
// **The base branch's check judges the change** (novox/hq issue 310), never the one the change brings.
judging, err := TheCheckThatJudges(ctx, tree, spec.Base)
if err != nil {
@@ -297,7 +341,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
v := CheckVerdict{Verdict: "pass", Summary: noModule, Took: time.Since(began),
Gate: &Layer{Verdict: "pass", Summary: noModule}, Repo: &Layer{Verdict: "warning", Summary: noScript}}
say("check", "%s; %s", noModule, noScript)
return v, nil
return named(v), nil
}
for dir, b := range spec.Beside {
if dir == name || !safeName.MatchString(dir) {
@@ -306,6 +350,13 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
if err := clone(b.Repository, b.Ref, dir); err != nil {
return CheckVerdict{}, fmt.Errorf("beside it, %w", err)
}
// The commit each one read is named in the verdict (ADR 0323 §3): its move is issue 568's question.
if at := headAt(ctx, run, filepath.Join(root, dir)); at != "" {
if judged.Beside == nil {
judged.Beside = map[string]string{}
}
judged.Beside[dir] = at
}
say("check", "beside it %s at %s", dir, short(b.Ref))
}
@@ -322,6 +373,26 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
return CheckVerdict{}, fmt.Errorf("a head of the group, %w", err)
}
say("check", "with it, of its group, %s/%s at %s", g.Owner, g.Repo, short(g.Ref))
// Each member is composed as its merge into its own trunk's head (ADR 0323 §6).
member := JudgedHead{Owner: g.Owner, Repo: g.Repo, Head: twelve(g.Ref)}
if g.Base != "" {
m, err := mergeIntoTrunk(ctx, run, filepath.Join(root, dir), g.Base, g.Ref)
if err != nil {
return CheckVerdict{}, fmt.Errorf("a head of the group, %s/%s: %w", g.Owner, g.Repo, err)
}
member.Head, member.Trunk = twelve(m.Head), twelve(m.Trunk)
if len(m.Conflicts) > 0 {
said := fmt.Sprintf("%s/%s: %s", g.Owner, g.Repo, conflictWords(m))
say("check", "%s", said)
judged.Members = append(judged.Members, member)
return named(CheckVerdict{Verdict: "fail", Summary: said, Report: said, Took: time.Since(began),
Gate: &Layer{Verdict: "fail", Summary: said}, Repo: &Layer{Verdict: "fail", Summary: said}}), nil
}
if changed := changedByMerge(ctx, run, filepath.Join(root, dir), m); len(changed) > 0 {
g.Paths = changed
}
}
judged.Members = append(judged.Members, member)
heads = append(heads, map[string]any{"repository": g.Owner + "/" + g.Repo,
"tree": filepath.Join(root, dir), "changed": g.Paths})
}
@@ -433,7 +504,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
verdictFile := filepath.Join(root, "verdict.json")
env := []string{EnvFacts + "=" + factsFile, EnvGate + "=" + gate, EnvGateStore + "=" + storeURL,
EnvTestStore + "=" + storeURL, EnvTestBus + "=nats://" + bus, EnvRepository + "=" + spec.Owner + "/" + spec.Repo,
EnvChanged + "=" + strings.Join(spec.Paths, ","), EnvBeside + "=" + root,
EnvChanged + "=" + strings.Join(paths, ","), EnvBeside + "=" + root,
EnvModules + "=" + strings.Join(append(append([]string{}, spec.Modules...), spec.New...), ","),
EnvGroup + "=" + groupFile,
"GOCACHE=" + filepath.Join(workspace, "go-cache"), "GOMODCACHE=" + filepath.Join(workspace, "go-modules")}
@@ -501,7 +572,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
v.Report, v.Took = redact.redact(withWhatFailed(out.String(), v.Repo)), time.Since(began)
say("check", "gate %s — %s; repository %s — %s (%s)", strings.ToUpper(v.Gate.Verdict), v.Gate.Summary,
strings.ToUpper(v.Repo.Verdict), v.Repo.Summary, v.Took.Round(time.Second))
return v, nil
return named(v), nil
}
// ownCheck runs the repository's own check, part by part, each in its toolchain's own container: the
+115
View File
@@ -0,0 +1,115 @@
package builder
import (
"context"
"fmt"
"strings"
)
// **A merge check judges the merge, not the head** (novox/hq ADR 0323, issue 341).
//
// On 2026-10-09 a node-engine pull request was checked green at its head and broke the trunk when it merged:
// a merge an hour earlier had added a field under the same JSON key on other lines, git merged the two edits
// without a conflict, and the result did not compile. The check had judged the head alone. So a check now
// takes the trunk the pull request merges into, its head at the moment the check clones it as the check's
// **trunk commit**, and makes in its own clone the merge of the head into that commit, as the forge's
// default merge would. The gate and the repository's own check judge that merge.
//
// **The merge is only checked** (ADR 0323 §2, ADR 0238 §1): it is a commit off the trunk, made in the
// check's own clone, never pushed to the forge and never registered; the clone is removed when the check
// ends. Nothing here writes to a remote.
//
// The forge keeps no merge reference for a pull request, only its head, so the builder makes it itself.
// TrunkMerge is the merge a check made: the head and the trunk commit it was merged into, whole, and the merge
// commit; or the files in conflict when the head does not merge cleanly.
type TrunkMerge struct {
Head, Trunk, Merge string
Conflicts []string
}
// JudgedHead is one head a check judged, with the trunk commit it merged that head into, each as twelve
// characters (ADR 0323 §3, §6).
type JudgedHead struct {
Owner, Repo string
Head, Trunk string
}
// mergeIdentity is who a check's merge is made as: it is never pushed, so it names nobody's account. No hook of the
// clone's or the build machine's git configuration runs, and no setting there changes the merge.
var mergeIdentity = []string{"-c", "user.name=mesh merge check", "-c", "user.email=merge-check@mesh.invalid",
"-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", "-c", "merge.verifySignatures=false"}
// mergeIntoTrunk makes, in the clone at tree, the merge of head into the head of origin/<base> as it stands in
// that fresh clone: the trunk commit. A head that does not merge cleanly answers the files in conflict and leaves
// the tree at the trunk commit; an error is that the merge could not be tried at all.
func mergeIntoTrunk(ctx context.Context, run Runner, tree, base, head string) (TrunkMerge, error) {
var m TrunkMerge
revParse := func(ref string) (string, error) {
out, err := run(ctx, tree, "git", "rev-parse", "--verify", "--quiet", ref+"^{commit}")
return strings.TrimSpace(out), err
}
var err error
if m.Head, err = revParse(head); err != nil || m.Head == "" {
return m, fmt.Errorf("the head %s cannot be read in the check's clone: %v", head, err)
}
if m.Trunk, err = revParse("refs/remotes/origin/" + base); err != nil || m.Trunk == "" {
return m, fmt.Errorf("the trunk %s cannot be read, and a check judges the head merged into it "+
"(novox/hq ADR 0323): %v", base, err)
}
if _, err := run(ctx, tree, "git", "checkout", "--quiet", "--detach", m.Trunk); err != nil {
return m, fmt.Errorf("the trunk commit %s cannot be checked out: %w", twelve(m.Trunk), err)
}
message := fmt.Sprintf("Check the merge of %s into %s at %s (made for a merge check, never pushed)",
twelve(m.Head), base, twelve(m.Trunk))
args := append(append([]string{}, mergeIdentity...), "merge", "--no-ff", "--no-edit", "--no-verify", "--quiet", "-m", message, m.Head)
if _, err := run(ctx, tree, "git", args...); err != nil {
out, _ := run(ctx, tree, "git", "diff", "--name-only", "--diff-filter=U")
m.Conflicts = strings.Fields(out)
_, _ = run(ctx, tree, "git", "merge", "--abort")
if len(m.Conflicts) == 0 {
return m, fmt.Errorf("the head %s could not be merged into trunk %s: %w", twelve(m.Head), twelve(m.Trunk), err)
}
return m, nil
}
if m.Merge, err = revParse("HEAD"); err != nil {
return m, fmt.Errorf("the merge made for the check cannot be read: %w", err)
}
return m, nil
}
// changedByMerge is every path the merge changes against the trunk commit: what the gate's delivery plan is
// computed from (ADR 0323 §1). Nil when it cannot be read; the caller keeps the pull request's own list.
func changedByMerge(ctx context.Context, run Runner, tree string, m TrunkMerge) []string {
if m.Merge == "" || m.Trunk == "" {
return nil
}
out, err := run(ctx, tree, "git", "diff", "--name-only", "--no-renames", m.Trunk, m.Merge)
if err != nil {
return nil // empty-on-error: the caller keeps the pull request's own list of changed files, the safe one
}
return strings.Fields(out)
}
// conflictWords is a head that does not merge cleanly, said as its check's failure.
func conflictWords(m TrunkMerge) string {
return fmt.Sprintf("head %s does not merge cleanly into trunk %s: in conflict %s", twelve(m.Head), twelve(m.Trunk),
strings.Join(m.Conflicts, ", "))
}
// headAt is the commit a clone is checked out at, as twelve characters; empty when it cannot be read.
func headAt(ctx context.Context, run Runner, tree string) string {
out, err := run(ctx, tree, "git", "rev-parse", "--verify", "--quiet", "HEAD^{commit}")
if err != nil {
return ""
}
return twelve(strings.TrimSpace(out))
}
// twelve is a commit as a verdict names it: its first twelve characters.
func twelve(commit string) string {
if len(commit) > 12 {
return commit[:12]
}
return commit
}
+212
View File
@@ -0,0 +1,212 @@
package builder
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// **A merge check judges the merge, not the head** (novox/hq ADR 0323, issue 341): the head merged into the trunk
// commit in the check's own clone, never pushed; a head that does not merge cleanly fails, naming the files; the
// verdict names the head, the trunk commit and the commits of the repositories read beside it.
// aTrunkAndABranch is a repository whose main moved after the branch was cut: main holds trunkFiles over the
// common files, the branch headFiles. It answers the repository, the branch's head and main's head.
func aTrunkAndABranch(t *testing.T, common, trunkFiles, headFiles map[string]string) (string, string, string) {
t.Helper()
dir := t.TempDir()
git := func(args ...string) string {
t.Helper()
cmd := exec.Command("git", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
return strings.TrimSpace(string(out))
}
write := func(files map[string]string) {
for name, body := range files {
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o755); err != nil {
t.Fatal(err)
}
}
git("add", "-A")
}
git("init", "--quiet", "-b", "main")
write(common)
git("commit", "--quiet", "-m", "common")
git("checkout", "--quiet", "-b", "change")
write(headFiles)
git("commit", "--quiet", "-m", "the change")
head := git("rev-parse", "HEAD")
git("checkout", "--quiet", "main")
write(trunkFiles)
git("commit", "--quiet", "-m", "the trunk moved")
return dir, head, git("rev-parse", "HEAD")
}
func TestACheckJudgesTheHeadMergedIntoTheTrunk(t *testing.T) {
// main's script passes only in a tree holding both the trunk's file and the head's: the merge.
repo, head, trunk := aTrunkAndABranch(t,
map[string]string{CheckScript: "test -f trunk-only && test -f head-only || { echo 'FAIL: not the merge'; exit 1; }\n"},
map[string]string{"trunk-only": "t"}, map[string]string{"head-only": "h"})
beside, besideHead := aCheckedRepository(t, map[string]string{"README": "the catalogue"})
registry := aFactsRegistry(t)
aRuntimeOnThisMachine(t)
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-341", Repository: repo, Ref: head, Owner: "novox",
Repo: "mesh-lab", Number: 1, Base: "main", Toolchain: "golang",
Beside: map[string]Beside{"mesh-catalog": {Repository: beside, Ref: besideHead}}}, t.TempDir(), registry,
GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Repo == nil || v.Repo.Verdict != "pass" {
t.Fatalf("the check did not judge the head merged into the trunk: %+v\n%s", v.Repo, v.Report)
}
if v.Head != head[:12] || v.Trunk != trunk[:12] {
t.Fatalf("the verdict names head %q and trunk %q, not %s and %s", v.Head, v.Trunk, head[:12], trunk[:12])
}
if v.Beside["mesh-catalog"] != besideHead[:12] {
t.Fatalf("the verdict names the catalogue read beside it as %q, not %s: %v", v.Beside["mesh-catalog"],
besideHead[:12], v.Beside)
}
if v.Merge == "" || v.Merge == head || v.Merge == trunk {
t.Fatalf("no merge of the head into the trunk was made for the check: %q", v.Merge)
}
}
func TestAHeadThatDoesNotMergeCleanlyFailsNamingTheFiles(t *testing.T) {
repo, head, trunk := aTrunkAndABranch(t,
map[string]string{"store.go": "package store\n\nconst Declared = 1\n", "other.go": "package store\n"},
map[string]string{"store.go": "package store\n\nconst Declared = 2\n"},
map[string]string{"store.go": "package store\n\nconst Declared = 3\n"})
// No artifact store and no container runtime: a head that does not merge is judged before either is needed.
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-341-conflict", Repository: repo, Ref: head,
Owner: "novox", Repo: "mesh-lab", Number: 2, Base: "main", Toolchain: "golang"}, t.TempDir(), "",
GitCredential{}, nil)
if err != nil {
t.Fatalf("a head that does not merge cleanly was not judged: %v", err)
}
want := "does not merge cleanly into trunk " + trunk[:12]
for name, l := range map[string]*Layer{"gate": v.Gate, "repository": v.Repo} {
if l == nil || l.Verdict != "fail" || !strings.Contains(l.Summary, want) || !strings.Contains(l.Summary, "store.go") {
t.Fatalf("its %s layer said %+v, not a failure saying %q and naming store.go", name, l, want)
}
}
if strings.Contains(v.Summary, "other.go") {
t.Fatalf("a file not in conflict is named: %s", v.Summary)
}
if v.Head != head[:12] || v.Trunk != trunk[:12] {
t.Fatalf("the failure names head %q and trunk %q", v.Head, v.Trunk)
}
}
// **The merge made for a check is only checked** (ADR 0323 §2): the forge never receives it.
func TestTheMergeMadeForACheckIsNeverPushed(t *testing.T) {
repo, head, trunk := aTrunkAndABranch(t, map[string]string{CheckScript: "true\n"},
map[string]string{"trunk-only": "t"}, map[string]string{"head-only": "h"})
registry := aFactsRegistry(t)
aRuntimeOnThisMachine(t)
refsBefore := gitOut(t, repo, "for-each-ref")
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-341-push", Repository: repo, Ref: head, Owner: "novox",
Repo: "mesh-lab", Number: 3, Base: "main", Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Merge == "" {
t.Fatal("the check made no merge of the head into the trunk")
}
if err := exec.Command("git", "-C", repo, "cat-file", "-e", v.Merge+"^{commit}").Run(); err == nil {
t.Fatalf("the merge %s made for the check reached the forge's repository", v.Merge)
}
if after := gitOut(t, repo, "for-each-ref"); after != refsBefore {
t.Fatalf("the check moved a ref of the forge's repository:\nbefore %s\nafter %s", refsBefore, after)
}
if got := gitOut(t, repo, "rev-parse", "main"); got != trunk {
t.Fatalf("the trunk moved to %s", got)
}
}
// A delivery group's composed check merges each member's head into its own trunk's head (ADR 0323 §6), and names
// both for each; a member that does not merge cleanly fails the composed check, naming it.
func TestAGroupsComposedCheckMergesEachMemberIntoItsTrunk(t *testing.T) {
repo, head, trunk := aTrunkAndABranch(t, map[string]string{CheckScript: "true\n"},
map[string]string{"trunk-only": "t"}, map[string]string{"head-only": "h"})
member, memberHead, memberTrunk := aTrunkAndABranch(t, map[string]string{"a": "1"},
map[string]string{"b": "trunk"}, map[string]string{"c": "head"})
registry := aFactsRegistry(t)
aRuntimeOnThisMachine(t)
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-341-group", Repository: repo, Ref: head, Owner: "novox",
Repo: "mesh-controller", Number: 4, Base: "main", Toolchain: "golang", Group: []GroupHead{{Owner: "novox",
Repo: "mesh-host", Repository: member, Ref: memberHead, Base: "main"}}}, t.TempDir(), registry,
GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Head != head[:12] || v.Trunk != trunk[:12] {
t.Fatalf("the composed check names its first head %q into %q", v.Head, v.Trunk)
}
if len(v.Members) != 1 || v.Members[0].Head != memberHead[:12] || v.Members[0].Trunk != memberTrunk[:12] {
t.Fatalf("the composed check names its members %+v, not mesh-host %s into %s", v.Members, memberHead[:12],
memberTrunk[:12])
}
conflicting, conflictHead, _ := aTrunkAndABranch(t, map[string]string{"a": "1"}, map[string]string{"a": "2"},
map[string]string{"a": "3"})
v, err = Check(t.Context(), Command, CheckSpec{ID: "check-341-group-conflict", Repository: repo, Ref: head,
Owner: "novox", Repo: "mesh-controller", Number: 4, Base: "main", Toolchain: "golang", Group: []GroupHead{{
Owner: "novox", Repo: "mesh-host", Repository: conflicting, Ref: conflictHead, Base: "main"}}}, t.TempDir(),
registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Verdict != "fail" || !strings.Contains(v.Summary, "novox/mesh-host") ||
!strings.Contains(v.Summary, "does not merge cleanly") {
t.Fatalf("a member that does not merge cleanly was answered %s: %s", v.Verdict, v.Summary)
}
}
func gitOut(t *testing.T, dir string, args ...string) string {
t.Helper()
out, err := exec.Command("git", append([]string{"-C", dir}, args...)...).CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
return strings.TrimSpace(string(out))
}
// No hook runs when a check makes its merge: not one in the clone, whatever the build machine's git configuration says.
func TestACheckMergeRunsNoHook(t *testing.T) {
repo, head, _ := aTrunkAndABranch(t, map[string]string{"a": "1"}, map[string]string{"b": "trunk"},
map[string]string{"c": "head"})
clone := filepath.Join(t.TempDir(), "clone")
if out, err := exec.Command("git", "clone", "--quiet", repo, clone).CombinedOutput(); err != nil {
t.Fatalf("%v\n%s", err, out)
}
ran := filepath.Join(t.TempDir(), "ran")
for _, hook := range []string{"pre-merge-commit", "commit-msg", "prepare-commit-msg", "post-merge"} {
if err := os.WriteFile(filepath.Join(clone, ".git", "hooks", hook),
[]byte("#!/bin/sh\necho "+hook+" >> "+ran+"\n"), 0o755); err != nil {
t.Fatal(err)
}
}
m, err := mergeIntoTrunk(t.Context(), Command, clone, "main", head)
if err != nil || m.Merge == "" {
t.Fatalf("the merge was not made: %+v, %v", m, err)
}
if said, err := os.ReadFile(ran); err == nil {
t.Fatalf("a hook ran when the check made its merge: %s", said)
}
}
+139
View File
@@ -0,0 +1,139 @@
package builder
import (
"net"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// TestReplay341 is the replay of novox/hq issue 341, written with only what the build agent had before its fix,
// so mesh-lab's prover can lay it over the commit before. On 2026-10-09 node-engine #64/#65 merged at 00:15 and
// added Applied.Declared, under the JSON key `declared`, to the store record. #66, checked at 00:49 at its head,
// had added its own Applied.Declared under the same key on other lines. Its check passed; git merged the two
// edits without a conflict, and the trunk no longer compiled: "Declared redeclared". Before the fix the check
// judged the head alone and passed. After it (novox/hq ADR 0323), the check judges the head merged into the
// trunk commit that held the other field, and fails saying the trunk does not compile.
//
// It runs the check whole, with a container runtime that runs each container's command on this machine
// (aRuntimeOnThisMachine): the repository's own check is `go build ./...`.
func TestReplay341(t *testing.T) {
origin, head := theNinthOfOctober(t)
registry := aFactsRegistry(t)
aRuntimeOnThisMachine(t)
v, err := Check(t.Context(), Command, CheckSpec{ID: "replay-341", Repository: origin, Ref: head, Owner: "novox",
Repo: "mesh-host", Number: 66, Base: "main", Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatalf("the check could not run: %v", err)
}
if v.Repo == nil || v.Repo.Verdict != "fail" {
t.Fatalf("node-engine #66 at its head was answered %+v: the merge it makes into the trunk that holds #65's "+
"field does not compile, and the check let it merge", v.Repo)
}
if said := v.Repo.Summary + "\n" + v.Repo.Failed + "\n" + v.Report; !strings.Contains(said, "Declared redeclared") {
t.Fatalf("the check failed without saying the trunk does not compile:\n%s", said)
}
}
// theNinthOfOctober is the node-engine's repository as it stood at 00:49 on 2026-10-09, in miniature: a trunk
// holding #65's Applied.Declared (a string, key `declared`), and #66's head, branched before it, holding its own
// Applied.Declared (a *FoundUnit, the same key) on other lines. Each compiles alone; their merge does not.
func theNinthOfOctober(t *testing.T) (string, string) {
t.Helper()
origin := t.TempDir()
git := func(args ...string) string {
t.Helper()
cmd := exec.Command("git", args...)
cmd.Dir = origin
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
return strings.TrimSpace(string(out))
}
write := func(name, body string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(filepath.Join(origin, name)), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(origin, name), []byte(body), 0o755); err != nil {
t.Fatal(err)
}
}
store := func(declared65, declared66 string) string {
return "package store\n\n// FoundUnit is a unit found on the machine.\ntype FoundUnit struct{ Name string }\n\n" +
"// Applied is what the node-engine applied.\ntype Applied struct {\n\tGeneration int `json:\"generation\"`\n" +
declared66 + "\tModules []string `json:\"modules\"`\n\tFiles []string `json:\"files\"`\n" +
"\tUnits []string `json:\"units\"`\n\tAt string `json:\"at\"`\n" + declared65 + "}\n"
}
git("init", "--quiet", "-b", "main")
write("go.mod", "module example.org/mesh-host\n\ngo 1.21\n")
write("internal/store/store.go", store("", ""))
write(CheckScript, "# mesh-check-toolchain: go\nset -e\nGOFLAGS= GOWORK=off GOTOOLCHAIN=local GOPROXY=off go build ./...\n")
git("add", "-A")
git("commit", "--quiet", "-m", "#63: the store record")
git("checkout", "--quiet", "-b", "adr-0269")
write("internal/store/store.go", store("", "\tDeclared *FoundUnit `json:\"declared\"`\n"))
git("commit", "--quiet", "-am", "#66: the unit a declaration names")
head := git("rev-parse", "HEAD")
git("checkout", "--quiet", "main")
write("internal/store/store.go", store("\tDeclared string `json:\"declared\"`\n", ""))
git("commit", "--quiet", "-am", "#65: the declaration applied")
return origin, head
}
// aRuntimeOnThisMachine puts a container runtime on PATH that runs a container's command on this machine, in the
// directory and with the environment the check gives it: a store and a bus that answer, a judge that is never
// built, and the repository's own check run as written. Only for a check whose code is the test's own.
func aRuntimeOnThisMachine(t *testing.T) {
t.Helper()
bus, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { bus.Close() })
go func() {
for {
c, err := bus.Accept()
if err != nil {
return
}
c.Close()
}
}()
dir := t.TempDir()
runtime := `#!/bin/sh
cmd="$1"; shift
case "$cmd" in
ps|rm|exec) exit 0 ;;
port) echo "` + bus.Addr().String() + `"; exit 0 ;;
run)
workdir=""; detached=""
while [ $# -gt 0 ]; do
case "$1" in
-d) detached=1; shift ;;
--rm) shift ;;
--workdir) workdir="$2"; shift 2 ;;
--env) export "$2"; shift 2 ;;
--network|--volume|--user|--label|--name|-p|-e) shift 2 ;;
*) break ;;
esac
done
if [ -n "$detached" ]; then echo replay-container; exit 0; fi
shift
if [ "$1" = "sh" ] && [ -n "$workdir" ]; then cd "$workdir" && exec "$@"; fi
echo "no such image on this machine" >&2
exit 1 ;;
esac
exit 1
`
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(runtime), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
}
+326
View File
@@ -0,0 +1,326 @@
package builder
import (
"archive/tar"
"bufio"
"bytes"
"compress/bzip2"
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path"
"path/filepath"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// An archive whose source is somebody else's published file, pinned by URL and sha256 (novox/hq ADR 0316
// §2, §6, to-be 56 §7): a mail client's release, an add-on's xpi.
//
// **Fetched by the build seat, checked against the pin before anything is read from it, and packed as any
// archive is packed.** What a machine receives is the mesh's own archive from the artifact store, by digest,
// exactly as for a directory of the repository; no machine ever asks the vendor. A file whose bytes are not
// the pin's fails the build and names both digests, because the address serving something else is the one
// thing the pin exists to catch.
//
// **Kept in the build seat's workspace under its sha256**, so it is fetched once: a vendor that serves only
// its newest version at one address (Owl) would otherwise break every later build of the same pin.
// maxUpstream is the largest upstream file fetched, and the largest a release may unpack to: the node-engine
// unpacks no archive larger than this either (mesh-host internal/apply/archive.go, maxArchive).
const maxUpstream = 512 << 20
// maxEntries is the most entries an upstream archive may hold: millions of empty ones would fill the build
// seat's inodes and the node's. Betterbird's release holds 64.
const maxEntries = 100_000
// noDowngrade refuses a redirect off https: the digest still guards the bytes, and a fetch in the clear says
// to whoever is on the path what the build seat fetches.
func noDowngrade(req *http.Request, via []*http.Request) error {
if req.URL.Scheme != "https" {
return fmt.Errorf("%s redirected to %s, which is not https", via[0].URL, req.URL)
}
if len(via) >= 10 {
return errors.New("more than 10 redirects")
}
return nil
}
// upstreamClient fetches upstream files: a variable so a test serves its own.
var upstreamClient = &http.Client{Timeout: 15 * time.Minute}
// upstreamDir is where the build seat keeps the upstream files it fetched, by sha256.
func upstreamDir(workspace string) string { return filepath.Join(workspace, "upstream") }
// upstreamArchive is the packed archive of an artifact pinned by URL: the file fetched (or kept from an
// earlier build) and checked, then laid out as the artifact says and packed.
func upstreamArchive(ctx context.Context, workspace string, a catalogue.Artifact,
say func(step, format string, args ...any)) ([]byte, error) {
file, err := upstreamFile(ctx, workspace, a, say)
if err != nil {
return nil, err
}
laid := filepath.Join(workspace, "upstream-"+a.Name)
if err := os.RemoveAll(laid); err != nil {
return nil, err
}
defer os.RemoveAll(laid)
if err := os.MkdirAll(laid, 0o755); err != nil {
return nil, err
}
if a.File != "" {
body, err := os.ReadFile(file)
if err != nil {
return nil, err
}
if err := os.WriteFile(filepath.Join(laid, a.File), body, 0o644); err != nil {
return nil, err
}
} else if err := unpackUpstream(ctx, file, laid, a.Within); err != nil {
return nil, fmt.Errorf("%s (%s): %w", a.URL, a.Version, err)
}
return pack(laid)
}
// upstreamFile is the path of the pinned file in the workspace: kept from an earlier build when its bytes
// are still the pin's, else fetched now and kept only when they are.
func upstreamFile(ctx context.Context, workspace string, a catalogue.Artifact,
say func(step, format string, args ...any)) (string, error) {
want := strings.ToLower(a.SHA256)
dir := upstreamDir(workspace)
kept := filepath.Join(dir, want)
if got, err := sumOf(kept); err == nil {
if got == want {
say("upstream", "%s %s: kept from an earlier build (sha256:%s)", a.Name, a.Version, short(want))
return kept, nil
}
// Changed on disk since it was kept: fetched again rather than trusted.
os.Remove(kept)
}
if err := os.MkdirAll(dir, 0o755); err != nil {
return "", err
}
say("upstream", "fetching %s (%s)", a.URL, a.Version)
request, err := http.NewRequestWithContext(ctx, http.MethodGet, a.URL, nil)
if err != nil {
return "", err
}
client := *upstreamClient
client.CheckRedirect = noDowngrade
response, err := client.Do(request)
if err != nil {
return "", fmt.Errorf("cannot fetch %s: %w", a.URL, err)
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return "", fmt.Errorf("%s answered %s", a.URL, response.Status)
}
part, err := os.CreateTemp(dir, want+".*.part")
if err != nil {
return "", err
}
defer os.Remove(part.Name())
hash := sha256.New()
n, err := io.Copy(io.MultiWriter(part, hash), io.LimitReader(response.Body, maxUpstream+1))
if cerr := part.Close(); err == nil {
err = cerr
}
if err != nil {
return "", fmt.Errorf("fetching %s: %w", a.URL, err)
}
if n > maxUpstream {
return "", fmt.Errorf("%s is larger than %d bytes, more than a machine unpacks; nothing was kept", a.URL, maxUpstream)
}
got := hex.EncodeToString(hash.Sum(nil))
if got != want {
return "", fmt.Errorf("%s was pinned as sha256:%s and what it served is sha256:%s (%d bytes); nothing "+
"was kept. The address serves another file than the one reviewed: compare it, then move the pin "+
"in a reviewed change (novox/hq ADR 0316)", a.URL, want, got, n)
}
if err := os.Rename(part.Name(), kept); err != nil {
return "", err
}
say("upstream", "%s %s: %d bytes, sha256:%s as pinned", a.Name, a.Version, n, short(want))
return kept, nil
}
// sumOf is the sha256 of a file, in hexadecimal.
func sumOf(file string) (string, error) {
f, err := os.Open(file)
if err != nil {
return "", err
}
defer f.Close()
hash := sha256.New()
if _, err := io.Copy(hash, f); err != nil {
return "", err
}
return hex.EncodeToString(hash.Sum(nil)), nil
}
// unpackUpstream unpacks a tar archive, compressed with gzip, bzip2 or xz (told by its first bytes), into a
// directory: of its entries only those within `within` (all, when empty), with that prefix taken off.
//
// **Only files and directories, and nothing outside.** A link, a device or an entry naming a path outside
// the archive, or outside `within`, refuses the build by name: the node-engine unpacks only files and
// directories, and an archive quietly unpacked without what it held would be a release that is not the one
// pinned.
func unpackUpstream(ctx context.Context, file, into, within string) error {
f, err := os.Open(file)
if err != nil {
return err
}
defer f.Close()
buffered := bufio.NewReader(f)
magic, _ := buffered.Peek(6)
var stream io.Reader
var wait func() error
switch {
case bytes.HasPrefix(magic, []byte{0x1f, 0x8b}):
zipped, err := gzip.NewReader(buffered)
if err != nil {
return err
}
defer zipped.Close()
stream = zipped
case bytes.HasPrefix(magic, []byte("BZh")):
stream = bzip2.NewReader(buffered)
case bytes.HasPrefix(magic, []byte{0xfd, '7', 'z', 'X', 'Z', 0x00}):
// Go's standard library reads no xz; the build seat's image carries xz (cmd/mesh-builder/Dockerfile).
if _, err := exec.LookPath("xz"); err != nil {
return errors.New("it is compressed with xz, and this build seat has no xz to read it")
}
cmd := exec.CommandContext(ctx, "xz", "-dc")
cmd.Stdin = buffered
var stderr bytes.Buffer
cmd.Stderr = &stderr
out, err := cmd.StdoutPipe()
if err != nil {
return err
}
if err := cmd.Start(); err != nil {
return err
}
stream = out
wait = func() error {
if err := cmd.Wait(); err != nil {
return fmt.Errorf("xz could not read it: %v %s", err, strings.TrimSpace(stderr.String()))
}
return nil
}
defer func() {
if wait != nil {
_ = cmd.Process.Kill()
_ = cmd.Wait()
}
}()
default:
return errors.New("it is not a tar archive compressed with gzip, bzip2 or xz; a single file is kept " +
"whole by naming it in `file`")
}
if err := untar(stream, into, within); err != nil {
return err
}
if wait != nil {
// Whatever follows the tar's end is read before xz is waited for: xz blocks on a full pipe otherwise.
if _, err := io.Copy(io.Discard, stream); err != nil {
return err
}
w := wait
wait = nil
return w()
}
return nil
}
// untar writes a tar stream's files and directories within `within` into a directory.
func untar(stream io.Reader, into, within string) error {
reader := tar.NewReader(stream)
prefix := ""
if within != "" {
prefix = path.Clean(within) + "/"
}
var total int64
files, entries := 0, 0
for {
header, err := reader.Next()
if err == io.EOF {
break
}
if err != nil {
return err
}
if entries++; entries > maxEntries {
return fmt.Errorf("it holds more than %d entries", maxEntries)
}
if header.Typeflag == tar.TypeXGlobalHeader {
// A pax global header (git archive writes one) names no file.
continue
}
name := path.Clean(strings.TrimPrefix(header.Name, "./"))
if path.IsAbs(name) || name == ".." || strings.HasPrefix(name, "../") {
return fmt.Errorf("%s names a path outside the archive", header.Name)
}
if name == "." || name+"/" == prefix ||
(header.Typeflag == tar.TypeDir && strings.HasPrefix(prefix, name+"/")) {
// The archive's root, the directory taken, or one of its parents.
continue
}
if prefix != "" {
if !strings.HasPrefix(name, prefix) {
return fmt.Errorf("%s is outside %s, the directory the artifact is taken from", header.Name, within)
}
name = strings.TrimPrefix(name, prefix)
}
target := filepath.Join(into, filepath.FromSlash(name))
switch header.Typeflag {
case tar.TypeDir:
if err := os.MkdirAll(target, 0o755); err != nil {
return err
}
case tar.TypeReg:
total += header.Size
if total > maxUpstream {
return fmt.Errorf("it unpacks to more than %d bytes, more than a machine unpacks", maxUpstream)
}
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
return err
}
mode := os.FileMode(0o644)
if header.Mode&0o111 != 0 {
mode = 0o755
}
out, err := os.OpenFile(target, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, mode)
if err != nil {
return err
}
if _, err := io.Copy(out, io.LimitReader(reader, header.Size)); err != nil {
out.Close()
return err
}
if err := out.Close(); err != nil {
return err
}
files++
default:
return fmt.Errorf("%s is a %c, and an archive carries only files and directories", header.Name,
header.Typeflag)
}
}
if files == 0 {
if within != "" {
return fmt.Errorf("it holds no file within %s", within)
}
return errors.New("it holds no file")
}
return nil
}
+385
View File
@@ -0,0 +1,385 @@
package builder
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"sort"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// An archive pinned by URL and sha256 (novox/hq ADR 0316 §2, §6; How it is checked: "an upstream artifact
// whose download does not match its sha256 fails the build and names both digests").
type tarEntry struct {
name, body string
mode int64
kind byte
}
func tarOf(t *testing.T, entries []tarEntry) []byte {
t.Helper()
var raw bytes.Buffer
w := tar.NewWriter(&raw)
for _, e := range entries {
kind := e.kind
if kind == 0 {
kind = tar.TypeReg
}
h := &tar.Header{Name: e.name, Mode: e.mode, Typeflag: kind, Size: int64(len(e.body))}
if kind == tar.TypeSymlink {
h.Linkname, h.Size = "/usr/share/hunspell", 0
}
if kind == tar.TypeDir {
h.Size = 0
}
if err := w.WriteHeader(h); err != nil {
t.Fatal(err)
}
if kind == tar.TypeReg {
if _, err := w.Write([]byte(e.body)); err != nil {
t.Fatal(err)
}
}
}
if err := w.Close(); err != nil {
t.Fatal(err)
}
return raw.Bytes()
}
func gzipped(t *testing.T, body []byte) []byte {
t.Helper()
var out bytes.Buffer
z := gzip.NewWriter(&out)
if _, err := z.Write(body); err != nil {
t.Fatal(err)
}
if err := z.Close(); err != nil {
t.Fatal(err)
}
return out.Bytes()
}
func sumHex(b []byte) string {
s := sha256.Sum256(b)
return hex.EncodeToString(s[:])
}
// vendor serves files by path over https and counts what it was asked.
type vendor struct {
files map[string][]byte
asked int
}
func serveVendor(t *testing.T, v *vendor) string {
t.Helper()
server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
v.asked++
body, ok := v.files[r.URL.Path]
if !ok {
http.NotFound(w, r)
return
}
w.Write(body)
}))
t.Cleanup(server.Close)
before := upstreamClient
// Under a public name, as a manifest must say it: example.com, which the test server's certificate
// names, dialled at the server.
transport := server.Client().Transport.(*http.Transport).Clone()
addr := server.Listener.Addr().String()
transport.DialContext = func(ctx context.Context, network, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, network, addr)
}
upstreamClient = &http.Client{Transport: transport}
t.Cleanup(func() { upstreamClient = before })
return "https://example.com"
}
// entriesOf lists a packed archive: each file with its mode and content.
func entriesOf(t *testing.T, packed []byte) map[string]string {
t.Helper()
z, err := gzip.NewReader(bytes.NewReader(packed))
if err != nil {
t.Fatal(err)
}
r := tar.NewReader(z)
out := map[string]string{}
for {
h, err := r.Next()
if err == io.EOF {
return out
}
if err != nil {
t.Fatal(err)
}
body, _ := io.ReadAll(r)
out[h.Name] = string(body) + " " + os.FileMode(h.Mode).String()
}
}
func release(t *testing.T) []byte {
return gzipped(t, tarOf(t, []tarEntry{
{name: "betterbird/", kind: tar.TypeDir, mode: 0o755},
{name: "betterbird/application.ini", body: "[App]\nVersion=153.4.0\n", mode: 0o644},
{name: "betterbird/betterbird", body: "#!/bin/sh\n", mode: 0o755},
{name: "betterbird/defaults/pref/channel-prefs.js", body: "pref(\"app.update.channel\", \"default\");\n", mode: 0o644},
}))
}
func pinnedManifest(base, sum string) string {
return `{"module":"betterbird","version":"1",
"build":{"artifacts":[
{"name":"archive","kind":"archive","url":"` + base + `/release.tar.gz","sha256":"` + sum + `","version":"153.4.0esr-bb10","within":"betterbird"}]},
"resources":[{"id":"installation","type":"archive","path":"/opt/betterbird","artifact":"archive"}]}`
}
func TestAnArchivePinnedByURLIsFetchedCheckedAndPackedWithoutItsOwnDirectory(t *testing.T) {
body := release(t)
v := &vendor{files: map[string][]byte{"/release.tar.gz": body}}
base := serveVendor(t, v)
r, workspace := aRepository(t, pinnedManifest(base, sumHex(body)), nil)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/catalogue.git", "", "", workspace, nil,
Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
digest, _ := got.Manifest.Resources[0]["digest"].(string)
if !strings.HasPrefix(digest, "sha256:") || r.archives["betterbird/archive"] != digest {
t.Fatalf("the release was not published and pinned as the module's archive: %v, published %v",
got.Manifest.Resources[0], r.archives)
}
packed, err := upstreamArchive(context.Background(), workspace, catalogue.Artifact{Name: "archive", Kind: "archive",
URL: base + "/release.tar.gz", SHA256: sumHex(body), Version: "153.4.0esr-bb10", Within: "betterbird"},
func(string, string, ...any) {})
if err != nil {
t.Fatal(err)
}
if "sha256:"+sumHex(packed) != digest {
t.Fatalf("a second packing of one pin gave another digest: %s and %s", digest, sumHex(packed))
}
entries := entriesOf(t, packed)
want := map[string]string{
"application.ini": "[App]\nVersion=153.4.0\n -rw-r--r--",
"betterbird": "#!/bin/sh\n -rwxr-xr-x",
"defaults/pref/channel-prefs.js": "pref(\"app.update.channel\", \"default\");\n -rw-r--r--",
}
if len(entries) != len(want) {
t.Fatalf("the archive holds %v, want %v", entries, want)
}
for name, w := range want {
if entries[name] != w {
t.Errorf("%s is %q in the archive, want %q", name, entries[name], w)
}
}
}
func TestAnUpstreamFileIsFetchedOnceAndKeptByItsDigest(t *testing.T) {
// The vendor of an add-on serves only its newest version at one address: a pin that was fetched once
// must still build after the address moved on.
body := release(t)
v := &vendor{files: map[string][]byte{"/release.tar.gz": body}}
base := serveVendor(t, v)
r, workspace := aRepository(t, pinnedManifest(base, sumHex(body)), nil)
first, err := Build(context.Background(), r.run, r, "https://forge.invalid/catalogue.git", "", "", workspace, nil,
Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
v.files["/release.tar.gz"] = []byte("the vendor's next release")
second, err := Build(context.Background(), r.run, r, "https://forge.invalid/catalogue.git", "", "", workspace, nil,
Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatalf("a pin fetched before failed to build once the address served something else: %v", err)
}
if v.asked != 1 {
t.Fatalf("the vendor was asked %d times for one pin, want once", v.asked)
}
if first.Manifest.Resources[0]["digest"] != second.Manifest.Resources[0]["digest"] {
t.Fatal("two builds of one pin gave two digests")
}
}
func TestAnUpstreamFileWhoseDownloadDoesNotMatchItsPinFailsAndNamesBothDigests(t *testing.T) {
pinned := release(t)
served := gzipped(t, tarOf(t, []tarEntry{{name: "betterbird/betterbird", body: "not what was reviewed", mode: 0o755}}))
v := &vendor{files: map[string][]byte{"/release.tar.gz": served}}
base := serveVendor(t, v)
r, workspace := aRepository(t, pinnedManifest(base, sumHex(pinned)), nil)
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/catalogue.git", "", "", workspace, nil,
Npmrc{}, GitCredential{}, nil)
if err == nil {
t.Fatal("a download that is not the pinned file was built")
}
for _, digest := range []string{sumHex(pinned), sumHex(served)} {
if !strings.Contains(err.Error(), "sha256:"+digest) {
t.Errorf("the refusal does not name sha256:%s: %v", digest, err)
}
}
if len(r.archives) != 0 {
t.Fatalf("something was published: %v", r.archives)
}
kept, _ := filepath.Glob(filepath.Join(upstreamDir(workspace), "*"))
if len(kept) != 0 {
t.Fatalf("the refused download was kept: %v", kept)
}
}
func TestAnUpstreamFileThatIsNotAnArchiveIsKeptWholeUnderItsName(t *testing.T) {
xpi := []byte("PK\x03\x04 an add-on")
v := &vendor{files: map[string][]byte{"/owl.xpi": xpi}}
base := serveVendor(t, v)
packed, err := upstreamArchive(context.Background(), t.TempDir(), catalogue.Artifact{Name: "owl", Kind: "archive",
URL: base + "/owl.xpi", SHA256: sumHex(xpi), Version: "1.5.6.1", File: "owl.xpi"}, func(string, string, ...any) {})
if err != nil {
t.Fatal(err)
}
entries := entriesOf(t, packed)
if len(entries) != 1 || entries["owl.xpi"] != string(xpi)+" -rw-r--r--" {
t.Fatalf("the add-on was not kept whole as owl.xpi: %v", entries)
}
}
func TestAnUpstreamArchiveHoldingALinkOrAnEntryOutsideItsDirectoryIsRefused(t *testing.T) {
cases := map[string][]tarEntry{
// The community package links the dictionaries; a release carrying a link is not one a machine unpacks.
"is a 2": {{name: "betterbird/betterbird", body: "x", mode: 0o755},
{name: "betterbird/dictionaries", kind: tar.TypeSymlink}},
"outside betterbird": {{name: "betterbird/betterbird", body: "x", mode: 0o755},
{name: "README", body: "x", mode: 0o644}},
"outside the archive": {{name: "../etc/passwd", body: "x", mode: 0o644}},
"no file within": {{name: "betterbird/", kind: tar.TypeDir, mode: 0o755}},
}
for want, entries := range cases {
body := gzipped(t, tarOf(t, entries))
v := &vendor{files: map[string][]byte{"/r.tar.gz": body}}
base := serveVendor(t, v)
_, err := upstreamArchive(context.Background(), t.TempDir(), catalogue.Artifact{Name: "archive", Kind: "archive",
URL: base + "/r.tar.gz", SHA256: sumHex(body), Version: "1", Within: "betterbird"}, func(string, string, ...any) {})
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("an archive whose entry is %q was not refused for it: %v", want, err)
}
}
}
func TestAnUpstreamArchiveCompressedWithXzIsRead(t *testing.T) {
// Betterbird publishes its Linux release as .tar.xz; the build seat's image carries xz for it.
if _, err := exec.LookPath("xz"); err != nil {
t.Skip("NOT TESTED here: no xz on this machine (the build seat's image carries it)")
}
plain := tarOf(t, []tarEntry{{name: "betterbird/betterbird", body: "#!/bin/sh\n", mode: 0o755}})
cmd := exec.Command("xz", "-zc")
cmd.Stdin = bytes.NewReader(plain)
compressed, err := cmd.Output()
if err != nil {
t.Fatal(err)
}
v := &vendor{files: map[string][]byte{"/r.tar.xz": compressed}}
base := serveVendor(t, v)
packed, err := upstreamArchive(context.Background(), t.TempDir(), catalogue.Artifact{Name: "archive", Kind: "archive",
URL: base + "/r.tar.xz", SHA256: sumHex(compressed), Version: "1", Within: "betterbird"}, func(string, string, ...any) {})
if err != nil {
t.Fatal(err)
}
var names []string
for n := range entriesOf(t, packed) {
names = append(names, n)
}
sort.Strings(names)
if strings.Join(names, ",") != "betterbird" {
t.Fatalf("the xz release unpacked to %v", names)
}
}
func TestAnUpstreamTarWithAGlobalHeaderNestedWithinAndTrailingDataIsRead(t *testing.T) {
// git archive writes a pax global header; a release may unpack under a/b with a/ listed first; xz may be
// followed by bytes after the tar's end, which must be read before xz is waited for.
var raw bytes.Buffer
w := tar.NewWriter(&raw)
w.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, Name: "pax_global_header", PAXRecords: map[string]string{"comment": "abc"}})
w.WriteHeader(&tar.Header{Name: "a/", Typeflag: tar.TypeDir, Mode: 0o755})
w.WriteHeader(&tar.Header{Name: "a/b/", Typeflag: tar.TypeDir, Mode: 0o755})
w.WriteHeader(&tar.Header{Name: "a/b/f", Typeflag: tar.TypeReg, Mode: 0o644, Size: 1})
w.Write([]byte("x"))
w.Close()
body := gzipped(t, raw.Bytes())
v := &vendor{files: map[string][]byte{"/r.tar.gz": body}}
base := serveVendor(t, v)
packed, err := upstreamArchive(context.Background(), t.TempDir(), catalogue.Artifact{Name: "a", Kind: "archive",
URL: base + "/r.tar.gz", SHA256: sumHex(body), Version: "1", Within: "a/b"}, func(string, string, ...any) {})
if err != nil {
t.Fatal(err)
}
if e := entriesOf(t, packed); len(e) != 1 || e["f"] == "" {
t.Fatalf("%v", e)
}
if _, err := exec.LookPath("xz"); err != nil {
t.Skip("NOT TESTED here: no xz")
}
// A megabyte after the tar's end: more than a pipe holds.
trailing := append(append([]byte{}, raw.Bytes()...), bytes.Repeat([]byte{0}, 1<<20)...)
cmd := exec.Command("xz", "-zc")
cmd.Stdin = bytes.NewReader(trailing)
compressed, err := cmd.Output()
if err != nil {
t.Fatal(err)
}
v.files["/r.tar.xz"] = compressed
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if _, err := upstreamArchive(ctx, t.TempDir(), catalogue.Artifact{Name: "a", Kind: "archive", URL: base + "/r.tar.xz",
SHA256: sumHex(compressed), Version: "1", Within: "a/b"}, func(string, string, ...any) {}); err != nil {
t.Fatal(err)
}
}
func TestAnUpstreamArchiveWithTooManyEntriesIsRefused(t *testing.T) {
var raw bytes.Buffer
w := tar.NewWriter(&raw)
for i := 0; i <= maxEntries; i++ {
w.WriteHeader(&tar.Header{Name: fmt.Sprintf("d%d/", i), Typeflag: tar.TypeDir, Mode: 0o755})
}
w.Close()
body := gzipped(t, raw.Bytes())
v := &vendor{files: map[string][]byte{"/r.tar.gz": body}}
base := serveVendor(t, v)
_, err := upstreamArchive(context.Background(), t.TempDir(), catalogue.Artifact{Name: "a", Kind: "archive",
URL: base + "/r.tar.gz", SHA256: sumHex(body), Version: "1"}, func(string, string, ...any) {})
if err == nil || !strings.Contains(err.Error(), "entries") {
t.Fatalf("%v", err)
}
}
func TestARedirectOffHttpsIsNotFollowed(t *testing.T) {
plain := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte("x")) }))
t.Cleanup(plain.Close)
secure := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, plain.URL+"/r", http.StatusFound)
}))
t.Cleanup(secure.Close)
before := upstreamClient
upstreamClient = secure.Client()
t.Cleanup(func() { upstreamClient = before })
_, err := upstreamFile(context.Background(), t.TempDir(), catalogue.Artifact{Name: "a", URL: secure.URL + "/r",
SHA256: sumHex([]byte("x")), Version: "1", File: "a"}, func(string, string, ...any) {})
if err == nil || !strings.Contains(err.Error(), "not https") {
t.Fatalf("%v", err)
}
}
+83 -1
View File
@@ -2,7 +2,10 @@ package catalogue
import (
"fmt"
"net"
"net/url"
"path"
"regexp"
"sort"
"strings"
)
@@ -312,7 +315,7 @@ func (b *Build) problems(module string) []string {
module, a.Name, a.System, spokenSystems()))
}
} else {
if a.From == "" {
if a.From == "" && a.URL == "" {
problems = append(problems, fmt.Sprintf(
"%s: %q says nothing about what it is built from", module, a.Name))
}
@@ -350,6 +353,85 @@ func (b *Build) problems(module string) []string {
problems = append(problems, fmt.Sprintf(
"%s: %q mirrors %q, which names no tag or digest", module, a.Name, a.From))
}
problems = append(problems, upstreamFileProblems(module, a)...)
}
return problems
}
// reservedSuffixes are names no public vendor publishes under: an installation's own and documentation's.
var reservedSuffixes = []string{".internal", ".local", ".lan", ".home", ".corp", ".localhost", ".home.arpa",
".test", ".invalid", ".example", ".intranet", ".private"}
// notAPublicHost says why a host is not a public vendor's name, or "".
func notAPublicHost(host string) string {
h := strings.ToLower(strings.TrimSuffix(host, "."))
if net.ParseIP(h) != nil {
return "an address rather than a name"
}
if !strings.Contains(h, ".") {
return "a name with no domain"
}
for _, s := range reservedSuffixes {
if strings.HasSuffix(h, s) {
return "a name under " + s + ", which is nobody's public domain"
}
}
return ""
}
// sha256Hex is a pinned sha256 as a manifest writes it: 64 lower-case hexadecimal characters.
var sha256Hex = regexp.MustCompile(`^[0-9a-f]{64}$`)
// upstreamFileProblems is what is wrong with an archive pinned by URL (novox/hq ADR 0316 §2, §6): it is an
// archive, fetched over https, pinned by a sha256 and a version, and it names no directory of the repository
// besides; a file name or a directory within is a plain path that stays where it is put.
func upstreamFileProblems(module string, a Artifact) []string {
pinned := a.URL != "" || a.SHA256 != "" || a.Version != "" || a.File != "" || a.Within != ""
if !pinned {
return nil
}
var problems []string
say := func(format string, args ...any) {
problems = append(problems, fmt.Sprintf("%s: %q ", module, a.Name)+fmt.Sprintf(format, args...))
}
if a.Kind != ArtifactArchive {
say("is a %q and names an upstream file (url, sha256, version, file, within): only an archive is "+
"mirrored from one (novox/hq ADR 0316)", a.Kind)
return problems
}
if a.URL == "" {
say("names a sha256, a version, a file or a directory within, and no url to fetch it from")
return problems
}
if a.From != "" {
say("is built from %q and fetched from %s: an archive is one or the other", a.From, a.URL)
}
if u, err := url.Parse(a.URL); err != nil || u.Scheme != "https" || u.Host == "" {
say("is fetched from %q, and an upstream file is fetched over https from a named host", a.URL)
} else if why := notAPublicHost(u.Hostname()); why != "" {
say("is fetched from %s, %s: an upstream file is somebody else's release, published under a public name", a.URL, why)
}
if !sha256Hex.MatchString(a.SHA256) {
say("is fetched from %s without a pinned sha256 (64 lower-case hexadecimal characters): what that "+
"address serves changes under the mesh, and only the digest says it is what was reviewed", a.URL)
}
if strings.TrimSpace(a.Version) == "" {
say("pins %s and says no version, so a person reading the pin cannot tell which release it is", a.URL)
}
if a.File != "" && a.Within != "" {
say("names a file (%q) and a directory within (%q): a file is kept whole, a directory is taken out "+
"of an archive", a.File, a.Within)
}
for field, p := range map[string]string{"file": a.File, "within": a.Within} {
if p == "" {
continue
}
clean := path.Clean(p)
if clean != p || strings.HasPrefix(p, "/") || clean == "." || clean == ".." || strings.HasPrefix(clean, "../") ||
(field == "file" && strings.Contains(p, "/")) {
say("names %s %q, which is not a plain relative %s", field, p, map[string]string{
"file": "file name", "within": "directory"}[field])
}
}
return problems
}
+9 -1
View File
@@ -645,6 +645,11 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
sealed = ForeseenSealed
*foreseen = append(*foreseen, m.Module+"/"+name)
}
if sealed == "" && m.OwnSecrets[name].IssuedBy == IssuedOutside {
// **Not given, and the mesh never makes it** (novox/hq issue 401, ADR 0228): no file, as for a
// family's member (ADR 0283), so the module reads its absence as not given.
continue
}
if sealed == "" {
// Declared and not made. Refused rather than skipped: a module whose own
// credential is silently absent starts, fails to authenticate, and the reason is
@@ -1002,7 +1007,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// content is where a setting lands, so a placeholder may only exist once the setting
// has been put in — filling secrets first would look at content that is not yet what
// the machine receives.
if err := intoFile(copied, sealed, m.Module); err != nil {
if err := intoFile(copied, sealed, outsideNotGiven(m, with.Needed[m.Module]), m.Module); err != nil {
return nil, err
}
// **Substituted here, not on the machine.** A bound value is not secret — the mesh
@@ -2431,6 +2436,9 @@ func secretsReadBy(resource map[string]any, m Manifest, given map[string]string)
}
paths := map[string]string{}
for name, own := range m.OwnSecrets.Plain() {
if own.IssuedBy == IssuedOutside && given[name] == "" {
continue // not given, so not placed: nothing to restart on (novox/hq issue 401)
}
paths[name] = own.Path
}
for name, sealed := range given {
+7 -1
View File
@@ -71,6 +71,9 @@ var hostname = regexp.MustCompile(
`(?:be|nl|de|fr|uk|eu|com|net|org|io|dev|app|cloud|site|online|me|co|ch|at|lu|` +
`internal|example|tld|test|invalid)`)
// upstreamVersion is where an artifact pinned by URL says which release it is (Artifact.Version).
var upstreamVersion = regexp.MustCompile(`^build\.artifacts\[[0-9]+\]\.version$`)
// address is a dotted quad.
var address = regexp.MustCompile(`(?:[0-9]{1,3}\.){3}[0-9]{1,3}`)
@@ -120,7 +123,10 @@ func InstallationProblems(m Manifest) []string {
problems = append(problems, fmt.Sprintf("%s names %s at %s", m.Module, name, at))
}
for _, ip := range addressesIn(value) {
if meant[ip] {
// An upstream pin's version may have four dotted numbers (Owl 1.5.6.1), which is no address; a name
// in it is still judged above. Its URL is judged whole: the vendor's host goes through the artifact's
// names-on-purpose like any name a definition means (novox/hq ADR 0316).
if meant[ip] || upstreamVersion.MatchString(at) {
continue
}
problems = append(problems, fmt.Sprintf("%s names the public address %s at %s", m.Module, ip, at))
+31
View File
@@ -976,6 +976,35 @@ type Artifact struct {
// success, and was invisible to the launcher (novox/hq 04-ISSUES/142).
Binary string `json:"binary,omitempty"`
// URL is where an archive's upstream file is fetched from, for an archive that is a release
// somebody else published rather than a directory of this repository (novox/hq ADR 0316 §2, §6).
// With it come SHA256 and Version, and From stays empty.
//
// **Pinned by the digest of what the vendor published, mirrored by the build seat.** A release of
// about 80 MB copied into a repository is kept for ever in its history at every upgrade, and an
// add-on whose licence forbids copying it to third parties may not be in a repository at all. So the
// manifest names the file, the build seat fetches it, refuses it when its sha256 is not the pin's and
// names both digests, and packs it as any archive is packed: what reaches a machine is the mesh's own
// archive in the artifact store, fetched by digest like every other, and a machine never asks the
// vendor. Upgrading is changing the pin, reviewed like any other change.
URL string `json:"url,omitempty"`
// SHA256 is the sha256 of the file at URL, as 64 lower-case hexadecimal characters.
SHA256 string `json:"sha256,omitempty"`
// Version is what the vendor calls the release pinned, said where a person reads the pin; the build
// does not read it.
Version string `json:"version,omitempty"`
// File is the name an upstream file that is not an archive is kept under: the archive built from it
// holds that one file (an add-on's xpi). Empty means the file is a tar archive (gzip, bzip2 or xz)
// whose files are the artifact.
File string `json:"file,omitempty"`
// Within is the directory inside an upstream tar archive whose content is the artifact, for a
// release that unpacks into a directory of its own name (`betterbird/`). An entry outside it refuses
// the build. Empty means the archive's root.
Within string `json:"within,omitempty"`
// NamesOnPurpose are the names this artifact means to name, each with why: the vendor's host an upstream
// file is fetched from, which the installation check judges like any name (InstallationProblems).
NamesOnPurpose map[string]string `json:"names-on-purpose,omitempty"`
// Entrypoints are the compiled files a tool host should load from this module, relative to the
// bundle's root.
//
@@ -1689,6 +1718,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
// facts about the catalogue and are checked at registration (CatalogueProblems).
problems = append(problems, declaredSeatProblems(m)...)
problems = append(problems, RunsAsProblems(m)...)
problems = append(problems, ReadsHolderProblems(m)...)
if m.Computed != "" && len(m.Resources) > 0 {
// One or the other. A module that both ships files and has them computed would leave
// nobody able to say where a given file came from.
@@ -1829,6 +1859,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, invokeProblems(m)...)
problems = append(problems, replacesProblems(m)...)
problems = append(problems, UnitSettingProblems(m)...)
problems = append(problems, OwnerSettingProblems(m)...)
problems = append(problems, endpointNameProblems(m)...)
problems = append(problems, RouteProblems(m)...)
for _, port := range m.Guards {
+34 -19
View File
@@ -68,9 +68,10 @@ func TestFailedIsAnOptionalVerbOfTheServiceManager(t *testing.T) {
holder := func(serves ...string) Manifest {
return Manifest{Module: "systemd", Version: "1", Claims: []Claim{{Name: ServiceManagerSeat, Scope: ScopeNode, Serves: serves}}}
}
eight := []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal"}
eight := []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal",
"reset-failed", "wanted-by", "unlink-dangling"}
if err := CanHold(holder(eight...), seat); err != nil {
t.Fatalf("today's holder, without failed, is refused: %v", err)
t.Fatalf("a holder without failed is refused: %v", err)
}
if err := CanHold(holder(append(eight, "failed")...), seat); err != nil {
t.Fatalf("a holder serving failed is refused: %v", err)
@@ -81,7 +82,7 @@ func TestFailedIsAnOptionalVerbOfTheServiceManager(t *testing.T) {
if err := CanHold(holder(eight[1:]...), seat); err == nil || !strings.Contains(err.Error(), "does not serve units") {
t.Fatalf("a holder missing a required verb was accepted: %v", err)
}
optional := map[string]bool{"failed": true, "reset-failed": true, "wanted-by": true, "unlink-dangling": true}
optional := map[string]bool{"failed": true}
for _, v := range seat.Serves {
if v.Optional != optional[v.Name] {
t.Errorf("%s optional: %v", v.Name, v.Optional)
@@ -89,26 +90,35 @@ func TestFailedIsAnOptionalVerbOfTheServiceManager(t *testing.T) {
}
}
// **`reset-failed` and `wanted-by` join the seat optional** (novox/hq issue 332, ADR 0246 step 1): the
// holder running today, serving the nine verbs, still holds the seat; a holder serving all eleven is not
// refused; each replaces the shell command it names; and read back from the store's row they stay optional.
func TestResetFailedAndWantedByAreOptionalVerbsOfTheServiceManager(t *testing.T) {
// **`reset-failed`, `wanted-by` and `unlink-dangling` are required** (novox/hq issue 332, ADR 0246 step 3):
// every holder serves them, so a holder serving only the nine verbs before them is refused by the verb it
// lacks; a holder serving all twelve holds the seat; each replaces the shell command it names; and read
// back from the store's row they stay required.
func TestResetFailedWantedByAndUnlinkDanglingAreRequiredVerbsOfTheServiceManager(t *testing.T) {
defer UseSeats(DefaultSeats())
nine := []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal", "failed"}
all := append(slices.Clone(nine), "reset-failed", "wanted-by", "unlink-dangling")
holder := func(serves ...string) Manifest {
return Manifest{Module: "systemd", Version: "1", Claims: []Claim{{Name: ServiceManagerSeat, Scope: ScopeNode, Serves: serves}}}
}
refusedWithout := func(seat Seat) {
t.Helper()
for _, verb := range []string{"reset-failed", "wanted-by", "unlink-dangling"} {
serves := slices.DeleteFunc(slices.Clone(all), func(v string) bool { return v == verb })
if err := CanHold(holder(serves...), seat); err == nil || !strings.Contains(err.Error(), "does not serve "+verb) {
t.Errorf("a holder not serving %s holds the seat: %v", verb, err)
}
}
}
seat, _ := SeatNamed(ServiceManagerSeat)
if err := CanHold(holder(nine...), seat); err != nil {
t.Fatalf("today's holder, without the two verbs, is refused: %v", err)
if err := CanHold(holder(nine...), seat); err == nil {
t.Fatal("a holder serving only the nine verbs before issue 332 holds the seat")
}
if err := CanHold(holder(append(nine, "reset-failed", "wanted-by")...), seat); err != nil {
t.Fatalf("a holder serving both is refused: %v", err)
refusedWithout(seat)
if err := CanHold(holder(all...), seat); err != nil {
t.Fatalf("a holder serving all twelve is refused: %v", err)
}
if err := CanHold(holder(append(nine, "reset-failed", "wanted-by", "unlink-dangling")...), seat); err != nil {
t.Fatalf("a holder serving unlink-dangling too is refused: %v", err)
}
if err := CanHold(holder(append(nine, "why-started")...), seat); err == nil || !strings.Contains(err.Error(), "does not promise") {
if err := CanHold(holder(append(slices.Clone(all), "why-started")...), seat); err == nil || !strings.Contains(err.Error(), "does not promise") {
t.Fatalf("a verb the seat does not promise was accepted: %v", err)
}
replaces := map[string]string{"reset-failed": "systemctl reset-failed", "wanted-by": "systemctl list-dependencies --reverse",
@@ -118,6 +128,9 @@ func TestResetFailedAndWantedByAreOptionalVerbsOfTheServiceManager(t *testing.T)
if !ok {
continue
}
if v.Optional {
t.Errorf("%s is still optional", v.Name)
}
if !slices.Contains(v.Replaces, want) {
t.Errorf("%s does not say it replaces %q: %v", v.Name, want, v.Replaces)
}
@@ -148,8 +161,9 @@ func TestResetFailedAndWantedByAreOptionalVerbsOfTheServiceManager(t *testing.T)
}
UseSeats(rows)
seat, _ = SeatNamed(ServiceManagerSeat)
if err := CanHold(holder(nine...), seat); err != nil {
t.Fatalf("read back from the row, the two verbs are required: %v", err)
refusedWithout(seat)
if err := CanHold(holder(all...), seat); err != nil {
t.Fatalf("read back from the row, a holder serving all twelve is refused: %v", err)
}
}
@@ -169,8 +183,9 @@ func TestAnOptionalVerbStaysOptionalInASetReadFromTheStore(t *testing.T) {
UseSeats(rows)
seat, _ := SeatNamed(ServiceManagerSeat)
holder := Manifest{Module: "systemd", Version: "1", Claims: []Claim{{Name: ServiceManagerSeat, Scope: ScopeNode,
Serves: []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal"}}}}
Serves: []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal",
"reset-failed", "wanted-by", "unlink-dangling"}}}}
if err := CanHold(holder, seat); err != nil {
t.Fatalf("a set read from rows refuses today's holder: %v", err)
t.Fatalf("a set read from rows refuses a holder without the optional failed: %v", err)
}
}
@@ -0,0 +1,106 @@
package catalogue
import (
"strings"
"testing"
)
// An own secret issued outside the mesh that nobody gave (novox/hq issue 401, ADR 0228): the mesh makes no value for it,
// so the composition places no file, a file of the module that names it says empty there, and nothing restarts on it.
// Given, it is placed and filled like any own secret.
func outsideManifest(t *testing.T) Manifest {
t.Helper()
m, err := ParseManifest([]byte(`{"module":"assistant","version":"1",
"own-secrets":{
"password":{"path":"/var/lib/assistant/password.secret","taken":"at-start"},
"vendor-key":{"path":"/var/lib/assistant/vendor-key.secret","taken":"at-start","issued-by":"outside"}},
"resources":[
{"id":"env","type":"file","path":"/var/lib/assistant/env","mode":"0600",
"content":"PASSWORD=${secret:password}\nVENDOR_KEY=${secret:vendor-key}\n"},
{"id":"server","type":"container","name":"assistant","image":"assistant:1",
"env-file":["/var/lib/assistant/env"],
"volumes":["/var/lib/assistant/vendor-key.secret:/run/vendor-key:ro"],
"secrets-in-environment":"reads its settings from the environment only"}]}`))
if err != nil {
t.Fatal(err)
}
return m
}
func resourceAt(out []map[string]any, path string) map[string]any {
for _, r := range out {
if r["path"] == path {
return r
}
}
return nil
}
func resourceNamed(out []map[string]any, name string) map[string]any {
for _, r := range out {
if r["name"] == name {
return r
}
}
return nil
}
func TestAnOutsideSecretNotGivenIsNoFileAndAFileNamingItSaysEmpty(t *testing.T) {
r := Resolution{Node: "server", Modules: []Manifest{outsideManifest(t)}}
out, err := r.Declaration(Rendering{Needed: map[string]map[string]string{"assistant": {"password": "sealed-password"}}})
if err != nil {
t.Fatalf("a module whose outside secret nobody gave did not compose: %v", err)
}
if got := resourceAt(out, "/var/lib/assistant/vendor-key.secret"); got != nil {
t.Fatalf("a secret nobody gave was placed: %v", got)
}
env := resourceAt(out, "/var/lib/assistant/env")
if env == nil {
t.Fatal("the module's own file is not composed")
}
if content := env["content"].(string); content != "PASSWORD=${secret:password}\nVENDOR_KEY=\n" {
t.Fatalf("the file names the secret nobody gave as %q", content)
}
secrets, _ := env["secrets"].(map[string]any)
if len(secrets) != 1 || secrets["password"] != "sealed-password" {
t.Fatalf("the file is filled from %v", secrets)
}
server := resourceNamed(out, "assistant")
if on := strings.Join(stringsIn(server["restart-on"]), " "); strings.Contains(on, "vendor-key") {
t.Fatalf("the container restarts on a secret nobody gave: %q", on)
}
}
func TestAnOutsideSecretGivenIsPlacedAndFilled(t *testing.T) {
r := Resolution{Node: "server", Modules: []Manifest{outsideManifest(t)}}
out, err := r.Declaration(Rendering{Needed: map[string]map[string]string{"assistant": {
"password": "sealed-password", "vendor-key": "sealed-vendor-key"}}})
if err != nil {
t.Fatal(err)
}
placed := resourceAt(out, "/var/lib/assistant/vendor-key.secret")
if placed == nil || placed["sealed"] != "sealed-vendor-key" {
t.Fatalf("the secret given is placed as %v", placed)
}
env := resourceAt(out, "/var/lib/assistant/env")
if content := env["content"].(string); content != "PASSWORD=${secret:password}\nVENDOR_KEY=${secret:vendor-key}\n" {
t.Fatalf("the file was changed: %q", content)
}
if secrets, _ := env["secrets"].(map[string]any); secrets["vendor-key"] != "sealed-vendor-key" {
t.Fatalf("the file is filled from %v", secrets)
}
server := resourceNamed(out, "assistant")
if on := strings.Join(stringsIn(server["restart-on"]), " "); !strings.Contains(on, "assistant."+NeedID("vendor-key")) {
t.Fatalf("the container does not restart on the secret given: %q", on)
}
}
// A secret the mesh makes is still refused when it was not made: only an outside one is left out.
func TestAnOwnSecretTheMeshMakesIsStillRefusedWhenNotMade(t *testing.T) {
r := Resolution{Node: "server", Modules: []Manifest{outsideManifest(t)}}
_, err := r.Declaration(Rendering{})
if nm, ok := err.(*NotMadeError); !ok || nm.Name != "password" {
t.Fatalf("a secret the mesh makes, not made: %v", err)
}
}
+220
View File
@@ -0,0 +1,220 @@
package catalogue
import (
"encoding/json"
"fmt"
"path/filepath"
"sort"
"strings"
)
// A node's files and accounts, read (novox/hq ADR 0310, extending ADR 0268 §5; hq to-be 41 WP8).
//
// **A seat of its own on every node.** Reading a file, searching one, listing a directory, the owner and mode
// of each component of a path, and looking up an account or a group are the reads ADR 0268 §5 has narrow tools
// take over from the login shell's `execute`, so an approval is asked for an act and never for a read. Every
// node needs them, the control-node included, and another module could offer them instead of the tool
// runner's (ADR 0126 §9): a seat, so the mesh finds them on every node by one name.
//
// **Read as an account that holds no secret.** The holder runs as an account of its own (runs-as, ADR 0259
// §8), so the node's file modes bound what it sees — never the operator's account, which sees the
// operator's keys and the agent's credentials. On top, it refuses by name the places a secret is kept,
// which the controller names for it (secretPlaces), and every answer passes the content rule's secret class
// (mesh-sdk's outward, ADR 0310 §4).
// ReadsSeat is the role that reads a node's files and accounts (novox/hq ADR 0310).
const ReadsSeat = "node-reads"
// ReadsSecretPlaces is the word the holder of node-reads is given: the places on its node where a secret is
// kept, as a JSON list of absolute paths, each refused by name before any read (novox/hq ADR 0310 §5).
const ReadsSecretPlaces = "MESH_SECRET_PLACES"
// readsVerbs is the contract every holder of node-reads serves (novox/hq ADR 0310 §2). The bounds are the
// build's to set and each verb's description to state (ADR 0310, Consequences); the holder states them in
// every answer it cuts.
func readsVerbs() []Verb {
return []Verb{
{Name: "stat", Description: "Each component of one path or several, from / down: its type, owner and group " +
"(name and number), mode, whether it is a link and to what, and whether it is a mount point and of which " +
"file system. A link is not followed unless follow is \"true\". Read as the holder's own account, which " +
"holds no secret: a component it may not reach says so. At most 200 paths in one call.",
Input: schema(map[string]string{
"paths": "one absolute path, or several separated by line breaks",
"follow": "\"true\": also say where the path leads once every link is followed",
}, []string{"paths"}, "follow"),
Replaces: []string{"stat", "ls -ld", "namei -l", "findmnt -T"}},
{Name: "readlink", Description: "What one link points to, as written, and where the path leads once every " +
"link is followed, and whether that exists.",
Input: schema(map[string]string{"path": "an absolute path"}, []string{"path"}),
Replaces: []string{"readlink", "readlink -f", "realpath"}},
{Name: "list", Description: "Every entry of one directory with its type, owner, group, mode, size and, for a " +
"link, its target. At most 2000 entries; a longer directory is cut and says so. A place a secret is kept " +
"is refused by name.",
Input: schema(map[string]string{"path": "an absolute path to a directory"}, []string{"path"}),
Replaces: []string{"ls", "ls -la"}},
{Name: "read", Description: "One text file, or a part of it by line: at most 256 KiB in one answer " +
"(64 KiB unless more is asked), cut at a line and said so. A binary file is refused; a place a secret is " +
"kept, or a known credential file, is refused by name; a line shaped like a secret is withheld and its " +
"number said. /proc and /sys are read as far as the holder's account may.",
Input: schema(map[string]string{
"path": "an absolute path to a file",
"from": "the first line to answer, counted from 1 (optional)",
"lines": "how many lines to answer (optional)",
"bytes": "the most bytes to answer, up to 262144 (optional, 65536 unless given)",
}, []string{"path"}),
Replaces: []string{"cat", "head", "tail", "less"}},
{Name: "search", Description: "Find a fixed string (never a pattern) in one file or under one directory: " +
"every matching line with its file and number. Bounded: at most 500 matches (100 unless more is asked), " +
"8 directories deep (4 unless more is asked), 5000 files and 4 MiB per file; links are not followed and " +
"binary files are skipped. A place a secret is kept is skipped and named; a matching line shaped like a " +
"secret is withheld. The answer says which bound it met.",
Input: schema(map[string]string{
"text": "the fixed string to find",
"path": "an absolute path to a file or a directory",
"depth": "how many directories deep, up to 8 (optional)",
"matches": "the most matches to answer, up to 500 (optional)",
"case": "\"false\": match whatever the case (optional)",
}, []string{"text", "path"}, "case"),
Replaces: []string{"grep", "grep -r", "grep -F"}},
{Name: "account", Description: "One account or group by name or number, as this node's account files say: " +
"an account's number, group, home, shell and every group it is in; a group's number and members. With " +
"neither, every account and group. Never a password field.",
Input: schema(map[string]string{
"name": "an account's name or number (optional)",
"group": "a group's name or number (optional)",
}, nil),
Replaces: []string{"getent passwd", "getent group", "id", "groups"}},
}
}
// ReadsHolderProblems is what a holder of node-reads is held to (novox/hq ADR 0310 §3): it reads as an
// account of its own, so the node's file modes bound what it sees. Carried by the machine's runtime it would
// read as the operator's account, which sees the operator's keys and the agent's credentials.
func ReadsHolderProblems(m Manifest) []string {
if !m.ClaimsSeat(ReadsSeat) {
return nil
}
if m.RunsAs == "" {
return []string{fmt.Sprintf("%s claims %s and must run as an account of its own (runs-as): its reads are "+
"bounded by that account's file modes, and the machine's runtime runs as the operator's account "+
"(novox/hq ADR 0310 §3)", m.Module, ReadsSeat)}
}
// In no group beside its own: a group is a way to a secret (docker's is root's), and the account's file modes
// are what bound its reads.
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "user" || fmt.Sprint(r["name"]) != m.RunsAs {
continue
}
if groups, said := r["groups"]; said {
if list, ok := groups.([]any); !ok || len(list) > 0 {
return []string{fmt.Sprintf("%s claims %s, and its account %s is put in groups %v: the account that "+
"reads is in no group beside its own (novox/hq ADR 0310 §3)", m.Module, ReadsSeat, m.RunsAs, groups)}
}
}
}
return nil
}
// secretPlaces is every place on this node a secret is kept, as the placed manifests say (novox/hq ADR 0310
// §5): the directory of every module's own secret, every credential file a module is given and every grant
// directory a provider is handed its consumers' credentials in, the place a module keeps the operator-sealed
// secrets, and every directory of the module that provides `secret` — the vault — wherever the node places it,
// a directory declared without a path included. Sorted, each once, and a place under another dropped, so the
// list says each refusal once.
//
// **Failure is loud.** A place still holding a placeholder, or not absolute, fails the composition with the
// module named: a list that silently shrank would be a refusal the decision promises and nothing does.
func (r Resolution) secretPlaces(with Rendering) ([]string, error) {
seen := map[string]bool{}
var problems []string
add := func(module, what, path string) {
path = strings.TrimSpace(path)
if path == "" {
return
}
if !strings.HasPrefix(path, "/") || strings.Contains(path, "${") {
problems = append(problems, fmt.Sprintf("%s's %s is %q, which is not a place on %s", module, what, path, r.Node))
return
}
path = filepath.Clean(path)
if path == "/" {
problems = append(problems, fmt.Sprintf("%s's %s is the root directory", module, what))
return
}
seen[path] = true
}
for _, m := range r.Modules {
for name, own := range m.OwnSecrets {
add(m.Module, "own secret "+name, own.Path)
if strings.HasPrefix(own.Path, "/") {
add(m.Module, "own secret "+name+"'s directory", filepath.Dir(own.Path))
}
}
for _, to := range m.SecretRequirements() {
for _, f := range m.SecretFiles(to) {
add(m.Module, "credential file for "+to, f.Path)
}
}
for to, where := range m.Grants {
add(m.Module, "grant directory for "+to, where)
}
add(m.Module, "place for the operator-sealed secrets (keeps)", m.Keeps)
if providesSecret(m) {
dirs := dirsFor(m, with)
for _, id := range sortedKeys(dirs) {
add(m.Module, "directory "+id, dirs[id])
}
}
}
if len(problems) > 0 {
sort.Strings(problems)
return nil, fmt.Errorf("the holder of %s on %s cannot be told where a secret is kept, so it would read "+
"one (novox/hq ADR 0310 §5): %s", ReadsSeat, r.Node, strings.Join(problems, "; "))
}
all := make([]string, 0, len(seen))
for p := range seen {
all = append(all, p)
}
sort.Strings(all)
var out []string
for _, p := range all {
if n := len(out); n > 0 && strings.HasPrefix(p, out[n-1]+"/") {
continue
}
out = append(out, p)
}
return out, nil
}
func providesSecret(m Manifest) bool {
for _, p := range m.Provides {
if p.Name == "secret" {
return true
}
}
return false
}
// readsWords is what the holder of node-reads is given beside its own words: the node's secret places.
func (r Resolution) readsWords(m Manifest, words map[string]string, with Rendering) (map[string]string, error) {
if !m.ClaimsSeat(ReadsSeat) {
return words, nil
}
places, err := r.secretPlaces(with)
if err != nil {
return nil, err
}
body, err := json.Marshal(places)
if err != nil {
return nil, err
}
if words == nil {
words = map[string]string{}
}
if _, has := words[ReadsSecretPlaces]; has {
return nil, fmt.Errorf("%s gives its own %s; the node's secret places are the mesh's to say (novox/hq ADR 0310 §5)",
m.Module, ReadsSecretPlaces)
}
words[ReadsSecretPlaces] = string(body)
return words, nil
}
+136
View File
@@ -0,0 +1,136 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// novox/hq ADR 0310 §1, §2: node-reads is a node seat, and its verbs say the commands they replace, so
// mesh_search finds them by `cat`, `grep` and `getent passwd` (ADR 0245 §3).
func TestNodeReadsIsANodeSeatWhoseVerbsSayWhatTheyReplace(t *testing.T) {
s, ok := SeatNamed(ReadsSeat)
if !ok {
t.Fatalf("the mesh defines no seat %s", ReadsSeat)
}
if s.Scope != ScopeNode || s.Delivers != "" || !strings.Contains(s.Decision, "0310") {
t.Errorf("%s: scope %q, delivers %q, decision %q", ReadsSeat, s.Scope, s.Delivers, s.Decision)
}
replaced := map[string]string{}
for _, v := range s.Serves {
if v.Optional {
t.Errorf("%s.%s is optional: a new seat's holder serves every verb", ReadsSeat, v.Name)
}
if len(v.Replaces) == 0 {
t.Errorf("%s.%s says nothing it replaces", ReadsSeat, v.Name)
}
for _, c := range v.Replaces {
replaced[c] = v.Name
}
}
for command, verb := range map[string]string{"cat": "read", "grep": "search", "getent passwd": "account",
"getent group": "account", "id": "account", "stat": "stat", "readlink": "readlink", "ls": "list"} {
if replaced[command] != verb {
t.Errorf("%q is replaced by %q, want %s.%s", command, replaced[command], ReadsSeat, verb)
}
}
}
// novox/hq ADR 0310 §3: a holder of node-reads reads as an account of its own; one the machine's runtime would
// carry — as the operator's account — is refused at parse.
func TestAHolderOfNodeReadsWithoutAnAccountOfItsOwnIsRefused(t *testing.T) {
_, err := ParseManifest([]byte(`{"module": "reads", "version": "1",
"claims": [{"name": "node-reads", "scope": "node"}]}`))
if err == nil || !strings.Contains(err.Error(), "runs-as") || !strings.Contains(err.Error(), "0310") {
t.Errorf("a holder of node-reads without runs-as was not refused for it: %v", err)
}
if p := ReadsHolderProblems(Manifest{Module: "reads", RunsAs: "mesh-reads",
Claims: []Claim{{Name: ReadsSeat}}}); len(p) != 0 {
t.Errorf("a holder running as its own account was refused: %v", p)
}
}
// novox/hq ADR 0310 §5: the holder is told, by the composition, where its node keeps a secret — the directory
// of every module's own secret, every credential file and grant directory, the vault's directories and the
// place the operator-sealed secrets are kept — and no other module is told.
func TestTheHolderOfNodeReadsIsToldWhereItsNodeKeepsASecret(t *testing.T) {
with := Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"},
"reads": {"broker": "own"}, "desk": {"broker": "own"}}}
goRuntime := Manifest{Module: RuntimeModule, Version: "1",
OwnSecrets: OwnSecrets{"broker": {Path: "/var/lib/mesh/" + RuntimeModule + "/broker"}},
Build: &Build{Artifacts: []Artifact{{Name: "runtime", Kind: ArtifactBundle, Language: "go",
System: "arch", From: "cmd/node-tools"}}}}
goRuntime, err := goRuntime.Resolve([]Built{{Name: "runtime", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + RuntimeModule + "/runtime/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
reads := aToolsModule(t, "reads", "tools/index.js")
reads.RunsAs, reads.SecretsOwner = "mesh-reads", "mesh-reads"
reads.Claims = []Claim{{Name: ReadsSeat}}
reads.OwnSecrets = OwnSecrets{"broker": {Path: "/var/lib/mesh-reads/broker"}}
desk := aToolsModule(t, "desk", "tools/index.js")
desk.RunsAs, desk.SecretsOwner = "desk", "desk"
desk.OwnSecrets = OwnSecrets{"broker": {Path: "/var/lib/desk/broker"}}
// The vault as the catalogue declares it: its directories have no path of their own, and the node places them.
vault, err := placedManifest(catalogueManifest(t, "mesh-vault"), with)
if err != nil {
t.Fatal(err)
}
app := Manifest{Module: "app", Version: "1", Secrets: map[string]string{"postgres-database": "/etc/app/db.secret"},
OwnSecrets: OwnSecrets{"token": {Path: "/home/ops/.config/app/token"}}}
r := Resolution{Node: "anchor", Account: "ops", Modules: []Manifest{reads, desk, vault, app, goRuntime}}
owns, err := r.ownRuntimes(with)
if err != nil {
t.Fatal(err)
}
var given map[string]map[string]string
for _, p := range owns {
if p["id"] == "reads."+OwnRuntimeID() {
if err := json.Unmarshal([]byte(p["env"].(map[string]string)[RuntimeToolEnv]), &given); err != nil {
t.Fatal(err)
}
}
if p["id"] == "desk."+OwnRuntimeID() && strings.Contains(p["env"].(map[string]string)[RuntimeToolEnv], ReadsSecretPlaces) {
t.Error("a module not holding node-reads is told the node's secret places")
}
}
var places []string
if err := json.Unmarshal([]byte(given["reads"][ReadsSecretPlaces]), &places); err != nil {
t.Fatalf("the holder is not told its node's secret places: %v (%v)", err, given)
}
want := []string{"/etc/app/db.secret", "/home/ops/.config/app", "/var/lib/desk", "/var/lib/mesh-reads",
"/var/lib/mesh-vault", "/var/lib/mesh/" + RuntimeModule}
if strings.Join(places, " ") != strings.Join(want, " ") {
t.Errorf("secret places:\n got %v\nwant %v", places, want)
}
}
// The review of #257: a secret place the controller cannot resolve fails the composition, naming the module,
// rather than leaving the list one place shorter.
func TestASecretPlaceThatCannotBeResolvedFailsTheComposition(t *testing.T) {
r := Resolution{Node: "anchor", Modules: []Manifest{
{Module: "app", Version: "1", Keeps: "${dir:keep}/sealed"},
}}
if _, err := r.secretPlaces(Rendering{}); err == nil || !strings.Contains(err.Error(), "app") {
t.Errorf("an unresolved place was dropped: %v", err)
}
r.Modules[0].Keeps = "relative/place"
if _, err := r.secretPlaces(Rendering{}); err == nil {
t.Error("a relative place was dropped")
}
}
// ADR 0310 §3: the account that reads is in no group beside its own; one put in a group is refused.
func TestTheReadsAccountIsInNoGroup(t *testing.T) {
m := Manifest{Module: "reads", RunsAs: "mesh-reads", Claims: []Claim{{Name: ReadsSeat}},
Resources: []map[string]any{{"id": "account", "type": "user", "name": "mesh-reads", "groups": []any{"docker"}}}}
if p := ReadsHolderProblems(m); len(p) != 1 || !strings.Contains(p[0], "docker") {
t.Errorf("an account in docker was not refused: %v", p)
}
m.Resources[0]["groups"] = []any{}
if p := ReadsHolderProblems(m); len(p) != 0 {
t.Errorf("an account in no group was refused: %v", p)
}
}
+4
View File
@@ -303,6 +303,10 @@ func (r Resolution) ownRuntimes(with Rendering) ([]map[string]any, error) {
if err != nil {
return nil, err
}
// The holder of node-reads is told where this node keeps its secrets (novox/hq ADR 0310 §5).
if words, err = r.readsWords(m, words, with); err != nil {
return nil, err
}
if len(words) > 0 {
body, err := json.Marshal(map[string]map[string]string{m.Module: words})
if err != nil {
+10 -7
View File
@@ -310,6 +310,9 @@ var defaultSeats = append([]Seat{
// node-mounts writes a mount and an automount unit per share on a machine that asks (shares.go).
{Name: NFSServerSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0263", Serves: nfsServerVerbs()},
{Name: MountsSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0263", Serves: mountsVerbs()},
// A node's files and accounts, read (novox/hq ADR 0310, reads.go): by an account of the holder's own that holds
// no secret, on every node, so the reads ADR 0268 §5 takes over from the login shell's execute have a tool.
{Name: ReadsSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0310", Serves: readsVerbs()},
},
// The graphical session's roles (novox/hq ADR 0208), last because they are a workstation's.
graphicalSessionSeats()...)
@@ -678,13 +681,13 @@ func serviceManagerVerbs() []Verb {
Replaces: []string{"systemctl --failed"}},
// **Clearing a failed record, and finding what starts a unit** (novox/hq issue 332): a unit whose
// file is gone stays failed in its manager until the record is reset, and what keeps asking for it
// is a dependency or an enable link no verb could show. Both optional while their holders catch up
// (ADR 0246 step 1); a later change requires them.
{Name: "reset-failed", Optional: true, Description: "Clear one unit's failed record in its manager — " +
// is a dependency or an enable link no verb could show. Required since every holder serves them
// (ADR 0246 step 3).
{Name: "reset-failed", Description: "Clear one unit's failed record in its manager — " +
"its failed state and its start-limit count — and answer its state after. It removes no file and " +
"starts nothing.",
Input: scoped(unit, []string{"unit"}), Replaces: []string{"systemctl reset-failed"}},
{Name: "wanted-by", Optional: true, Description: "What wants, requires or triggers one unit, read-only: the " +
{Name: "wanted-by", Description: "What wants, requires or triggers one unit, read-only: the " +
"manager's reverse dependencies (WantedBy, RequiredBy, UpheldBy, BoundBy, TriggeredBy and the rest) and " +
"every enable link naming it in the scope's configuration directories (*.wants, *.requires, *.upholds), " +
"each with where it points and whether it dangles — the account's own directory included in user scope.",
@@ -692,9 +695,9 @@ func serviceManagerVerbs() []Verb {
Replaces: []string{"systemctl list-dependencies --reverse", "systemctl show -p WantedBy",
"ls ~/.config/systemd/user/*.wants"}},
// **A dangling enable link removed on the person's word** (novox/hq issue 332): `systemctl disable`
// leaves a link whose unit file is gone, so the manager keeps asking for the unit. Optional until its
// holders serve it (ADR 0246 step 1).
{Name: "unlink-dangling", Optional: true, Description: "Remove the dangling enable links named for one " +
// leaves a link whose unit file is gone, so the manager keeps asking for the unit. Required since
// every holder serves it (ADR 0246 step 3).
{Name: "unlink-dangling", Description: "Remove the dangling enable links named for one " +
"unit — links in a *.wants, *.requires or *.upholds directory of the account's or the machine's own " +
"configuration whose target does not exist, which `disable` leaves once the unit's file is gone — on " +
"the person's word: `why` is required. Each is recorded (its path, its target, the reason) before it is " +
+3 -3
View File
@@ -48,7 +48,7 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
delivered[s.Delivers] = s.Name
}
}
// Forty with node-nfs-server and node-mounts (novox/hq ADR 0263); thirty-eight with node-resolver (novox/hq ADR 0247); thirty-seven with mesh-delivery (novox/hq ADR 0239); thirty-six since node-resolver-config retired
// Forty-one with node-reads (novox/hq ADR 0310); forty with node-nfs-server and node-mounts (novox/hq ADR 0263); thirty-eight with node-resolver (novox/hq ADR 0247); thirty-seven with mesh-delivery (novox/hq ADR 0239); thirty-six since node-resolver-config retired
// into node-uplink (novox/hq ADR 0223); thirty-seven
// since the retired node-dns-resolver went (novox/hq ADR 0220); thirty-eight with
// node-backup (novox/hq ADR 0214); thirty-seven with node-message-bus (novox/hq ADR 0215);
@@ -59,8 +59,8 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
// node-container-runtime (ADR 0207); nineteen with node-environment and node-login-shell (ADR 0203,
// ADR 0204); seventeen with node-build-agent (ADR 0190). One fewer once the retired
// mesh-build-machine row goes, when no registered manifest claims it.
if len(Seats()) != 40 {
t.Errorf("the mesh defines %d seats rather than 40; the set is closed, so a change here is "+
if len(Seats()) != 41 {
t.Errorf("the mesh defines %d seats rather than 41; the set is closed, so a change here is "+
"a decision (novox/hq ADR 0110): %s", len(Seats()), seatNames())
}
}
+25 -2
View File
@@ -156,7 +156,11 @@ func secretFilesOf(resources []map[string]any) map[string]bool {
// A name the module never declared is refused here rather than on the machine. The host would
// refuse it too — but it would do so having already been handed a declaration, which reads as the
// mesh sending something broken, and the name it could not find is a manifest's typo.
func intoFile(resource map[string]any, sealed map[string]string, module string) error {
//
// **An own secret issued outside the mesh that nobody gave is filled with nothing** (novox/hq issue 401): the mesh
// never makes one (ADR 0228), and a file is where the module reads it, so the place says empty — the module sees it
// was not given, where a value the mesh made looked given and wrong. notGiven names those secrets.
func intoFile(resource map[string]any, sealed map[string]string, notGiven map[string]bool, module string) error {
if fmt.Sprint(resource["type"]) != "file" {
return nil
}
@@ -171,6 +175,10 @@ func intoFile(resource map[string]any, sealed map[string]string, module string)
into := map[string]any{}
for _, name := range used {
value := sealed[name]
if value == "" && notGiven[name] {
content = strings.ReplaceAll(content, "${secret:"+name+"}", "")
continue
}
if value == "" {
return fmt.Errorf(
"%s has a file that says ${secret:%s}, and %s has no secret of its own by that "+
@@ -179,10 +187,25 @@ func intoFile(resource map[string]any, sealed map[string]string, module string)
}
into[name] = value
}
resource["secrets"] = into
resource["content"] = content
if len(into) > 0 {
resource["secrets"] = into
}
return nil
}
// outsideNotGiven is each own secret of a module issued outside the mesh that this composition holds no value for
// (novox/hq issue 401): what intoFile fills with nothing, and what nothing restarts on.
func outsideNotGiven(m Manifest, given map[string]string) map[string]bool {
out := map[string]bool{}
for name, own := range m.OwnSecrets.Plain() {
if own.IssuedBy == IssuedOutside && given[name] == "" {
out[name] = true
}
}
return out
}
// namesOr says what a module could have written, because the answer to "that name is wrong" is
// almost always one of two or three right ones.
func namesOr(sealed map[string]string) string {
+181
View File
@@ -0,0 +1,181 @@
package catalogue
import (
"strings"
"testing"
)
// A declared directory may be owned by the account its module runs as (novox/hq issue 279): a media
// manager runs as the puid and pgid its assignment sets per node, and a folder it must write was owned
// by the manifest's literal, so the app refused it on a machine whose identity was not the image's.
func recycler() Manifest {
return Manifest{Module: "sonarr",
Resources: []map[string]any{
{"id": "recycle-bin", "type": "directory", "path": "/srv/sonarr/recycle", "mode": "0755",
"owner": "${setting:puid}:${setting:pgid}"},
{"id": "identity", "type": "file", "path": "/srv/sonarr/identity.env", "mode": "0644",
"content": "PUID=${setting:puid}\nPGID=${setting:pgid}\n"},
},
}
}
func copied(r map[string]any) map[string]any {
out := map[string]any{}
for k, v := range r {
out[k] = v
}
return out
}
// The node's identity reaches the directory's owner, the node's layer over the mesh's, a number given
// as a string or as a JSON number alike.
func TestASettingOwnsADirectory(t *testing.T) {
m := recycler()
layers := []Layer{
{From: "the mesh", Values: map[string]any{"puid": "1000", "pgid": "1000"}},
{From: "this node", Values: map[string]any{"puid": float64(1001), "pgid": "2000"}},
}
r := copied(m.Resources[0])
if err := settingInto(r, layers, m.Module); err != nil {
t.Fatal(err)
}
if r["owner"] != "1001:2000" {
t.Fatalf("owned by %q, not 1001:2000", r["owner"])
}
user := copied(m.Resources[0])
user["owner"] = "${setting:puid}"
if err := settingInto(user, layers, m.Module); err != nil || user["owner"] != "1001" {
t.Fatalf("user alone: %v, %v", user["owner"], err)
}
}
// Composed for a machine, the way a push writes it: the directory reaches the node-engine owned by the
// number the node's settings name, never by the placeholder.
func TestAComposedMachineGetsTheOwnerTheSettingNames(t *testing.T) {
r := anAdoptedAnchor()
r.Modules = append(r.Modules, recycler())
with := anchorRendering(false)
with.Settings["sonarr"] = []Layer{{From: "anchor", Values: map[string]any{"puid": "1001", "pgid": "2000"}}}
composed, err := r.Compose(with)
if err != nil {
t.Fatal(err)
}
if why, left := composed.LeftOut["sonarr"]; left {
t.Fatalf("left out: %s", why)
}
got := byID(composed.Resources)["sonarr.recycle-bin"]
if got["owner"] != "1001:2000" {
t.Fatalf("composed owned by %v, not 1001:2000", got["owner"])
}
}
// Only a whole number owns a directory: an account's name, text, a sign, a fraction or a value past the
// kernel's ids is refused by name, and the directory is left as it was. So is a key nothing sets.
func TestAnOwnerSettingThatIsNoNumberIsRefused(t *testing.T) {
m := recycler()
for _, bad := range []any{"abc", "root", "", "-1", "+5", "1000:1000", "10 00", "1000\nx", float64(1.5),
float64(-3), "4294967295", "99999999999", true} {
r := copied(m.Resources[0])
err := settingInto(r, []Layer{{From: "ace", Values: map[string]any{"puid": bad, "pgid": "1000"}}}, m.Module)
if err == nil || !strings.Contains(err.Error(), `"puid"`) || !strings.Contains(err.Error(), "whole number") {
t.Errorf("%v: %v", bad, err)
}
if r["owner"] != m.Resources[0]["owner"] {
t.Errorf("%v: the owner was changed on refusal: %v", bad, r["owner"])
}
}
r := copied(m.Resources[0])
if err := settingInto(r, nil, m.Module); err == nil || !strings.Contains(err.Error(), "${setting:puid}") {
t.Errorf("nothing set: %v", err)
}
if err := JudgeSettings(m, []Layer{{From: "ace", Values: map[string]any{"puid": "media", "pgid": "1000"}}}, false); err == nil ||
!strings.Contains(err.Error(), "whole number") {
t.Errorf("a stored name is not refused when judged: %v", err)
}
}
// Where the manifest is read: an owner holding a setting is a whole placeholder or a whole number on
// each side of the colon, and every key is one the module asks for elsewhere or declares a default for.
func TestAnOwnerSettingIsJudgedWhereTheManifestIsRead(t *testing.T) {
ok := []string{"${setting:puid}:${setting:pgid}", "${setting:puid}", "${setting:puid}:1000", "1000:${setting:pgid}",
"1000:1000", "media:media"}
bad := []string{
"${setting:puid}x",
"x${setting:puid}",
"${setting:puid}:${setting:pgid}:1",
"${setting:puid}${setting:pgid}",
"media:${setting:pgid}",
"${setting:puid}:",
"${setting:other}:${setting:pgid}",
}
for _, owner := range append(ok, bad...) {
m := recycler()
m.Resources[0] = copied(m.Resources[0])
m.Resources[0]["owner"] = owner
err := parsed(t, m)
refused := err != nil && strings.Contains(err.Error(), "owner")
want := false
for _, b := range bad {
want = want || b == owner
}
if refused != want {
t.Errorf("%s: refused %v, want %v (%v)", owner, refused, want, err)
}
}
// A key with a default is declared, though nothing else asks for it.
m := recycler()
m.Resources = m.Resources[:1]
m.Resources[0] = copied(m.Resources[0])
m.Resources[0]["owner"] = "${setting:workers}"
m.Settings = map[string]SettingDeclaration{"workers": {Kind: KindPreference, Default: float64(4), Why: "a test"}}
if err := parsed(t, m); err != nil && strings.Contains(err.Error(), "owner") {
t.Errorf("a declared key was refused: %v", err)
}
// A file's owner takes no setting: the sweep says so.
f := recycler()
f.Resources[1] = copied(f.Resources[1])
f.Resources[1]["owner"] = "${setting:puid}"
if err := parsed(t, f); err == nil || !strings.Contains(err.Error(), "not filled in this field") {
t.Errorf("a file's owner setting passed: %v", err)
}
}
// A key a directory's owner asks for is a destination, so setting it is not called stray.
func TestASettingAnOwnerAsksForIsNotStray(t *testing.T) {
m := recycler()
m.Resources = m.Resources[:1]
stray := strings.Join(UnusedSettings(m, []Layer{{From: "ace", Values: map[string]any{"puid": "1001", "pgid": "2000"}}}), "; ")
if strings.Contains(stray, "puid") || strings.Contains(stray, "pgid") {
t.Errorf("called stray: %s", stray)
}
}
// An owner is written as the number parsed: "0001" and "00" reach the node-engine as 1 and 0, never as text.
// Uid 0 is allowed, because a key an owner asks for is the terminal's alone.
func TestAnOwnerSettingIsWrittenAsTheNumber(t *testing.T) {
m := recycler()
r := copied(m.Resources[0])
if err := settingInto(r, []Layer{{From: "ace", Values: map[string]any{"puid": "0001", "pgid": "00"}}}, m.Module); err != nil {
t.Fatal(err)
}
if r["owner"] != "1:0" {
t.Fatalf("owned by %q, not 1:0", r["owner"])
}
}
// Every key a directory's owner asks for is the controller's terminal's alone (novox/hq issue 279, issue 339):
// root chowns the folder to it, so it is terminal-only whatever else asks for it — a file nothing trusts, a
// default, or nothing at all.
func TestAnOwnersKeyIsTheTerminals(t *testing.T) {
m := recycler()
m.Resources[1] = copied(m.Resources[1])
m.Resources[1]["trusted"] = false
m.Settings = map[string]SettingDeclaration{"pgid": {Kind: KindPreference, Default: "1000", Why: "a test"}}
keys := strings.Join(TerminalKeys(m), ",")
for _, k := range []string{"puid", "pgid"} {
if !strings.Contains(","+keys+",", ","+k+",") {
t.Errorf("%s is not the terminal's: %s", k, keys)
}
}
}
+125 -3
View File
@@ -4,6 +4,7 @@ import (
"fmt"
"regexp"
"sort"
"strconv"
"strings"
)
@@ -15,7 +16,8 @@ import (
// (novox/hq issues 122, 134). ADR 0112 names the operator as one of the four providers; this is the
// operator answering.
//
// `${setting:<key>}` in a file's content, and in a service's unit name, is filled from the module's settings layers — the mesh's,
// `${setting:<key>}` in a file's content, in a service's unit name and in a directory's owner (a uid or
// gid only, issue 279) is filled from the module's settings layers — the mesh's,
// then this node's — the same layers a mergeable JSON file and a contribution already take, so
// `settings set <module>` is the one place a person's values go. **Refused when no layer sets it**,
// naming the key and the remedy: a definition that carried a default for a mail domain would be
@@ -59,6 +61,9 @@ func settingInto(resource map[string]any, layers []Layer, module string) error {
if fmt.Sprint(resource["type"]) == "service" {
return settingIntoUnit(resource, layers, module)
}
if fmt.Sprint(resource["type"]) == "directory" {
return settingIntoOwner(resource, layers, module)
}
if fmt.Sprint(resource["type"]) != "file" {
return nil
}
@@ -143,6 +148,107 @@ func settingIntoUnit(resource map[string]any, layers []Layer, module string) err
return nil
}
// A declared directory may be owned by the account its module runs as (novox/hq issue 279).
//
// A containerised module runs as the identity its assignment sets per node — its puid and pgid
// settings, written into the environment file its image reads (ADR 0112, issue 153) — and a directory
// its app must write, beyond the one its image re-owns at start, was owned by whatever the manifest
// spelt: right on a machine whose identity matched the image's default, refused by the app on one whose
// did not. So a directory's `owner` may name the settings that identity is: `${setting:<key>}` for the
// user alone, `${setting:<user>}:${setting:<group>}` for both, filled from the same layers at
// composition.
//
// **Whole numbers only.** An owner a setting fills is a uid or a gid — never an account's name, never
// text — so no value can put anything into the ownership the node-engine applies but a number; one that
// is not is refused at composition, naming the key, and the directory is left as it was. The shape is
// judged where the manifest is read (OwnerSettingProblems): each side of the colon a whole placeholder
// or a whole number, and every key one the module itself asks for elsewhere — in a file, a unit, a
// contribution or a served fact — or declares a default for, so the owner follows the identity its
// process is told it runs as, never a key that reaches nothing else.
// ownerSide is one side of an owner a setting may fill: a whole placeholder or a whole number.
var ownerSide = `(?:\$\{setting:[a-z0-9][a-z0-9_.-]*\}|[0-9]+)`
// ownerWithSetting is the only shape an owner holding a setting may have: user, or user:group.
var ownerWithSetting = regexp.MustCompile(`^` + ownerSide + `(?::` + ownerSide + `)?$`)
// wholeID is a value as a uid or a gid, and whether it is one: a whole number below the kernel's "no id"
// (2^32-1). Uid 0 is one: a key an owner asks for is the controller's terminal's alone (TerminalKeys).
func wholeID(v string) (uint64, bool) {
if v == "" || strings.TrimLeft(v, "0123456789") != "" {
return 0, false
}
n, err := strconv.ParseUint(v, 10, 32)
return n, err == nil && n < 1<<32-1
}
// OwnerSettingProblems refuses, where a manifest is read, a directory whose owner carries a setting
// in any shape but a whole placeholder or number on each side of the colon, or names a key the module
// asks for nowhere else and declares no default for.
func OwnerSettingProblems(m Manifest) []string {
var problems []string
asked := settingKeysAskedOutsideOwners(m)
for k := range m.Settings {
asked[k] = true
}
for _, r := range m.Resources {
owner, _ := r["owner"].(string)
if !strings.Contains(owner, "${setting:") {
continue
}
if fmt.Sprint(r["type"]) != "directory" {
// Said by the placeholder sweep: only a directory's owner takes a setting.
continue
}
if !ownerWithSetting.MatchString(owner) {
problems = append(problems, fmt.Sprintf("%s: the directory %v's owner %q carries a setting in a shape "+
"the mesh does not fill; an owner names the account by number only, each side of the colon a whole "+
"${setting:key} or a whole number (${setting:puid}:${setting:pgid}) (novox/hq issue 279)",
m.Module, r["id"], owner))
continue
}
for _, key := range settingsUsed(owner) {
if !asked[key] {
problems = append(problems, fmt.Sprintf("%s: the directory %v's owner says ${setting:%s}, and the "+
"module asks for %q nowhere else and declares no default for it: an owner follows the identity "+
"the module's process is told it runs as (its identity file's ${setting:%s}), never a key of its "+
"own (novox/hq issue 279)", m.Module, r["id"], key, key, key))
}
}
}
return problems
}
// settingIntoOwner fills ${setting:…} in a directory's owner, refusing — with the key and why, the
// resource left as it was — a key nothing sets and a value that is not a whole number.
func settingIntoOwner(resource map[string]any, layers []Layer, module string) error {
owner, ok := resource["owner"].(string)
if !ok {
return nil
}
for _, key := range settingsUsed(owner) {
value, set := settingValue(layers, key)
if !set {
return &UnsetSettingError{Module: module, Setting: key, said: fmt.Sprintf(
"%s has a directory %v whose owner says ${setting:%s}, and nothing sets %q for it — an operator's "+
"value is the assignment's, never the definition's (novox/hq ADR 0112): "+
"`settings set %s <file>` with {%q: …}%s",
module, resource["id"], key, key, module, key, orNoSettings(layers))}
}
v := plainly(value)
id, whole := wholeID(v)
if !whole {
return fmt.Errorf("%s: the setting %q is %q, which cannot own the directory %v: an owner a setting "+
"names is a uid or a gid, a whole number, never an account's name or other text (novox/hq issue 279)",
module, key, v, resource["id"])
}
// Written as the number parsed, so "0001" reaches the node-engine as 1, never as text.
owner = strings.ReplaceAll(owner, "${setting:"+key+"}", strconv.FormatUint(id, 10))
}
resource["owner"] = owner
return nil
}
func settingValue(layers []Layer, key string) (any, bool) {
var value any
set := false
@@ -171,9 +277,25 @@ func orNoSettings(layers []Layer) string {
return "; set today: " + strings.Join(keys, ", ")
}
// settingKeysUsedBy is every key a module's files, contributions and served facts ask for, so a
// setting that lands in one is not called stray.
// settingKeysUsedBy is every key a module's files, units, directory owners, contributions and served
// facts ask for, so a setting that lands in one is not called stray.
func settingKeysUsedBy(m Manifest) map[string]bool {
used := settingKeysAskedOutsideOwners(m)
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
if owner, ok := r["owner"].(string); ok {
for _, k := range settingsUsed(owner) {
used[k] = true
}
}
}
}
return used
}
// settingKeysAskedOutsideOwners is every key a module's files, units, contributions and served facts
// ask for: what a directory's owner may follow (OwnerSettingProblems).
func settingKeysAskedOutsideOwners(m Manifest) map[string]bool {
used := map[string]bool{}
note := func(s string) {
for _, k := range settingsUsed(s) {
+12 -2
View File
@@ -33,6 +33,9 @@ import (
// 4. **A module's whole layer, when it has a mergeable file not marked `"trusted": false`** (novox/hq issue 340,
// TrustedMergeable). A mergeable file takes any key a layer sets, not only those its content names, so no list
// of keys covers it: an empty runtime configuration a provider reads would take a `url` of the caller's.
// 5. **Every setting a directory's owner asks for** (novox/hq issue 279): root chowns the directory to the number
// at the next push, so through a verb any caller could hand itself — or uid 0 — a module's folder. Whatever
// else asks for the key, a file marked `"trusted": false` included, the owner makes it the terminal's.
//
// Derived from the manifest, never listed by hand, so a provider or a trusted file added tomorrow is covered.
@@ -42,8 +45,8 @@ import (
const TrustedField = "trusted"
// TerminalKeys are the settings keys of a module that are set at the controller's terminal alone: places and
// accesses, every key its provisions serve and every setting a served value asks for, and every setting a file
// asks for unless it says `"trusted": false`. Places and accesses first, then the rest sorted.
// accesses, every key its provisions serve and every setting a served value asks for, every setting a file
// asks for unless it says `"trusted": false`, and every setting a directory's owner asks for. Places and accesses first, then the rest sorted.
func TerminalKeys(m Manifest) []string {
keys := map[string]bool{}
for _, served := range m.Serves {
@@ -57,6 +60,13 @@ func TerminalKeys(m Manifest) []string {
}
}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
owner, _ := r["owner"].(string)
for _, asked := range settingsUsed(owner) {
keys[asked] = true
}
continue
}
if fmt.Sprint(r["type"]) != "file" {
continue
}
+1 -1
View File
@@ -56,7 +56,7 @@ type filler struct {
// seatInto, machineInto and contributionsInto. A pass that comes to read another field adds it here,
// or the sweep refuses the placeholder it would have filled.
var fillers = []filler{
{namespace: "setting", patterns: []*regexp.Regexp{settingRef}, fields: map[string][]string{"file": {"content"}, "service": {"unit"}}},
{namespace: "setting", patterns: []*regexp.Regexp{settingRef}, fields: map[string][]string{"file": {"content"}, "service": {"unit"}, "directory": {"owner"}}},
{namespace: "dir", patterns: []*regexp.Regexp{dirRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
{namespace: "access", patterns: []*regexp.Regexp{accessRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
{namespace: "secret", patterns: []*regexp.Regexp{placeholder}, fields: map[string][]string{"file": {"content"}},
+101
View File
@@ -0,0 +1,101 @@
package catalogue
import (
"strings"
"testing"
)
// An archive pinned by URL and sha256 (novox/hq ADR 0316 §2, §6): the vendor's file, mirrored by the build
// seat, never a copy in the repository.
const pinnedSum = "38dd24890a66b4cf9a45bc24e15192e00c26b929e1bb8df7974c16a4539129e2"
func upstreamManifest(artifact string) string {
return `{"module":"a","version":"1","build":{"artifacts":[` + artifact + `]}}`
}
func TestAnArchivePinnedByURLAndSha256IsAccepted(t *testing.T) {
for _, a := range []string{
`{"name":"archive","kind":"archive","url":"https://vendor.example.org/release-1.tar.xz","sha256":"` + pinnedSum +
`","version":"1.0","within":"release"}`,
`{"name":"addon","kind":"archive","url":"https://vendor.example.org/addon.xpi","sha256":"` + pinnedSum +
`","version":"1.5.6.1","file":"addon.xpi"}`,
} {
if _, err := ParseManifest([]byte(upstreamManifest(a))); err != nil {
t.Errorf("%s was refused: %v", a, err)
}
}
}
func TestAnUpstreamFileThatIsNotPinnedIsRefused(t *testing.T) {
cases := map[string]string{
// No digest: what the address serves changes under the mesh.
`{"name":"x","kind":"archive","url":"https://vendor.invalid/r.tar.xz","version":"1"}`: "without a pinned sha256",
`{"name":"x","kind":"archive","url":"https://vendor.invalid/r.tar.xz","sha256":"ABC","version":"1"}`: "without a pinned sha256",
// No version: a person reading the pin cannot tell which release.
`{"name":"x","kind":"archive","url":"https://vendor.invalid/r.tar.xz","sha256":"` + pinnedSum + `"}`: "says no version",
// Not https.
`{"name":"x","kind":"archive","url":"http://vendor.invalid/r.tar.xz","sha256":"` + pinnedSum + `","version":"1"}`: "over https",
// Both a directory of the repository and a URL.
`{"name":"x","kind":"archive","from":"files","url":"https://vendor.invalid/r.tar.xz","sha256":"` + pinnedSum +
`","version":"1"}`: "one or the other",
// A pin without a URL.
`{"name":"x","kind":"archive","from":"files","sha256":"` + pinnedSum + `"}`: "no url",
// Only an archive is mirrored from a file.
`{"name":"x","kind":"image","from":"Dockerfile","url":"https://vendor.invalid/r.tar.xz","sha256":"` + pinnedSum +
`","version":"1"}`: "only an archive",
// A file name that is a path, a directory within that leaves the archive, and both at once.
`{"name":"x","kind":"archive","url":"https://vendor.invalid/a.xpi","sha256":"` + pinnedSum +
`","version":"1","file":"../a.xpi"}`: "not a plain relative file name",
`{"name":"x","kind":"archive","url":"https://vendor.invalid/r.tar.xz","sha256":"` + pinnedSum +
`","version":"1","within":"../up"}`: "not a plain relative directory",
`{"name":"x","kind":"archive","url":"https://vendor.invalid/r.tar.xz","sha256":"` + pinnedSum +
`","version":"1","within":"r","file":"r.tar.xz"}`: "a file is kept whole",
}
for artifact, want := range cases {
_, err := ParseManifest([]byte(upstreamManifest(artifact)))
if err == nil {
t.Errorf("%s was accepted", artifact)
continue
}
if !strings.Contains(err.Error(), want) {
t.Errorf("%s was refused for another reason than %q: %v", artifact, want, err)
}
}
}
func TestAnUpstreamFilesVendorIsNamedOnPurposeAndAnInstallationsIsRefused(t *testing.T) {
// The vendor's host is judged like any name, and passes through names-on-purpose; a version of four dotted
// numbers is not an address, and a name in it is still judged.
m, err := ParseManifest([]byte(upstreamManifest(`{"name":"owl","kind":"archive","url":"https://vendor.be/owl/owl.xpi","sha256":"` +
pinnedSum + `","version":"1.5.6.1","file":"owl.xpi","names-on-purpose":{"vendor.be":"Owl's vendor publishes it there"}}`)))
if err != nil {
t.Fatal(err)
}
if problems := InstallationProblems(m); len(problems) != 0 {
t.Fatalf("the declared vendor and the version were taken for an installation: %v", problems)
}
m.Build.Artifacts[0].NamesOnPurpose = nil
if problems := InstallationProblems(m); len(problems) != 1 || !strings.Contains(problems[0], "vendor.be") {
t.Fatalf("an undeclared vendor host is judged: %v", problems)
}
m.Build.Artifacts[0].Version = "git.novox.be 10.0.0.1"
if problems := InstallationProblems(m); len(problems) != 2 {
t.Fatalf("a name in the version is judged: %v", problems)
}
m.Build.Artifacts[0].URL, m.Build.Artifacts[0].Version = "https://mail.novox.be/owl.xpi", "1"
if problems := InstallationProblems(m); len(problems) != 1 || !strings.Contains(problems[0], "mail.novox.be") {
t.Fatalf("an installation's host in the url is judged: %v", problems)
}
}
func TestAnUpstreamFileFromAnAddressOrAReservedNameIsRefused(t *testing.T) {
for _, u := range []string{"https://git.novox.internal/r.tar.xz", "https://10.1.2.3/r.tar.xz",
"https://203.0.113.9/r.tar.xz", "https://[2001:db8::1]/r.tar.xz", "https://nas.lan/r.tar.xz", "https://vendor/r.tar.xz"} {
_, err := ParseManifest([]byte(upstreamManifest(`{"name":"x","kind":"archive","url":"` + u + `","sha256":"` +
pinnedSum + `","version":"1"}`)))
if err == nil || !strings.Contains(err.Error(), "published under a public name") {
t.Errorf("%s: %v", u, err)
}
}
}
+18 -3
View File
@@ -116,9 +116,15 @@ var ControllerVerbs = []Verb{
"limit": "how many builds to list (default 20); not with log",
}, nil)},
{Name: "plans", Description: "What the last merges produced and where each stands (novox/hq ADR 0162): " +
"the tiers, the tier a plan is at, what it waits for and since when; one plan whole, given its id.",
"the tiers, the tier a plan is at, what it waits for and since when; one plan whole, given its id; " +
"or, given wait, one plan followed until it moves.",
Input: schema(map[string]string{
"id": "a plan's id (as `plans` lists them): that plan, tier by tier",
"id": "a plan's id (as `plans` lists them): that plan, tier by tier",
"wait": "a plan's id: wait until it moves, then answer it as data — the plan, and what ended the wait " +
"(moved, tier, done, or bound); ask again to follow it further (novox/hq issue 229)",
"until": "with wait: moved (any change, the default), tier (it reaches another tier) or done (it ends)",
"bound": "with wait: how long to wait at most, as 15s or seconds — 20s, the default, is the longest, " +
"inside what a call is answered within",
"stop": "a plan's id: stop it — what was asked still builds, nothing further is asked",
"close": "a plan's id: close a plan that will not move again, as failed by hand (novox/hq issue 254)",
"retry": "a failed plan's id: ask its failed builds again under new ids, and carry the plan on from that tier (novox/hq ADR 0219)",
@@ -333,7 +339,9 @@ var ControllerVerbs = []Verb{
// The build queue, controlled by hand (novox/hq ADR 0219). Every verb that drops an ask leaves a
// failed outcome for it, so a plan waiting on it fails visibly instead of hanging.
{Name: "queue", Description: "Every ask in the build queue: waiting, in flight (on which machine, for how long), " +
"and dead (handed out as often as allowed and never settled) — each with its id, repository, path, ref and when it was asked.",
"and dead (handed out as often as allowed and never settled) — each with its id, repository, path, ref and when it was asked. " +
"Answers the table as text and, as data, each ask's state, kind (check or build), repository, commit, machine, since " +
"and position among the waiting (novox/hq issue 455).",
Input: schema(nil, nil)},
{Name: "cancel", Description: "Drop one waiting or dead ask from the build queue; its outcome is recorded failed, " +
"cancelled by hand, and a plan that asked for it fails. One in flight is refused: `kill` ends it where it runs.",
@@ -490,6 +498,13 @@ var ControllerVerbs = []Verb{
"why": "with consumer or older-than: why — required, and recorded in the hand-act log",
"cause": "with consumer or older-than: the cause in a word (cleanup-waiting when absent)",
}, nil, "confirm")},
{Name: "judge-directories", Description: "Read-only: which of the directories one machine is declared its " +
"node-engine's walk that follows no link another account could make would refuse (novox/hq issue 358), one line " +
"each: placed, would be refused and why (the link and who could replace it), refused already by the placement " +
"guard, or below a home. The engine judges the declaration it keeps and makes, chmods and chowns nothing.",
Input: schema(map[string]string{
"node": "the machine's name",
}, []string{"node"})},
// What a consumer gave up on (novox/hq issue 330): kept in DEAD_LETTERS until a person acts on it.
{Name: "root-free", Description: "Whether each machine named is root-free now (novox/hq ADR 0259 §8): no agent " +
"there can become root without a person. Judged when asked, never from a condition: free only when the machine " +
+45
View File
@@ -0,0 +1,45 @@
package inventory
import (
"strings"
"testing"
"time"
)
// **A plan's id is a value, never SQL** (novox/hq issue 561): the id reaches PlanByID from a caller of the
// controller's plans verb, so an id holding a quote is read as a plain id — no error from the store, and no
// other plan answered for it.
func TestAPlanIdHoldingAQuoteIsReadAsAPlainId(t *testing.T) {
inv := ForTest(t)
ctx := t.Context()
p := Plan{ID: "plan-561", Repository: "novox/mesh-controller", Commit: "abc", Created: time.Now().UTC(),
State: PlanBuilding, Tiers: [][]string{{"a"}}, Modules: map[string]*PlanModule{"a": {}}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
for _, id := range []string{
"x' or '1'='1", // every plan, when pasted into the query
"plan-561' or id like '%", // the plan above, by a pattern
"it's", // a quote alone: a syntax error, when pasted
} {
got, err := inv.PlanByID(ctx, id)
if err == nil {
t.Fatalf("PlanByID(%q) answered %s, a plan of another id", id, got.ID)
}
if want := "no plan " + id; err.Error() != want {
t.Fatalf("PlanByID(%q) failed with %q, not as a plain id no plan has (%q)", id, err, want)
}
}
// A plan whose id holds a quote is found by it, and by nothing else.
quoted := p
quoted.ID, quoted.Revision = "plan-'561'", 0
if err := inv.SavePlan(ctx, &quoted); err != nil {
t.Fatal(err)
}
if got, err := inv.PlanByID(ctx, quoted.ID); err != nil || got.ID != quoted.ID {
t.Fatalf("the plan with a quote in its id was not read back: %v %+v", err, got.ID)
}
if recent, err := inv.RecentPlans(ctx, 1); err != nil || len(recent) != 1 || !strings.HasPrefix(recent[0].ID, "plan-") {
t.Fatalf("RecentPlans with a parameter: %v %+v", err, recent)
}
}
+44 -11
View File
@@ -61,6 +61,10 @@ type Plan struct {
// Phases is the walk's time from its merge, phase by phase (ADR 0282 decision 6): never kept, worked out
// from the walk's own moments by whoever says the walk (`delivery walks`, `plan-moved`).
Phases *WalkPhases `json:"phases,omitempty"`
// WalkSaid is, on the save that started or ended the walk, which of the two it was (novox/hq ADR 0308
// decision 1): set by the save, never kept, read by whoever says the save (`plan-moved`). Empty on every
// other save.
WalkSaid string `json:"-"`
}
// PlanTimes are a walk's own moments beside its modules' (novox/hq ADR 0282 decision 6).
@@ -73,6 +77,10 @@ type PlanTimes struct {
// Class is the walk's module class, read at its cut: core when it moves a module on the controller's own
// path or the mesh's resolver, leaf otherwise (ADR 0282 decision 1).
Class string `json:"class,omitempty"`
// Started is when the walk was said started, and Ended when it was said ended (novox/hq ADR 0308 decision
// 1): each said once, on the save that marked it.
Started *time.Time `json:"started,omitempty"`
Ended *time.Time `json:"ended,omitempty"`
}
// PlanCommit is one repository's commit a walk carries: the latest merge of its branch in the batch.
@@ -117,6 +125,14 @@ type PlanBatch struct {
// the batch's walk does not build them.
Closed bool `json:"closed,omitempty"`
Withheld []string `json:"withheld,omitempty"`
// CutAtOnce says the batch's window closed when it was made: its merge shares nothing with a batch
// assembling and has no delivery group member still to merge, so it waits for no window (novox/hq ADR 0304
// decision 1, ADR 0306 decisions 2 and 3). A merge that joins it later opens its window again. It is not
// Closed, which is a stop's.
CutAtOnce bool `json:"cut_at_once,omitempty"`
// Waits says why a batch made for one merge waits for the window: its delivery group has a member still to
// merge, or mesh-delivery could not say whether it has (novox/hq ADR 0306 decision 2).
Waits string `json:"waits,omitempty"`
}
// OwnPath says the record is a batch of merges on the controller's own path.
@@ -183,8 +199,8 @@ type PlanDelivery struct {
Go *time.Time `json:"go,omitempty"`
By string `json:"by,omitempty"`
Why string `json:"why,omitempty"`
// Stopped is who ended the walk through the delivery's owner, and StoppedWhy why: the walk is failed,
// and the delivery reads it as stopped, not as a build that failed.
// Stopped is who ended the walk, through the delivery's owner or by hand (`plans stop|close`), and StoppedWhy
// why: the walk is failed, and the delivery reads it as stopped, not as a build that failed.
Stopped string `json:"stopped,omitempty"`
StoppedWhy string `json:"stopped_why,omitempty"`
// Merges is every merge the walk or batch answers (novox/hq ADR 0276): its own commits, and the earlier
@@ -236,10 +252,12 @@ func (p Plan) Waiting() bool {
return p.Delivery != nil && p.Delivery.Awaits != "" && p.Delivery.Go == nil
}
// PlanSaved is told every plan this process kept, after it is kept (novox/hq ADR 0239): the serving
// controller says it on the bus as `plan-moved`. Nil in a command, which says nothing; whoever follows a
// walk also asks for it, so a save made by a command is found by comparison.
var PlanSaved func(Plan)
// PlanSaved is told every plan this process kept, after it is kept, with the inventory that kept it (novox/hq
// ADR 0239): the serving controller says it on the bus as `plan-moved`, and so does a command (the controller's
// run installs it for every verb, novox/hq ADR 0308 decision 1), so the save that starts or ends a walk is said
// whichever process made it. Nil says nothing; whoever follows a walk also asks for it, and finds an unsaid save
// by comparison.
var PlanSaved func(*Inventory, Plan)
// ErrPlanMoved is a save against a plan written by somebody else since it was read.
var ErrPlanMoved = errors.New("the plan was written by somebody else since it was read")
@@ -451,7 +469,7 @@ type MergeMove struct {
// leaves a merge answered by two plans, nor a walk stopped whose merges were not moved). A plan written by
// somebody else since it was read refuses the whole act with ErrPlanMoved. Each plan is said (PlanSaved) once
// the transaction is committed, in order.
func (i *Inventory) SavePlansMoving(ctx context.Context, plans []*Plan, moves []MergeMove) error {
func (i *Inventory) SavePlansMoving(ctx context.Context, plans []*Plan, moves []MergeMove) (err error) {
epoch, err := i.actingEpoch(ctx)
if err != nil {
if len(plans) > 0 {
@@ -472,6 +490,20 @@ func (i *Inventory) SavePlansMoving(ctx context.Context, plans []*Plan, moves []
entered time.Time
}
done := make([]written, len(plans))
// **The save that starts or ends a walk is marked** (novox/hq ADR 0308 decision 1), in the same write, and
// unmarked should the write not be made, so the next save marks it again.
marked := time.Now()
undo := make([]func(), 0, len(plans))
defer func() {
if err != nil {
for _, u := range undo {
u()
}
}
}()
for _, p := range plans {
undo = append(undo, p.markWalk(marked))
}
for n, p := range plans {
revision, entered, err := writePlan(ctx, tx, p, epoch)
if err != nil {
@@ -497,7 +529,7 @@ func (i *Inventory) SavePlansMoving(ctx context.Context, plans []*Plan, moves []
p.Epoch = 0
}
if PlanSaved != nil {
PlanSaved(*p)
PlanSaved(i, *p)
}
}
return nil
@@ -591,7 +623,7 @@ func (i *Inventory) PlansSince(ctx context.Context, since time.Time) ([]Plan, er
// RecentPlans is the last few plans, newest first, open or not — what the overview shows.
func (i *Inventory) RecentPlans(ctx context.Context, limit int) ([]Plan, error) {
return i.plans(ctx, fmt.Sprintf(`order by created desc limit %d`, limit))
return i.plans(ctx, `order by created desc limit $1`, limit)
}
// NewestMergeOf is the plan, in any state, of the newest merge into a repository's branch that the mesh
@@ -606,9 +638,10 @@ func (i *Inventory) NewestMergeOf(ctx context.Context, repository, branch string
return plans[0], true, nil
}
// PlanByID is one plan.
// PlanByID is one plan. The id is a parameter, never part of the query (novox/hq issue 561): it comes from a
// caller of the plans verb.
func (i *Inventory) PlanByID(ctx context.Context, id string) (Plan, error) {
plans, err := i.plans(ctx, `where id = '`+id+`'`)
plans, err := i.plans(ctx, `where id = $1`, id)
if err != nil {
return Plan{}, err
}
+4 -3
View File
@@ -202,14 +202,15 @@ func TestASeatsVerbGainsTheArgumentsTheBinaryNames(t *testing.T) {
t.Fatal("failed was not added to the row")
}
// The optional mark is not stored; the working set read from the rows takes it from the compiled seat,
// so today's holder, which does not serve failed, still holds the seat.
// so a holder that does not serve the optional failed still holds the seat.
catalogue.UseSeats(seats)
defer catalogue.UseSeats(catalogue.DefaultSeats())
live, _ := catalogue.SeatNamed(catalogue.ServiceManagerSeat)
holder := catalogue.Manifest{Module: "systemd", Version: "1", Claims: []catalogue.Claim{{Name: catalogue.ServiceManagerSeat,
Scope: catalogue.ScopeNode, Serves: []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal"}}}}
Scope: catalogue.ScopeNode, Serves: []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal",
"reset-failed", "wanted-by", "unlink-dangling"}}}}
if err := catalogue.CanHold(holder, live); err != nil {
t.Fatalf("the seat read from the store refuses today's holder: %v", err)
t.Fatalf("the seat read from the store refuses a holder without the optional failed: %v", err)
}
// Seeding again changes nothing more.
before := verbs["journal"]
+31
View File
@@ -325,6 +325,8 @@ func (i *Inventory) ModuleSecretIfIssued(
return sealed, true, nil
}
// SecretForModule makes what is not held, so it is never asked for an own secret issued outside the mesh: the
// composition reads only a value a person gave for one (GivenOwnSecret; novox/hq issue 401, ADR 0228).
func (i *Inventory) SecretForModule(ctx context.Context, node, module, name string) (string, error) {
key, err := i.SealingKeyOf(ctx, node)
if err != nil {
@@ -655,6 +657,35 @@ func (i *Inventory) GivenMembers(ctx context.Context, node, module, family strin
return out, rows.Err()
}
// GivenOwnSecret is an own secret issued outside the mesh as a person gave it on a machine (novox/hq issue 401, ADR
// 0228): its value sealed to the machine and whether that is the key the machine holds now. Not given is false: a
// value the mesh made for it is not one a person gave, and is never read as one; a machine with no sealing key
// holds none either.
func (i *Inventory) GivenOwnSecret(ctx context.Context, node, module, name string) (GivenMember, bool, error) {
key, err := i.SealingKeyOf(ctx, node)
if err != nil || key == "" {
return GivenMember{}, false, err
}
record, err := i.NodeByName(ctx, node)
if err != nil {
return GivenMember{}, false, err
}
g := GivenMember{Name: name}
var against string
err = i.store.Pool().QueryRow(ctx,
`select sealed, node_key from module_secret
where node = $1 and module = $2 and name = $3 and origin = 'accepted'`,
record.ID, module, name).Scan(&g.Sealed, &against)
if errors.Is(err, pgx.ErrNoRows) {
return GivenMember{}, false, nil
}
if err != nil {
return GivenMember{}, false, err
}
g.Current = against == key
return g, true, nil
}
// GivenOwnSecrets is every own secret of a module a person gave on a machine, by name, with when (novox/hq ADR
// 0283): what the controller checks a module's wait for a secret against. A value the mesh made is not one.
func (i *Inventory) GivenOwnSecrets(ctx context.Context, node, module string) (map[string]time.Time, error) {
+81
View File
@@ -0,0 +1,81 @@
package inventory
import "time"
// A walk says when it starts and when it ends (novox/hq ADR 0308 decision 1): the save that starts it and the
// save that ends it are marked, so `plan-moved` names what the walk touches, its bound and its outcome on those
// two saves, and on no other. The marks are kept in the walk's times, so a controller replaced mid-walk (a core
// walk replaces it) neither says a start again nor misses the end.
// The moments of a walk `plan-moved` names.
const (
WalkStarted = "started"
WalkEnded = "ended"
)
// The outcomes of an ended walk (ADR 0308 decision 1).
const (
WalkDelivered = "delivered"
WalkFailed = "failed"
WalkStopped = "stopped"
)
// Live says the walk is walking: cut, not waiting for its delivery's word, and not ended.
func (p Plan) Live() bool { return p.Open() && !p.Batch() && !p.Waiting() }
// Ended says the walk is in a state it does not leave by itself: done, failed or superseded.
func (p Plan) Ended() bool {
return p.State == PlanDone || p.State == PlanFailed || p.State == PlanSuperseded
}
// Outcome is how an ended walk ended: delivered when done; stopped when a person or the delivery's owner stopped
// it, or a newer walk took it over; failed otherwise. Empty for a walk not ended.
func (p Plan) Outcome() string {
switch {
case p.State == PlanDone:
return WalkDelivered
case p.State == PlanSuperseded:
return WalkStopped
case p.State == PlanFailed && p.Delivery != nil && p.Delivery.Stopped != "":
return WalkStopped
case p.State == PlanFailed:
return WalkFailed
}
return ""
}
// markWalk marks the save that starts or ends the walk, in its times and in WalkSaid, and returns what undoes
// the mark should the save not be written. A walk starts on the first save that finds it live — when it is cut
// and needs no word, or when it is let go — which is before its first tier is asked, and so before anything is
// sent: what it restarts is held from its first send on. It starts again when a failed walk is retried. It ends
// on the first save that finds it ended after it started; one that ends without ever starting (a batch that
// moved nothing, a walk stopped while it waited for its word) touched nothing and says neither. A walk kept
// before its times were (ADR 0282 decision 6) is not marked; a release plan is, its times holding only these.
func (p *Plan) markWalk(now time.Time) (undo func()) {
p.WalkSaid = ""
if p.Batch() || (p.Times == nil && p.Release == nil) {
return func() {}
}
var was *PlanTimes
if p.Times != nil {
kept := *p.Times
was = &kept
}
times := PlanTimes{}
if p.Times != nil {
times = *p.Times
}
at := now.UTC()
switch {
case p.Live() && (times.Started == nil || times.Ended != nil):
times.Started, times.Ended = &at, nil
p.WalkSaid = WalkStarted
case p.Ended() && times.Started != nil && times.Ended == nil:
times.Ended = &at
p.WalkSaid = WalkEnded
default:
return func() {}
}
p.Times = &times
return func() { p.Times, p.WalkSaid = was, "" }
}
+130
View File
@@ -0,0 +1,130 @@
package inventory
import (
"testing"
"time"
)
// A walk is said started on the save that finds it live, ended on the save that finds it ended after it started,
// and neither on any other save (novox/hq ADR 0308 decision 1).
func TestAWalkIsSaidStartedOnceAndEndedOnce(t *testing.T) {
now := time.Date(2026, 10, 12, 11, 0, 0, 0, time.UTC)
cut := now.Add(-time.Second)
p := Plan{ID: "plan-1", State: PlanBuilding, Tiers: [][]string{{"messenger"}},
Modules: map[string]*PlanModule{"messenger": {}}, Times: &PlanTimes{Cut: &cut, Class: ClassLeaf}}
p.markWalk(now)
if p.WalkSaid != WalkStarted || p.Times.Started == nil || !p.Times.Started.Equal(now) {
t.Fatalf("a live walk's first save starts it: %q %+v", p.WalkSaid, p.Times)
}
p.State = PlanRolling
p.markWalk(now.Add(time.Minute))
if p.WalkSaid != "" || !p.Times.Started.Equal(now) {
t.Fatalf("a later save says nothing and keeps the start: %q %+v", p.WalkSaid, p.Times)
}
p.State = PlanDone
p.markWalk(now.Add(2 * time.Minute))
if p.WalkSaid != WalkEnded || p.Times.Ended == nil || p.Outcome() != WalkDelivered {
t.Fatalf("the save that finds it done ends it, delivered: %q %+v %q", p.WalkSaid, p.Times, p.Outcome())
}
p.markWalk(now.Add(3 * time.Minute))
if p.WalkSaid != "" {
t.Fatalf("an ended walk saved again says nothing: %q", p.WalkSaid)
}
// Retried, it walks again: a new start.
p.State = PlanBuilding
p.markWalk(now.Add(4 * time.Minute))
if p.WalkSaid != WalkStarted || p.Times.Ended != nil || !p.Times.Started.Equal(now.Add(4*time.Minute)) {
t.Fatalf("a retried walk starts again: %q %+v", p.WalkSaid, p.Times)
}
}
// What is never said: a batch, a walk waiting for its word, a walk that ends without having started, and a walk
// kept before its times were. A release plan is said.
func TestWhatIsNotAWalkStartIsNotSaid(t *testing.T) {
now := time.Now()
for name, p := range map[string]Plan{
"a batch": {State: PlanAssembling, Times: &PlanTimes{}},
"waiting for a word": {State: PlanBuilding, Times: &PlanTimes{}, Delivery: &PlanDelivery{Awaits: "mesh-delivery"}},
"ended, never begun": {State: PlanDone, Times: &PlanTimes{}},
"kept before times": {State: PlanBuilding},
"stopped while waits": {State: PlanFailed, Times: &PlanTimes{}, Delivery: &PlanDelivery{Awaits: "mesh-delivery", Stopped: "x"}},
} {
p.markWalk(now)
if p.WalkSaid != "" {
t.Errorf("%s: said %q", name, p.WalkSaid)
}
}
r := Plan{State: PlanRolling, Release: &PlanRelease{Order: []string{"ace"}}}
r.markWalk(now)
if r.WalkSaid != WalkStarted || r.Times == nil || r.Times.Started == nil || r.Times.Class != "" {
t.Fatalf("a release plan is said started, its times holding only that: %q %+v", r.WalkSaid, r.Times)
}
// Let go, the walk starts.
go1 := now
w := Plan{State: PlanBuilding, Times: &PlanTimes{}, Delivery: &PlanDelivery{Awaits: "mesh-delivery", Go: &go1}}
w.markWalk(now)
if w.WalkSaid != WalkStarted {
t.Fatalf("a walk let go starts: %q", w.WalkSaid)
}
}
// An ended walk's outcome: delivered, failed, or stopped by a person or the delivery's owner, or by a newer walk.
func TestAWalksOutcome(t *testing.T) {
for want, p := range map[string]Plan{
WalkDelivered: {State: PlanDone},
WalkFailed: {State: PlanFailed},
WalkStopped: {State: PlanFailed, Delivery: &PlanDelivery{Stopped: "mesh-delivery"}},
"": {State: PlanRolling},
} {
if got := p.Outcome(); got != want {
t.Errorf("%s: outcome %q, want %q", p.State, got, want)
}
}
if (Plan{State: PlanSuperseded}).Outcome() != WalkStopped {
t.Error("a superseded walk is stopped")
}
}
// The marks are kept with the walk, so a controller replaced mid-walk neither says its start again nor misses
// its end; and a save refused leaves the walk unmarked, so the save that is made says it.
func TestAWalksStartIsKeptAndARefusedSaveLeavesItUnsaid(t *testing.T) {
inv := ForTest(t)
ctx := t.Context()
cut := time.Now().UTC().Add(-time.Minute)
p := Plan{ID: "plan-308", Repository: "novox/mesh-catalog", Commit: "abc", Created: cut, State: PlanBuilding,
Tiers: [][]string{{"messenger"}}, Modules: map[string]*PlanModule{"messenger": {}},
Times: &PlanTimes{Cut: &cut, Class: ClassLeaf}}
var said []string
was := PlanSaved
PlanSaved = func(_ *Inventory, p Plan) { said = append(said, p.WalkSaid) }
t.Cleanup(func() { PlanSaved = was })
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
back, err := inv.PlanByID(ctx, p.ID)
if err != nil || back.Times == nil || back.Times.Started == nil || back.Times.Class != ClassLeaf {
t.Fatalf("the start is kept with the walk: %v %+v", err, back.Times)
}
// Read back by another controller and saved: not said again.
back.State = PlanRolling
if err := inv.SavePlan(ctx, &back); err != nil {
t.Fatal(err)
}
// A stale copy ending it is refused, and unmarked.
stale := p
stale.Revision = 1
stale.State = PlanDone
if err := inv.SavePlan(ctx, &stale); err == nil {
t.Fatal("a stale save is refused")
}
if stale.WalkSaid != "" || stale.Times.Ended != nil {
t.Fatalf("a refused save is unmarked: %q %+v", stale.WalkSaid, stale.Times)
}
back.State = PlanDone
if err := inv.SavePlan(ctx, &back); err != nil {
t.Fatal(err)
}
if len(said) != 3 || said[0] != WalkStarted || said[1] != "" || said[2] != WalkEnded {
t.Fatalf("said: %q, want started, nothing, ended", said)
}
}
+11
View File
@@ -138,6 +138,9 @@ type GroupMember struct {
Repository string `json:"repository"`
Ref string `json:"ref"`
Paths []string `json:"paths,omitempty"`
// Base is the branch it merges into: the build seat merges the head into that trunk's head before the
// group is composed (novox/hq ADR 0323 §6). A build seat from before it composes the head alone.
Base string `json:"base,omitempty"`
}
// Who judges a merge check's gate.
@@ -164,6 +167,14 @@ type CheckOutcome struct {
// Gate and RepoCheck are its two layers; Verdict and Summary above are the gate's.
Gate *CheckLayer `json:"gate,omitempty"`
RepoCheck *CheckLayer `json:"repo-check,omitempty"`
// Head and Trunk are what was judged (novox/hq ADR 0323 §3): the pull request's head merged into the trunk
// commit, each as twelve characters. Trunk is empty from a build seat from before it, which judged the head
// alone. Beside are the commits of the repositories read beside it, by directory, named and not judged
// against (issue 568); Members a delivery group's other heads, each with the trunk commit it merged into.
Head string `json:"head,omitempty"`
Trunk string `json:"trunk,omitempty"`
Beside map[string]string `json:"beside,omitempty"`
Members []CheckedMember `json:"members,omitempty"`
}
// SourceOnSeat names a repository by the seat whose holder serves it and its path there.
+47 -10
View File
@@ -601,6 +601,24 @@ func Acknowledge(ctx context.Context) {
}
}
// LongestAnswer is the longest a handler that waits on purpose may hold its caller's one answer
// (AnswersWithin, novox/hq issue 229): inside the console's thirty seconds with room for the trip back,
// and inside the bus's window for an answer (broker.ResponseTTL). A variable so a test need not wait.
var LongestAnswer = 25 * time.Second
type answerWithinKey struct{}
// AnswersWithin asks that the call's caller be answered in full if the handler ends within d of the
// call's start, rather than within AnswerWithin: for a handler that waits on purpose, with a bound —
// `plans` waiting for a plan to move (novox/hq issue 229). Past d, or past LongestAnswer whichever is
// shorter, its caller is told it is running, as for any call. Never shortens the window; without effect
// outside a served call, and once the caller was answered.
func AnswersWithin(ctx context.Context, d time.Duration) {
if f, ok := ctx.Value(answerWithinKey{}).(func(time.Duration)); ok {
f(d)
}
}
// running is the interim answer: what a caller reads when the call has not finished.
func running(c *Call, within time.Duration, acknowledged bool, follow string) []byte {
why := fmt.Sprintf("it has not finished in %s", within)
@@ -645,6 +663,13 @@ func (l *CallLog) serveCallWithin(seat, verb string, args json.RawMessage, reply
acknowledged := make(chan struct{})
var once sync.Once
ctx = context.WithValue(ctx, answerNowKey{}, func() { once.Do(func() { close(acknowledged) }) })
longer := make(chan time.Duration, 1)
ctx = context.WithValue(ctx, answerWithinKey{}, func(d time.Duration) {
select {
case longer <- d:
default:
}
})
type outcome struct {
body []byte
@@ -703,17 +728,29 @@ func (l *CallLog) serveCallWithin(seat, verb string, args json.RawMessage, reply
}
}
}
timer := time.NewTimer(AnswerWithin)
within, start := AnswerWithin, time.Now()
timer := time.NewTimer(within)
defer timer.Stop()
select {
case o := <-done:
l.finish(c, o.body, o.failed, false)
say(o.body)
return
case <-acknowledged:
say(running(c, AnswerWithin, true, l.Follow))
case <-timer.C:
say(running(c, AnswerWithin, false, l.Follow))
waiting:
for {
select {
case o := <-done:
l.finish(c, o.body, o.failed, false)
say(o.body)
return
case <-acknowledged:
say(running(c, within, true, l.Follow))
break waiting
case d := <-longer:
// A handler waiting on purpose (AnswersWithin): its window is stretched, never past LongestAnswer.
if d = min(d, LongestAnswer); d > within {
within = d
timer.Reset(time.Until(start.Add(d)))
}
case <-timer.C:
say(running(c, within, false, l.Follow))
break waiting
}
}
o := <-done
l.finish(c, o.body, o.failed, true)
+55
View File
@@ -0,0 +1,55 @@
package link
import (
"context"
"encoding/json"
"testing"
"time"
)
// **A handler that waits on purpose is answered in full when it ends inside the window it asked for**
// (novox/hq issue 229): `plans` asked to wait for a plan holds its caller's one answer until the plan
// moves or its bound passes, rather than being answered "still running" at AnswerWithin.
func TestAHandlerThatWaitsOnPurposeIsAnsweredInFullInsideItsWindow(t *testing.T) {
shortWindow(t, 20*time.Millisecond)
l, a := NewCallLog(), newAnswers(t)
l.serveCall("mesh-controller", "plans", json.RawMessage(`{"wait":"plan-1"}`), "_INBOX.x.3",
func(ctx context.Context, _ json.RawMessage) (any, error) {
AnswersWithin(ctx, 200*time.Millisecond)
time.Sleep(80 * time.Millisecond) // past AnswerWithin, inside what it asked for
return "plan-1 moved", nil
}, a.respond, nil)
if got := a.only()["result"]; got != "plan-1 moved" {
t.Fatalf("a waiting handler inside its window was answered %v, not in full", got)
}
}
// What a handler may ask for is bounded: past LongestAnswer its caller is told it is running, as any
// call is, so no answer is ever later than the console waits.
func TestAWaitIsNeverLongerThanTheLongestAnswer(t *testing.T) {
shortWindow(t, 10*time.Millisecond)
was := LongestAnswer
LongestAnswer = 30 * time.Millisecond
t.Cleanup(func() { LongestAnswer = was })
l, a := NewCallLog(), newAnswers(t)
release := make(chan struct{})
finished := make(chan struct{})
go func() {
l.serveCall("mesh-controller", "plans", nil, "_INBOX.x.4", func(ctx context.Context, _ json.RawMessage) (any, error) {
AnswersWithin(ctx, time.Hour)
<-release
return "late", nil
}, a.respond, nil)
close(finished)
}()
select {
case <-a.sent:
case <-time.After(2 * time.Second):
t.Fatal("a handler asking for an hour held its caller past LongestAnswer")
}
if got, _ := a.only()["result"].(map[string]any); got["running"] != true {
t.Fatalf("past LongestAnswer the caller was answered %v, not that it is running", a.only())
}
close(release)
<-finished
}
+25 -8
View File
@@ -221,13 +221,17 @@ type SourceMoved struct {
// PullUpdated is what the forge announces when an open pull request's head moves — opened, or pushed
// to (novox/hq to-be 45 §9): what the controller asks the build seat to check before it merges.
type PullUpdated struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Number int `json:"number"`
Title string `json:"title,omitempty"`
Base string `json:"base"`
Head string `json:"head"`
Commit string `json:"head_sha"`
Owner string `json:"owner"`
Repo string `json:"repo"`
Number int `json:"number"`
Title string `json:"title,omitempty"`
Base string `json:"base"`
Head string `json:"head"`
Commit string `json:"head_sha"`
// BaseSHA is the base branch's head when the forge announced the head (novox/hq ADR 0323): the trunk commit a
// verdict the controller gives without the build seat — a change that runs no build — is judged against. Empty
// from a forge module from before it.
BaseSHA string `json:"base_sha,omitempty"`
CloneURL string `json:"clone_url"`
HTMLURL string `json:"html_url,omitempty"`
// Paths are the files the pull request changes; PathsTruncated says there were more.
@@ -278,14 +282,24 @@ type Checked struct {
// status from it — the group's verdict is its owner's, mesh-delivery, to say on each member's head.
Group string `json:"group,omitempty"`
Members []CheckedMember `json:"members,omitempty"`
// Trunk is the trunk commit the head was merged into and judged as, twelve characters (novox/hq ADR 0323
// §3): the verdict holds for that merge only, and goes out of date when the trunk moves. Empty from a build
// seat that judged the head alone. Beside are the commits of the other repositories the check read, by
// directory, named and not judged against (issue 568). Took is how long the check ran, for the merge tool
// to say how long the next one may take (ADR 0323 §5).
Trunk string `json:"trunk,omitempty"`
Beside map[string]string `json:"beside,omitempty"`
Took string `json:"took,omitempty"`
}
// CheckedMember is one head a group's composed check judged.
// CheckedMember is one head a group's composed check judged, and the trunk commit it was merged into (novox/hq
// ADR 0323 §6), twelve characters; empty from a build seat that composed the head alone.
type CheckedMember struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Number int `json:"number,omitempty"`
Commit string `json:"commit"`
Trunk string `json:"trunk,omitempty"`
}
// ChangePlan is what a change does to the mesh, computed from its diffset — a repository, the branch it
@@ -311,6 +325,9 @@ type ChangePlan struct {
Steps []string `json:"steps,omitempty"`
// Summary is the plan in one line, for a commit status.
Summary string `json:"summary"`
// Trunk is the trunk commit the head was merged into when it was judged (novox/hq ADR 0323 §3), twelve
// characters: set on the plan a verdict carries, which mesh-delivery keeps with its other inputs.
Trunk string `json:"trunk,omitempty"`
}
// MachinePlan is one machine's part of a change plan.
+22
View File
@@ -148,6 +148,28 @@ func AskSetuidSearch(ctx context.Context, conn *nats.Conn, signer Signer, node,
"there", record: "a start"})
}
// JudgeDirectoriesContext is prefixed to a directory judgement ask's bytes before signing (novox/hq issue 358): never
// another ask's signature, nor a declaration's. The engine holds the same words.
const JudgeDirectoriesContext = "novox-mesh judge-directories v1\n"
// AskJudgeDirectories asks one machine's node-engine to judge, read-only, which of the directories it is declared
// the walk that follows no link another account could make would refuse (novox/hq issue 358): the ask a hand-over
// is, naming no path, signed under JudgeDirectoriesContext. The engine changes nothing and answers one line per
// directory.
func AskJudgeDirectories(ctx context.Context, conn *nats.Conn, signer Signer, node, by string,
timeout time.Duration) (HandOverAnswer, error) {
if conn == nil {
return HandOverAnswer{}, errors.New("this controller is not on the bus")
}
body, err := signAsk(ctx, signer, JudgeDirectoriesContext, HandOverAsk{Node: node, By: by})
if err != nil {
return HandOverAnswer{}, err
}
return askSigned(ctx, conn, broker.AskJudgeDirectoriesSubject(node), body, node, timeout, askWords{
what: "a judgement of its directories", nothing: "nothing was judged, and nothing is changed by one",
issue: "issue 358", unknown: "ask again: a judgement changes nothing", record: "a judgement"})
}
// SignHandOver fills the ask's expiry and nonce and signs it with the mesh's key, over HandOverContext and the
// ask's bytes exactly as they travel.
func SignHandOver(ctx context.Context, signer Signer, ask HandOverAsk) ([]byte, error) {
+60
View File
@@ -0,0 +1,60 @@
package link
import (
"context"
"crypto/ed25519"
"encoding/json"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/testbus"
)
// A judgement of a machine's directories is asked of the machine on its own subject, signed under its own context
// (never a hand-over's), naming no path, and the engine's answer comes back (novox/hq issue 358). The engine holds the
// same subject and context (mesh-host internal/link, TestTheDirectoryJudgementAskKeepsItsSubjectAndContext).
func TestADirectoryJudgementIsAskedOfTheMachineSignedUnderItsOwnContext(t *testing.T) {
if broker.AskJudgeDirectoriesSubject("laptop") != "mesh.node.laptop.ask.judge-directories" ||
JudgeDirectoriesContext != "novox-mesh judge-directories v1\n" {
t.Fatalf("the subject %q, the context %q", broker.AskJudgeDirectoriesSubject("laptop"), JudgeDirectoriesContext)
}
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
signer, public := testSigner(t)
var heard HandOverAsk
engine, err := conn.Subscribe(broker.AskJudgeDirectoriesSubject("laptop"), func(msg *nats.Msg) {
var signed SignedHandOver
_ = json.Unmarshal(msg.Data, &signed)
if ed25519.Verify(public, append([]byte(HandOverContext), signed.Ask...), signed.Signature) ||
ed25519.Verify(public, append([]byte(SetuidSearchContext), signed.Ask...), signed.Signature) ||
!ed25519.Verify(public, append([]byte(JudgeDirectoriesContext), signed.Ask...), signed.Signature) {
_ = msg.Respond([]byte(`{"refused":"not signed as a judgement"}`))
return
}
_ = json.Unmarshal(signed.Ask, &heard)
body, _ := json.Marshal(HandOverAnswer{Said: "3 directories declared, asked by " + heard.By})
_ = msg.Respond(body)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = engine.Unsubscribe() })
a, err := AskJudgeDirectories(context.Background(), conn, signer, "laptop", "jo", 5*time.Second)
if err != nil || a.Said != "3 directories declared, asked by jo" {
t.Fatalf("answered %+v, %v", a, err)
}
if heard.Node != "laptop" || heard.Path != "" || heard.Nonce == "" || heard.Expires.IsZero() {
t.Fatalf("the engine heard %+v", heard)
}
if _, err := AskJudgeDirectories(context.Background(), conn, signer, "anchor", "jo", 5*time.Second); err == nil ||
!strings.Contains(err.Error(), "nothing was judged") {
t.Fatalf("a machine with no engine listening: %v", err)
}
}
+25
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"regexp"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
@@ -268,6 +269,30 @@ func ReadQueue(ctx context.Context, js *broker.JetStream, seat string) (Queue, e
return q, nil
}
// WithoutPassword is a repository as a listing may say it: a URL's password, should one ever be written
// into it, left out (novox/hq issue 455). Only for what is shown: the ask itself keeps what it was given,
// which a cancel's outcome and a rebuild's match are read from.
func WithoutPassword(repository string) string {
scheme, rest, found := strings.Cut(repository, "://")
if !found {
return repository
}
authority, path, _ := strings.Cut(rest, "/")
at := strings.LastIndex(authority, "@")
if at < 0 {
return repository
}
user, _, password := strings.Cut(authority[:at], ":")
if !password {
return repository
}
out := scheme + "://" + user + authority[at:]
if len(rest) > len(authority) {
out += "/" + path
}
return out
}
// ReadBuildEvents is every start and outcome the seat announced since a moment, from the events
// stream, with a consumer of its own that is gone when this returns.
func ReadBuildEvents(ctx context.Context, js *broker.JetStream, seat string, since time.Time) (BuildHeard, error) {

Some files were not shown because too many files have changed in this diff Show More