Compare commits

..
Author SHA1 Message Date
mesh-admin 0a2e58c070 Merge pull request 'Deliver again an ask the controller made, removing the original from its queue (issue 334, part)' (#219) from fix/334-a-given-up-ask-can-be-delivered-again into main 2026-10-11 02:36:26 +00:00
mesh-admin 45b4ae92bc Merge pull request 'A provider whose wait fails its check lists who waits on it (issue 450)' (#221) from fix/450-a-provider-whose-wait-fails-lists-its-waiters into main 2026-10-11 02:19:02 +00:00
mesh-admin 9d6a12eb53 Merge pull request 'Say an unanswered merge check's time in the operator's zone (issue 443)' (#218) from fix/443-a-stalled-lines-times-are-local into main 2026-10-11 01:51:53 +00:00
jschoubben 984d85865d Give the words' zone through a seam, so no test writes time.Local under the race detector (novox/hq issue 443)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/443-a-stalled-lines-times-are-local delivered: every member is delivered
2026-10-11 03:41:14 +02:00
jschoubben ed45cc6415 Check a provider's waits before saying who waits on it (issue 450)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A provider whose wait fails the check raises unhealthy, but the holding read
its stored statement with the wait unchecked: sayWaiters built the
needs-operator key, found it not open and listed no held consumer. The
holding now reads each statement as judged (ADR 0283 decision 3), in
sayWaiters and in the provider's state its consumers are held by.
2026-10-11 03:29:10 +02:00
jschoubben ef551fdfb6 Remove an ask's original only when its sequence still holds it, and only with a worker (issue 334)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A seat's queue made again numbers from one, so an old dead letter's sequence
can name a live ask; deleting by number alone would drop it silently. An ask
with no worker would wait unseen while counted as delivered.
2026-10-11 03:24:52 +02:00
mesh-admin 7493bd8f18 Merge pull request 'A provider waiting for the operator holds its consumers, and a wait beside another condition is said (issue 405)' (#216) from fix/405-a-waiting-provider-holds-its-consumers into main 2026-10-11 01:23:35 +00:00
mesh-admin 8305e4e3d1 Merge pull request 'A test that reads another repository judges what the check clones, never the desktop's checkout (issue 432)' (#217) from fix/432-a-test-reads-what-it-carries into main 2026-10-11 01:21:14 +00:00
mesh-admin fbc7bc976b Merge pull request 'make check runs merge-check.sh, so it and the gate agree (issue 431)' (#215) from fix/431-make-check-runs-what-the-gate-runs into main 2026-10-11 01:20:19 +00:00
jschoubben f510b46319 Deliver again an ask the controller made, removing the original from its queue (issue 334)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A build ask its seat's worker gave up on could only be dropped, though the
controller already holds the publish on that seat's accepts and the stream
API to remove the original. Asks to other seats stay refused: delivering them
needs a grant ADR 0264 withholds, in the asker's name ADR 0259 protects.
2026-10-11 03:18:34 +02:00
jschoubben 926fde2fda Say an unanswered merge check's time in the operator's zone, from the checks given as data (novox/hq issue 443)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: FAIL github.com/novox/mesh-controller/cmd/mesh-controller 268.072s
mesh/delivery-group group fix/443-a-stalled-lines-times-are-local rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-11 03:10:30 +02:00
jschoubben d5cf3b42fe Say a waiting provider in the operator's words, and which state it is when a consumer is held (issue 405 review)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
2026-10-11 03:05:34 +02:00
mesh-admin 47c7877e8d Merge pull request 'A consumer's max-deliveries condition has one watcher and counts what was given up (issue 440)' (#214) from fix/440-one-key-one-watcher into main 2026-10-11 01:04:28 +00:00
jschoubben 68fbd99e89 Say how a check's clones are chosen and what the captured copy carries (issue 432 review)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-11 03:00:28 +02:00
jschoubben 9a37c143e4 Keep a waiting provider's hold within ADR 0283 (issue 405 review)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
A consumer held under a provider that only waits for the operator now
passes its gate as a wait for a person, carrying the wait, so no walk
waits on the operator's secret. Only consumers of the waiting part are
held; one that cannot be matched is raised on its own and says its
provider waits. needs-operator stays a warning while consumers wait.
2026-10-11 03:00:27 +02:00
jschoubben eb72075104 Judge tests that read another repository against what the check clones, never the desktop's checkout (issue 432)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Tests that read ../../../mesh-catalog or ../../../mesh-host gave a verdict
that depended on what sat beside the checkout: a stale or dirty sibling
failed them on a desktop, and a missing one skipped them unseen. They now
read the clone the build seat puts in MESH_CHECK_BESIDE, failing when it is
absent there, and elsewhere a copy captured at a named commit.

The skip had hidden that the builder test read a module retired by ADR 0190.
The systemd reading test no longer counts the machine's own environment.d.
2026-10-11 02:55:04 +02:00
jschoubben d6b867a87a Restart what reads a secret family member when the member is given again (issue 405)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheSecondControllerWaitsAndTakesAHigherEpochOnHandover (4.67s)
mesh/delivery superseded: a newer head of the same pull request
secretsReadBy looked only at secrets declared by name, so a container
mounting a member kept the value it started with.
2026-10-11 02:51:11 +02:00
jschoubben 1dc9961c43 Hold consumers under a provider waiting for the operator, and say a wait beside another condition (issue 405)
A provider whose only part not healthy waits for the operator's secret or
setting let its consumers raise their own unhealthy conditions, and a
wait beside a relogin, a directory used as found or a fault was not said
until the other cleared.
2026-10-11 02:51:11 +02:00
mesh-admin 11ffab887b Merge pull request 'Say an unanswered merge check in its own words, with no action it cannot take (issue 438)' (#213) from fix/438-a-check-that-never-answered-is-said into main 2026-10-11 00:47:10 +00:00
jschoubben cfbbad8209 Count a dead letter by when it was kept, so one kept late still counts (issue 440)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The give-up time is not newer for every letter kept: a notice that could
not be kept is offered again a minute later, so a later give-up can be kept
first and the one after it read as not fresh. The stored time rises with the
stream's sequence. A consumer whose letters vanish between the two reads is
left out of the look instead of counted as a look, and the skip of a
token-less max-deliveries advisory now has a test of its own.
2026-10-11 02:36:58 +02:00
jschoubben d2e9dc6159 Inline the check's teardown so make -n check stays a dry run
mesh/merge-gate pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery published: its walk waits for its turn
GNU make runs a recipe line holding $(MAKE) even under -n, so a dry run of
check ran the whole suite. novox/hq issue 431.
2026-10-11 02:35:37 +02:00
jschoubben 5557a01f06 Make check run merge-check.sh, so a developer's check and the gate cannot drift
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
make check listed its own steps, and its gofmt walked vendor/, failing on
third-party files no change could fix; the steps after it never ran. It now
raises the throwaway database and runs the script repo-check runs.
novox/hq issue 431.
2026-10-11 02:33:30 +02:00
mesh-admin 5bddb16514 Merge pull request 'A misspelled placeholder is refused, never written out as text (issue 231)' (#211) from fix/231-a-misspelled-placeholder-is-refused into main 2026-10-11 00:31:46 +00:00
jschoubben 671e350f56 Name the build queue and the merge check as the glossary does (issue 438)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/438-a-check-that-never-answered-is-said delivered: every member is delivered
Review of #213: the queue is the controller's build queue, not the build seat's,
and the merge check is the pair, so the headline says the pull request's merge
check has not answered.
2026-10-11 02:30:19 +02:00
jschoubben f83fcdc15f Give a consumer's max-deliveries key one watcher, counting what was given up (issue 440)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The dead-letter row and a max-deliveries advisory without a token both said
bus.<stream>.<consumer>.max-deliveries: each look added an observation and a
line of evidence through the row, and the two overwrote each other's words.
The row owns the key since issue 330, so the advisory watcher leaves it, and
the row now says when the newest held message was given up, so a letter held
for a day no longer reads as observed every 30 seconds. Times without Happened
is refused, since the raise ignored it and an update counted it.
2026-10-11 02:30:09 +02:00
jschoubben 585caa4371 Say an unanswered merge check in its own words, with no action it cannot take (issue 438)
mesh/delivery-group group fix/438-a-check-that-never-answered-is-said checking: 0 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh-delivery now says a head whose merge check started and never answered as a
stalled line in state unanswered. Through the generic delivery words it would read
as a delivery and offer a Stop that mesh-delivery refuses, since no delivery of
the head exists; it now names the checks waited on and what the operator can do.
2026-10-11 02:25:11 +02:00
mesh-admin f8d08b0637 Merge pull request 'A bus refusal is counted by what the bus said, not by the controller's looks (issue 402)' (#212) from fix/402-a-refusal-is-counted-once into main 2026-10-11 00:21:00 +00:00
jschoubben dc62fd0075 Let a shell parameter operator after a known word pass, so ${PORT:-8080} is not refused (issue 231)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The case-insensitive rule for the mesh's namespace words took ${PORT:-8080},
${SHELL:-/bin/sh}, ${SECRET:?unset} and ${dir:-/tmp} for misspellings, though they
are among the commonest lines of a script or env file. A :-, :=, :+ or :? after the
colon is the shell's, whatever the word's case.
2026-10-11 02:20:43 +02:00
jschoubben 525b2c1a11 Sweep the definition, not the filled values, and judge each field alike at check and composition (issue 231)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
The first sweep ran at composition over the resource after settings, bindings and
machine facts were filled, so an operator's value such as ${labels:instance} was
refused as a misspelling its author never wrote, and the whole machine got nothing.
Both points now sweep the resource as the manifest wrote it, against one table of
which fields each pass fills, so the check and composition refuse the same things.
Any ${<known namespace>: its pattern does not take is refused whatever its case or key.

Issue 231's undeclared-setting half is not met here: refusing a key the module does
not declare (ADR 0164) is not built and is handed to issue 398.
2026-10-11 02:15:37 +02:00
jschoubben c11222026a Count a bus refusal by what the bus said, not by how often the controller looked (issue 402)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
S9 reads the one remembered advisory again on every look for an hour, and
each look was kept as an observation and a line of evidence, so a single
refusal read as one repeated every 30 seconds. An observation may now say
when what it reads happened and how often; the keeper counts that, and a
look with nothing newer adds nothing. Sources that look at the present
keep counting their looks.
2026-10-11 02:10:48 +02:00
jschoubben 360c318e85 Refuse a placeholder no pass consumes, so a misspelling never reaches a machine as text (issue 231)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-11 02:06:14 +02:00
mesh-admin f008fab9d8 Merge pull request 'A kept call holds a command's first word, never the line (issue 397)' (#210) from fix/397-a-kept-call-holds-no-command-line into main 2026-10-10 23:23:49 +00:00
jschoubben 15a9919df5 A kept command's first word is parted by any whitespace, and capped (review of issue 397)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The review of #210 found that the fix left the hole open and widened
another.

A command's words were cut on a space alone, while the verb's own
splitter parts them on a space, a tab or a newline. The same line
written with tabs found no space, so all of it was kept — the very hole
this closes. Its words are now parted as the splitter parts them.

And because the command arm sits above the arm that caps a string at
120 bytes, a command kept whole was no longer capped: for a 300-byte
single token the change kept more than the code it replaced. The first
word now carries the same cap.

A one-word command is still kept whole, but the comment no longer
claims it carries nothing to withhold: the record is written before the
verb judges the line, so one word may be a token or compact JSON. The
cap is what bounds that.

The test now says the whole of what is kept for each line, which is
also what proves the rest is gone, and looks for forbidden words as
whole words rather than as substrings — so "set" is back in the list,
and "show" cannot hide in a word such as "shown". All eight cases fail
against the unfixed code.
2026-10-11 01:04:40 +02:00
jschoubben 1390836b73 A kept call holds a command's first word, never the line (issue 397)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
kept() withheld the arguments line, values, secret and stdin, and kept
the command argument of the generic command verb verbatim. A line such
as `settings set <module> '<value>' --node <node>` therefore put the
value into the calls record, which answers anyone who may call the seat.

It is now kept as a mesh-cli line is: its first word, with the rest said
to have been given. A command of one word is kept whole, since there is
nothing after it to withhold, and one that is not a string is kept as
"(given, not kept)".

The record as it stands holds no such line: its whole answer, read at
2026-10-10 22:52 UTC, carries no call of the command verb. That says
nothing of calls older than its window.
2026-10-11 00:53:37 +02:00
mesh-admin dac7e5f7f6 Merge pull request 'Say a part waiting for the operator as needs-operator and pass its gate; add secret families (issue 386, ADR 0283)' (#209) from feat/386-a-wait-for-the-operator into main 2026-10-10 19:49:29 +00:00
mesh-admin c06a2cd972 Merge pull request 'Say a walk's phases out of order unknown, never a negative time (hq issue 382)' (#208) from fix/382-phases-out-of-order into main 2026-10-10 19:36:52 +00:00
jochen d5f6b67b94 Say a part waiting for the operator as needs-operator and pass its gate; add secret families (issue 386, hq ADR 0283)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A module waiting for the operator's secret failed its first-node gate, held
every later walk and was said as 'nothing for you to do'. The node-engine's
new waiting state is checked against the manifest and the secrets given,
read by the gate as a wait for a person, and raised as needs-operator naming
the act. A secret family gives each part its own one-line secret, which the
mesh never makes, so the desk prompt can take each password.
2026-10-10 21:19:58 +02:00
jochen 12d4025d30 Say a walk's phases out of order unknown, never a negative time (hq issue 382)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
#206's own walk kept its first send 783 ms before its build, and delivery
walks said send-first measured at -783 ms. A moment recorded before the
one before it now makes both phases unknown: the earlier one's time joins
the unknown time, the later one has none, and the walk goes on from the
later moment. Measured phases and unknown time still add up to the total.
2026-10-10 21:15:48 +02:00
mesh-admin 1ca4d8ce60 Merge pull request 'Test that times is optional on the delivery seat (hq issue 382)' (#207) from test/382-times-optional into main 2026-10-10 19:13:56 +00:00
jochen a4f93b52a8 Test that a delivery seat holder without times still holds the seat, and one serving it is not refused (hq issue 382, review of #206)
mesh/merge-gate pass: builds build-agent, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 o…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 20:55:43 +02:00
mesh-admin a2a671adb7 Merge pull request 'Keep each walk's phases and say a delivery over its budget (hq ADR 0282 slice 1, issue 382)' (#206) from feat/382-walk-phases into main 2026-10-10 18:49:18 +00:00
jochen 412f11b079 Mark times optional on the delivery seat until mesh-delivery serves it (hq ADR 0282, design 33 §7)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 20:35:24 +02:00
jochen 5d4eb974ab Promise times among the mesh-delivery seat's verbs, so its holder may serve it (hq ADR 0282)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.75s)
mesh/delivery superseded: a newer head of the same pull request
2026-10-10 20:22:34 +02:00
jochen 6805e2bcb3 Walk phases: a report past the silent bound never moves a walk's end; keep phases outside the hold on the plans; first-node gate in words (review of #206)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.71s)
2026-10-10 20:20:07 +02:00
jochen c640341c50 Keep each walk's phases and say a delivery over its budget (hq ADR 0282 slice 1, issue 382)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.62s)
The operator's budget (a leaf module running everywhere within five minutes
of its merge, a core module within ten) cannot be held to without knowing
where a walk's time goes. A walk now keeps when its batch's window closed,
when it was cut and its class (migration 0093), each gate reading, and what
each machine of the rest was sent; 'delivery walks' and plan-moved say its
phases from the merge to every machine of the rest reporting the build
applied, and ended walks keep them as walk-phase durations per class. Probe
D16 reads mesh-delivery's 'times' and raises delivery.<class>.over-budget.
A phase that cannot be measured is said unknown, never zero. Measurement
only: no walk is held, sent or judged differently.
2026-10-10 20:15:01 +02:00
mesh-admin 690b75f659 Merge pull request 'Say where a command line's quote is left open and how a quote is written (hq issue 294)' (#205) from fix/294-quote-in-a-quoted-value into main 2026-10-10 17:21:57 +00:00
jochen d52de218c8 Quote none of the line in the unclosed-quote refusal: a refusal is kept on the bus as the call's answer (hq issue 294)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 18:39:20 +02:00
jochen 6188e6b1da Say where a command line's quote is left open and how a quote is written (hq issue 294)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
An apostrophe inside a single-quoted value ends that quote, and the line then failed as a bare
'unclosed quote' with no position and no way out named. The splitter's rules stay the shell's; the
refusal now names the character, shows the line from there and gives the two ways to write a quote.
2026-10-10 18:36:47 +02:00
mesh-admin 798738cc12 Merge pull request 'A shrink is read against the item's own path: a moved directory starts its size history again (hq issue 368)' (#204) from fix/368-data-shrank-path-change into main 2026-10-10 15:56:12 +00:00
jochen babfef4f3a Keep one reading when two paths are measured at one moment, rather than failing the machine's record (hq issue 368 review)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 17:48:49 +02:00
jochen 96b0966ad8 Read a shrink against the item's own path, so a moved directory starts its size history again (hq issue 368)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A reading now keeps the path it was measured at, and the peak is read only from
readings at the item's path now. Before, an agent's home that moved to its own
account was compared with the operator's home it left, and data-shrank was
raised for data that was never lost. Existing readings take their item's path
now, so a shrink that is real stays raised.
2026-10-10 17:42:46 +02:00
215 changed files with 18445 additions and 839 deletions
+18 -10
View File
@@ -99,16 +99,22 @@ proxy-image:
@echo @echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes' @docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The whole gate. Raises a database, runs everything against it, and takes it down again -- # The whole check. Raises a database, runs the repository's own check against it -- merge-check.sh,
# including when the tests fail, which is why the teardown is not conditional. # the very script `mesh/repo-check` runs -- and takes the database down again, including when the
# check fails, which is why the teardown is not conditional and the script's exit status is the
# target's.
# #
# **Packages in parallel, under the race detector, each test on a bus of its own** (internal/testbus). # **It calls the script rather than restating it** (novox/hq issue 431): this target used to list its
# It was one package at a time against one shared bus, because the live tests assert, read and remove # own steps, its formatting step walked vendor/ where the script's does not, and it failed on
# the mesh's own objects by their fixed names, and two packages at once deleted what the other read; the # third-party code no change could fix -- so a developer's check and the gate disagreed, and the steps
# suite was red run as Go runs it and read as noise. A bus per test, of the release the mesh runs, made # after formatting never ran through it. The script is the one list of steps; this target only gives it
# it the same in any order. The timeout bounds a hang to a failure with a stack, never a stalled gate. # a database (MESH_TEST_POSTGRES, exported above).
check: fmt vet postgres #
@go test -race -timeout 15m ./... ; status=$$? ; $(MAKE) postgres-stop ; exit $$status # **Packages in parallel, under the race detector, each test on a bus of its own** (internal/testbus),
# as the script runs them: a bus per test, of the release the mesh runs, makes the suite the same in any
# order, and the timeout bounds a hang to a failure with a stack, never a stalled gate.
check: postgres
@sh merge-check.sh ; status=$$? ; docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
# Without a database the store's tests skip rather than fail, so this is the honest subset and not # Without a database the store's tests skip rather than fail, so this is the honest subset and not
# the gate. # the gate.
@@ -118,8 +124,10 @@ test:
vet: vet:
go vet ./... go vet ./...
# Quick steps for a developer, not part of `check`. The directories gofmt reads must be the ones
# merge-check.sh lists, never `.`, which walks vendor/ (novox/hq issue 431).
fmt: fmt:
@unformatted=$$(gofmt -l . 2>/dev/null) ; \ @unformatted=$$(gofmt -l cmd internal examples) ; \
if [ -n "$$unformatted" ] ; then echo "not gofmt'd:" ; echo "$$unformatted" ; exit 1 ; fi if [ -n "$$unformatted" ] ; then echo "not gofmt'd:" ; echo "$$unformatted" ; exit 1 ; fi
postgres: postgres:
@@ -0,0 +1,180 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **A refusal the bus said once is counted once** (novox/hq issue 402). The bus refused the controller
// one publish at 18:52:59 UTC on 2026-10-10 and never again, yet S9 looked at the one remembered
// advisory every half minute for the hour it is kept, and the condition said "observed 15 time(s), last
// 13s ago" with a refusal in its evidence every 30 seconds: two of us read it as a refusal going on and
// went looking for a missing grant. The condition counts what the bus said, with when it last said it,
// never how often the controller looked.
func TestARefusalSaidOnceIsCountedOnceHoweverOftenItIsLookedAt(t *testing.T) {
refused := time.Date(2026, 10, 10, 18, 52, 59, 0, time.UTC)
now := refused.Add(10 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
// The words the live condition bus.controller.refused carried on 2026-10-10.
said := "the bus refused the controller: nats: permissions violation: Permissions Violation for " +
`Publish to "mesh.seat.mesh-delivery.tool.times"`
advisory := link.Advisory{Kind: link.AdvisoryRefused, ID: "controller", Said: said,
First: refused, Last: refused, Count: 1}
look := func() conditions.Condition {
t.Helper()
f := &signalFacts{now: now, host: "novox", advisories: []link.Advisory{advisory}}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
c, found, err := conditions.ReadOne(t.Context(), store, "bus.controller.refused")
if err != nil || !found {
t.Fatalf("bus.controller.refused: found %v, %v", found, err)
}
return c
}
var c conditions.Condition
for range 15 {
c = look()
now = now.Add(30 * time.Second)
}
if c.Observations != 1 || len(c.Evidence) != 1 {
t.Fatalf("one refusal, looked at 15 times, reads as observed %d time(s) with %d evidence lines: %+v",
c.Observations, len(c.Evidence), c.Evidence)
}
if !c.LastObserved.Equal(refused) || !c.Evidence[0].At.Equal(refused) {
t.Fatalf("last observed %s, evidence at %s: the refusal was at %s", c.LastObserved, c.Evidence[0].At, refused)
}
// The bus refuses it three times more between two looks: the condition counts four refusals, says
// the newest, and adds one line of evidence for the look that saw them.
again := now.Add(-5 * time.Second)
advisory.Last, advisory.Count = again, 4
c = look()
if c.Observations != 4 || len(c.Evidence) != 2 || !c.LastObserved.Equal(again) || !c.Evidence[0].At.Equal(again) {
t.Fatalf("four refusals read as observed %d time(s), last %s, evidence %+v", c.Observations, c.LastObserved, c.Evidence)
}
if !strings.Contains(c.Summary, "(4 times since 18:52 UTC)") {
t.Fatalf("summary %q", c.Summary)
}
now = now.Add(30 * time.Second)
if c = look(); c.Observations != 4 || len(c.Evidence) != 2 {
t.Fatalf("a look with nothing new counted: observed %d time(s), evidence %+v", c.Observations, c.Evidence)
}
}
// **A condition that looks at the present still counts its looks** (novox/hq issue 402 changes only
// what reads a record): a machine silent for three looks was observed three times, and says so.
func TestAConditionOfThePresentCountsItsLooks(t *testing.T) {
now := time.Date(2026, 10, 10, 19, 0, 0, 0, time.UTC)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
var c conditions.Condition
for range 3 {
var err error
if c, err = k.Observe(t.Context(), conditions.Observation{Scope: conditions.ScopeMachine, ID: "ace",
Kind: "silent", Machine: "ace", Severity: conditions.Warning, Summary: "ace has not been heard from",
Source: "S1"}); err != nil {
t.Fatal(err)
}
now = now.Add(30 * time.Second)
}
if c.Observations != 3 || len(c.Evidence) != 3 || !c.LastObserved.Equal(now.Add(-30*time.Second)) {
t.Fatalf("observed %d time(s), %d evidence lines, last %s", c.Observations, len(c.Evidence), c.LastObserved)
}
}
// **One key, one watcher** (novox/hq issue 440). A consumer's max-deliveries condition is said from
// DEAD_LETTERS while it holds a message the consumer gave up on (issue 330). A max-deliveries advisory
// without a token names the same key; read beside it, each look added an observation and a line of
// evidence through the dead-letter half, and the two summaries overwrote each other on every look. The
// dead-letter row alone says that key, and counts the messages given up on, never the looks.
func TestAHeldDeadLetterIsCountedByWhatWasGivenUpNotByTheLooks(t *testing.T) {
gaveUp := time.Date(2026, 10, 10, 21, 4, 0, 0, time.UTC)
now := gaveUp.Add(20 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
const key = "bus.EVENTS.media_sonarr.max-deliveries"
held := map[string]int{"EVENTS.media_sonarr": 1}
newest := map[string]time.Time{"EVENTS.media_sonarr": gaveUp}
advisory := link.Advisory{Kind: link.AdvisoryMaxDeliveries, ID: "EVENTS.media_sonarr", Stream: "EVENTS",
Consumer: "media_sonarr", Said: "sonarr on media handed message 7 over 5 times and gave up on it",
First: gaveUp, Last: gaveUp, Count: 1}
look := func() conditions.Condition {
t.Helper()
f := &signalFacts{now: now, host: "novox", advisories: []link.Advisory{advisory},
deadLetters: held, deadLettersNewest: newest}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
c, found, err := conditions.ReadOne(t.Context(), store, key)
if err != nil || !found {
t.Fatalf("%s: found %v, %v", key, found, err)
}
if !strings.Contains(c.Summary, "dead-letters verb") {
t.Fatalf("the summary is not the dead-letter row's: %q", c.Summary)
}
return c
}
var c conditions.Condition
for range 5 {
c = look()
now = now.Add(30 * time.Second)
}
if c.Observations != 1 || len(c.Evidence) != 1 || !c.LastObserved.Equal(gaveUp) {
t.Fatalf("one message given up on, looked at five times, reads as observed %d time(s), last %s, "+
"with %d evidence lines: %+v", c.Observations, c.LastObserved, len(c.Evidence), c.Evidence)
}
// The consumer gives up on a second message between two looks: two given up on, one more line.
again := now.Add(-10 * time.Second)
held["EVENTS.media_sonarr"], newest["EVENTS.media_sonarr"] = 2, again
c = look()
if c.Observations != 2 || len(c.Evidence) != 2 || !c.LastObserved.Equal(again) {
t.Fatalf("two given up on read as observed %d time(s), last %s, evidence %+v", c.Observations,
c.LastObserved, c.Evidence)
}
now = now.Add(30 * time.Second)
if c = look(); c.Observations != 2 || len(c.Evidence) != 2 {
t.Fatalf("a look with nothing new counted: observed %d time(s), evidence %+v", c.Observations, c.Evidence)
}
}
// The advisory still says the max-deliveries it alone knows: a message given up on that could not be
// kept, under a key of its own (issue 330), which issue 440's change leaves as it was.
func TestAMessageThatCouldNotBeKeptIsStillSaidFromTheAdvisory(t *testing.T) {
at := time.Date(2026, 10, 10, 21, 4, 0, 0, time.UTC)
f := &signalFacts{now: at, host: "novox", advisories: []link.Advisory{{Kind: link.AdvisoryMaxDeliveries,
ID: "EVENTS.media_sonarr", Stream: "EVENTS", Consumer: "media_sonarr", Token: link.AdvisoryNotKept,
Said: "sonarr on media gave up on message 7, and it could not be kept", First: at, Last: at, Count: 1}}}
said := watchAdvisories(f)
if len(said) != 1 || said[0].Key() != "bus.EVENTS.media_sonarr.not-kept" {
t.Fatalf("%+v", said)
}
}
// A max-deliveries advisory without a token names a consumer's dead-letter key, which only DEAD_LETTERS
// says (novox/hq issues 330 and 440): with nothing held, the advisory alone raises nothing.
func TestAMaxDeliveriesAdvisoryAloneRaisesNothing(t *testing.T) {
at := time.Date(2026, 10, 10, 21, 4, 0, 0, time.UTC)
f := &signalFacts{now: at, host: "novox", advisories: []link.Advisory{{Kind: link.AdvisoryMaxDeliveries,
ID: "EVENTS.media_sonarr", Stream: "EVENTS", Consumer: "media_sonarr",
Said: "sonarr on media handed message 7 over 5 times and gave up on it", First: at, Last: at, Count: 1}}}
if said := watchAdvisories(f); len(said) != 0 {
t.Fatalf("said %+v", said)
}
}
+48 -1
View File
@@ -478,7 +478,8 @@ func spelledOf(m inventory.BatchedMerge) string {
// namedOf is one merge as a walk or batch names it: its commit, and its pull request and moves as announced. // namedOf is one merge as a walk or batch names it: its commit, and its pull request and moves as announced.
func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge { func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge {
k := announcedOf(m) k := announcedOf(m)
return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves} return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves,
Merged: m.Merged, Heard: m.Heard}
} }
// laterMerge says a was merged after b: by the forge's merge time, then by when each was heard. // laterMerge says a was merged after b: by the forge's merge time, then by when each was heard.
@@ -772,6 +773,8 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
} }
plan.Delivery.Merges = named plan.Delivery.Merges = named
plan.Delivery.Alone = batch.Delivery != nil && batch.Delivery.Alone plan.Delivery.Alone = batch.Delivery != nil && batch.Delivery.Alone
// **Its moments and its class** (novox/hq ADR 0282 decision 6): measured, never acted on.
plan.Times = walkTimesAtCut(*batch, plan, entries, now)
if len(moved) == 0 { if len(moved) == 0 {
plan.State = inventory.PlanDone plan.State = inventory.PlanDone
plan.Tiers = [][]string{} plan.Tiers = [][]string{}
@@ -820,6 +823,50 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil return nil
} }
// walkTimesAtCut is a walk's own moments as it is cut (novox/hq ADR 0282 decision 6): when its batch's window
// closed — no merge for the window's length, or its maximum, whichever came first — when it was cut, and its
// class. A merge walked alone had no window.
func walkTimesAtCut(batch, walk inventory.Plan, entries []inventory.Entry, now time.Time) *inventory.PlanTimes {
cut := now
t := &inventory.PlanTimes{Cut: &cut, Class: classOf(walk, entries)}
if batch.Delivery != nil && batch.Delivery.Batch != nil {
w := batch.Delivery.Batch
closed := w.ClosesAt
if !w.AtMost.IsZero() && w.AtMost.Before(closed) {
closed = w.AtMost
}
if !closed.IsZero() {
if closed.After(now) {
closed = now
}
t.WindowClosed = &closed
}
}
return t
}
// resolverSeat is the seat of the mesh's resolver: a module claiming it is a core module (ADR 0282 decision 1).
const resolverSeat = "mesh-dns-resolver"
// classOf is a walk's class (novox/hq ADR 0282 decision 1): core when it walks a module on the controller's own
// path or one holding the mesh's resolver, leaf otherwise.
func classOf(walk inventory.Plan, entries []inventory.Entry) string {
resolvers := map[string]bool{}
for _, e := range entries {
for _, c := range e.Manifest.Claims {
if c.Name == resolverSeat {
resolvers[e.Manifest.Module] = true
}
}
}
for m := range walk.Modules {
if _, own := onTheControllersPath[m]; own || resolvers[m] {
return inventory.ClassCore
}
}
return inventory.ClassLeaf
}
// combinedMerges is a batch's merges as one merge per repository's branch: the latest, with every file the // combinedMerges is a batch's merges as one merge per repository's branch: the latest, with every file the
// merges of it changed and said to be a module's — on a linear trunk the latest contains the others. A file is // merges of it changed and said to be a module's — on a linear trunk the latest contains the others. A file is
// removed when the last merge of the batch that changed it removed it. merges are in the order they were made. // removed when the last merge of the batch that changed it removed it. merges are in the order they were made.
+2 -9
View File
@@ -151,13 +151,6 @@ func busCommand(ctx context.Context, args []string) error {
if len(args) > 0 && !strings.HasPrefix(args[0], "-") { if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
sub, args = args[0], args[1:] sub, args = args[0], args[1:]
} }
// The view's credential, a terminal line like a person's (bus_view.go).
switch sub {
case "view-credential":
return busViewCredential(ctx, args)
case "view-revoke":
return busViewRevoke(ctx, args)
}
set := flag.NewFlagSet("bus", flag.ContinueOnError) set := flag.NewFlagSet("bus", flag.ContinueOnError)
snapshot := set.String("snapshot-taken", "", "where the streams' snapshot a person took is, while the mesh takes none itself") snapshot := set.String("snapshot-taken", "", "where the streams' snapshot a person took is, while the mesh takes none itself")
reversible := set.Bool("reversible", false, "the new version can be undone by putting the old one back") reversible := set.Bool("reversible", false, "the new version can be undone by putting the old one back")
@@ -167,14 +160,14 @@ func busCommand(ctx context.Context, args []string) error {
if rest, err := parseAround(set, args); err != nil { if rest, err := parseAround(set, args); err != nil {
return err return err
} else if len(rest) > 0 { } else if len(rest) > 0 {
return errors.New(busUsage) return errors.New("bus [upgrade --why … --reversible|--irreversible [--snapshot-taken <where>]]")
} }
switch sub { switch sub {
case "": case "":
return busStatus(ctx) return busStatus(ctx)
case "upgrade": case "upgrade":
default: default:
return fmt.Errorf("bus says what a bus upgrade would do, or `bus upgrade`, `bus view-credential`, `bus view-revoke` — not %q", sub) return fmt.Errorf("bus says what a bus upgrade would do, or `bus upgrade` — not %q", sub)
} }
// Everything refused before anything is done. // Everything refused before anything is done.
if err := why.require("bus upgrade"); err != nil { if err := why.require("bus upgrade"); err != nil {
-117
View File
@@ -1,117 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
"strconv"
"strings"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// The view's credential: the one read-only user a page in a browser connects to the bus as, over the
// bus module's WebSocket listener (novox/hq research 036, gap G1; broker.KindView).
//
// **A terminal line, like a person's credential** (operator.go): printed once, never stored — the mesh
// keeps a hash — and revoked by forgetting the row, which the next composition of the user list makes
// real. There is one view; issuing it again rotates its password.
const busUsage = "bus [upgrade --why … --reversible|--irreversible [--snapshot-taken <where>] | view-credential | view-revoke]"
// busWebSocketPort is the port the bus module's WebSocket listener is published on, mirrored from the
// nats module's manifest (its `bus-websocket` opening), because the credential names where to connect
// and the controller does not read the module's configuration. Reached across the overlay only: the
// opening is from the mesh, and the mesh's filter admits nothing else.
const busWebSocketPort = 4223
func busViewCredential(ctx context.Context, args []string) error {
if len(args) != 0 {
return errors.New("bus view-credential takes nothing: there is one view, and this prints its credential once")
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
// Refused here rather than at the next composition, where it would stop the whole file.
if _, err := broker.PermissionsFor(broker.Principal{Kind: broker.KindView, PasswordHash: "x"}); err != nil {
return err
}
password, err := inv.MintBusPassword(ctx, inventory.BusUser{Username: broker.ViewUser, Kind: inventory.BusView})
if err != nil {
return err
}
where, err := broker.FromEnvironment()
if err != nil && !errors.Is(err, broker.ErrNotConfigured) {
return err
}
host := where.Address
if h, _, err := net.SplitHostPort(where.Address); err == nil {
host = h
}
websocket := ""
if host != "" {
websocket = "ws://" + net.JoinHostPort(host, strconv.Itoa(busWebSocketPort))
}
held, err := json.Marshal(struct {
WebSocket string `json:"websocket,omitempty"`
URL string `json:"url,omitempty"`
Fingerprint string `json:"fingerprint,omitempty"`
User string `json:"user"`
Password string `json:"password"`
InboxPrefix string `json:"inbox_prefix"`
Hears []string `json:"hears"`
Reads string `json:"reads"`
HowToRead string `json:"how_to_read"`
}{
WebSocket: websocket, URL: "nats://" + where.Address, Fingerprint: where.Fingerprint,
User: broker.ViewUser, Password: password,
// The client must make its inboxes under the view's own prefix: its subscribe grant is
// `_INBOX.view.>` and no wider (design 25 §4), and a client's default inbox is not under it.
InboxPrefix: "_INBOX." + broker.ViewUser,
Hears: broker.ViewHears, Reads: broker.ViewBucket,
HowToRead: "direct reads only, no watch (a consumer is refused): list with a request to $JS.API.DIRECT.GET.KV_" +
broker.ViewBucket + ` carrying {"multi_last":["$KV.` + broker.ViewBucket + `.>"]}, answered until a 204 status; ` +
"read one key with $JS.API.DIRECT.GET.KV_" + broker.ViewBucket + ".$KV." + broker.ViewBucket + ".<number> " +
"(nats.js: kvm.open(bucket, {allow_direct: true}), never create); re-read the key an event's number names",
})
if err != nil {
return err
}
fmt.Printf("issued the view, which hears %s and reads the bucket %s, and nothing else\n",
strings.Join(broker.ViewHears, ", "), broker.ViewBucket)
fmt.Println(" this is the only time the credential is printed; the mesh keeps a hash")
fmt.Println(" it works once the bus has been told, which is the next push to the machine holding mesh-broker —")
fmt.Println(" and while a new build of the bus module waits for that machine, the next `bus upgrade` a person starts,")
fmt.Println(" which is also what brings the WebSocket listener it connects through")
fmt.Println()
fmt.Println(string(held))
return nil
}
func busViewRevoke(ctx context.Context, args []string) error {
if len(args) != 0 {
return errors.New("bus view-revoke takes nothing: there is one view")
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
if err := open.inventory.ForgetBusUser(ctx, broker.ViewUser); err != nil {
return err
}
// **Revoked at the next composition, not now** — as a person is (operator revoke): the bus's users
// are a file, and the credential stops working when the file no longer names it.
fmt.Println("the view is forgotten, and its credential stops working at the next composition — " +
"push the machine holding mesh-broker to make it so")
return nil
}
+4 -5
View File
@@ -6,8 +6,10 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"github.com/novox/mesh-controller/internal/catalogue"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
"github.com/novox/mesh-controller/internal/catalogue"
) )
// The check anybody can run is the check registration runs (novox/hq issue 148, ADR 0037): a manifest // The check anybody can run is the check registration runs (novox/hq issue 148, ADR 0037): a manifest
@@ -56,10 +58,7 @@ func TestModuleCheckJudgesBetweenTheManifestsGiven(t *testing.T) {
// The real catalogue passes the command, the way it passes the test that used to be the only check. // The real catalogue passes the command, the way it passes the test that used to be the only check.
func TestModuleCheckPassesTheCatalogue(t *testing.T) { func TestModuleCheckPassesTheCatalogue(t *testing.T) {
root := filepath.Join("..", "..", "..", "mesh-catalog", "modules") root := beside.Catalogue(t)
if _, err := os.Stat(root); err != nil {
t.Skipf("catalogue sibling not present: %v", err)
}
paths, err := manifestsUnder(root) paths, err := manifestsUnder(root)
if err != nil || len(paths) == 0 { if err != nil || len(paths) == 0 {
t.Fatalf("no manifests under %s: %v", root, err) t.Fatalf("no manifests under %s: %v", root, err)
+38
View File
@@ -169,6 +169,44 @@ func TestAShrinkOfMoreThanHalfIsUrgent(t *testing.T) {
} }
} }
// THE FALSE ALARM (issue 368), replayed through the store D13 reads: an agent's home moved from the
// operator's own home (94.7 MB) to the agent account's fresh one (490 B), and `data-shrank` was raised
// for data that was never lost. A moved item is read against its new path only, so nothing is raised —
// and a genuine shrink at the new path, a week of history later, still is.
func TestAMovedPathIsNoShrinkAndAShrinkThereStillIs(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
shelf := shelfFor(t, houseManifest)
declared := []inventory.DeclaredData{{Module: "house", Item: "config", Class: "irreplaceable", Owned: true}}
start := time.Now().Add(-6 * time.Hour)
measure := func(at time.Time, path string, size int64) []conditions.Observation {
t.Helper()
if _, err := inv.RecordData(ctx, "home", declared, map[string]map[string]inventory.Measurement{"house": {
"config": {Path: path, Size: bytesOf(size), MeasuredAt: when(at), LastWrite: when(at),
LastBackup: when(at)}}}, "", at); err != nil {
t.Fatal(err)
}
records, err := inv.Data(ctx)
if err != nil {
t.Fatal(err)
}
peaks, err := inv.DataPeaks(ctx, at.Add(-shrinkWindow))
if err != nil {
t.Fatal(err)
}
return dataFindings(records, peaks, shelf, nil, nil, at)
}
measure(start, "/home/operator/.claude", 94_700_000)
if got := findingsByKind(measure(start.Add(10*time.Minute), "/home/agent/.claude", 490)); got[kindDataShrank].Kind != "" {
t.Fatalf("a moved path raised a shrink: %+v", got[kindDataShrank])
}
measure(start.Add(2*time.Hour), "/home/agent/.claude", 300<<20)
got := findingsByKind(measure(start.Add(4*time.Hour), "/home/agent/.claude", 1<<20))[kindDataShrank]
if got.Severity != conditions.Urgent || !strings.Contains(got.Summary, "shrank") {
t.Fatalf("a genuine shrink at the new path was not raised: %+v", got)
}
}
// Data said to be written all the time and not written; data with no backup or an old one — urgent when // Data said to be written all the time and not written; data with no backup or an old one — urgent when
// irreplaceable, a warning when valuable; and a new item given its bound before it is said. // irreplaceable, a warning when valuable; and a new item given its bound before it is said.
func TestQuietDataAndMissingBackupsAreSaidByClass(t *testing.T) { func TestQuietDataAndMissingBackupsAreSaidByClass(t *testing.T) {
+5 -1
View File
@@ -144,11 +144,15 @@ func actOnDeadLetter(ctx context.Context, on *busHandles, act string, id uint64,
} }
switch act { switch act {
case "deliver": case "deliver":
_, to, err := link.DeliverAgain(on.js, id) delivered, to, err := link.DeliverAgain(on.js, id)
if err != nil { if err != nil {
return nil, err return nil, err
} }
answer["delivered_on"] = to answer["delivered_on"] = to
if delivered.Original != "" {
// What became of the ask in its seat's queue (novox/hq issue 334): removed, or left, and why.
answer["original"] = delivered.Original
}
answer["done"] = fmt.Sprintf("dead letter %d was delivered again to %s, and nobody else; it is no longer kept", answer["done"] = fmt.Sprintf("dead letter %d was delivered again to %s, and nobody else; it is no longer kept",
id, consumerWho(d.Stream, d.Consumer)) id, consumerWho(d.Stream, d.Consumer))
case "drop": case "drop":
+72
View File
@@ -596,12 +596,81 @@ func deliveryCommand(ctx context.Context, args []string) error {
if err != nil { if err != nil {
return err return err
} }
// Each walk's phases, with the rest's reports (novox/hq ADR 0282 decision 6).
now := time.Now()
for i := range walks {
walks[i].Phases = walks[i].WalkPhases(now, appliedFrom(ctx, inv))
}
return answer(map[string]any{"held": deliverySeatHeld(entries), "walks": walks, return answer(map[string]any{"held": deliverySeatHeld(entries), "walks": walks,
"own-path": sortedKeysOf(ownPathWords())}) "own-path": sortedKeysOf(ownPathWords())})
} }
return fmt.Errorf("delivery %s: plan, order, check, go, stop or walks", sub) return fmt.Errorf("delivery %s: plan, order, check, go, stop or walks", sub)
} }
// appliedFrom answers a machine's first report after a send from the controller's `apply` durations; a lookup
// that fails is a report not read, which leaves the walk's end unknown rather than wrong.
func appliedFrom(ctx context.Context, inv *inventory.Inventory) inventory.AppliedLookup {
return func(node string, sent time.Time) (inventory.AppliedReport, bool) {
r, ok, err := inv.FirstAppliedAfter(ctx, node, sent)
if err != nil {
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, sent.Format(time.RFC3339), err)
return inventory.AppliedReport{}, false
}
return r, ok
}
}
// recordWalkPhases keeps, once, each phase of every walk ended lately whose end is known, as a duration of kind
// walk-phase per class (novox/hq ADR 0282 decision 6): what `durations` summarises. A walk whose end is unknown
// past ApplySilentAfter keeps its measured phases without its total.
func recordWalkPhases(ctx context.Context, inv *inventory.Inventory, now time.Time) error {
recent, err := inv.RecentPlans(ctx, 30)
if err != nil {
return err
}
for _, p := range recent {
if p.State != inventory.PlanDone || p.Release != nil || now.Sub(p.Updated) > 2*time.Hour {
continue
}
anyKept, totalKept, err := inv.WalkPhasesKept(ctx, p.ID)
if err != nil {
return err
}
if totalKept {
continue
}
ph := p.WalkPhases(now, appliedFrom(ctx, inv))
if ph != nil && ph.End == nil && anyKept {
continue // kept without its end; kept again only once its end is known
}
if ph == nil || (ph.End == nil && now.Sub(p.Updated) < inventory.ApplySilentAfter+time.Minute) {
continue
}
class := ph.Class
if class == "" {
class = "unclassed"
}
for _, x := range ph.Phases {
if x.State != inventory.PhaseMeasured || x.Start == nil {
continue
}
if err := inv.RecordDuration(ctx, inventory.Duration{Kind: inventory.DurationWalkPhase,
Subject: class + "/" + x.Name, Ref: fmt.Sprintf("%s/%s/%d", p.ID, x.Name, x.Tier), Started: *x.Start,
Took: time.Duration(x.TookMS) * time.Millisecond, Detail: p.Named()}); err != nil {
return err
}
}
if ph.End != nil && ph.From != nil {
if err := inv.RecordDuration(ctx, inventory.Duration{Kind: inventory.DurationWalkPhase,
Subject: class + "/total", Ref: p.ID + "/total", Started: *ph.From,
Took: time.Duration(ph.TotalMS) * time.Millisecond, Detail: ph.Said}); err != nil {
return err
}
}
}
return nil
}
// ownPathWords is the controller's own path as words, for an answer. // ownPathWords is the controller's own path as words, for an answer.
func ownPathWords() map[string]string { return onTheControllersPath } func ownPathWords() map[string]string { return onTheControllersPath }
@@ -728,6 +797,9 @@ func sayPlanMoved(ctx context.Context, bus link.Bus, p inventory.Plan) {
} }
func publishPlanMoved(ctx context.Context, bus link.Bus, p inventory.Plan) { func publishPlanMoved(ctx context.Context, bus link.Bus, p inventory.Plan) {
// Its phases so far (novox/hq ADR 0282 decision 6): the rest's reports come after the walk ends, and are
// read by whoever asks for the walk (`delivery walks`).
p.Phases = p.WalkPhases(time.Now(), nil)
body, err := json.Marshal(p) body, err := json.Marshal(p)
if err != nil { if err != nil {
return return
+26 -4
View File
@@ -37,12 +37,34 @@ type stalledLine struct {
// Number is the pull request's, for a line of a head the forge never announced (novox/hq issue 347). // Number is the pull request's, for a line of a head the forge never announced (novox/hq issue 347).
Number int `json:"number,omitempty"` Number int `json:"number,omitempty"`
State string `json:"state"` State string `json:"state"`
For string `json:"for"` // Waiting are the merge checks a line of the state `unanswered` waits on, as mesh-delivery says each:
Bound string `json:"bound"` // "mesh/merge-gate pending since <UTC time>", "mesh/repo-check never set" (novox/hq issue 438).
H2 string `json:"h2"` Waiting []string `json:"waiting,omitempty"`
Says string `json:"says"` // Checks are the same merge checks as data, which the controller words in the operator's own time (novox/hq
// issue 443): a time inside a finished sentence cannot be said again in another zone. A mesh-delivery from before
// says none, and Waiting is said as it reads.
Checks []waitingCheck `json:"checks,omitempty"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
} }
// waitingCheck is one merge check a stalled line waits on, as mesh-delivery gives it: its context, its state —
// "pending", or "never-set" — and, for a pending one, since when in RFC 3339, empty when the forge did not say.
type waitingCheck struct {
Context string `json:"context"`
State string `json:"state"`
Since string `json:"since,omitempty"`
}
// wordsNow is the time the words are said at, which says whether a time needs its day; a seam a test replaces.
var wordsNow = time.Now
// wordsZone is the zone a stalled line's times are said in: the controller's local zone, as every other time in its
// messages. A seam a test replaces, so that no test writes time.Local, which every goroutine of the package reads.
var wordsZone = func() *time.Location { return time.Local }
// operatorsOnly is whether the table leaves H2 nothing to do for the line: the state is the operator's. // operatorsOnly is whether the table leaves H2 nothing to do for the line: the state is the operator's.
func (l stalledLine) operatorsOnly() bool { return l.H2 == "" || strings.HasPrefix(l.H2, "none") } func (l stalledLine) operatorsOnly() bool { return l.H2 == "" || strings.HasPrefix(l.H2, "none") }
+5
View File
@@ -116,6 +116,11 @@ var probeRegistry = []probe{
{ID: probeDeliveriesID, Asserts: "no delivery is held past its state's bound unsaid: mesh-delivery's " + {ID: probeDeliveriesID, Asserts: "no delivery is held past its state's bound unsaid: mesh-delivery's " +
"`stalled`, each with the transition its table lets healer H2 take", From: "ADR 0239", "`stalled`, each with the transition its table lets healer H2 take", From: "ADR 0239",
Kind: kindDeliveryStalled, Phase: 3, run: probeDeliveries}, Kind: kindDeliveryStalled, Phase: 3, run: probeDeliveries},
// The delivery budgets (novox/hq ADR 0282 decision 7): the newest delivery of each class within its budget,
// read from mesh-delivery's `times`; a measurement said, never a delivery held.
{ID: probeBudgetsID, Asserts: "the newest delivery of a leaf module ran on every machine within five minutes of " +
"its merge, and of a core module within ten: mesh-delivery's `times`", From: "ADR 0282",
Kind: kindOverBudget, Phase: 3, run: probeBudgets},
// A client of the bus reconnecting in a loop (novox/hq issue 327), from the server's record of closed // A client of the bus reconnecting in a loop (novox/hq issue 327), from the server's record of closed
// connections, which the bus's own module reads. // connections, which the bus's own module reads.
{ID: probeReconnectsID, Asserts: "no user of the bus had its connection dropped more than twelve times in the " + {ID: probeReconnectsID, Asserts: "no user of the bus had its connection dropped more than twelve times in the " +
@@ -0,0 +1,66 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A provider whose wait fails the controller's check lists who waits on it (novox/hq issue 450).
//
// A wait that does not check out is judged unhealthy (ADR 0283 decision 3), so the provider raises its unhealthy
// condition and its consumers are held under it (ADR 0240 rule 5). What is stored is what the machine said, the
// wait unchecked: read as said, the provider seems to wait for the operator, and the held consumers were never
// listed on the condition that is open.
// refusedWait is a wait for a secret the database's manifest does not declare: it fails the check.
var refusedWait = inventory.Wait{Part: "postgres-database", Secret: "certificate", What: "the database's certificate"}
// judgedRefused is the provider's resource as the controller judges it: unhealthy, saying why.
func judgedRefused() inventory.ResourceHealth {
r := waitingDatabaseFor(refusedWait)
r.State, r.Waits = link.StateUnhealthy, nil
r.Reason = "says it waits for the secret certificate, which db does not declare"
return r
}
// The consumer's statement arrives after the provider's, so it is the consumer's judging (sayWaiters) that must list
// it at the provider's unhealthy condition, made urgent by who waits on it.
func TestAProviderWhoseWaitFailedItsCheckListsWhoWaitsOnIt(t *testing.T) {
k, _ := withConditionsInMemory(t)
stored := waitingDatabaseFor(refusedWait)
open := judgeBoth(t, k, []inventory.ResourceHealth{judgedRefused()},
map[string]inventory.NodeHealth{"anchor": {Node: "anchor", Resources: []inventory.ResourceHealth{stored}}},
shopFailingBeside(stored))
provider := conditionOf(open, moduleUnhealthyKey("db", "anchor"))
if len(open) != 1 || provider == nil {
t.Fatalf("a provider whose wait failed its check and a consumer of it raised %v; want the provider's "+
"unhealthy alone", openKeysOf(open))
}
if said := provider.Evidence[0].Said; !strings.Contains(said, "shop on laptop") {
t.Fatalf("the provider's unhealthy condition does not list shop on laptop as waiting on it: %s", said)
}
if provider.Severity != conditions.Urgent {
t.Fatalf("the provider's unhealthy condition is %s with a consumer waiting on it; want urgent", provider.Severity)
}
}
// A provider that waits for a secret already given is unhealthy to its consumers too, before its own condition opens:
// they are held under it as under any unhealthy provider, never as waiting for the operator, and its condition, when
// open, is said as unhealthy with who waits on it.
func TestAProviderWaitingForASecretAlreadyGivenIsUnhealthyToItsConsumers(t *testing.T) {
given := holdingOf(nil, shopFailingBeside(waitingDatabase()), shopOnTheDatabase)
given.waits.given["db@anchor"] = map[string]time.Time{"licence": time.Now().Add(-time.Hour)}
by, held := given.heldWith("laptop", "shop", []inventory.ResourceHealth{failingConsumer("shop")})
if !held || by.provider != theDatabase || len(by.waits) > 0 {
t.Fatalf("shop under a database waiting for a licence already given: held %v under %v with waits %v; want "+
"held under db on anchor as unhealthy, no waits", held, by.provider, by.waits)
}
if _, uncovered := given.waitingUncovered("laptop", "shop", []inventory.ResourceHealth{failingConsumer("shop")}); uncovered {
t.Fatalf("a provider whose wait failed its check is said as waiting for another part")
}
}
+79 -11
View File
@@ -121,10 +121,13 @@ type gateFacts struct {
health map[string]inventory.NodeHealth health map[string]inventory.NodeHealth
healthErr error healthErr error
// heldOn is, per "<module>@<machine>", the provider its findings are held under (ADR 0240 rule 5). // heldOn is, per "<module>@<machine>", the provider its findings are held under (ADR 0240 rule 5).
heldOn map[string]string heldOn map[string]heldReading
// groupsAdded is, per module, whether the move judged puts an account in a group its previous build did // groupsAdded is, per module, whether the move judged puts an account in a group its previous build did
// not (issue 318 review): the only move whose wait for a new login is excused. // not (issue 318 review): the only move whose wait for a new login is excused.
groupsAdded map[string]bool groupsAdded map[string]bool
// waits is what the controller holds to check a module's wait for the operator (novox/hq ADR 0283): the
// manifest of each module's build judged, and the secrets given on each machine.
waits operatorWaitFacts
// sent is, per machine, the declaration the gate's own send carried there (novox/hq issue 352): a // sent is, per machine, the declaration the gate's own send carried there (novox/hq issue 352): a
// report is held against it, never against the send made last. sentBuilds is what each machine was // report is held against it, never against the send made last. sentBuilds is what each machine was
// last sent of every module, and judged the commit of each module this gate judges: a machine last // last sent of every module, and judged the commit of each module this gate judges: a machine last
@@ -134,6 +137,29 @@ type gateFacts struct {
commits map[string]string commits map[string]string
} }
// heldReading is the provider a module's findings are held under, as "<module> on <machine>", and, when that
// provider only waits for the operator for the part the module needs, its wait in one sentence (novox/hq issue
// 405): the module's gate then reads as a wait for a person (ADR 0254), a pass carrying the wait, as ADR 0283
// decision 4 reads the waiting provider itself. A walk never waits on the operator's secret, there or here.
type heldReading struct {
on, waits string
}
// heldReadings is, per "<module>@<machine>" in every machine's newest statement, what its findings are held under.
func heldReadings(hold *holding) map[string]heldReading {
out := map[string]heldReading{}
for machine := range hold.healths {
for module, by := range hold.heldModules(machine) {
reading := heldReading{on: by.provider.Module + " on " + by.provider.Node}
if len(by.waits) > 0 {
reading.waits = operatorWaitSaid(by.provider.Module, by.provider.Node, by.waits)
}
out[module+"@"+machine] = reading
}
}
return out
}
// reportedOn says a machine's last report is on what the gate sent it (novox/hq issue 352): on that // reportedOn says a machine's last report is on what the gate sent it (novox/hq issue 352): on that
// declaration, or one it was sent after it — or, for a gate kept before sends were kept on it, on the // declaration, or one it was sent after it — or, for a gate kept before sends were kept on it, on the
// declaration last sent. On 2026-10-09 a release's gate read the control node's report against a newer // declaration last sent. On 2026-10-09 a release's gate read the control node's report against a newer
@@ -199,12 +225,7 @@ var gatherGateFacts = func(ctx context.Context, open *stores, component string)
// Whose findings wait on an unhealthy provider (ADR 0240 rule 5): their gates wait, not fail. // Whose findings wait on an unhealthy provider (ADR 0240 rule 5): their gates wait, not fail.
if f.healthErr == nil && f.openErr == nil { if f.healthErr == nil && f.openErr == nil {
if hold, err := readHolding(ctx, inv, f.open); err == nil { if hold, err := readHolding(ctx, inv, f.open); err == nil {
f.heldOn = map[string]string{} f.heldOn = heldReadings(hold)
for machine := range f.health {
for module, p := range hold.heldModules(machine) {
f.heldOn[module+"@"+machine] = p.Module + " on " + p.Node
}
}
} }
} }
if theLease != nil { if theLease != nil {
@@ -257,10 +278,11 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
firstLine(f.openErr.Error()) firstLine(f.openErr.Error())
} }
for _, c := range f.open { for _, c := range f.open {
// A wait for a person's new login, or for a directory used as found to be handed over, is the module's // A wait for a person's new login, for a directory used as found to be handed over, or for the
// reading, not a fault raised since the send: the gate reads it from the statement below (ADR 0254, // operator's secret or setting, is the module's reading, not a fault raised since the send: the gate
// novox/hq issue 339). // reads it from the statement below (ADR 0254, novox/hq issue 339, ADR 0283).
if c.Source == gateProbe || c.OpenAt(since) || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound { if c.Source == gateProbe || c.OpenAt(since) || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound ||
c.Kind == kindNeedsOperator {
continue continue
} }
onIt := c.Subject.Machine == machine || slices.Contains(c.Subject.Also, machine) || onIt := c.Subject.Machine == machine || slices.Contains(c.Subject.Also, machine) ||
@@ -477,6 +499,7 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
return "", err return "", err
} }
facts.groupsAdded = movesAddingGroups(ctx, open.inventory, g, pairs, shelf) facts.groupsAdded = movesAddingGroups(ctx, open.inventory, g, pairs, shelf)
facts.waits = gateWaitFacts(ctx, open.inventory, g, pairs, shelf, facts.health)
facts.sent = g.Sent facts.sent = g.Sent
facts.commits, facts.sentBuilds = judgedCommits(g, pairs), map[string]map[string]string{} facts.commits, facts.sentBuilds = judgedCommits(g, pairs), map[string]map[string]string{}
// A module this gate put back at once (putBackBroken) was sent its earlier build by the gate itself: // A module this gate put back at once (putBackBroken) was sent its earlier build by the gate itself:
@@ -603,6 +626,7 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
pastBound := now.Sub(*g.Since) > gateBound pastBound := now.Sub(*g.Since) > gateBound
switch { switch {
case worst == healthBroken: case worst == healthBroken:
g.Read(now, false, g.BrokenWhy)
var judging []string var judging []string
for _, m := range modules { for _, m := range modules {
if reading[m] != healthBroken && !passedAlone(m) { if reading[m] != healthBroken && !passedAlone(m) {
@@ -621,8 +645,10 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
// Waiting on a provider that is unhealthy: not a pass, and not a failure at the bound either — // Waiting on a provider that is unhealthy: not a pass, and not a failure at the bound either —
// the provider's own condition says what is wrong (ADR 0240 rule 5). // the provider's own condition says what is wrong (ADR 0240 rule 5).
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
g.Read(now, false, why)
case worst == healthNotYet: case worst == healthNotYet:
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
g.Read(now, false, why)
if pastBound { if pastBound {
fail(fmt.Sprintf("not healthy within %s of its apply: %s", gateBound, why)) fail(fmt.Sprintf("not healthy within %s of its apply: %s", gateBound, why))
} }
@@ -630,6 +656,7 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
// Healthy, or waiting for a person (ADR 0254): a pass, the wait carried along in the verdict. // Healthy, or waiting for a person (ADR 0254): a pass, the wait carried along in the verdict.
g.Passes++ g.Passes++
g.LastPass, g.Last, g.Failing = &now, "", nil g.LastPass, g.Last, g.Failing = &now, "", nil
g.Read(now, true, "")
if g.Passes >= gatePasses && settled { if g.Passes >= gatePasses && settled {
decide(g, inventory.GatePassed, fmt.Sprintf("healthy %d times over %s", g.Passes, decide(g, inventory.GatePassed, fmt.Sprintf("healthy %d times over %s", g.Passes,
now.Sub(*g.Since).Round(time.Second))+waitsSaid(g.Waits), now) now.Sub(*g.Since).Round(time.Second))+waitsSaid(g.Waits), now)
@@ -747,6 +774,47 @@ func movesAddingGroups(ctx context.Context, inv *inventory.Inventory, g *invento
return out return out
} }
// gateWaitFacts reads what checks the waits for the operator of the modules a gate judges (novox/hq ADR 0283): the
// manifest of the build judged — the one it moves to, else the catalogue's — and the secrets given on each machine
// that says a module of them waits.
func gateWaitFacts(ctx context.Context, inv *inventory.Inventory, g *inventory.PlanGate, pairs []judged,
shelf map[string]catalogue.Manifest, health map[string]inventory.NodeHealth) operatorWaitFacts {
var f operatorWaitFacts
judgedManifests := map[string]catalogue.Manifest{}
byMachine := map[string][]string{}
for _, j := range pairs {
waiting := false
for _, r := range health[j.node].Resources {
if r.Module == j.module && r.State == link.StateWaiting {
waiting = true
}
}
if !waiting {
continue
}
byMachine[j.node] = append(byMachine[j.node], j.module)
if _, done := judgedManifests[j.module]; done {
continue
}
to := g.To
for _, c := range g.Carried {
if c.Module == j.module {
to = c.To
break
}
}
if m, found, err := inv.ManifestAt(ctx, j.module, to); err == nil && found {
judgedManifests[j.module] = m
} else if m, known := shelf[j.module]; known {
judgedManifests[j.module] = m
}
}
for machine, modules := range byMachine {
readWaitFacts(ctx, inv, machine, modules, judgedManifests, &f)
}
return f
}
// decide sets a gate's verdict. // decide sets a gate's verdict.
func decide(g *inventory.PlanGate, verdict, why string, now time.Time) { func decide(g *inventory.PlanGate, verdict, why string, now time.Time) {
g.Verdict, g.Why, g.JudgedAt = verdict, why, &now g.Verdict, g.Why, g.JudgedAt = verdict, why, &now
+1 -1
View File
@@ -311,7 +311,7 @@ func usage() {
the self-check: the last verdict, a run now, the probes, the signals' ages the self-check: the last verdict, a run now, the probes, the signals' ages
healers [--days N] [--json] the healers, what they did lately, and their brake (to-be 45 §7) healers [--days N] [--json] the healers, what they did lately, and their brake (to-be 45 §7)
durations [--kind K] [--days N] [--json] durations [--kind K] [--days N] [--json]
apply, heartbeat, plan-tier and build durations, per machine or module apply, heartbeat, plan-tier, build and walk-phase durations
collection [--json] kept archives held/unheld by a manifest, and what the sweep may let go collection [--json] kept archives held/unheld by a manifest, and what the sweep may let go
builder issue <name> a broker account for a build machine, scoped to build work, builder issue <name> a broker account for a build machine, scoped to build work,
delivered as the builder module's broker secret (module add it first) delivered as the builder module's broker secret (module add it first)
+166 -36
View File
@@ -74,9 +74,21 @@ func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Ke
kept := inventory.ResourceHealth{Module: r.Module, Resource: r.Resource, Kind: r.Kind, Target: r.Target, kept := inventory.ResourceHealth{Module: r.Module, Resource: r.Resource, Kind: r.Kind, Target: r.Target,
State: r.State, Reason: r.Reason, Since: r.Since, Streak: r.Streak, Restarts: r.Restarts, State: r.State, Reason: r.Reason, Since: r.Since, Streak: r.Streak, Restarts: r.Restarts,
Check: r.Check, Needs: r.Needs, Account: r.Account, Root: r.Root} Check: r.Check, Needs: r.Needs, Account: r.Account, Root: r.Root}
for _, w := range r.Waits {
kept.Waits = append(kept.Waits, inventory.Wait{Part: w.Part, Secret: w.Secret, Setting: w.Setting, What: w.What})
}
resources = append(resources, kept) resources = append(resources, kept)
if r.State == link.StateUnhealthy && r.Module != "" { }
unhealthy[r.Module] = append(unhealthy[r.Module], kept) // **A wait for the operator is checked before it is excused** (novox/hq ADR 0283): a waiting resource whose
// wait does not check out is judged unhealthy, saying why; one that does is kept beside the unhealthy ones, so
// judgeModuleHealth can say it as needs-operator. What is stored is what the machine said.
var wf operatorWaitFacts
if mods := waitingModules(resources); len(mods) > 0 {
readWaitFacts(ctx, inv, node, mods, nil, &wf)
}
for _, r := range checkWaiting(node, resources, wf) {
if (r.State == link.StateUnhealthy || r.State == link.StateWaiting) && r.Module != "" {
unhealthy[r.Module] = append(unhealthy[r.Module], r)
} }
} }
streaks := map[string]int{} streaks := map[string]int{}
@@ -135,16 +147,15 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
} }
standing := map[string]conditions.Condition{} standing := map[string]conditions.Condition{}
for _, c := range open { for _, c := range open {
if (c.Kind == kindModuleUnhealthy || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound) && if (c.Kind == kindModuleUnhealthy || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound ||
c.Kind == kindNeedsOperator) &&
c.Subject.Machine == node { c.Subject.Machine == node {
standing[c.Key] = c standing[c.Key] = c
} }
} }
var hold *holding hold, err := readHoldingFor(ctx, inv, open)
if inv != nil { if err != nil {
if hold, err = readHolding(ctx, inv, open); err != nil { return err
return err
}
} }
var problems []string var problems []string
modules := make([]string, 0, len(unhealthy)) modules := make([]string, 0, len(unhealthy))
@@ -159,6 +170,39 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
heldOn := map[string]string{} heldOn := map[string]string{}
providers := map[catalogue.Chosen]bool{} providers := map[catalogue.Chosen]bool{}
for _, m := range modules { for _, m := range modules {
// **A part that waits for the operator is said as that** (novox/hq ADR 0283): its waits already checked,
// the operator's, never urgent, its words naming the act.
if waits, waiting := operatorWait(m, unhealthy[m]); waiting {
o := needsOperatorObservation(m, node, waits, unhealthy[m])
// **A provider waiting for the operator says who waits on it** (novox/hq issue 405), as one waiting for a
// login does: its consumers are held under it.
if hold != nil {
sayWaitingOn(&o, hold.waitersOn(catalogue.Chosen{Node: node, Module: m}))
}
seen[o.Key()] = true
became[m] = kindNeedsOperator
if _, isOpen := standing[o.Key()]; streaks[m] < moduleUnhealthyAfter && !isOpen {
continue
}
if _, err := k.Observe(ctx, o); err != nil {
problems = append(problems, err.Error())
}
continue
}
// **A wait for the operator beside anything else is said too** (novox/hq issue 405): a module with a checked
// wait beside a new login owed, a directory used as found or a fault of its own is said as that, and the
// operator's secret or setting it waits for was not mentioned until the other cleared. Its needs-operator
// condition stands beside the other, on the same looks; what follows judges the rest without the wait.
if waits, rest := besideAWait(m, unhealthy[m]); len(waits) > 0 {
o := needsOperatorObservation(m, node, waits, waitingOf(m, unhealthy[m]))
seen[o.Key()] = true
if _, isOpen := standing[o.Key()]; streaks[m] >= moduleUnhealthyAfter || isOpen {
if _, err := k.Observe(ctx, o); err != nil {
problems = append(problems, err.Error())
}
}
unhealthy[m] = rest
}
// **A directory used as found is said as that** (novox/hq issue 339): the operator's to hand over at the // **A directory used as found is said as that** (novox/hq issue 339): the operator's to hand over at the
// machine, never urgent — nothing is broken by the wait that a person was not told of — and its own kind, // machine, never urgent — nothing is broken by the wait that a person was not told of — and its own kind,
// so the gate never reads it as a fault of the build that happened to be sent beside it. // so the gate never reads it as a fault of the build that happened to be sent beside it.
@@ -195,13 +239,24 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
} }
o := moduleUnhealthyObservation(m, node, unhealthy[m]) o := moduleUnhealthyObservation(m, node, unhealthy[m])
if hold != nil { if hold != nil {
if p, held := hold.heldUnder(node, m, unhealthy[m]); held { if by, held := hold.heldWith(node, m, unhealthy[m]); held {
// Held under the provider's condition: nothing of its own, and the provider's says it waits. // Held under the provider's condition: nothing of its own, and the provider's says it waits. The
heldOn[o.Key()] = p.Module + " on " + p.Node // clearing line says which the provider is: unhealthy, or waiting for the operator (issue 405).
p := by.provider
heldOn[o.Key()] = p.Module + " on " + p.Node + ", which is unhealthy"
if len(by.waits) > 0 {
heldOn[o.Key()] = p.Module + " on " + p.Node + ", which waits for you"
}
providers[p] = true providers[p] = true
continue continue
} }
sayWaitingOn(&o, hold.waitersOn(catalogue.Chosen{Node: node, Module: m})) sayWaitingOn(&o, hold.waitersOn(catalogue.Chosen{Node: node, Module: m}))
// A provider that waits for the operator for a part this finding cannot be matched to does not hold it
// (novox/hq issue 405): raised as its own, and saying the provider waits, so neither is hidden.
if p, waiting := hold.waitingUncovered(node, m, unhealthy[m]); waiting {
o.Said += fmt.Sprintf("; %s on %s waits for you, but not for anything %s is known to need, so %s's "+
"fault is said on its own", p.Module, p.Node, m, m)
}
} }
seen[o.Key()] = true seen[o.Key()] = true
became[m] = kindModuleUnhealthy became[m] = kindModuleUnhealthy
@@ -228,8 +283,11 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
if c.Kind == kindUsedAsFound { if c.Kind == kindUsedAsFound {
why = fmt.Sprintf("%s says no directory of %s is used as found any more", node, module) why = fmt.Sprintf("%s says no directory of %s is used as found any more", node, module)
} }
if c.Kind == kindNeedsOperator {
why = fmt.Sprintf("%s says %s no longer waits for the operator", node, module)
}
if on, held := heldOn[key]; held { if on, held := heldOn[key]; held {
why = fmt.Sprintf("what %s finds on %s waits on %s, which is unhealthy: held under its condition", module, node, on) why = fmt.Sprintf("what %s finds on %s waits on %s: held under its condition", module, node, on)
} }
// **A condition that became the other kind** is not "working again" (issue 318 review): its clearing // **A condition that became the other kind** is not "working again" (issue 318 review): its clearing
// line says what it became. // line says what it became.
@@ -238,6 +296,12 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
case c.Kind == kindModuleUnhealthy && became[module] == kindReloginNeeded: case c.Kind == kindModuleUnhealthy && became[module] == kindReloginNeeded:
why = fmt.Sprintf("%s on %s now waits only for a new login", module, node) why = fmt.Sprintf("%s on %s now waits only for a new login", module, node)
resolved = fmt.Sprintf("%s on %s now waits only for a new login", module, node) resolved = fmt.Sprintf("%s on %s now waits only for a new login", module, node)
case c.Kind == kindModuleUnhealthy && became[module] == kindNeedsOperator:
why = fmt.Sprintf("%s on %s now only waits for the operator", module, node)
resolved = fmt.Sprintf("%s on %s now only waits for you", module, node)
case c.Kind == kindNeedsOperator && became[module] == kindModuleUnhealthy:
why = fmt.Sprintf("%s on %s no longer only waits for the operator, and is not healthy", module, node)
resolved = fmt.Sprintf("What %s on %s waited for is given, and it still does not work", module, node)
case c.Kind == kindReloginNeeded && became[module] == kindModuleUnhealthy: case c.Kind == kindReloginNeeded && became[module] == kindModuleUnhealthy:
why = fmt.Sprintf("%s on %s no longer waits for a new login, and is not healthy", module, node) why = fmt.Sprintf("%s on %s no longer waits for a new login, and is not healthy", module, node)
resolved = fmt.Sprintf("The new login on %s is done, and %s still does not work", node, module) resolved = fmt.Sprintf("The new login on %s is done, and %s still does not work", node, module)
@@ -262,36 +326,74 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
// sayWaiters observes a provider's open condition again, with who waits on it, from its machine's newest // sayWaiters observes a provider's open condition again, with who waits on it, from its machine's newest
// statement. Nothing when its condition is not open: it is raised by its own statements, on its own looks. // statement. Nothing when its condition is not open: it is raised by its own statements, on its own looks.
func sayWaiters(ctx context.Context, k *conditions.Keeper, hold *holding, p catalogue.Chosen, now time.Time) error { func sayWaiters(ctx context.Context, k *conditions.Keeper, hold *holding, p catalogue.Chosen, now time.Time) error {
var raisedAt *conditions.Condition // Its statement as it was judged, its waits checked (novox/hq issue 450): a wait that failed the check is
for i, c := range hold.open { // unhealthy, so the condition built here is the one its own statement raised, and who waits on it is listed.
if c.Key == moduleUnhealthyKey(p.Module, p.Node) || c.Key == reloginKey(p.Module, p.Node) {
raisedAt = &hold.open[i]
}
}
if raisedAt == nil {
return nil
}
var rs []inventory.ResourceHealth var rs []inventory.ResourceHealth
for _, r := range hold.healths[p.Node].Resources { for _, r := range hold.checked(p.Node) {
if r.Module == p.Module && r.State == link.StateUnhealthy { if r.Module == p.Module && (r.State == link.StateUnhealthy || r.State == link.StateWaiting) {
rs = append(rs, r) rs = append(rs, r)
} }
} }
if len(rs) == 0 { if len(rs) == 0 {
return nil return nil
} }
o := moduleUnhealthyObservation(p.Module, p.Node, rs) // The condition its own statement says it under: needs-operator while it only waits for the operator (novox/hq
if said, waits := personWait(p.Module, p.Node, rs); waits { // issue 405), else that for the rest of it, a wait beside it said on its own.
o = reloginObservation(p.Module, p.Node, said, operatorOn(ctx, hold.inv, p.Node), rs) var o conditions.Observation
if waits, waiting := operatorWait(p.Module, rs); waiting {
o = needsOperatorObservation(p.Module, p.Node, waits, rs)
} else {
_, rest := besideAWait(p.Module, rs)
o = moduleUnhealthyObservation(p.Module, p.Node, rest)
if said, waits := personWait(p.Module, p.Node, rest); waits {
o = reloginObservation(p.Module, p.Node, said, operatorOn(ctx, hold.inv, p.Node), rest)
}
} }
if o.Key() != raisedAt.Key { raised := false
return nil // its own statement says it next for _, c := range hold.open {
raised = raised || c.Key == o.Key()
}
if !raised {
return nil // not open yet, or open as another kind: its own statement says it next
} }
sayWaitingOn(&o, hold.waitersOn(p)) sayWaitingOn(&o, hold.waitersOn(p))
_, err := k.Observe(ctx, o) _, err := k.Observe(ctx, o)
return err return err
} }
// besideAWait is, for a module with something not healthy beside a part waiting for the operator, the waits (checked
// already) and the rest without the waiting parts (novox/hq issue 405); no waits when nothing waits, or when the
// module only waits (operatorWait says that whole). Pure.
func besideAWait(module string, rs []inventory.ResourceHealth) ([]inventory.Wait, []inventory.ResourceHealth) {
if _, only := operatorWait(module, rs); only {
return nil, rs
}
var waits []inventory.Wait
var rest []inventory.ResourceHealth
for _, r := range rs {
if r.Module == module && r.State == link.StateWaiting {
waits = append(waits, r.Waits...)
continue
}
rest = append(rest, r)
}
if len(waits) == 0 {
return nil, rs
}
return waits, rest
}
// waitingOf is a module's waiting resources: the evidence of its wait.
func waitingOf(module string, rs []inventory.ResourceHealth) []inventory.ResourceHealth {
var out []inventory.ResourceHealth
for _, r := range rs {
if r.Module == module && r.State == link.StateWaiting {
out = append(out, r)
}
}
return out
}
// reloginKey is a module's relogin-needed condition on a machine. // reloginKey is a module's relogin-needed condition on a machine.
func reloginKey(module, node string) string { func reloginKey(module, node string) string {
return conditions.Key(conditions.ScopeModule, module+"."+node, kindReloginNeeded) return conditions.Key(conditions.ScopeModule, module+"."+node, kindReloginNeeded)
@@ -363,12 +465,15 @@ func waitingAccounts(module string, rs []inventory.ResourceHealth) []string {
} }
// sayWaitingOn adds to a module's condition the consumers held under it (to-be 48 §6): urgent while anyone // sayWaitingOn adds to a module's condition the consumers held under it (to-be 48 §6): urgent while anyone
// waits on it, whether it is not working or waits for a new login. // waits on it, whether it is not working or waits for a new login. **A wait for the operator's secret or setting
// stays a warning** (ADR 0283 decision 5, novox/hq issue 405): who waits on it is listed, and nothing escalates it.
func sayWaitingOn(o *conditions.Observation, waiters []string) { func sayWaitingOn(o *conditions.Observation, waiters []string) {
if len(waiters) == 0 { if len(waiters) == 0 {
return return
} }
o.Severity = conditions.Urgent if o.Kind != kindNeedsOperator {
o.Severity = conditions.Urgent
}
o.Said += "; " + waitingWords(waiters) o.Said += "; " + waitingWords(waiters)
o.Summary += fmt.Sprintf("; %d consumer(s) wait on it", len(waiters)) o.Summary += fmt.Sprintf("; %d consumer(s) wait on it", len(waiters))
o.Explanation += fmt.Sprintf(" %d module(s) that depend on it wait for it.", len(waiters)) o.Explanation += fmt.Sprintf(" %d module(s) that depend on it wait for it.", len(waiters))
@@ -420,6 +525,11 @@ func reasonWords(r inventory.ResourceHealth) string {
case "": case "":
return "is unhealthy" return "is unhealthy"
} }
// A wait for the operator that did not check out is said as the controller found it (ADR 0283): names of
// secrets and settings only, never what the check itself said.
if strings.HasPrefix(r.Reason, waitRefusedPrefix) {
return r.Reason
}
// What a declared check found says an endpoint, a path or an address: evidence, never the summary the // What a declared check found says an endpoint, a path or an address: evidence, never the summary the
// operator's channel carries (ADR 0234 §6). The summary names the check. // operator's channel carries (ADR 0234 §6). The summary names the check.
if r.Check != "" { if r.Check != "" {
@@ -511,7 +621,9 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
if h.HeardAt.Before(since) { if h.HeardAt.Before(since) {
return healthNotYet, fmt.Sprintf("%s has not said how what %s runs is since it was sent", machine, module) return healthNotYet, fmt.Sprintf("%s has not said how what %s runs is since it was sent", machine, module)
} }
wait, waits := personWait(module, machine, h.Resources) // **A wait for the operator is checked first** (novox/hq ADR 0283): one that does not check out is unhealthy.
resources := checkWaiting(machine, h.Resources, f.waits)
wait, waits := personWait(module, machine, resources)
// **Only a build whose own send put the account in a new group is excused** (issue 318 review): read from // **Only a build whose own send put the account in a new group is excused** (issue 318 review): read from
// what the controller sent, never from when the machine says the wait began — that time is the engine's // what the controller sent, never from when the machine says the wait began — that time is the engine's
// memory, reset by its restart and moved by a change of words. A build that adds no account group cannot // memory, reset by its restart and moved by a change of words. A build that adds no account group cannot
@@ -519,11 +631,18 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
if waits && !f.groupsAdded[module] { if waits && !f.groupsAdded[module] {
waits = false waits = false
} }
var found []string var found, onWaiting []string
for _, r := range h.Resources { var forOperator []inventory.Wait
for _, r := range resources {
if r.Module != module { if r.Module != module {
continue continue
} }
// **Any build is excused while its wait for the operator checks out** (ADR 0283 decision 4): a secret not
// given is owed by every build alike, so it is no fault of this one, and the verdict carries it.
if r.State == link.StateWaiting {
forOperator = append(forOperator, r.Waits...)
continue
}
if waits && r.State == link.StateUnhealthy { if waits && r.State == link.StateUnhealthy {
continue continue
} }
@@ -541,8 +660,16 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
return healthNotYet, fmt.Sprintf("its %s %s on %s is still starting", r.Kind, r.Resource, machine) return healthNotYet, fmt.Sprintf("its %s %s on %s is still starting", r.Kind, r.Resource, machine)
case link.StateUnhealthy: case link.StateUnhealthy:
if on, held := f.heldOn[module+"@"+machine]; held { if on, held := f.heldOn[module+"@"+machine]; held {
// **Held under a provider that only waits for the operator** (novox/hq issue 405): a wait for a
// person, a pass carrying the wait (ADR 0254, ADR 0283 decision 4) — the walk never waits for the
// operator's secret, whichever module owes it.
if on.waits != "" {
onWaiting = append(onWaiting, fmt.Sprintf("its %s %s on %s waits on %s, which waits for you: %s",
r.Kind, r.Resource, machine, on.on, on.waits))
continue
}
return healthWaiting, fmt.Sprintf("its %s %s on %s waits on %s, which is unhealthy", r.Kind, return healthWaiting, fmt.Sprintf("its %s %s on %s waits on %s, which is unhealthy", r.Kind,
r.Resource, machine, on) r.Resource, machine, on.on)
} }
return healthNotYet, fmt.Sprintf("its %s %s on %s %s", r.Kind, r.Resource, machine, reasonWords(r)) return healthNotYet, fmt.Sprintf("its %s %s on %s %s", r.Kind, r.Resource, machine, reasonWords(r))
default: default:
@@ -550,11 +677,14 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
reasonAfter(r.Reason)) reasonAfter(r.Reason))
} }
} }
if waits || len(found) > 0 { if waits || len(found) > 0 || len(forOperator) > 0 || len(onWaiting) > 0 {
var said []string said := onWaiting
if waits { if waits {
said = append(said, wait) said = append(said, wait)
} }
if len(forOperator) > 0 {
said = append(said, operatorWaitSaid(module, machine, forOperator))
}
if len(found) > 0 { if len(found) > 0 {
said = append(said, fmt.Sprintf("on %s, %s uses %s as found and waits for the operator to hand it over "+ said = append(said, fmt.Sprintf("on %s, %s uses %s as found and waits for the operator to hand it over "+
"(`nox node hand-over %s <directory>` on the control-node)", machine, module, strings.Join(found, ", "), machine)) "(`nox node hand-over %s <directory>` on the control-node)", machine, module, strings.Join(found, ", "), machine))
+5 -4
View File
@@ -3,10 +3,12 @@ package main
import ( import (
"context" "context"
"os" "os"
"path/filepath"
"reflect" "reflect"
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
"github.com/novox/mesh-controller/internal/catalogue" "github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory" "github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay" "github.com/novox/mesh-controller/internal/overlay"
@@ -239,13 +241,12 @@ func TestATakeoverIsNotComposedForAHubPlacedOffItsTunnel(t *testing.T) {
} }
} }
// theResolver is the catalogue's dnsmasq module as it is, or the test is skipped where the // theResolver is the catalogue's dnsmasq module as it is (internal/beside).
// catalogue is not beside this checkout.
func theResolver(t *testing.T) catalogue.Manifest { func theResolver(t *testing.T) catalogue.Manifest {
t.Helper() t.Helper()
raw, err := os.ReadFile("../../../mesh-catalog/modules/dnsmasq/module.json") raw, err := os.ReadFile(filepath.Join(beside.Catalogue(t), "dnsmasq", "module.json"))
if err != nil { if err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err) t.Fatal(err)
} }
m, err := catalogue.ParseManifest(raw) m, err := catalogue.ParseManifest(raw)
if err != nil { if err != nil {
+261
View File
@@ -0,0 +1,261 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A part that waits for the operator's secret or setting (novox/hq ADR 0283, issue 386).
//
// **A module's tool check may say it waits**: nothing of it is wrong but a part that cannot work until the operator
// gives one of its own secrets or one of its settings. The node-engine states such a resource `waiting`, with what
// it waits for. A module saying so is an assertion, so **the controller checks each wait before it excuses it**:
//
// - a wait for a secret names an own secret the module's manifest declares — by its name, or as a member of a
// secret family — said `"issued-by": "outside"`, and the store holds no value a person gave for it on that
// machine;
// - a wait for a setting names a setting the manifest declares (checked by name only: the controller cannot tell
// whether a free-form value covers a part, and the needs-operator condition is where a false one shows).
//
// An excused wait is read by the first-node gate as *waits for a person* (ADR 0254), a pass carried in the verdict,
// for any build of the module — a secret not given is owed by every build alike. It is said to the operator as
// `module.<module>.<machine>.needs-operator`, naming the act. A wait that fails the check is judged unhealthy, saying
// why, and raises the module's `unhealthy` condition.
// kindNeedsOperator is a module's condition while a part of it waits for the operator's secret or setting.
const kindNeedsOperator = "needs-operator"
// needsOperatorKey is a module's needs-operator condition on a machine.
func needsOperatorKey(module, node string) string {
return conditions.Key(conditions.ScopeModule, module+"."+node, kindNeedsOperator)
}
// operatorWaitFacts is what the controller holds to check a module's waits: the manifest judged per module, and per
// "<module>@<machine>" the own secrets a person gave there, with when. A module or a machine absent is not known,
// and no wait of it is excused.
type operatorWaitFacts struct {
manifests map[string]catalogue.Manifest
given map[string]map[string]time.Time
}
// checkWait is nil when a wait is excused, and otherwise why not, in words. Pure.
func checkWait(module, machine string, w inventory.Wait, f operatorWaitFacts) error {
m, known := f.manifests[module]
if !known {
return fmt.Errorf("says it waits for %s, and the mesh holds no manifest of %s to check it against", waitNames(w), module)
}
switch {
case w.Secret != "" && w.Setting != "", w.Secret == "" && w.Setting == "":
return fmt.Errorf("says it waits, naming %s, where a wait names one secret or one setting", waitNames(w))
case w.Setting != "":
if _, declared := m.Settings[w.Setting]; !declared {
return fmt.Errorf("says it waits for the setting %s, which %s does not declare", w.Setting, module)
}
return nil
}
own, _, declared := m.OwnSecrets.Lookup(w.Secret)
if !declared {
return fmt.Errorf("says it waits for the secret %s, which %s does not declare", w.Secret, module)
}
if own.IssuedBy != catalogue.IssuedOutside {
return fmt.Errorf("says it waits for the secret %s, which the mesh makes itself: only a secret issued outside "+
"the mesh waits for the operator", w.Secret)
}
given, readable := f.given[module+"@"+machine]
if !readable {
return fmt.Errorf("says it waits for the secret %s, and what was given on %s could not be read", w.Secret, machine)
}
if at, was := given[w.Secret]; was {
return fmt.Errorf("says it waits for the secret %s, which was given at %s", w.Secret,
at.UTC().Format("2006-01-02 15:04 MST"))
}
return nil
}
// waitRefusedPrefix opens every reason checkWait gives, so the words of a refused wait are told from a check's own.
const waitRefusedPrefix = "says it waits"
// waitNames is what a wait names, as "the secret x" or "the setting y".
func waitNames(w inventory.Wait) string {
switch {
case w.Secret != "" && w.Setting != "":
return "the secret " + w.Secret + " and the setting " + w.Setting
case w.Secret != "":
return "the secret " + w.Secret
case w.Setting != "":
return "the setting " + w.Setting
}
return "nothing"
}
// checkWaiting reads one machine's resources against the facts: every waiting resource whose waits all check out is
// kept as said; one with a wait that does not, or with no wait at all, is answered as unhealthy with why. Pure; the
// statement as kept is not changed.
func checkWaiting(machine string, rs []inventory.ResourceHealth, f operatorWaitFacts) []inventory.ResourceHealth {
out := make([]inventory.ResourceHealth, 0, len(rs))
for _, r := range rs {
if r.State == link.StateWaiting {
var why error
if len(r.Waits) == 0 {
why = fmt.Errorf("says it waits, and names nothing it waits for")
}
for _, w := range r.Waits {
if why == nil {
why = checkWait(r.Module, machine, w, f)
}
}
if why != nil {
r.State, r.Reason = link.StateUnhealthy, why.Error()
}
}
out = append(out, r)
}
return out
}
// operatorWait is whether everything not healthy of a module on a machine is waiting with its waits checked
// (checkWaiting already applied), and those waits. A module with anything unhealthy, starting or unknown beside it
// does not wait: it is judged as before.
func operatorWait(module string, rs []inventory.ResourceHealth) ([]inventory.Wait, bool) {
var waits []inventory.Wait
for _, r := range rs {
if r.Module != module {
continue
}
switch r.State {
case link.StateHealthy:
case link.StateWaiting:
waits = append(waits, r.Waits...)
default:
return nil, false
}
}
return waits, len(waits) > 0
}
// operatorWaitSaid is a module's wait for the operator in one sentence, for the gate's verdict and the condition's
// summary: what the operator gives and what it names, and for a secret the line that opens the desk prompt.
func operatorWaitSaid(module, machine string, waits []inventory.Wait) string {
var parts []string
for _, w := range waits {
part := fmt.Sprintf("%s (%s", w.What, waitNames(w))
if w.Secret != "" {
part += fmt.Sprintf(", given with `nox secret ask %s %s %s`", machine, module, w.Secret)
}
parts = append(parts, part+")")
}
return fmt.Sprintf("%s on %s waits for the operator: %s", module, machine, strings.Join(parts, "; "))
}
// needsOperatorObservation is a module whose only parts not healthy wait for the operator (ADR 0283): the operator's,
// a warning however long it stands, its plain words naming the act and never saying there is nothing to do.
func needsOperatorObservation(module, node string, waits []inventory.Wait, rs []inventory.ResourceHealth) conditions.Observation {
o := moduleUnhealthyObservation(module, node, rs)
o.Token, o.Kind, o.Resolver, o.Severity = kindNeedsOperator, kindNeedsOperator, conditions.ResolverOperator, conditions.Warning
o.Summary = operatorWaitSaid(module, node, waits)
w := needsOperatorWords(module, node, waits)
o.Headline, o.Explanation, o.Needs, o.Resolved, o.Actions = w.Headline, w.Explanation, w.Needs, w.Resolved, nil
return o
}
// needsOperatorWords is what the operator reads of a module waiting for them (ADR 0253, ADR 0283): the act, for a
// secret typed at the machine's desk prompt and for a setting approved when an agent proposes it. The secret's and
// the setting's names, and the line, are in the summary for whoever looks closer.
func needsOperatorWords(module, node string, waits []inventory.Wait) words {
var acts []string
seen := map[string]bool{}
secret := false
for _, w := range waits {
var act string
switch {
case w.Secret != "":
act, secret = fmt.Sprintf("type %s at %s's desk prompt", w.What, node), true
case w.Setting != "":
act = fmt.Sprintf("approve %s of %s on %s when it is proposed to you", w.Setting, module, node)
}
if act != "" && !seen[act] {
seen[act] = true
acts = append(acts, act)
}
}
needs := strings.Join(acts, "; and ") + "."
// Plain words hold one sentence of at most conditions.NeedsMax characters: several acts are named in the
// summary instead.
if len(acts) == 0 || len(needs) > conditions.NeedsMax {
needs = fmt.Sprintf("give what %s waits for on %s; the details name each secret and setting.", module, node)
}
explanation := fmt.Sprintf("Part of %s on %s cannot work until you give what it waits for.", module, node)
if secret {
explanation += " A hidden prompt opens at the desk when the secret is asked for, and what you type there " +
"is sealed to the machine."
}
explanation += " Its update is in place and nothing was undone; it carries on by itself once it is given."
return words{
Headline: fmt.Sprintf("%s waits for you on %s", module, node),
Needs: needs,
Explanation: explanation,
Resolved: fmt.Sprintf("%s on %s no longer waits for you", module, node),
}
}
// readWaitFacts reads what the controller holds to check the waits of the modules named on one machine: the
// manifests (the catalogue's, or those given) and the secrets given there. A read that fails leaves that module
// unknown, so none of its waits is excused.
func readWaitFacts(ctx context.Context, inv *inventory.Inventory, machine string, modules []string,
manifests map[string]catalogue.Manifest, f *operatorWaitFacts) {
if f.manifests == nil {
f.manifests = map[string]catalogue.Manifest{}
}
if f.given == nil {
f.given = map[string]map[string]time.Time{}
}
if inv == nil {
return
}
var shelf map[string]catalogue.Manifest
sort.Strings(modules)
for _, module := range modules {
if _, has := f.manifests[module]; !has {
if m, given := manifests[module]; given {
f.manifests[module] = m
} else {
if shelf == nil {
var err error
if shelf, err = inv.Catalogue(ctx); err != nil {
shelf = map[string]catalogue.Manifest{}
}
}
if m, known := shelf[module]; known {
f.manifests[module] = m
}
}
}
if _, read := f.given[module+"@"+machine]; read {
continue
}
if given, err := inv.GivenOwnSecrets(ctx, machine, module); err == nil {
f.given[module+"@"+machine] = given
}
}
}
// waitingModules is every module with a waiting resource in a statement.
func waitingModules(rs []inventory.ResourceHealth) []string {
seen := map[string]bool{}
var out []string
for _, r := range rs {
if r.State == link.StateWaiting && r.Module != "" && !seen[r.Module] {
seen[r.Module] = true
out = append(out, r.Module)
}
}
return out
}
+264
View File
@@ -0,0 +1,264 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A part that waits for the operator's secret or setting (novox/hq ADR 0283, issue 386).
var mountsManifest = catalogue.Manifest{Module: "mounts", Version: "1",
Settings: map[string]catalogue.SettingDeclaration{"smb-users": {}, "sources": {}},
OwnSecrets: catalogue.OwnSecrets{
"smb-password-*": {Path: "/s/smb-password-*.secret", IssuedBy: catalogue.IssuedOutside},
"broker": {Path: "/s/broker"},
"made": {Path: "/s/made", Taken: catalogue.TakenAtStart},
"licence": {Path: "/s/licence", IssuedBy: catalogue.IssuedOutside},
}}
var passwordWait = inventory.Wait{Part: "the source games", Secret: "smb-password-games",
What: "the password of the source games"}
var usernameWait = inventory.Wait{Part: "the source games", Setting: "smb-users", What: "the username of the source games"}
func waitingResource(waits ...inventory.Wait) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: "mounts", Resource: "mounts.watch", Kind: "process",
Target: "mesh-mounts-watch.service", State: link.StateWaiting, Check: "tool",
Reason: "the source games waits for its password", Waits: waits}
}
func factsGiven(given map[string]time.Time) operatorWaitFacts {
return operatorWaitFacts{manifests: map[string]catalogue.Manifest{"mounts": mountsManifest},
given: map[string]map[string]time.Time{"mounts@workstation": given}}
}
// Rule 3: a wait is excused only when what it names is the module's, issued outside the mesh, and not given there.
func TestAWaitIsExcusedOnlyWhenItChecksOut(t *testing.T) {
at := time.Date(2026, 10, 10, 15, 8, 0, 0, time.UTC)
for _, c := range []struct {
name string
w inventory.Wait
f operatorWaitFacts
says string // "" when excused
}{
{"a member of an outside family, not given", passwordWait, factsGiven(nil), ""},
{"an outside secret by name, not given", inventory.Wait{Part: "p", Secret: "licence", What: "w"}, factsGiven(nil), ""},
{"a declared setting", usernameWait, factsGiven(nil), ""},
{"a member given", passwordWait, factsGiven(map[string]time.Time{"smb-password-games": at}), "was given at 2026-10-10 15:08"},
{"a secret not declared", inventory.Wait{Part: "p", Secret: "smb-credentials", What: "w"}, factsGiven(nil), "does not declare"},
{"a secret the mesh makes", inventory.Wait{Part: "p", Secret: "made", What: "w"}, factsGiven(nil), "mesh makes itself"},
{"the bus account", inventory.Wait{Part: "p", Secret: "broker", What: "w"}, factsGiven(nil), "mesh makes itself"},
{"a setting not declared", inventory.Wait{Part: "p", Setting: "logins", What: "w"}, factsGiven(nil), "does not declare"},
{"both", inventory.Wait{Part: "p", Secret: "licence", Setting: "smb-users", What: "w"}, factsGiven(nil), "one secret or one setting"},
{"neither", inventory.Wait{Part: "p", What: "w"}, factsGiven(nil), "one secret or one setting"},
{"no manifest known", passwordWait, operatorWaitFacts{}, "no manifest"},
{"what was given cannot be read", passwordWait,
operatorWaitFacts{manifests: map[string]catalogue.Manifest{"mounts": mountsManifest}}, "could not be read"},
} {
err := checkWait("mounts", "workstation", c.w, c.f)
switch {
case c.says == "" && err != nil:
t.Errorf("%s: refused: %v", c.name, err)
case c.says != "" && (err == nil || !strings.Contains(err.Error(), c.says)):
t.Errorf("%s: %v; want a refusal saying %q", c.name, err, c.says)
}
}
}
// A waiting resource whose wait does not check out, or that names nothing, is judged unhealthy, saying why.
func TestAWaitThatFailsItsCheckIsUnhealthy(t *testing.T) {
given := factsGiven(map[string]time.Time{"smb-password-games": time.Now()})
got := checkWaiting("workstation", []inventory.ResourceHealth{waitingResource(passwordWait)}, given)
if got[0].State != link.StateUnhealthy || !strings.Contains(got[0].Reason, "was given") {
t.Fatalf("a wait for a secret given: %+v", got[0])
}
got = checkWaiting("workstation", []inventory.ResourceHealth{waitingResource()}, factsGiven(nil))
if got[0].State != link.StateUnhealthy || !strings.Contains(got[0].Reason, "names nothing") {
t.Fatalf("a wait naming nothing: %+v", got[0])
}
got = checkWaiting("workstation", []inventory.ResourceHealth{waitingResource(passwordWait, usernameWait)}, factsGiven(nil))
if got[0].State != link.StateWaiting {
t.Fatalf("two waits that check out: %+v", got[0])
}
}
// Rule 4: the gate passes a module whose only parts not healthy wait for the operator, carrying the wait; anything
// else beside it is judged as before; and any build is excused, not only one that added something.
func TestTheGatePassesAWaitForTheOperatorCarriedAlong(t *testing.T) {
now := time.Now()
since := now.Add(-time.Minute)
healthy := inventory.ResourceHealth{Module: "mounts", Resource: "mounts.apply", Kind: "process",
Target: "mesh-mounts-apply.service", State: link.StateHealthy}
f := gateFacts{now: now, waits: factsGiven(nil), groupsAdded: map[string]bool{"mounts": false},
health: map[string]inventory.NodeHealth{"workstation": {Node: "workstation", HeardAt: now,
Resources: []inventory.ResourceHealth{healthy, waitingResource(passwordWait)}}}}
h, why := moduleHealthWord("mounts", "workstation", since, f)
if h != healthPerson || !strings.Contains(why, "waits for the operator: the password of the source games") ||
!strings.Contains(why, "nox secret ask workstation mounts smb-password-games") {
t.Fatalf("an excused wait reads %v %q; want a wait for a person naming the act", h, why)
}
// A second resource unhealthy beside it: not yet, as before.
down := healthy
down.State, down.Reason = link.StateUnhealthy, "down"
f.health["workstation"] = inventory.NodeHealth{Node: "workstation", HeardAt: now,
Resources: []inventory.ResourceHealth{down, waitingResource(passwordWait)}}
if h, why := moduleHealthWord("mounts", "workstation", since, f); h != healthNotYet {
t.Fatalf("a resource down beside the wait reads %v %q", h, why)
}
// The password given and the module still saying it waits: not excused.
f.waits = factsGiven(map[string]time.Time{"smb-password-games": now})
f.health["workstation"] = inventory.NodeHealth{Node: "workstation", HeardAt: now,
Resources: []inventory.ResourceHealth{healthy, waitingResource(passwordWait)}}
if h, why := moduleHealthWord("mounts", "workstation", since, f); h != healthNotYet || !strings.Contains(why, "was given") {
t.Fatalf("a wait for a secret given reads %v %q", h, why)
}
// Facts never read (no manifest): never excused.
f.waits = operatorWaitFacts{}
if h, _ := moduleHealthWord("mounts", "workstation", since, f); h != healthNotYet {
t.Fatalf("a wait nothing could check reads %v", h)
}
}
func needsOperatorOpen(t *testing.T, k *conditions.Keeper) (*conditions.Condition, []conditions.Condition) {
t.Helper()
open, err := k.Open(t.Context())
if err != nil {
t.Fatal(err)
}
for i, c := range open {
if c.Key == needsOperatorKey("mounts", "workstation") {
return &open[i], open
}
}
return nil, open
}
// Rule 5: two statements of an excused wait raise needs-operator, the operator's, a warning however long, naming the
// act; a statement without it clears it.
func TestTheNeedsOperatorConditionNamesTheAct(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
rs := map[string][]inventory.ResourceHealth{"mounts": {waitingResource(passwordWait)}}
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": 1}, time.Now()); err != nil {
t.Fatal(err)
}
if got, _ := needsOperatorOpen(t, k); got != nil {
t.Fatal("raised on one statement")
}
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": 2}, time.Now()); err != nil {
t.Fatal(err)
}
got, open := needsOperatorOpen(t, k)
if got == nil {
t.Fatalf("not raised on two statements: %+v", open)
}
for _, c := range open {
if c.Kind == kindModuleUnhealthy {
t.Fatalf("raised as a fault too: %+v", c)
}
}
if got.Kind != kindNeedsOperator || got.Resolver != conditions.ResolverOperator || got.Severity != conditions.Warning {
t.Fatalf("the condition: %+v", got)
}
if !strings.Contains(got.Needs, "type the password of the source games at workstation's desk prompt") ||
!strings.Contains(got.Explanation, "hidden prompt opens at the desk") {
t.Fatalf("its needs do not name the act: %q", got.Needs)
}
if strings.Contains(strings.ToLower(got.Explanation), "nothing for you") || !strings.Contains(got.Explanation, "nothing was undone") {
t.Fatalf("its explanation: %q", got.Explanation)
}
if !strings.Contains(got.Summary, "smb-password-games") || !strings.Contains(got.Summary, "nox secret ask workstation mounts smb-password-games") {
t.Fatalf("its summary does not name the secret and the line: %q", got.Summary)
}
// Long open is still a warning: only the operator can end it.
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": 3}, time.Now().Add(48*time.Hour)); err != nil {
t.Fatal(err)
}
if got, _ := needsOperatorOpen(t, k); got == nil || got.Severity == conditions.Urgent {
t.Fatalf("after two days: %+v", got)
}
// Given: the next statement does not say it, and it clears.
if err := judgeModuleHealth(ctx, nil, k, "workstation", map[string][]inventory.ResourceHealth{}, nil, time.Now()); err != nil {
t.Fatal(err)
}
if got, _ := needsOperatorOpen(t, k); got != nil {
t.Fatal("not cleared once given")
}
}
func TestASettingsWaitAsksForTheApproval(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
rs := map[string][]inventory.ResourceHealth{"mounts": {waitingResource(usernameWait)}}
for i := 1; i <= 2; i++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": i}, time.Now()); err != nil {
t.Fatal(err)
}
}
got, _ := needsOperatorOpen(t, k)
if got == nil || !strings.Contains(got.Needs, "approve smb-users of mounts on workstation when it is proposed to you") {
t.Fatalf("the condition: %+v", got)
}
}
// A wait that fails its check is the module's own fault: unhealthy, with why, and no needs-operator.
func TestAWaitThatFailsItsCheckRaisesUnhealthy(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
checked := checkWaiting("workstation", []inventory.ResourceHealth{waitingResource(passwordWait)},
factsGiven(map[string]time.Time{"smb-password-games": time.Now()}))
rs := map[string][]inventory.ResourceHealth{"mounts": checked}
for i := 1; i <= 2; i++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", rs, map[string]int{"mounts": i}, time.Now()); err != nil {
t.Fatal(err)
}
}
got, open := needsOperatorOpen(t, k)
if got != nil {
t.Fatalf("a wait for a secret given raised needs-operator: %+v", got)
}
unhealthy := false
for _, c := range open {
unhealthy = unhealthy || c.Key == moduleUnhealthyKey("mounts", "workstation")
}
if !unhealthy {
t.Fatalf("not raised as unhealthy: %+v", open)
}
}
// A module that waited and is then broken says so when the wait clears, and the other way round.
func TestANeedsOperatorThatBecameUnhealthySaysSo(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
waiting := map[string][]inventory.ResourceHealth{"mounts": {waitingResource(passwordWait)}}
for i := 1; i <= 2; i++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", waiting, map[string]int{"mounts": i}, time.Now()); err != nil {
t.Fatal(err)
}
}
broken := waitingResource()
broken.State, broken.Reason, broken.Waits = link.StateUnhealthy, "the source games refused its login", nil
for i := 3; i <= 4; i++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", map[string][]inventory.ResourceHealth{"mounts": {broken}},
map[string]int{"mounts": i}, time.Now()); err != nil {
t.Fatal(err)
}
}
got, open := needsOperatorOpen(t, k)
if got != nil {
t.Fatal("needs-operator still open after it became a fault")
}
found := false
for _, c := range open {
found = found || c.Key == moduleUnhealthyKey("mounts", "workstation")
}
if !found {
t.Fatalf("the fault is not raised: %+v", open)
}
}
+124
View File
@@ -0,0 +1,124 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A delivery over its budget is loud (novox/hq ADR 0282 decision 7, issue 382): the self-check reads
// mesh-delivery's `times` and raises the warning `delivery.<class>.over-budget` when the newest delivery of a
// class whose delivery time is known took longer than its class's budget — five minutes for a leaf module, ten
// for a core module — naming the delivery and its longest phase. It clears when the next delivery of that class
// lands within its budget. Measurement only: the condition says, it never holds or acts on a delivery.
// probeBudgetsID is the probe that reads the delivery times.
const probeBudgetsID = "D16"
// kindOverBudget is what a delivery over its class's budget raises.
const kindOverBudget = "over-budget"
// deliveryBudgets are the budgets by class (ADR 0282 decision 1); the probe raises nothing for another class.
var deliveryBudgets = map[string]time.Duration{inventory.ClassLeaf: 5 * time.Minute, inventory.ClassCore: 10 * time.Minute}
// timesAnswer is what mesh-delivery's `times` answers, as far as the probe reads it.
type timesAnswer struct {
Classes []timesClass `json:"classes"`
}
// timesClass is one class's line of `times`.
type timesClass struct {
Class string `json:"class"`
Latest *timesLatest `json:"latest,omitempty"`
}
// timesLatest is the newest delivery of a class whose delivery time is known.
type timesLatest struct {
ID string `json:"id"`
TookMS int64 `json:"took_ms"`
Longest string `json:"longest,omitempty"`
Landed string `json:"landed,omitempty"`
}
// deliveryTimes is what the delivery's owner says of its delivery times; nothing when no holder is on record or
// none answers (D3 says that one).
func deliveryTimes(ctx context.Context, conn *nats.Conn, held bool) (*timesAnswer, error) {
if !held {
return nil, nil
}
raw, err := askDeliveryOwner(ctx, conn, "times", map[string]any{})
if errors.Is(err, link.ErrNothingServes) {
return nil, nil
}
if err != nil {
return nil, err
}
var a timesAnswer
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("%s.times answered something unreadable: %w", catalogue.DeliverySeat, err)
}
return &a, nil
}
// overBudgetObservations are the conditions of the classes whose newest delivery took longer than its budget:
// strictly longer, so a delivery of exactly its budget is within it.
func overBudgetObservations(a *timesAnswer) []conditions.Observation {
if a == nil {
return nil
}
var out []conditions.Observation
for _, c := range a.Classes {
budget, ok := deliveryBudgets[c.Class]
if !ok || c.Latest == nil {
continue
}
took := time.Duration(c.Latest.TookMS) * time.Millisecond
if took <= budget {
continue
}
longest := c.Latest.Longest
if longest == "" {
longest = "not known"
}
out = append(out, conditions.Observation{Scope: conditions.ScopeDelivery, ID: c.Class, Kind: kindOverBudget,
Severity: conditions.Warning,
Summary: fmt.Sprintf("the %s delivery %s took %s from its merge to running everywhere, over its budget of %s; "+
"its longest phase: %s — `mesh-delivery.times`", c.Class, c.Latest.ID, humanDuration(took),
humanDuration(budget), longest),
Said: fmt.Sprintf("%s took %s (budget %s), longest phase %s", c.Latest.ID, took.Round(time.Second), budget, longest),
Headline: fmt.Sprintf("A %s delivery took %s, over its %s budget", c.Class, humanDuration(took),
humanDuration(budget)),
Explanation: fmt.Sprintf("The delivery %s took %s from its merge until every machine ran it; a %s module is "+
"held to %s (ADR 0282). Most of the time went to %s. Nothing was held or changed because of this: it is "+
"a measurement.", c.Latest.ID, humanDuration(took), c.Class, humanDuration(budget), longest),
Resolved: fmt.Sprintf("the next %s delivery lands within %s", c.Class, humanDuration(budget)),
})
}
return out
}
// probeBudgets is D16: the newest delivery of each class lands within its class's budget.
func probeBudgets(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
entries, err := d.open.inventory.Catalogued(ctx)
if err != nil {
return nil, err
}
var conn *nats.Conn
if d.js != nil {
conn = d.js.Conn()
}
a, err := deliveryTimes(ctx, conn, deliverySeatHeld(entries))
if err != nil {
return nil, err
}
return overBudgetObservations(a), nil
}
+111
View File
@@ -0,0 +1,111 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A leaf delivery of 5 minutes 10 seconds raises delivery.leaf.over-budget naming its longest phase; one of
// exactly five minutes, a core one of nine and a class without a budget raise nothing (ADR 0282 decision 7).
func TestADeliveryOverItsBudgetIsLoud(t *testing.T) {
a := &timesAnswer{Classes: []timesClass{
{Class: inventory.ClassLeaf, Latest: &timesLatest{ID: "novox/mesh-catalog@abc", TookMS: (5*time.Minute + 10*time.Second).Milliseconds(),
Longest: "judgement 2m40s"}},
{Class: inventory.ClassCore, Latest: &timesLatest{ID: "novox/mesh-controller@def", TookMS: (9 * time.Minute).Milliseconds(),
Longest: "build 1m"}},
{Class: "unclassed", Latest: &timesLatest{ID: "x@y", TookMS: time.Hour.Milliseconds()}},
}}
obs := overBudgetObservations(a)
if len(obs) != 1 {
t.Fatalf("one class over its budget, got %d: %+v", len(obs), obs)
}
o := obs[0]
if o.Key() != "delivery.leaf.over-budget" || !strings.Contains(o.Summary, "judgement 2m40s") ||
!strings.Contains(o.Summary, "novox/mesh-catalog@abc") {
t.Fatalf("the condition names its delivery and longest phase: %s — %s", o.Key(), o.Summary)
}
// The next leaf delivery within its budget clears it: the probe raises nothing for the class.
a.Classes[0].Latest = &timesLatest{ID: "novox/mesh-catalog@ghi", TookMS: (5 * time.Minute).Milliseconds()}
if obs := overBudgetObservations(a); len(obs) != 0 {
t.Fatalf("a delivery of exactly its budget is within it: %+v", obs)
}
a.Classes[1].Latest.TookMS = (10*time.Minute + time.Second).Milliseconds()
if obs := overBudgetObservations(a); len(obs) != 1 || obs[0].Key() != "delivery.core.over-budget" {
t.Fatalf("a core delivery over ten minutes: %+v", obs)
}
if obs := overBudgetObservations(nil); obs != nil {
t.Fatal("no answer raises nothing")
}
// No delivery of a class with a known time yet: nothing said of it.
if obs := overBudgetObservations(&timesAnswer{Classes: []timesClass{{Class: inventory.ClassLeaf}}}); len(obs) != 0 {
t.Fatalf("a class with no delivery: %+v", obs)
}
}
// The probe's question is one the controller's grant names: a question the bus refuses checks nothing.
func TestTheControllerMayAskForTheDeliveryTimes(t *testing.T) {
found := false
for _, v := range broker.VerbsTheControllerAsksTheDeliveryOwner {
found = found || (v.Seat == catalogue.DeliverySeat && v.Verb == "times")
}
if !found {
t.Fatal("the grant does not name mesh-delivery.times")
}
}
// A walk's class is core when it walks a module of the controller's own path or one holding the mesh's resolver,
// leaf otherwise; its window closed at its batch's window, or its maximum, never after its cut.
func TestAWalksClassAndWindowAreReadAtItsCut(t *testing.T) {
entries := []inventory.Entry{
{Manifest: catalogue.Manifest{Module: "dnsmasq", Claims: []catalogue.Claim{{Name: resolverSeat}}}},
{Manifest: catalogue.Manifest{Module: "gitea"}},
}
walk := func(modules ...string) inventory.Plan {
p := inventory.Plan{Modules: map[string]*inventory.PlanModule{}}
for _, m := range modules {
p.Modules[m] = &inventory.PlanModule{}
}
return p
}
for want, w := range map[string]inventory.Plan{
inventory.ClassLeaf: walk("gitea"), inventory.ClassCore: walk("gitea", "mesh-controller"),
} {
if got := classOf(w, entries); got != want {
t.Errorf("%v: %s, want %s", w.Modules, got, want)
}
}
if got := classOf(walk("dnsmasq"), entries); got != inventory.ClassCore {
t.Errorf("the resolver's holder is core: %s", got)
}
now := time.Date(2026, 10, 10, 18, 0, 0, 0, time.UTC)
batch := inventory.Plan{Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{
ClosesAt: now.Add(-20 * time.Second), AtMost: now.Add(5 * time.Minute)}}}
times := walkTimesAtCut(batch, walk("gitea"), entries, now)
if times.Cut == nil || !times.Cut.Equal(now) || times.WindowClosed == nil || !times.WindowClosed.Equal(now.Add(-20*time.Second)) ||
times.Class != inventory.ClassLeaf {
t.Fatalf("times at the cut: %+v", times)
}
batch.Delivery.Batch.AtMost = now.Add(-time.Minute)
if times := walkTimesAtCut(batch, walk("gitea"), entries, now); !times.WindowClosed.Equal(now.Add(-time.Minute)) {
t.Fatalf("a window closed at its maximum: %v", times.WindowClosed)
}
if times := walkTimesAtCut(inventory.Plan{Delivery: &inventory.PlanDelivery{Alone: true}}, walk("gitea"), entries, now); times.WindowClosed != nil {
t.Fatalf("a merge walked alone had no window: %v", times.WindowClosed)
}
}
// What each machine of the rest was sent is kept only for the machines the send reached.
func TestTheRestIsKeptForTheMachinesTheSendReached(t *testing.T) {
got := restOf([]string{"ace", "g14"}, []string{"ace", "shanks"}, map[string]inventory.SentDeclaration{"ace": {Digest: "d1"}})
if len(got) != 1 || got["ace"].Digest != "d1" {
t.Fatalf("rest: %+v", got)
}
if restOf([]string{"g14"}, []string{"ace"}, nil) != nil {
t.Fatal("no machine reached: nothing kept")
}
}
+81
View File
@@ -221,6 +221,16 @@ var plainWordings = map[string]func(conditions.Observation) words{
w := usedAsFoundObservation(orModule(module), machineOr(o, "a machine"), o.Summary, nil) w := usedAsFoundObservation(orModule(module), machineOr(o, "a machine"), o.Summary, nil)
return words{Headline: w.Headline, Explanation: w.Explanation, Needs: w.Needs, Resolved: w.Resolved} return words{Headline: w.Headline, Explanation: w.Explanation, Needs: w.Needs, Resolved: w.Resolved}
}), }),
kindNeedsOperator: worded(func(o conditions.Observation) words {
// The observation carries the act itself (ADR 0283); these are its words when only the kind is known.
module := ""
if o.Scope == conditions.ScopeModule && o.Machine != "" {
module = strings.TrimSuffix(o.ID, "."+o.Machine)
}
node := machineOr(o, "a machine")
w := needsOperatorWords(orModule(module), node, nil)
return w
}),
kindProviderFailing: worded(func(o conditions.Observation) words { kindProviderFailing: worded(func(o conditions.Observation) words {
thing, consumer := conditions.ThingWords(o), idPart(o, 2) thing, consumer := conditions.ThingWords(o), idPart(o, 2)
if consumer == "" { if consumer == "" {
@@ -818,6 +828,9 @@ func stalledWords(l stalledLine, o conditions.Observation) (headline, explanatio
"to check it: the forge never announced it. It cannot merge until it is checked.", pull, long), "to check it: the forge never announced it. It cannot merge until it is checked.", pull, long),
fmt.Sprintf("%s has a merge check now, or is closed", pull), needs, nil fmt.Sprintf("%s has a merge check now, or is closed", pull), needs, nil
} }
if l.State == "unanswered" {
return unansweredWords(l, o)
}
held := l.State held := l.State
if held == "" { if held == "" {
held = "held" held = "held"
@@ -847,6 +860,74 @@ func stalledWords(l stalledLine, o conditions.Observation) (headline, explanatio
fmt.Sprintf("Delivery of %s is no longer %s", name, held), needs, actions fmt.Sprintf("Delivery of %s is no longer %s", name, held), needs, actions
} }
// unansweredWords are the plain words of a pull request's head whose merge check was asked and has not answered
// within its bound (novox/hq issue 438): mesh-delivery holds no delivery of it, so there is nothing to stop, release
// or close, and no action is offered. The operator's acts are a new commit, which asks the check again, or a look
// at the build queue, where a check that never ran may still wait.
func unansweredWords(l stalledLine, o conditions.Observation) (headline, explanation, resolved, needs string,
actions []conditions.Action) {
repository, _, _ := strings.Cut(l.ID, "@")
pull := "A pull request of " + repoName(repository)
if l.Number > 0 {
pull = fmt.Sprintf("Pull request %s #%d", repoName(repository), l.Number)
}
long, limit := "for too long", ""
if d, err := time.ParseDuration(l.For); err == nil {
long = "for " + humanDuration(d)
}
if d, err := time.ParseDuration(l.Bound); err == nil {
limit = ", past its limit of " + humanDuration(d)
}
which := "Its merge check"
if said := uncheckedWaitWords(l); len(said) > 0 {
which = "Its merge check (" + strings.Join(said, ", ") + ")"
}
if o.Resolver == conditions.ResolverOperator {
needs = "push a new commit to its branch, which asks the check again, or see whether its check still waits " +
"in the build queue: mesh-controller.queue."
}
return fmt.Sprintf("%s's merge check has not answered %s", pull, long),
fmt.Sprintf("%s is open on a branch that requires the merge check. %s was asked and has not answered %s%s. "+
"It cannot merge until its check answers.", pull, which, long, limit),
fmt.Sprintf("%s has an answer from its merge check, or is closed", pull), needs, nil
}
// uncheckedWaitWords are the merge checks an unanswered line waits on, as the operator reads them: from the checks
// given as data, each time in the controller's local zone, as every other time in a message — "mesh/merge-gate
// pending since 02:11" — never the UTC time inside mesh-delivery's finished words, which cannot be said again in
// another zone (novox/hq issue 443). A line from a mesh-delivery that gives no such data is said by its words.
func uncheckedWaitWords(l stalledLine) []string {
if len(l.Checks) == 0 {
return l.Waiting
}
out := make([]string, 0, len(l.Checks))
for _, c := range l.Checks {
switch c.State {
case "never-set":
out = append(out, c.Context+" never set")
case "pending":
out = append(out, c.Context+" pending"+sinceWords(c.Since))
default:
out = append(out, c.Context+" not answered")
}
}
return out
}
// sinceWords is " since 02:11" in the controller's local zone, with its day when that is not today ("since 23:05 on
// 9 Oct"), so the time stays absolute; a time not given, or not readable, is said so and never made up.
func sinceWords(since string) string {
at, err := time.Parse(time.RFC3339, since)
if err != nil {
return ", since a time the forge did not say"
}
local, today := at.In(wordsZone()), wordsNow().In(wordsZone())
if local.YearDay() == today.YearDay() && local.Year() == today.Year() {
return " since " + local.Format("15:04")
}
return " since " + local.Format("15:04 on 2 Jan")
}
// causeWords is a hand-act's cause as a person says it. // causeWords is a hand-act's cause as a person says it.
func causeWords(cause string) string { func causeWords(cause string) string {
return strings.NewReplacer(".", " ", "_", " ").Replace(cause) return strings.NewReplacer(".", " ", "_", " ").Replace(cause)
+105
View File
@@ -1,6 +1,7 @@
package main package main
import ( import (
"encoding/json"
"regexp" "regexp"
"strings" "strings"
"testing" "testing"
@@ -347,3 +348,107 @@ func TestAnUnannouncedPullRequestSaysToPushANewCommit(t *testing.T) {
t.Fatalf("it reads %q / %q", o.Headline, o.Explanation) t.Fatalf("it reads %q / %q", o.Headline, o.Explanation)
} }
} }
// **A pull request whose merge check never answered says which check, since when, and what to do** (novox/hq issue
// 438): mesh-delivery says one as a stalled line in state `unanswered`, the line below exactly as its test makes it.
// No delivery of the head is held, so there is nothing to stop, release or close: no action is offered, and the
// operator's acts are a new commit, which asks the check again, or a look at the build queue.
func TestAnUnansweredMergeCheckSaysWhichCheckAndWhatToDo(t *testing.T) {
var l stalledLine
if err := json.Unmarshal([]byte(`{"id":"novox/mesh-controller@c11222026a3b","number":212,"state":"unanswered",`+
`"waiting":["mesh/merge-gate pending since 2026-10-11T00:11:39Z","mesh/repo-check never set"],"for":"1h1m0s",`+
`"bound":"1h0m0s","h2":"none: no delivery of it is held here, so there is none to close — the operator's",`+
`"says":"novox/mesh-controller#212 is open on main, which requires the merge check, and its head's check `+
`started and never answered: mesh/merge-gate pending since 2026-10-11T00:11:39Z, mesh/repo-check never set. `+
`No delivery of it is held here, so nothing asks it again. A new commit on its branch announces it and asks `+
`its check"}`), &l); err != nil {
t.Fatal(err)
}
obs := stalledObservations([]stalledLine{l})
if len(obs) != 1 || obs[0].Resolver != conditions.ResolverOperator {
t.Fatalf("an unanswered merge check is not the operator's: %+v", obs)
}
o := obs[0]
t.Logf("headline: %s\nexplanation: %s\nneeds: %s\nresolved: %s", o.Headline, o.Explanation, o.Needs, o.Resolved)
if len(o.Actions) != 0 {
t.Fatalf("it offers an action a head with no delivery cannot take: %+v", o.Actions)
}
if strings.Contains(o.Needs, "stop") || strings.Contains(o.Needs, "release") || !strings.Contains(o.Needs, "commit") ||
!strings.Contains(o.Needs, "in the build queue: mesh-controller.queue") || strings.Contains(o.Needs, "build seat") {
t.Fatalf("it says to %q", o.Needs)
}
if o.Headline != "Pull request mesh-controller #212's merge check has not answered for 61 minutes" {
t.Fatalf("its headline reads %q", o.Headline)
}
for _, want := range []string{"mesh/merge-gate pending since 2026-10-11T00:11:39Z", "mesh/repo-check never set",
"past its limit of 60 minutes"} {
if !strings.Contains(o.Explanation, want) {
t.Fatalf("its explanation does not say %q: %q", want, o.Explanation)
}
}
if strings.Contains(o.Explanation, "never asked") || strings.Contains(o.Explanation, "never announced") {
t.Fatalf("it says the mesh was never asked, of a head whose check started: %q", o.Explanation)
}
// A line from a mesh-delivery that names no checks still says the check did not answer, and offers nothing.
var bare stalledLine
if err := json.Unmarshal([]byte(`{"id":"novox/mesh-controller@c11222026a3b","number":212,"state":"unanswered",`+
`"for":"1h1m0s","bound":"1h0m0s","h2":"none: no delivery of it is held here, so there is none to close — the operator's"}`),
&bare); err != nil {
t.Fatal(err)
}
o = stalledObservations([]stalledLine{bare})[0]
if len(o.Actions) != 0 || !strings.Contains(o.Explanation, "has not answered") || strings.Contains(o.Headline, "Delivery of") {
t.Fatalf("a line naming no checks reads %q / %q, actions %+v", o.Headline, o.Explanation, o.Actions)
}
}
// **An unanswered merge check's time reaches the operator in their own time, never as raw UTC** (novox/hq issue
// 443): mesh-delivery says each check waited on as data — its context, its state and since when, in RFC 3339 — beside
// the finished words it gave before, and the controller words it in its local zone, as every other time in a message.
// The line is the one mesh-delivery says, as in the report of issue 443, with the checks it now carries.
func TestAnUnansweredMergeChecksTimeIsSaidInLocalTime(t *testing.T) {
// The operator's zone given through the seam, never by writing time.Local: other tests' goroutines read it, and
// the build seat runs the suite under the race detector.
wasZone, wasNow := wordsZone, wordsNow
wordsZone = func() *time.Location { return time.FixedZone("CEST", 2*60*60) }
wordsNow = func() time.Time { return time.Date(2026, 10, 11, 1, 12, 39, 0, time.UTC) }
t.Cleanup(func() { wordsZone, wordsNow = wasZone, wasNow })
var l stalledLine
if err := json.Unmarshal([]byte(`{"id":"novox/mesh-controller@c11222026a3b","number":212,"state":"unanswered",`+
`"waiting":["mesh/merge-gate pending since 2026-10-11T00:11:39Z","mesh/repo-check never set"],`+
`"checks":[{"context":"mesh/merge-gate","state":"pending","since":"2026-10-11T00:11:39Z"},`+
`{"context":"mesh/repo-check","state":"never-set"}],"for":"1h1m0s",`+
`"bound":"1h0m0s","h2":"none: no delivery of it is held here, so there is none to close — the operator's",`+
`"says":"novox/mesh-controller#212 is open on main, which requires the merge check, and its head's check `+
`started and never answered: mesh/merge-gate pending since 2026-10-11T00:11:39Z, mesh/repo-check never set. `+
`No delivery of it is held here, so nothing asks it again. A new commit on its branch announces it and asks `+
`its check"}`), &l); err != nil {
t.Fatal(err)
}
o := stalledObservations([]stalledLine{l})[0]
t.Logf("explanation: %s", o.Explanation)
for _, raw := range []string{"2026-10-11T00:11:39Z", "00:11", "UTC"} {
if strings.Contains(o.Explanation, raw) || strings.Contains(o.Headline, raw) {
t.Fatalf("the operator reads %q: %q / %q", raw, o.Headline, o.Explanation)
}
}
for _, want := range []string{"mesh/merge-gate pending since 02:11", "mesh/repo-check never set"} {
if !strings.Contains(o.Explanation, want) {
t.Fatalf("its explanation does not say %q: %q", want, o.Explanation)
}
}
// A check pending since another day than today says which day, so the time stays absolute.
l.Checks[0].Since = "2026-10-09T21:05:00Z"
if o = stalledObservations([]stalledLine{l})[0]; !strings.Contains(o.Explanation, "mesh/merge-gate pending since 23:05 on 9 Oct") {
t.Fatalf("a check pending since an earlier day reads %q", o.Explanation)
}
// A pending check whose time the forge did not say is said so, with no time made up.
l.Checks[0].Since = ""
if o = stalledObservations([]stalledLine{l})[0]; !strings.Contains(o.Explanation, "mesh/merge-gate pending, since a time the forge did not say") {
t.Fatalf("a pending check without its time reads %q", o.Explanation)
}
}
+27 -1
View File
@@ -664,7 +664,33 @@ func renderingFor(ctx context.Context, open *stores, node string,
needed := map[string]map[string]string{} needed := map[string]map[string]string{}
foreseen := map[string]map[string]bool{} foreseen := map[string]map[string]bool{}
for _, m := range plan.Modules { for _, m := range plan.Modules {
for name := range m.OwnSecrets { // **A secret family is never made** (novox/hq ADR 0283): each member a person gave on this machine is
// placed, and one not given is nothing — the module says it waits for it.
for _, family := range m.OwnSecrets.Families() {
members, err := inv.GivenMembers(ctx, node, m.Module, family)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
for _, g := range members {
if _, fam, ok := m.OwnSecrets.Lookup(g.Name); !ok || fam != family {
continue // a longer family's member, or a name no longer of this family
}
if !g.Current {
if choosing == Allocating {
return catalogue.Rendering{}, inventory.Node{}, fmt.Errorf(
"%s on %s holds %q, which was given to the mesh rather than made by it, and %s has "+
"since generated a new sealing key. The mesh cannot make another; give it again",
m.Module, node, g.Name, node)
}
continue
}
if needed[m.Module] == nil {
needed[m.Module] = map[string]string{}
}
needed[m.Module][g.Name] = g.Sealed
}
}
for name := range m.OwnSecrets.Plain() {
// Minted on the send path and only read on every other. Making one is an insert, and // Minted on the send path and only read on every other. Making one is an insert, and
// a question that writes is a question that can block against the machine it is about. // a question that writes is a question that can block against the machine it is about.
var sealed string var sealed string
+151 -21
View File
@@ -37,6 +37,32 @@ type holding struct {
shelf map[string]catalogue.Manifest shelf map[string]catalogue.Manifest
// providers memoises providerFor by machine, consumer and provision. // providers memoises providerFor by machine, consumer and provision.
providers map[string]providerLookup providers map[string]providerLookup
// waits is what the waits of a statement are checked against, read as they are asked for (novox/hq issue 450).
waits operatorWaitFacts
}
// checked is a machine's newest statement as the controller judges it: each wait checked (ADR 0283 decision 3), so
// a wait that does not check out reads as unhealthy, as it did when the statement was judged (novox/hq issue 450).
// What is stored is what the machine said, the waits unchecked; read as stored, a provider whose wait failed its
// check seems to wait for the operator while its unhealthy condition is open.
func (h *holding) checked(machine string) []inventory.ResourceHealth {
rs := h.healths[machine].Resources
mods := waitingModules(rs)
if len(mods) == 0 {
return rs
}
if h.waits.manifests == nil {
h.waits.manifests = map[string]catalogue.Manifest{}
}
for _, module := range mods {
if _, has := h.waits.manifests[module]; !has {
if m, ok := h.manifestOf(module); ok {
h.waits.manifests[module] = m
}
}
}
readWaitFacts(h.ctx, h.inv, machine, mods, nil, &h.waits)
return checkWaiting(machine, rs, h.waits)
} }
type providerLookup struct { type providerLookup struct {
@@ -53,6 +79,15 @@ func readHolding(ctx context.Context, inv *inventory.Inventory, open []condition
return &holding{ctx: ctx, inv: inv, healths: healths, open: open, providers: map[string]providerLookup{}}, nil return &holding{ctx: ctx, inv: inv, healths: healths, open: open, providers: map[string]providerLookup{}}, nil
} }
// readHoldingFor is how a judging reads its holding: nothing without a store. A variable so a test can hand a
// judging the record it holds under (novox/hq issue 405).
var readHoldingFor = func(ctx context.Context, inv *inventory.Inventory, open []conditions.Condition) (*holding, error) {
if inv == nil {
return nil, nil
}
return readHolding(ctx, inv, open)
}
// heldFinding says a resource's state is a finding of its declared check that names a provision: what // heldFinding says a resource's state is a finding of its declared check that names a provision: what
// may be held. Down and restarting are liveness, the resource's own. // may be held. Down and restarting are liveness, the resource's own.
func heldFinding(r inventory.ResourceHealth) bool { func heldFinding(r inventory.ResourceHealth) bool {
@@ -106,42 +141,137 @@ func (h *holding) lookUpProvider(machine, consumer, provision string) (catalogue
return catalogue.Chosen{}, false return catalogue.Chosen{}, false
} }
// unhealthy says a provider is unhealthy on the record: its condition is open, or its machine's newest // providerState is how a provider stands on the record: unhealthy when its unhealthy condition is open or its
// statement says a resource of it is unhealthy. // machine's newest statement says a resource of it is unhealthy; else waiting for the operator when that statement
func (h *holding) unhealthy(p catalogue.Chosen) bool { // says a resource of it waits, with the waits it names, or its needs-operator condition is open (waits then
key := moduleUnhealthyKey(p.Module, p.Node) // unknown); else healthy.
func (h *holding) providerState(p catalogue.Chosen) (unhealthy, waiting bool, waits []inventory.Wait) {
for _, c := range h.open { for _, c := range h.open {
if c.Key == key { if c.Key == moduleUnhealthyKey(p.Module, p.Node) {
return true return true, false, nil
} }
} }
for _, r := range h.healths[p.Node].Resources { for _, r := range h.checked(p.Node) {
if r.Module == p.Module && r.State == link.StateUnhealthy { if r.Module != p.Module {
return true continue
}
switch r.State {
case link.StateUnhealthy:
return true, false, nil
case link.StateWaiting:
waiting = true
waits = append(waits, r.Waits...)
} }
} }
return false if !waiting {
for _, c := range h.open {
waiting = waiting || c.Key == needsOperatorKey(p.Module, p.Node)
}
}
return false, waiting, waits
}
// manifestOf is a module's manifest from the catalogue, read once; false when it cannot be read.
func (h *holding) manifestOf(module string) (catalogue.Manifest, bool) {
if h.shelf == nil && h.inv != nil {
shelf, err := h.inv.Catalogue(h.ctx)
if err != nil {
return catalogue.Manifest{}, false
}
h.shelf = shelf
}
m, ok := h.shelf[module]
return m, ok
}
// covering is the waits of a provider that cover a provision it gives (novox/hq issue 405): a module that waits
// says nothing else of it is wrong and names each part that waits (ADR 0283 decision 1), so only a consumer of
// the waiting part waits on it. A wait covers a provision when its part is that provision, or the secret it waits
// for is the provision's shared credential (ADR 0158). Nothing when no wait can be matched: a consumer failing
// then is not held, since holding it would hide a fault that may be its own.
func (h *holding) covering(p catalogue.Chosen, provision string, waits []inventory.Wait) []inventory.Wait {
credential := ""
if m, ok := h.manifestOf(p.Module); ok {
credential, _ = m.SharedCredentialOf(provision)
}
var out []inventory.Wait
for _, w := range waits {
if w.Part == provision || (w.Secret != "" && w.Secret == credential) {
out = append(out, w)
}
}
return out
}
// heldUnderProvider is the provider a consumer's findings are held under, and — when that provider only waits for the
// operator, for the part the consumer needs — the waits that hold it.
type heldUnderProvider struct {
provider catalogue.Chosen
// waits is set when every finding held waits on a provider that only waits for the operator: the consumer's
// gate then reads as ADR 0254's waits for a person, a pass with the wait carried (ADR 0283 decision 4).
waits []inventory.Wait
} }
// heldUnder is the provider a consumer's unhealthy resources wait on: when every one of them is a finding // heldUnder is the provider a consumer's unhealthy resources wait on: when every one of them is a finding
// of a check naming a provision whose provider for this consumer is unhealthy on the record. False when any // of a check naming a provision whose provider for this consumer is unhealthy on the record, or waits for the
// is the consumer's own. // operator for the part that gives it (novox/hq issue 405). False when any is the consumer's own.
func (h *holding) heldUnder(machine, module string, rs []inventory.ResourceHealth) (catalogue.Chosen, bool) { func (h *holding) heldUnder(machine, module string, rs []inventory.ResourceHealth) (catalogue.Chosen, bool) {
var on catalogue.Chosen by, held := h.heldWith(machine, module, rs)
return by.provider, held
}
func (h *holding) heldWith(machine, module string, rs []inventory.ResourceHealth) (heldUnderProvider, bool) {
var on heldUnderProvider
onlyWaits := true
for _, r := range rs { for _, r := range rs {
if r.State != link.StateUnhealthy { if r.State != link.StateUnhealthy {
continue continue
} }
if !heldFinding(r) { if !heldFinding(r) {
return catalogue.Chosen{}, false return heldUnderProvider{}, false
} }
p, ok := h.providerFor(machine, module, r.Needs) p, ok := h.providerFor(machine, module, r.Needs)
if !ok || (p.Node == machine && p.Module == module) || !h.unhealthy(p) { if !ok || (p.Node == machine && p.Module == module) {
return catalogue.Chosen{}, false return heldUnderProvider{}, false
} }
on = p unhealthy, waiting, waits := h.providerState(p)
switch {
case unhealthy:
onlyWaits = false
case waiting:
covered := h.covering(p, r.Needs, waits)
if len(covered) == 0 {
return heldUnderProvider{}, false
}
on.waits = append(on.waits, covered...)
default:
return heldUnderProvider{}, false
}
on.provider = p
} }
return on, on.Module != "" if !onlyWaits {
on.waits = nil
}
return on, on.provider.Module != ""
}
// waitingUncovered is a provider of a consumer's failing finding that waits for the operator for a part the
// finding cannot be matched to (novox/hq issue 405): the consumer is raised on its own, and its condition says
// the provider waits, so neither is hidden.
func (h *holding) waitingUncovered(machine, module string, rs []inventory.ResourceHealth) (catalogue.Chosen, bool) {
for _, r := range rs {
if r.State != link.StateUnhealthy || !heldFinding(r) {
continue
}
p, ok := h.providerFor(machine, module, r.Needs)
if !ok || (p.Node == machine && p.Module == module) {
continue
}
if unhealthy, waiting, waits := h.providerState(p); !unhealthy && waiting && len(h.covering(p, r.Needs, waits)) == 0 {
return p, true
}
}
return catalogue.Chosen{}, false
} }
// waitersOn is every consumer held under a provider, as "<module> on <machine>", sorted. // waitersOn is every consumer held under a provider, as "<module> on <machine>", sorted.
@@ -165,8 +295,8 @@ func (h *holding) waitersOn(p catalogue.Chosen) []string {
} }
// heldModules is, for one machine's statement, each module whose finding is held, with the provider. // heldModules is, for one machine's statement, each module whose finding is held, with the provider.
func (h *holding) heldModules(machine string) map[string]catalogue.Chosen { func (h *holding) heldModules(machine string) map[string]heldUnderProvider {
out := map[string]catalogue.Chosen{} out := map[string]heldUnderProvider{}
byModule := map[string][]inventory.ResourceHealth{} byModule := map[string][]inventory.ResourceHealth{}
for _, r := range h.healths[machine].Resources { for _, r := range h.healths[machine].Resources {
if r.Module != "" && r.State == link.StateUnhealthy { if r.Module != "" && r.State == link.StateUnhealthy {
@@ -174,7 +304,7 @@ func (h *holding) heldModules(machine string) map[string]catalogue.Chosen {
} }
} }
for module, rs := range byModule { for module, rs := range byModule {
if on, held := h.heldUnder(machine, module, rs); held { if on, held := h.heldWith(machine, module, rs); held {
out[module] = on out[module] = on
} }
} }
+28
View File
@@ -510,6 +510,14 @@ func advancePlans(ctx context.Context, open *stores) {
advanceHeld(ctx, open) advanceHeld(ctx, open)
} }
// keepWalkPhases keeps where the walks that ended lately spent their time (novox/hq ADR 0282), outside the hold
// on the plans so it never lengthens it: measured, never acted on, and an error only said.
func keepWalkPhases(ctx context.Context, open *stores) {
if err := recordWalkPhases(ctx, open.inventory, time.Now()); err != nil {
fmt.Printf("plans: the phases of the walks ended lately could not be kept: %v\n", err)
}
}
// advanceHeld is advancePlans for a caller already holding the plans. // advanceHeld is advancePlans for a caller already holding the plans.
func advanceHeld(ctx context.Context, open *stores) { func advanceHeld(ctx context.Context, open *stores) {
inv := open.inventory inv := open.inventory
@@ -864,8 +872,12 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
strings.Join(machines, ", "), p.Tier, err) strings.Join(machines, ", "), p.Tier, err)
} }
now := time.Now().UTC() now := time.Now().UTC()
// What each machine of the rest was sent, kept for when it reports it applied (novox/hq ADR 0282 decision
// 6): measured, never acted on.
sentWhat := sentNow(ctx, open.inventory, sent)
for _, m := range rest { for _, m := range rest {
p.Modules[m].SentAt = &now p.Modules[m].SentAt = &now
p.Modules[m].Rest = restOf(restTo[m], sent, sentWhat)
} }
fmt.Printf("%s: tier %d built; sent %s to %s, one send each\n", p.ID, p.Tier, strings.Join(rest, ", "), fmt.Printf("%s: tier %d built; sent %s to %s, one send each\n", p.ID, p.Tier, strings.Join(rest, ", "),
strings.Join(sent, ", ")) strings.Join(sent, ", "))
@@ -946,6 +958,20 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
return true, nil return true, nil
} }
// restOf is, for one module, every machine of its rest that the send reached and what it carried there.
func restOf(to, sent []string, what map[string]inventory.SentDeclaration) map[string]inventory.SentDeclaration {
out := map[string]inventory.SentDeclaration{}
for _, n := range to {
if slices.Contains(sent, n) {
out[n] = what[n]
}
}
if len(out) == 0 {
return nil
}
return out
}
// firstSend sends one machine, in one send, every module of the plan's tier whose first machine it is // firstSend sends one machine, in one send, every module of the plan's tier whose first machine it is
// (novox/hq issue 281), and records the send on each: what that machine ran of it before — read once, // (novox/hq issue 281), and records the send on each: what that machine ran of it before — read once,
// before the send, so a module the same send carries is never read as already moved — the machines it // before the send, so a module the same send carries is never read as already moved — the machines it
@@ -1193,6 +1219,7 @@ func sayUnsent(p *inventory.Plan, rollsOut func(string) bool) {
// planTicker advances open plans on a timer, for the steps outcomes alone cannot take. // planTicker advances open plans on a timer, for the steps outcomes alone cannot take.
func planTicker(ctx context.Context, open *stores) { func planTicker(ctx context.Context, open *stores) {
advancePlans(ctx, open) advancePlans(ctx, open)
keepWalkPhases(ctx, open)
tick := time.NewTicker(30 * time.Second) tick := time.NewTicker(30 * time.Second)
defer tick.Stop() defer tick.Stop()
for { for {
@@ -1201,6 +1228,7 @@ func planTicker(ctx context.Context, open *stores) {
return return
case <-tick.C: case <-tick.C:
advancePlans(ctx, open) advancePlans(ctx, open)
keepWalkPhases(ctx, open)
} }
} }
} }
+14 -3
View File
@@ -1366,7 +1366,7 @@ func splitCommandLine(line string) ([]string, error) {
var words []string var words []string
var cur strings.Builder var cur strings.Builder
inWord := false inWord := false
quote := rune(0) quote, opened := rune(0), 0
runes := []rune(line) runes := []rune(line)
for i := 0; i < len(runes); i++ { for i := 0; i < len(runes); i++ {
r := runes[i] r := runes[i]
@@ -1387,7 +1387,7 @@ func splitCommandLine(line string) ([]string, error) {
cur.WriteRune(r) cur.WriteRune(r)
} }
case r == '\'' || r == '"': case r == '\'' || r == '"':
quote = r quote, opened = r, i
inWord = true inWord = true
case r == '\\' && i+1 < len(runes): case r == '\\' && i+1 < len(runes):
i++ i++
@@ -1405,7 +1405,7 @@ func splitCommandLine(line string) ([]string, error) {
} }
} }
if quote != 0 { if quote != 0 {
return nil, fmt.Errorf("command has an unclosed %c quote", quote) return nil, unclosedQuote(quote, opened)
} }
if inWord { if inWord {
words = append(words, cur.String()) words = append(words, cur.String())
@@ -1413,6 +1413,17 @@ func splitCommandLine(line string) ([]string, error) {
return words, nil return words, nil
} }
// unclosedQuote is the refusal of a line whose quote is never closed. Most often an apostrophe inside
// a single-quoted value ended that quote early and a later quote was left open, so the refusal says
// where the open quote is and how a quote is written inside a quoted value — the shell's own two
// ways, which this splitter already reads (novox/hq issue 294). It quotes none of the line: a refusal
// is kept on the bus as the call's answer, and the line may carry a setting's value.
func unclosedQuote(quote rune, opened int) error {
return fmt.Errorf("command has an unclosed %c quote, opened at character %d — an apostrophe "+
"inside a single-quoted value ends it; write a ' inside single quotes as '\\'' (it'\\''s), or use "+
"double quotes and write \\\" for a \" and \\\\ for a \\ inside them", quote, opened+1)
}
// seatAnnouncement is what the controller says it serves on the bus (novox/hq ADR 0197): the // seatAnnouncement is what the controller says it serves on the bus (novox/hq ADR 0197): the
// mesh-controller seat, one endpoint per verb it answers, each with the seat's own description and // mesh-controller seat, one endpoint per verb it answers, each with the seat's own description and
// argument schema — the same facts `tools` answers from the records, as NATS's services format. // argument schema — the same facts `tools` answers from the records, as NATS's services format.
+45
View File
@@ -415,3 +415,48 @@ func TestNoVerbSetsTheOperatorsKeyOrRevealsASecret(t *testing.T) {
t.Fatalf("behind %v: %v", behind, err) t.Fatalf("behind %v: %v", behind, err)
} }
} }
// A value with a quote in it can be said on a `command` line, as in a shell, and a line whose quote
// is left open is refused naming where it opened and how a quote is written, quoting none of it (novox/hq issue 294: an
// apostrophe inside a single-quoted JSON value cut the line, and the refusal named no cause).
func TestAQuotedValueCanHoldAQuote(t *testing.T) {
for _, c := range []struct{ line, want string }{
{`settings set claude-code '{"role":"the operator'\''s laptop"}'`, `{"role":"the operator's laptop"}`},
{`settings set claude-code "{\"role\":\"the operator's laptop\"}"`, `{"role":"the operator's laptop"}`},
{"x \"a \\\\ b\nc\"", "a \\ b\nc"},
{`x 'a\b'`, `a\b`},
} {
argv, err := splitCommandLine(c.line)
if err != nil || argv[len(argv)-1] != c.want {
t.Errorf("%s: read back %q %v, want %q", c.line, argv, err, c.want)
}
}
_, err := splitCommandLine(`settings set claude-code '{"role":"the operator's laptop"}' --node g14`)
if err == nil {
t.Fatal("an apostrophe that leaves a quote open was accepted")
}
for _, want := range []string{"character 57", `'\''`, `\"`} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
if strings.Contains(err.Error(), "laptop") || strings.Contains(err.Error(), "g14") {
t.Errorf("the refusal quotes the line, which may carry a setting's value: %v", err)
}
}
// Every value reads back as it was when written the way the refusal says: single-quoted with '\”
// for each quote, or double-quoted with \ before each " and \ — newlines and backslashes included.
func TestAQuotedValueRoundTrips(t *testing.T) {
for _, v := range []string{`plain`, `it's`, `say "hi"`, `back\slash\`, "two\nlines", `'"\'\"`, `''`, ``} {
single := "x '" + strings.ReplaceAll(v, "'", `'\''`) + "'"
double := `x "` + strings.NewReplacer(`\`, `\\`, `"`, `\"`).Replace(v) + `"`
for _, line := range []string{single, double} {
argv, err := splitCommandLine(line)
if err != nil || len(argv) != 2 || argv[1] != v {
t.Errorf("%s: read back %q %v, want %q", line, argv, err, v)
}
}
}
}
+22 -1
View File
@@ -523,6 +523,14 @@ func watchAdvisories(f *signalFacts) []conditions.Observation {
if a.Kind == link.AdvisoryConsumerLost && !f.lostConsumers[a.Stream+"."+a.Consumer] { if a.Kind == link.AdvisoryConsumerLost && !f.lostConsumers[a.Stream+"."+a.Consumer] {
continue // it exists again, or the mesh no longer expects it: a removal, not a loss continue // it exists again, or the mesh no longer expects it: a removal, not a loss
} }
if a.Kind == link.AdvisoryMaxDeliveries && a.Token == "" {
// A consumer's max-deliveries key is the dead-letter row's (novox/hq issue 330): said while
// DEAD_LETTERS holds what it gave up on, cleared when that is delivered again or dropped. The
// listener records no such advisory today; one read here as well added a look to the count and
// overwrote the row's words on every look (novox/hq issue 440). Only one that could not be kept
// (AdvisoryNotKept), which DEAD_LETTERS cannot say, is said from here.
continue
}
severity := conditions.Warning severity := conditions.Warning
times := "" times := ""
if a.Count > 1 { if a.Count > 1 {
@@ -533,7 +541,10 @@ func watchAdvisories(f *signalFacts) []conditions.Observation {
machine = f.host machine = f.host
} }
o := conditions.Observation{Scope: conditions.ScopeBus, ID: a.ID, Kind: a.Kind, Token: a.Token, o := conditions.Observation{Scope: conditions.ScopeBus, ID: a.ID, Kind: a.Kind, Token: a.Token,
Machine: machine, Severity: severity, Summary: a.Said + times, Said: a.Said} Machine: machine, Severity: severity, Summary: a.Said + times, Said: a.Said,
// The advisory is remembered and read again on every look for an hour: the condition counts
// what the bus said and when, not the looks (novox/hq issue 402).
Happened: a.Last, Times: a.Count}
if a.Kind == link.AdvisoryMaxDeliveries && a.Token == link.AdvisoryNotKept { if a.Kind == link.AdvisoryMaxDeliveries && a.Token == link.AdvisoryNotKept {
o.Machine = consumerMachine(a.Stream, a.Consumer) o.Machine = consumerMachine(a.Stream, a.Consumer)
o.Headline = clip(conditions.Capital(fmt.Sprintf("%s gave up on a message, not kept", o.Headline = clip(conditions.Capital(fmt.Sprintf("%s gave up on a message, not kept",
@@ -565,7 +576,17 @@ func watchDeadLetters(f *signalFacts) []conditions.Observation {
messages, them = fmt.Sprintf("%d messages", n), "them" messages, them = fmt.Sprintf("%d messages", n), "them"
} }
who := consumerWho(stream, consumer) who := consumerWho(stream, consumer)
// DEAD_LETTERS is a record of what was given up on: the condition says when the newest held message
// was kept, and its count is a running total of the messages given up on while it is open, never
// how often the controller looked (novox/hq issues 402 and 440). It is at least the number held now,
// and does not go down when one is delivered again or dropped. Without that time it counts its
// looks, as a source of the present does.
happened, times := f.deadLettersNewest[key], 0
if !happened.IsZero() {
times = n
}
out = append(out, conditions.Observation{Scope: conditions.ScopeBus, ID: key, Kind: link.AdvisoryMaxDeliveries, out = append(out, conditions.Observation{Scope: conditions.ScopeBus, ID: key, Kind: link.AdvisoryMaxDeliveries,
Happened: happened, Times: times,
Machine: consumerMachine(stream, consumer), Severity: conditions.Warning, Machine: consumerMachine(stream, consumer), Severity: conditions.Warning,
Summary: fmt.Sprintf("%s gave up on %s; %s kept in %s until delivered again or dropped, with why, "+ Summary: fmt.Sprintf("%s gave up on %s; %s kept in %s until delivered again or dropped, with why, "+
"through the controller's dead-letters verb", link.ConsumerInWords(stream, consumer), messages, "through the controller's dead-letters verb", link.ConsumerInWords(stream, consumer), messages,
+5 -4
View File
@@ -100,14 +100,15 @@ var suppressions = map[string]suppression{
inside: func(f *signalFacts) { f.standings = []conditions.Condition{standingSaid(f.now.Add(-29 * time.Minute))} }, inside: func(f *signalFacts) { f.standings = []conditions.Condition{standingSaid(f.now.Add(-29 * time.Minute))} },
past: func(f *signalFacts) { f.standings = []conditions.Condition{standingSaid(f.now.Add(-31 * time.Minute))} }, past: func(f *signalFacts) { f.standings = []conditions.Condition{standingSaid(f.now.Add(-31 * time.Minute))} },
}, },
// A message given up on and not kept: the one max-deliveries advisory S9 says itself (issue 440).
"S9": { "S9": {
inside: func(f *signalFacts) { inside: func(f *signalFacts) {
f.advisories = []link.Advisory{{Kind: link.AdvisoryMaxDeliveries, ID: "EVENTS.anchor_shop", Said: "gave up", f.advisories = []link.Advisory{{Kind: link.AdvisoryMaxDeliveries, ID: "EVENTS.anchor_shop",
First: f.now.Add(-2 * time.Hour), Last: f.now.Add(-61 * time.Minute), Count: 1}} Token: link.AdvisoryNotKept, Said: "gave up, not kept", First: f.now.Add(-2 * time.Hour), Last: f.now.Add(-61 * time.Minute), Count: 1}}
}, },
past: func(f *signalFacts) { past: func(f *signalFacts) {
f.advisories = []link.Advisory{{Kind: link.AdvisoryMaxDeliveries, ID: "EVENTS.anchor_shop", Said: "gave up", f.advisories = []link.Advisory{{Kind: link.AdvisoryMaxDeliveries, ID: "EVENTS.anchor_shop",
First: f.now.Add(-2 * time.Hour), Last: f.now.Add(-59 * time.Minute), Count: 1}} Token: link.AdvisoryNotKept, Said: "gave up, not kept", First: f.now.Add(-2 * time.Hour), Last: f.now.Add(-59 * time.Minute), Count: 1}}
}, },
}, },
"S10": { "S10": {
@@ -0,0 +1,384 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A provider waiting for the operator holds its consumers, and a wait beside another wait is said (novox/hq
// issue 405, found in the review of ADR 0283's controller change).
//
// The shapes are those of issue 386 (mounts waiting for smb-password-games: waitingResource, passwordWait) and of
// the openrazer wait on the workstation of 2026-10-11 (relogin, userUnit): an account in its group whose session
// began before it was, and its unit failed in that account's own service manager.
// waitingDatabase is a provider whose only part not healthy waits for the operator's secret: a database's
// process stated waiting, as the node-engine states a tool check answering waits (ADR 0283 decision 2). Its wait
// names the part that waits by the provision it gives.
func waitingDatabase() inventory.ResourceHealth {
return waitingDatabaseFor(inventory.Wait{Part: "postgres-database", Secret: "licence",
What: "the licence key of the database"})
}
func waitingDatabaseFor(waits ...inventory.Wait) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: "db", Resource: "db.server", Kind: "process", Target: "db.service",
State: link.StateWaiting, Check: "tool", Reason: "the database waits for its licence", Waits: waits}
}
// backupsWait is a wait of the same provider for another part than the one its consumers need.
var backupsWait = inventory.Wait{Part: "the nightly backups", Secret: "backup-key", What: "the key of the backups"}
// failingConsumer is a consumer whose check that needs the database fails.
func failingConsumer(module string) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: module, Resource: module + ".web", Kind: "container", Target: module,
State: link.StateUnhealthy, Reason: "http /health on web: answered 500", Check: "http", Needs: "postgres-database"}
}
// dbSecrets is what the database's waits name: own secrets issued outside the mesh, so a wait for one checks out
// while nobody gave it (ADR 0283 decision 3, novox/hq issue 450).
var dbSecrets = catalogue.OwnSecrets{
"licence": {Path: "/s/licence", IssuedBy: catalogue.IssuedOutside},
"backup-key": {Path: "/s/backup-key", IssuedBy: catalogue.IssuedOutside},
}
// holdingOf is one reading of the record without a store: the newest statements, the open conditions, the
// catalogue, what was given on the provider's machine (nothing), and each consumer's provider already looked up,
// as providerFor memoises it.
func holdingOf(open []conditions.Condition, healths map[string]inventory.NodeHealth, bound map[[3]string]catalogue.Chosen) *holding {
h := &holding{ctx: context.Background(), healths: healths, open: open, providers: map[string]providerLookup{},
shelf: map[string]catalogue.Manifest{"db": {Module: "db", Version: "1", OwnSecrets: dbSecrets,
Provides: []catalogue.Offer{{Name: "postgres-database", Scope: catalogue.ScopeMesh}}}},
waits: operatorWaitFacts{given: map[string]map[string]time.Time{"db@anchor": {}}}}
for k, p := range bound {
h.providers[k[0]+"\x00"+k[1]+"\x00"+k[2]] = providerLookup{p, true}
}
return h
}
var theDatabase = catalogue.Chosen{Node: "anchor", Module: "db"}
var shopOnTheDatabase = map[[3]string]catalogue.Chosen{{"laptop", "shop", "postgres-database"}: theDatabase}
func shopFailingBeside(provider ...inventory.ResourceHealth) map[string]inventory.NodeHealth {
return map[string]inventory.NodeHealth{
"anchor": {Node: "anchor", Resources: provider},
"laptop": {Node: "laptop", Resources: []inventory.ResourceHealth{failingConsumer("shop")}},
}
}
// (1) A provider whose only part not healthy waits for the operator holds the findings of the consumers of the
// waiting part under it (ADR 0240 rule 5); a consumer of another part, or one whose part cannot be matched, is
// its own (ADR 0283 decision 1: a module that waits says nothing else of it is wrong).
func TestAProviderWaitingForTheOperatorHoldsOnlyTheConsumersOfTheWaitingPart(t *testing.T) {
shop := []inventory.ResourceHealth{failingConsumer("shop")}
unhealthyDB := waitingDatabase()
unhealthyDB.State, unhealthyDB.Waits, unhealthyDB.Reason = link.StateUnhealthy, nil, "its tool check: refused"
healthyDB := waitingDatabase()
healthyDB.State, healthyDB.Waits = link.StateHealthy, nil
byCredential := holdingOf(nil, shopFailingBeside(waitingDatabaseFor(inventory.Wait{Part: "the server",
Secret: "licence", What: "the licence key"})), shopOnTheDatabase)
byCredential.shelf["db"] = catalogue.Manifest{Module: "db", Version: "1", OwnSecrets: dbSecrets, Provides: []catalogue.Offer{{
Name: "postgres-database", Credential: &catalogue.OfferCredential{Own: "licence"}}}}
for _, c := range []struct {
name string
hold *holding
held bool
onlyWaits bool
uncovered bool
}{
{"the waiting part is the provision", holdingOf(nil, shopFailingBeside(waitingDatabase()), shopOnTheDatabase), true, true, false},
{"the wait is for the provision's shared credential", byCredential, true, true, false},
{"the wait is for another part", holdingOf(nil, shopFailingBeside(waitingDatabaseFor(backupsWait)), shopOnTheDatabase), false, false, true},
{"only the needs-operator condition, its parts not said", holdingOf(
[]conditions.Condition{{Key: needsOperatorKey("db", "anchor"), Kind: kindNeedsOperator}},
shopFailingBeside(), shopOnTheDatabase), false, false, true},
{"an unhealthy provider holds as before", holdingOf(nil, shopFailingBeside(unhealthyDB), shopOnTheDatabase), true, false, false},
{"a healthy provider holds nothing", holdingOf(nil, shopFailingBeside(healthyDB), shopOnTheDatabase), false, false, false},
} {
by, held := c.hold.heldWith("laptop", "shop", shop)
if held != c.held || (held && by.provider != theDatabase) || (len(by.waits) > 0) != c.onlyWaits {
t.Errorf("%s: held %v under %v with waits %v; want held %v, only waits %v", c.name, held, by.provider,
by.waits, c.held, c.onlyWaits)
}
if _, uncovered := c.hold.waitingUncovered("laptop", "shop", shop); uncovered != c.uncovered {
t.Errorf("%s: said as a provider waiting for another part %v; want %v", c.name, uncovered, c.uncovered)
}
}
}
// (1) The consumer's first-node gate under a provider that only waits for the operator passes as a wait for a
// person (ADR 0254), carrying the wait (ADR 0283 decision 4); under an unhealthy provider it waits, as before.
func TestAConsumersGateUnderAWaitingProviderPassesCarryingTheWait(t *testing.T) {
now := time.Now()
since := now.Add(-time.Minute)
for _, c := range []struct {
name string
provider inventory.ResourceHealth
want health
says []string
}{
{"a provider that only waits", waitingDatabase(), healthPerson,
[]string{"waits on db on anchor, which waits for you", "the licence key of the database", "nox secret ask anchor db licence"}},
{"an unhealthy provider", func() inventory.ResourceHealth {
r := waitingDatabase()
r.State, r.Waits, r.Reason = link.StateUnhealthy, nil, "its tool check: refused"
return r
}(), healthWaiting, []string{"waits on db on anchor, which is unhealthy"}},
} {
healths := shopFailingBeside(c.provider)
for m, h := range healths {
h.HeardAt = now
healths[m] = h
}
f := gateFacts{now: now, health: healths, heldOn: heldReadings(holdingOf(nil, healths, shopOnTheDatabase))}
h, why := moduleHealthWord("shop", "laptop", since, f)
if h != c.want {
t.Errorf("%s: the consumer's gate reads %v %q; want %v", c.name, h, why, c.want)
continue
}
for _, s := range c.says {
if !strings.Contains(why, s) {
t.Errorf("%s: the gate's reading %q does not say %q", c.name, why, s)
}
}
}
}
// judgeBoth judges the provider's statement and then the consumer's, two looks each, over a record that changes
// as the statements do.
func judgeBoth(t *testing.T, k *conditions.Keeper, provider []inventory.ResourceHealth,
byProvider, byConsumer map[string]inventory.NodeHealth) []conditions.Condition {
t.Helper()
ctx := t.Context()
was := readHoldingFor
t.Cleanup(func() { readHoldingFor = was })
healths := byProvider
readHoldingFor = func(_ context.Context, _ *inventory.Inventory, open []conditions.Condition) (*holding, error) {
return holdingOf(open, healths, shopOnTheDatabase), nil
}
for look := 1; look <= 2; look++ {
healths = byProvider
if err := judgeModuleHealth(ctx, nil, k, "anchor", map[string][]inventory.ResourceHealth{"db": provider},
map[string]int{"db": look}, time.Now()); err != nil {
t.Fatal(err)
}
}
for look := 1; look <= 2; look++ {
healths = byConsumer
if err := judgeModuleHealth(ctx, nil, k, "laptop", map[string][]inventory.ResourceHealth{"shop": {failingConsumer("shop")}},
map[string]int{"shop": look}, time.Now()); err != nil {
t.Fatal(err)
}
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
return open
}
func conditionOf(open []conditions.Condition, key string) *conditions.Condition {
for i, c := range open {
if c.Key == key {
return &open[i]
}
}
return nil
}
// (1) The consumer raises nothing of its own; the provider's needs-operator condition lists who waits on it and
// stays a warning (ADR 0283 decision 5: never escalated). The consumer's statement arrives after the provider's,
// so it is the consumer's judging (sayWaiters) that lists it at the provider — no store involved.
func TestAWaitingProvidersConditionListsWhoWaitsOnItAndStaysAWarning(t *testing.T) {
k, _ := withConditionsInMemory(t)
open := judgeBoth(t, k, []inventory.ResourceHealth{waitingDatabase()},
map[string]inventory.NodeHealth{"anchor": {Node: "anchor", Resources: []inventory.ResourceHealth{waitingDatabase()}}}, shopFailingBeside(waitingDatabase()))
provider := conditionOf(open, needsOperatorKey("db", "anchor"))
if len(open) != 1 || provider == nil {
t.Fatalf("a provider waiting for the operator and a consumer of its waiting part raised %v; want the "+
"provider's needs-operator alone", openKeysOf(open))
}
if said := provider.Evidence[0].Said; !strings.Contains(said, "shop on laptop") {
t.Fatalf("the provider's needs-operator does not list shop on laptop as waiting on it: %s", said)
}
if provider.Severity != conditions.Warning {
t.Fatalf("the provider's needs-operator became %s with a consumer waiting; it stays a warning", provider.Severity)
}
if provider.Resolver != conditions.ResolverOperator || !strings.Contains(provider.Needs, "desk prompt") {
t.Fatalf("the provider's condition: %+v", provider)
}
}
// (1) A consumer of another part than the one waiting is raised on its own, and says its provider waits.
func TestAConsumerOfAnotherPartIsRaisedOnItsOwn(t *testing.T) {
k, _ := withConditionsInMemory(t)
backups := waitingDatabaseFor(backupsWait)
open := judgeBoth(t, k, []inventory.ResourceHealth{backups}, shopFailingBeside(backups), shopFailingBeside(backups))
consumer := conditionOf(open, moduleUnhealthyKey("shop", "laptop"))
if consumer == nil || conditionOf(open, needsOperatorKey("db", "anchor")) == nil {
t.Fatalf("raised %v; want shop's own unhealthy beside db's needs-operator", openKeysOf(open))
}
if said := consumer.Evidence[0].Said; !strings.Contains(said, "db on anchor waits for you, but not for anything shop is known to need, so shop's fault is said on its own") {
t.Fatalf("the consumer's condition does not say its provider waits: %s", said)
}
}
// relogin and userUnit are person_wait_test.go's; mounts is said here with the same account and unit beside its
// wait for the password.
func reloginBesideAWait() []inventory.ResourceHealth {
return []inventory.ResourceHealth{relogin("mounts", "operator"), userUnit("mounts", "operator"),
waitingResource(passwordWait)}
}
// (2) A module with a checked wait beside a relogin-needed account says both: the new login, and the act the
// operator owes it.
func TestAWaitBesideAReloginIsSaid(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
for look := 1; look <= 2; look++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", map[string][]inventory.ResourceHealth{"mounts": reloginBesideAWait()},
map[string]int{"mounts": look}, time.Now()); err != nil {
t.Fatal(err)
}
}
got, open := needsOperatorOpen(t, k)
if got == nil {
t.Fatalf("a wait for the operator beside a relogin is not said: %v", openKeysOf(open))
}
if !strings.Contains(got.Summary, "smb-password-games") || got.Severity != conditions.Warning {
t.Fatalf("the needs-operator beside the relogin: %+v", got)
}
relogged := false
for _, c := range open {
relogged = relogged || c.Key == reloginKey("mounts", "workstation")
}
if !relogged {
t.Fatalf("the relogin is no longer said beside the wait: %v", openKeysOf(open))
}
// The login done, the wait stays said and the relogin clears.
if err := judgeModuleHealth(ctx, nil, k, "workstation", map[string][]inventory.ResourceHealth{"mounts": {waitingResource(passwordWait)}},
map[string]int{"mounts": 3}, time.Now()); err != nil {
t.Fatal(err)
}
got, open = needsOperatorOpen(t, k)
if got == nil || len(open) != 1 {
t.Fatalf("after the new login: %v", openKeysOf(open))
}
}
// (2) The same beside a directory used as found.
func TestAWaitBesideADirectoryUsedAsFoundIsSaid(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
found := foundDirectory("mounts", time.Now().Add(-time.Hour))
for look := 1; look <= 2; look++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", map[string][]inventory.ResourceHealth{
"mounts": {found, waitingResource(passwordWait)}}, map[string]int{"mounts": look}, time.Now()); err != nil {
t.Fatal(err)
}
}
got, open := needsOperatorOpen(t, k)
if got == nil {
t.Fatalf("a wait for the operator beside a directory used as found is not said: %v", openKeysOf(open))
}
asFound := false
for _, c := range open {
asFound = asFound || c.Key == usedAsFoundKey("mounts", "workstation")
}
if !asFound {
t.Fatalf("the directory used as found is no longer said beside the wait: %v", openKeysOf(open))
}
}
// (2) Beside a fault of its own, the wait is said too, and the fault's words do not count the waiting part among
// what fails.
func TestAWaitBesideAFaultIsSaidAndTheFaultIsTheFaultAlone(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
down := inventory.ResourceHealth{Module: "mounts", Resource: "mounts.apply", Kind: "process",
Target: "mesh-mounts-apply.service", State: link.StateUnhealthy, Reason: "down"}
for look := 1; look <= 2; look++ {
if err := judgeModuleHealth(ctx, nil, k, "workstation", map[string][]inventory.ResourceHealth{
"mounts": {down, waitingResource(passwordWait)}}, map[string]int{"mounts": look}, time.Now()); err != nil {
t.Fatal(err)
}
}
got, open := needsOperatorOpen(t, k)
if got == nil {
t.Fatalf("a wait for the operator beside a fault is not said: %v", openKeysOf(open))
}
var fault *conditions.Condition
for i, c := range open {
if c.Key == moduleUnhealthyKey("mounts", "workstation") {
fault = &open[i]
}
}
if fault == nil {
t.Fatalf("the fault is not said: %v", openKeysOf(open))
}
if strings.Contains(fault.Summary, "mounts.watch") {
t.Fatalf("the fault's summary counts the waiting part as failing: %q", fault.Summary)
}
}
func openKeysOf(cs []conditions.Condition) []string {
var out []string
for _, c := range cs {
out = append(out, c.Key)
}
return out
}
// A consumer raised on its own, whose provider then waits for the operator for the part it needs, is cleared saying
// which the provider is: it waits for you, not that it is unhealthy (novox/hq issue 405 review).
func TestAConsumerHeldOnceItsProviderWaitsSaysWhichItIs(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
start := time.Now().Add(-time.Second)
healthy := waitingDatabase()
healthy.State, healthy.Waits = link.StateHealthy, nil
healths := shopFailingBeside(healthy)
was := readHoldingFor
t.Cleanup(func() { readHoldingFor = was })
readHoldingFor = func(_ context.Context, _ *inventory.Inventory, open []conditions.Condition) (*holding, error) {
return holdingOf(open, healths, shopOnTheDatabase), nil
}
shop := map[string][]inventory.ResourceHealth{"shop": {failingConsumer("shop")}}
for look := 1; look <= 2; look++ {
if err := judgeModuleHealth(ctx, nil, k, "laptop", shop, map[string]int{"shop": look}, time.Now()); err != nil {
t.Fatal(err)
}
}
if open, _ := k.Open(ctx); conditionOf(open, moduleUnhealthyKey("shop", "laptop")) == nil {
t.Fatalf("shop beside a healthy provider is not raised: %v", openKeysOf(open))
}
healths = shopFailingBeside(waitingDatabase())
if err := judgeModuleHealth(ctx, nil, k, "laptop", shop, map[string]int{"shop": 3}, time.Now()); err != nil {
t.Fatal(err)
}
var why string
for deadline := time.Now().Add(2 * time.Second); why == "" && time.Now().Before(deadline); {
events, err := k.HistorySince(ctx, start)
if err != nil {
t.Fatal(err)
}
for _, e := range events {
if e.Key == moduleUnhealthyKey("shop", "laptop") && e.Change == conditions.ChangeCleared {
why = e.Why
}
}
if why == "" {
time.Sleep(10 * time.Millisecond)
}
}
if !strings.Contains(why, "waits on db on anchor, which waits for you") {
t.Fatalf("shop's clearing says %q; want it to say db on anchor waits for you", why)
}
}
+15 -2
View File
@@ -79,8 +79,11 @@ type signalFacts struct {
lostConsumers map[string]bool lostConsumers map[string]bool
// deadLetters are how many messages DEAD_LETTERS holds per consumer, by `<stream>.<consumer>` // deadLetters are how many messages DEAD_LETTERS holds per consumer, by `<stream>.<consumer>`
// (novox/hq issue 330): each consumer's max-deliveries condition is open while it holds any. // (novox/hq issue 330): each consumer's max-deliveries condition is open while it holds any.
deadLetters map[string]int deadLetters map[string]int
advisoriesErr error // deadLettersNewest is when DEAD_LETTERS kept the newest message it holds for each of those
// consumers: the condition counts the messages given up on, not the looks (novox/hq issue 440).
deadLettersNewest map[string]time.Time
advisoriesErr error
selfCheck selfCheckFacts selfCheck selfCheckFacts
@@ -350,6 +353,16 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
if f.advisoriesErr == nil && w.js != nil { if f.advisoriesErr == nil && w.js != nil {
f.deadLetters, f.advisoriesErr = link.HeldDeadLetters(w.js.Context()) f.deadLetters, f.advisoriesErr = link.HeldDeadLetters(w.js.Context())
} }
if f.advisoriesErr == nil && len(f.deadLetters) > 0 {
f.deadLettersNewest, f.advisoriesErr = link.NewestDeadLetters(w.js.Context(), f.deadLetters)
for key := range f.deadLetters {
if _, ok := f.deadLettersNewest[key]; !ok && f.advisoriesErr == nil {
// Delivered again or dropped between the two reads: left out of this look, rather than
// observed without a time and counted as a look (novox/hq issue 440).
delete(f.deadLetters, key)
}
}
}
f.handActs, f.handActsErr = w.gatherHandActs(ctx, now) f.handActs, f.handActsErr = w.gatherHandActs(ctx, now)
f.facts.taken, _, f.facts.began, f.facts.err = exportedFacts.last() f.facts.taken, _, f.facts.began, f.facts.err = exportedFacts.last()
return f return f
+94
View File
@@ -0,0 +1,94 @@
// Package beside is where a test finds another repository of the mesh it reads: the catalogue's manifests,
// the node-engine's genesis template (novox/hq issue 432).
//
// A test used to read the checkout beside this one, `../../../mesh-catalog`, so its verdict depended on
// whatever sat on the machine running it: a stale or dirty checkout failed it on a desktop, and where none
// was beside it skipped and said nothing. Now there are two inputs, both named, and no third:
//
// - In a merge check, the build seat clones each core repository beside the one checked (the catalogue
// at its main, the node-engine at the commit the mesh runs) and says where in MESH_CHECK_BESIDE. A
// test judges against those clones, so agreement with the other repository is checked where
// `mesh/repo-check` runs; a repository missing there fails the test, never skips it. Which clones a
// check gets is chosen from the inventory: mesh-catalog by the source of the `nats` module, mesh-host
// by the source of `mesh-host`. In a mesh where either module has no source repository nothing is
// cloned, and these tests fail loudly with "not beside this check": a cause in the setup, not in the
// change. And in a delivery group that holds a mesh-catalog pull request, these tests read the
// catalogue's main, not the group's head.
// - Anywhere else, the test judges against the copy captured in this repository's testdata/beside, at the
// commit testdata/beside/CAPTURED names. Never against a developer's own checkout: to judge one, set
// MESH_CHECK_BESIDE to the directory holding it, as the check does.
//
// The captured manifests are named module.json.captured, and put back as module.json in a directory of
// the test's own: a module.json anywhere in this repository is a module of it to the forge's announcer and
// the planner, and a merge would build and register a hundred copies of the catalogue's.
package beside
import (
"io/fs"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
)
// Env is where a merge check says the repositories cloned beside the one checked are (internal/builder's
// EnvBeside, repeated here so a test does not import the builder).
const Env = "MESH_CHECK_BESIDE"
// CapturedSuffix is what a captured file's name carries that the repository's own does not.
const CapturedSuffix = ".captured"
// Dir is the named repository a test reads — the clone beside a merge check, or the captured copy — and
// says which in the test's log.
func Dir(t testing.TB, repository string) string {
t.Helper()
if root := os.Getenv(Env); root != "" {
dir := filepath.Join(root, repository)
if _, err := os.Stat(dir); err != nil {
t.Fatalf("%s is not beside this check in %s=%s, so its agreement with this repository cannot be "+
"judged here: %v", repository, Env, root, err)
}
t.Logf("judged against %s as cloned beside this check", dir)
return dir
}
from := filepath.Join(Captured(), repository)
if _, err := os.Stat(from); err != nil {
t.Fatalf("no captured copy of %s at %s: %v", repository, from, err)
}
dir := filepath.Join(t.TempDir(), repository)
err := filepath.WalkDir(from, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, _ := filepath.Rel(from, path)
into := filepath.Join(dir, strings.TrimSuffix(rel, CapturedSuffix))
if d.IsDir() {
return os.MkdirAll(into, 0o755)
}
raw, err := os.ReadFile(path)
if err != nil {
return err
}
return os.WriteFile(into, raw, 0o644)
})
if err != nil {
t.Fatalf("the captured copy of %s could not be laid out: %v", repository, err)
}
t.Logf("judged against the copy of %s captured in testdata/beside (see CAPTURED); a merge check "+
"judges it against the repository's own clone", repository)
return dir
}
// Catalogue is the catalogue's modules directory, as Dir finds the catalogue.
func Catalogue(t testing.TB) string {
t.Helper()
return filepath.Join(Dir(t, "mesh-catalog"), "modules")
}
// Captured is this repository's testdata/beside, found from this file rather than from the working
// directory, so a test in any package reaches it.
func Captured() string {
_, file, _, _ := runtime.Caller(0)
return filepath.Join(filepath.Dir(file), "..", "..", "testdata", "beside")
}
+4 -3
View File
@@ -8,6 +8,7 @@ import (
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
"github.com/novox/mesh-controller/internal/catalogue" "github.com/novox/mesh-controller/internal/catalogue"
) )
@@ -74,10 +75,10 @@ func TestTheAgreementCheckCatchesASubscriptionThatMatchesNothing(t *testing.T) {
func theCataloguesEvents(t *testing.T) ([]AnEmitter, []AConsumer, []DeclaredSeat) { func theCataloguesEvents(t *testing.T) ([]AnEmitter, []AConsumer, []DeclaredSeat) {
t.Helper() t.Helper()
root := filepath.Join("..", "..", "..", "mesh-catalog", "modules") root := beside.Catalogue(t)
entries, err := os.ReadDir(root) entries, err := os.ReadDir(root)
if err != nil { if err != nil {
t.Skipf("catalogue sibling not present: %v", err) t.Fatal(err)
} }
var emitters []AnEmitter var emitters []AnEmitter
var consumers []AConsumer var consumers []AConsumer
@@ -119,7 +120,7 @@ func theCataloguesEvents(t *testing.T) ([]AnEmitter, []AConsumer, []DeclaredSeat
} }
} }
if len(emitters) == 0 { if len(emitters) == 0 {
t.Skip("no manifests found beside this checkout") t.Fatalf("no module under %s emits anything, so this proved nothing", root)
} }
return emitters, consumers, seats return emitters, consumers, seats
} }
+6 -3
View File
@@ -10,6 +10,8 @@ import (
"testing" "testing"
"golang.org/x/crypto/bcrypt" "golang.org/x/crypto/bcrypt"
"github.com/novox/mesh-controller/internal/beside"
) )
// **The first user list the installer carries must be the one the controller would compose.** // **The first user list the installer carries must be the one the controller would compose.**
@@ -76,13 +78,14 @@ func theCarriedAccounts(t *testing.T) string {
return "" return ""
} }
// theTemplate is the installer's bundle, as resources. // theTemplate is the installer's bundle, as resources: the node-engine's, as a merge check clones it at
// the commit the mesh runs, or as captured in testdata/beside (internal/beside, novox/hq issue 432).
func theTemplate(t *testing.T) []map[string]any { func theTemplate(t *testing.T) []map[string]any {
t.Helper() t.Helper()
path := filepath.Join("..", "..", "..", "mesh-host", "examples", "foundation-first-node-nats.lock") path := filepath.Join(beside.Dir(t, "mesh-host"), "examples", "foundation-first-node-nats.lock")
raw, err := os.ReadFile(path) raw, err := os.ReadFile(path)
if err != nil { if err != nil {
t.Skipf("the host's checkout is not beside this one: %v", err) t.Fatal(err)
} }
// The template is JSON with line comments, which is how every one of them is written. // The template is JSON with line comments, which is how every one of them is written.
var lines []string var lines []string
+20 -68
View File
@@ -42,64 +42,8 @@ const (
// Its authority is the union of what the modules it carries would each have had for their // Its authority is the union of what the modules it carries would each have had for their
// tools — and nothing of what they consume, because tools are what it runs, not reactions. // tools — and nothing of what they consume, because tools are what it runs, not reactions.
KindNodeTools Kind = "node-tools" KindNodeTools Kind = "node-tools"
// KindView is the one read-only principal a view onto the bus connects as (novox/hq research 036,
// gap G1): a page in a browser, over the bus module's WebSocket listener, watching the issue tracker.
// Fixed, and derived from no declaration: what it hears is ViewHears, what it reads is ViewBucket,
// and it publishes nothing but direct reads of that one bucket (ViewReads), each answered in its own
// inbox. Composed like every other user, into the same
// file, once its credential is minted (`bus view-credential`); forgotten like every other user
// (`bus view-revoke`), at the next composition.
KindView Kind = "view"
) )
// ViewUser is the view's one username: there is one view, and it is nobody's machine or module.
const ViewUser = "view"
// ViewBucket is the state the view reads: the issue tracker's issues, as the bus names the bucket
// (mesh-issues's state `issues`, novox/hq ADR 0201).
var ViewBucket = BucketName("mesh-issues", "issues")
// ViewHears are the events the view subscribes, each named: the issue tracker's own, the controller's
// walks and conditions, and the delivery owner's — what a page about issues shows beside them. Subscribe
// only, and no stream or consumer of its own: a page hears what happens while it is open, and reads the
// bucket for everything before.
var ViewHears = []string{
moduleEventSubject("mesh-issues", "opened"),
moduleEventSubject("mesh-issues", "moved"),
moduleEventSubject("mesh-issues", "noted"),
moduleEventSubject("mesh-issues", "linked"),
seatEventSubject(ControllerSeat, "plan-moved"),
seatEventSubject(ControllerSeat, "condition-raised"),
seatEventSubject(ControllerSeat, "condition-changed"),
seatEventSubject(ControllerSeat, "condition-cleared"),
moduleEventSubject("mesh-delivery", "transition"),
moduleEventSubject("mesh-delivery", "group"),
}
// ViewReads are the JetStream API requests the view makes, on ViewBucket's stream and no other: binding
// (STREAM.INFO), and direct reads — one key by its subject (`DIRECT.GET.<stream>.$KV.<bucket>.<key>`), and
// the batch form on the bare subject, which answers the newest value of every key (`multi_last`) into the
// asker's inbox. The page lists the bucket with the batch, and re-reads one key when the tracker's event
// names it (every event carries the issue's `number`).
//
// **No consumer, deliberately, and so no watch.** A KV watch is a push consumer, and a push consumer's
// deliver subject is the creator's choice, delivered by the server's own client — which the server does
// not hold to the creator's permissions. Measured on 2.11.17 (2026-10-10): the view, granted
// CONSUMER.CREATE on this stream, made a consumer delivering to `mesh.mod.mesh-issues.event.opened`, and
// a module subscribed there received the bucket's entry as the tracker's event. A grant of CONSUMER.CREATE
// is a publish to any subject in the account; the view publishes nothing, so it has none (nor
// CONSUMER.DELETE, which would let it delete a module's consumer). Nothing here is a write either: no
// `$KV.<bucket>.>`, which is what a put or a delete publishes to, and no STREAM.* that defines, purges or
// deletes.
func ViewReads() []string {
stream := "KV_" + ViewBucket
return []string{
"$JS.API.STREAM.INFO." + stream,
"$JS.API.DIRECT.GET." + stream,
"$JS.API.DIRECT.GET." + stream + ".>",
}
}
// RuntimeModule is the module that IS the node's tool runtime (novox/hq ADR 0175). Where it is // RuntimeModule is the module that IS the node's tool runtime (novox/hq ADR 0175). Where it is
// assigned, the mesh composes one runtime principal for the machine in place of that module's own, // assigned, the mesh composes one runtime principal for the machine in place of that module's own,
// and the per-module containers that served tools until then stop being the way tools reach a node. // and the per-module containers that served tools until then stop being the way tools reach a node.
@@ -219,6 +163,21 @@ type Principal struct {
// goes with the retired seat row. // goes with the retired seat row.
var seatsTheControllerAsks = []string{"node-build-agent", "mesh-build-machine"} var seatsTheControllerAsks = []string{"node-build-agent", "mesh-build-machine"}
// TheControllersAsk says whether a message of stream, published on subject, is an ask the controller
// itself makes: one on the accept subject of a seat in seatsTheControllerAsks, kept in that seat's own
// work queue. Such an ask, given up on by the seat's worker, can be delivered again with the authority
// the controller already holds — the publish on that seat's accepts and the stream API to remove the
// original — and no other can (novox/hq issue 334, ADR 0264's consequences: no grant over the seats'
// queues).
func TheControllersAsk(stream, subject string) bool {
for _, seat := range seatsTheControllerAsks {
if stream == seatStreamName(seat) && strings.HasPrefix(subject, "mesh.seat."+seat+".accept.") {
return true
}
}
return false
}
// SeatVerb is one verb of one seat, on every machine holding it. // SeatVerb is one verb of one seat, on every machine holding it.
type SeatVerb struct{ Seat, Verb string } type SeatVerb struct{ Seat, Verb string }
@@ -288,8 +247,11 @@ func MaySubscribe(perms Permissions, subject string) bool {
// //
// And, since novox/hq ADR 0259, `release` and `stop`: the controller asks the operator for them about a // And, since novox/hq ADR 0259, `release` and `stop`: the controller asks the operator for them about a
// delivery held past its bound, and calls them on the operator's warrant, with its why. // delivery held past its bound, and calls them on the operator's warrant, with its why.
//
// And, since novox/hq ADR 0282, `times`: the self-check reads the delivery times to say a delivery over its budget.
var VerbsTheControllerAsksTheDeliveryOwner = []SeatVerb{{Seat: "mesh-delivery", Verb: "stalled"}, var VerbsTheControllerAsksTheDeliveryOwner = []SeatVerb{{Seat: "mesh-delivery", Verb: "stalled"},
{Seat: "mesh-delivery", Verb: "close"}, {Seat: "mesh-delivery", Verb: "release"}, {Seat: "mesh-delivery", Verb: "stop"}} {Seat: "mesh-delivery", Verb: "close"}, {Seat: "mesh-delivery", Verb: "release"}, {Seat: "mesh-delivery", Verb: "stop"},
{Seat: "mesh-delivery", Verb: "times"}}
// VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant // VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant
// chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the // chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the
@@ -317,8 +279,6 @@ func (p Principal) Username() string {
switch p.Kind { switch p.Kind {
case KindPerson: case KindPerson:
return "person." + p.Module return "person." + p.Module
case KindView:
return ViewUser
case KindModule, KindNodeTools: case KindModule, KindNodeTools:
// The runtime is named exactly as the module it stands for would have been: the mesh // The runtime is named exactly as the module it stands for would have been: the mesh
// issues its credential through the same path a module's takes (`module issue`), and // issues its credential through the same path a module's takes (`module issue`), and
@@ -591,14 +551,6 @@ func PermissionsFor(p Principal) (Permissions, error) {
// itself, its replies to the asker's own inbox. // itself, its replies to the asker's own inbox.
pub = append(pub, discovering()...) pub = append(pub, discovering()...)
case KindView:
// Hears what it is for and reads one bucket, and nothing else (ViewHears, ViewReads): no tool,
// no event of its own, no stream, no bucket written. Its requests are answered in its own
// inbox, granted below with the person's; a reply to anything is never permitted, because
// nothing is ever asked of it.
sub = append(sub, ViewHears...)
pub = append(pub, ViewReads()...)
case KindEnrolment: case KindEnrolment:
// A leaked token is useless for anything but enrolling: it cannot read a declaration, hear // A leaked token is useless for anything but enrolling: it cannot read a declaration, hear
// an event, or subscribe any inbox but the one its own token derives (design 25 §6). // an event, or subscribe any inbox but the one its own token derives (design 25 §6).
@@ -900,7 +852,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
pub = unique(pub) pub = unique(pub)
} }
if p.Kind == KindPerson || p.Kind == KindView { if p.Kind == KindPerson {
// An inbox to hear answers in, and nothing else. No ack subject: a person has no durable // An inbox to hear answers in, and nothing else. No ack subject: a person has no durable
// consumer, because nothing is delivered to a person — they ask and are answered. // consumer, because nothing is delivered to a person — they ask and are answered.
sub = append(sub, p.inbox()) sub = append(sub, p.inbox())
-2
View File
@@ -31,8 +31,6 @@ func TestTheComposedConfigMatchesTheGolden(t *testing.T) {
// The bus's own module: the snapshot API and its inbox, nothing else (novox/hq ADR 0235). // The bus's own module: the snapshot API and its inbox, nothing else (novox/hq ADR 0235).
{Kind: KindModule, Node: "one", Module: "nats", SnapshotsTheBus: true, {Kind: KindModule, Node: "one", Module: "nats", SnapshotsTheBus: true,
Serves: []string{"nats_streams"}, PasswordHash: "$2a$11$bbbbbbbbbbbbbbbbbbbbbb"}, Serves: []string{"nats_streams"}, PasswordHash: "$2a$11$bbbbbbbbbbbbbbbbbbbbbb"},
// The view: hears the issue tracker and reads its bucket, writes nothing (research 036).
{Kind: KindView, PasswordHash: "$2a$11$vvvvvvvvvvvvvvvvvvvvvv"},
}) })
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
+1 -5
View File
@@ -24,7 +24,7 @@ accounts {
jetstream: enabled jetstream: enabled
users = [ users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: { { user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*"] } publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] } subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
allow_responses: { max: 1, ttl: "1m" } allow_responses: { max: 1, ttl: "1m" }
} } } }
@@ -56,10 +56,6 @@ accounts {
subscribe: { allow: ["$SRV.INFO", "$SRV.INFO.shop", "$SRV.INFO.shop.>", "$SRV.PING", "$SRV.PING.shop", "$SRV.PING.shop.>", "$SRV.STATS", "$SRV.STATS.shop", "$SRV.STATS.shop.>", "_INBOX.two.shop.>", "mesh.assignment.two.shop", "mesh.mod.shop.tool.>"] } subscribe: { allow: ["$SRV.INFO", "$SRV.INFO.shop", "$SRV.INFO.shop.>", "$SRV.PING", "$SRV.PING.shop", "$SRV.PING.shop.>", "$SRV.STATS", "$SRV.STATS.shop", "$SRV.STATS.shop.>", "_INBOX.two.shop.>", "mesh.assignment.two.shop", "mesh.mod.shop.tool.>"] }
allow_responses: { max: 1, ttl: "1m" } allow_responses: { max: 1, ttl: "1m" }
} } } }
{ user: "view", password: "$2a$11$vvvvvvvvvvvvvvvvvvvvvv", permissions: {
publish: { allow: ["$JS.API.DIRECT.GET.KV_mesh-issues_issues", "$JS.API.DIRECT.GET.KV_mesh-issues_issues.>", "$JS.API.STREAM.INFO.KV_mesh-issues_issues"] }
subscribe: { allow: ["_INBOX.view.>", "mesh.mod.mesh-delivery.event.group", "mesh.mod.mesh-delivery.event.transition", "mesh.mod.mesh-issues.event.linked", "mesh.mod.mesh-issues.event.moved", "mesh.mod.mesh-issues.event.noted", "mesh.mod.mesh-issues.event.opened", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.plan-moved"] }
} }
] ]
} }
} }
-6
View File
@@ -67,9 +67,6 @@ type Records struct {
Enrolling []string Enrolling []string
// People is each person's name against the tools they may invoke, `*` for an administrator. // People is each person's name against the tools they may invoke, `*` for an administrator.
People map[string][]string People map[string][]string
// View says the mesh minted the view's credential (`bus view-credential`), so the one read-only
// view principal is composed (KindView); forgotten, it is left out, like a person.
View bool
// Interchangeable is each module whose definition says its instances are the same anywhere // Interchangeable is each module whose definition says its instances are the same anywhere
// (ADR 0160), which decides whether the module's plain subject is issued to every instance. // (ADR 0160), which decides whether the module's plain subject is issued to every instance.
Interchangeable map[string]bool Interchangeable map[string]bool
@@ -145,9 +142,6 @@ func Users(r Records) ([]Principal, error) {
for _, person := range sortedNames(r.People) { for _, person := range sortedNames(r.People) {
out = append(out, Principal{Kind: KindPerson, Module: person, Invokes: r.People[person]}) out = append(out, Principal{Kind: KindPerson, Module: person, Invokes: r.People[person]})
} }
if r.View {
out = append(out, Principal{Kind: KindView})
}
// Refused here rather than discovered by the server. Two users with one name is a file the // Refused here rather than discovered by the server. Two users with one name is a file the
// server reads as one of them, and which one depends on the order — so a module assigned to a // server reads as one of them, and which one depends on the order — so a module assigned to a
-230
View File
@@ -1,230 +0,0 @@
package broker
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"golang.org/x/crypto/bcrypt"
)
// The view against a real server, over WebSocket (novox/hq research 036): the composed user list is
// what the server reads, the listener is the shape the bus module declares (no TLS, compression on,
// reached across the overlay only), and the view with its credential binds the issue tracker's bucket,
// lists it with one batch read, follows a tracker event to re-read the key it names — and is refused
// every write: a put, a delete, an event, and a consumer delivering onto the tracker's event subject.
//
// go test ./internal/broker/ -run TestTheView
func TestTheViewReadsTheIssuesOverWebSocketAndWritesNothing(t *testing.T) {
hash := func(password string) string {
h, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
return string(h)
}
const node = "anchor"
tracker := Principal{Kind: KindModule, Node: node, Module: "mesh-issues", State: []string{"issues"},
Emits: []string{"opened", "moved"}, PasswordHash: hash("tracker")}
accounts, err := ComposeAccounts([]Principal{
{Kind: KindController, PasswordHash: hash("controller")},
tracker,
{Kind: KindView, PasswordHash: hash("view")},
})
if err != nil {
t.Fatal(err)
}
conf := filepath.Join(t.TempDir(), "accounts.conf")
if err := os.WriteFile(conf, []byte(accounts), 0o600); err != nil {
t.Fatal(err)
}
// The server reads the composed file as the bus does — through its own parser — and listens as the
// bus module's configuration says: a WebSocket listener without TLS and with compression, beside
// the client port. Ports chosen by the system, so this runs beside a live bus.
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the server refused the composed user list: %v", err)
}
opts.Host, opts.Port = "127.0.0.1", server.RANDOM_PORT
opts.JetStream, opts.StoreDir = true, t.TempDir()
opts.NoLog, opts.NoSigs = true, true
opts.Websocket = server.WebsocketOpts{Host: "127.0.0.1", Port: server.RANDOM_PORT, NoTLS: true, Compression: true}
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(30 * time.Second) {
s.Shutdown()
t.Fatal("the server did not come up")
}
t.Cleanup(func() { s.Shutdown(); s.WaitForShutdown() })
dial := func(url, user, password string, refused chan<- string) *nats.Conn {
t.Helper()
nc, err := nats.Connect(url, nats.UserInfo(user, password), nats.CustomInboxPrefix("_INBOX."+user),
nats.Compression(true), nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) {
if refused != nil && errors.Is(err, nats.ErrPermissionViolation) {
refused <- err.Error()
}
}))
if err != nil {
t.Fatalf("%s could not connect to %s: %v", user, url, err)
}
t.Cleanup(nc.Close)
return nc
}
// The controller defines the bucket, as it does for every module's state; the tracker writes it.
controller := dial(s.ClientURL(), "controller", "controller", nil)
cjs, _ := controller.JetStream()
if _, err := cjs.CreateKeyValue(&nats.KeyValueConfig{Bucket: ViewBucket, History: 8}); err != nil {
t.Fatalf("the controller could not define %s: %v", ViewBucket, err)
}
trackerConn := dial(s.ClientURL(), tracker.Username(), "tracker", nil)
tjs, _ := trackerConn.JetStream()
tkv, err := tjs.KeyValue(ViewBucket)
if err != nil {
t.Fatal(err)
}
if _, err := tkv.Put("365", []byte(`{"number":365,"status":"open"}`)); err != nil {
t.Fatalf("the tracker could not write its own bucket: %v", err)
}
// The view, over WebSocket with its credential.
refused := make(chan string, 8)
view := dial(s.WebsocketURL(), ViewUser, "view", refused)
if !strings.HasPrefix(view.ConnectedUrl(), "ws://") {
t.Fatalf("the view is connected to %s, not over WebSocket", view.ConnectedUrl())
}
vjs, _ := view.JetStream(nats.MaxWait(3 * time.Second))
vkv, err := vjs.KeyValue(ViewBucket)
if err != nil {
t.Fatalf("the view could not bind %s: %v", ViewBucket, err)
}
if got, err := vkv.Get("365"); err != nil {
t.Fatalf("the view could not read a key: %v", err)
} else if !strings.Contains(string(got.Value()), `"number":365`) {
t.Fatalf("the view read %q", got.Value())
}
if _, err := tkv.Put("366", []byte(`{"number":366,"status":"open"}`)); err != nil {
t.Fatal(err)
}
// The list: one batch read, the newest value of every key, into the view's own inbox, ended by the
// server's end-of-batch status (204).
listed := map[string]string{}
inbox := view.NewRespInbox()
batch, err := view.SubscribeSync(inbox)
if err != nil {
t.Fatal(err)
}
if err := view.PublishRequest("$JS.API.DIRECT.GET.KV_"+ViewBucket, inbox,
[]byte(`{"multi_last":["$KV.`+ViewBucket+`.>"]}`)); err != nil {
t.Fatal(err)
}
for {
m, err := batch.NextMsg(5 * time.Second)
if err != nil {
t.Fatalf("the view's batch read ended without its end-of-batch (%d keys so far): %v", len(listed), err)
}
if m.Header.Get("Status") == "204" {
break
}
if status := m.Header.Get("Status"); status != "" {
t.Fatalf("the batch read answered %s %s", status, m.Header.Get("Description"))
}
listed[m.Header.Get("Nats-Subject")] = string(m.Data)
}
_ = batch.Unsubscribe()
for _, key := range []string{"365", "366"} {
if !strings.Contains(listed["$KV."+ViewBucket+"."+key], `"number":`+key) {
t.Errorf("the batch read did not list %s: %v", key, listed)
}
}
// A change followed: the tracker moves 365 and says so; the view hears the event and reads the key
// it names.
moved, err := view.SubscribeSync("mesh.mod.mesh-issues.event.moved")
if err != nil {
t.Fatal(err)
}
_ = view.Flush()
if _, err := tkv.Put("365", []byte(`{"number":365,"status":"located"}`)); err != nil {
t.Fatal(err)
}
if err := trackerConn.Publish("mesh.mod.mesh-issues.event.moved", []byte(`{"number":365,"to":"located"}`)); err != nil {
t.Fatal(err)
}
if _, err := moved.NextMsg(5 * time.Second); err != nil {
t.Fatalf("the view did not hear the tracker's event: %v", err)
}
if got, err := vkv.Get("365"); err != nil || !strings.Contains(string(got.Value()), "located") {
t.Fatalf("after the event the view read %v (%v)", got, err)
}
// **The hole a watch would open, shut** (ViewReads): a consumer delivering onto the tracker's event
// subject would have the server republish the bucket there, as the tracker. Refused, and nothing
// reaches a module listening on that subject.
listener, err := trackerConn.SubscribeSync("mesh.mod.mesh-issues.event.opened")
if err != nil {
t.Fatal(err)
}
_ = trackerConn.Flush()
if _, err := vjs.AddConsumer("KV_"+ViewBucket, &nats.ConsumerConfig{Name: "w", DeliverSubject: "mesh.mod.mesh-issues.event.opened",
AckPolicy: nats.AckNonePolicy, FilterSubject: "$KV." + ViewBucket + ".>"}); err == nil {
t.Error("the view made a consumer")
}
if _, err := vkv.WatchAll(); err == nil {
t.Error("the view made a watch, which is a consumer")
}
if m, err := listener.NextMsg(2 * time.Second); err == nil {
t.Errorf("a message reached the tracker's event subject from the view: %q", m.Data)
}
// The refusals of the consumer create land as permission violations too; drained before the writes.
drain := time.After(500 * time.Millisecond)
for draining := true; draining; {
select {
case <-refused:
case <-drain:
draining = false
}
}
// And every write is refused: the server says so, and the bucket is unchanged.
if _, err := vkv.Put("367", []byte(`{"number":367}`)); err == nil {
t.Error("the view put a key")
}
if err := vkv.Delete("365"); err == nil {
t.Error("the view deleted a key")
}
if err := view.Publish("mesh.mod.mesh-issues.event.opened", []byte(`{"number":367}`)); err != nil {
t.Fatal(err)
}
_ = view.Flush()
violations := map[string]bool{}
deadline := time.After(10 * time.Second)
for len(violations) < 3 {
select {
case v := <-refused:
for _, subject := range []string{"$KV." + ViewBucket + ".367", "$KV." + ViewBucket + ".365", "mesh.mod.mesh-issues.event.opened"} {
if strings.Contains(v, subject) {
violations[subject] = true
}
}
case <-deadline:
t.Fatalf("the server refused %d of the view's 3 writes as permission violations", len(violations))
}
}
if _, err := tkv.Get("367"); !errors.Is(err, nats.ErrKeyNotFound) {
t.Errorf("after the view's put, 367 is %v", err)
}
if _, err := tkv.Get("365"); err != nil {
t.Errorf("after the view's delete, 365 is gone: %v", err)
}
}
-144
View File
@@ -1,144 +0,0 @@
package broker
import (
"reflect"
"sort"
"testing"
)
// The view (novox/hq research 036): one read-only user, composed like every other, whose whole
// authority is a list here — so a grant that is not on the list fails a test, not a review.
// Exactly what it hears, exactly what it asks, and nothing it could write or answer. A mutation that
// adds a publish grant — `$KV.<bucket>.>`, an event, a tool — fails here.
func TestTheViewHearsAndReadsAndCanPublishNothingElse(t *testing.T) {
perms, err := PermissionsFor(Principal{Kind: KindView})
if err != nil {
t.Fatal(err)
}
wantSub := append(append([]string(nil), ViewHears...), "_INBOX.view.>")
sort.Strings(wantSub)
if !reflect.DeepEqual(perms.Subscribe, wantSub) {
t.Errorf("the view subscribes\n %v\nand should subscribe exactly\n %v", perms.Subscribe, wantSub)
}
wantPub := ViewReads()
sort.Strings(wantPub)
if !reflect.DeepEqual(perms.Publish, wantPub) {
t.Errorf("the view publishes\n %v\nand should publish exactly\n %v", perms.Publish, wantPub)
}
if len(perms.PublishDeny) != 0 {
t.Errorf("the view needs no deny, because nothing it may publish reaches the controller's own: %v", perms.PublishDeny)
}
if perms.AllowResponses {
t.Error("the view may answer, and nothing is ever asked of it")
}
// Every publish grant is binding the one bucket's stream or reading it directly. **The mutation this
// holds against**: a write grant of any shape, and a consumer of any shape — a consumer's deliver
// subject is the creator's choice, so creating one is publishing anywhere (ViewReads).
stream := "KV_" + ViewBucket
for _, p := range perms.Publish {
readOnly := p == "$JS.API.STREAM.INFO."+stream ||
p == "$JS.API.DIRECT.GET."+stream ||
p == "$JS.API.DIRECT.GET."+stream+".>"
if !readOnly {
t.Errorf("the view is granted a publish on %q, which is not a read of %s", p, ViewBucket)
}
}
for _, refused := range []string{
"$KV." + ViewBucket + ".365", // a put or a delete
"$KV.mesh-controller_conditions.x", // another bucket
"$JS.API.STREAM.CREATE." + stream, // defining the stream
"$JS.API.STREAM.PURGE." + stream, // emptying it
"$JS.API.STREAM.DELETE." + stream, // deleting it
"$JS.API.STREAM.MSG.DELETE." + stream, // deleting a message
"$JS.API.CONSUMER.CREATE.KV_mesh-controller_conditions.x", // reading another bucket
"$JS.API.CONSUMER.CREATE." + stream + ".w.$KV." + ViewBucket + ".>", // a watch: delivers anywhere
"$JS.API.CONSUMER.CREATE." + stream, // an unnamed consumer
"$JS.API.CONSUMER.DELETE." + stream + ".a_mesh-issues", // a module's consumer
"$JS.FC." + stream + ".x",
"$JS.API.DIRECT.GET.KV_mesh-controller_conditions", // another bucket, directly
"$JS.API.STREAM.INFO.EVENTS", // the events stream
"$JS.API.INFO", // the account
"mesh.mod.mesh-issues.event.opened", // claiming the tracker said something
"mesh.mod.mesh-issues.tool.open", // opening an issue
"mesh.seat.issue-tracker.tool.open", // through the seat
"mesh.seat.issue-tracker.tool.open.novox", // on one machine
"mesh.seat.mesh-controller.tool.status", // the controller's verbs
"mesh.seat.mesh-controller.event.plan-moved",
"$SRV.PING",
"_INBOX.controller.x",
} {
if MayPublish(perms, refused) {
t.Errorf("the view may publish %q", refused)
}
}
for _, refused := range []string{
"mesh.mod.mesh-issues.tool.open", // a tool asked of the tracker
"mesh.mod.telegram.event.received", // another module's events
"mesh.seat.mesh-controller.event.applied",
"mesh.control.novox.report",
"_INBOX.controller.x",
"_INBOX.person.jochen.x",
"_DELIVER.controller.EVENTS",
} {
if MaySubscribe(perms, refused) {
t.Errorf("the view may subscribe %q", refused)
}
}
for _, heard := range []string{
"mesh.mod.mesh-issues.event.opened",
"mesh.mod.mesh-issues.event.moved",
"mesh.mod.mesh-issues.event.noted",
"mesh.mod.mesh-issues.event.linked",
"mesh.seat.mesh-controller.event.plan-moved",
"mesh.seat.mesh-controller.event.condition-raised",
"mesh.seat.mesh-controller.event.condition-changed",
"mesh.seat.mesh-controller.event.condition-cleared",
"mesh.mod.mesh-delivery.event.transition",
"mesh.mod.mesh-delivery.event.group",
"_INBOX.view.abc",
} {
if !MaySubscribe(perms, heard) {
t.Errorf("the view cannot subscribe %q", heard)
}
}
}
// Composed once the mesh minted its credential, and not before: its row is the whole record of it.
func TestTheViewIsComposedOnlyOnceItsCredentialIsMinted(t *testing.T) {
without, err := Users(Records{Nodes: []string{"anchor"}})
if err != nil {
t.Fatal(err)
}
for _, p := range without {
if p.Kind == KindView {
t.Fatal("the view is composed before its credential was minted")
}
}
with, err := Users(Records{Nodes: []string{"anchor"}, View: true})
if err != nil {
t.Fatal(err)
}
views := 0
for _, p := range with {
if p.Kind == KindView {
views++
if p.Username() != ViewUser {
t.Errorf("the view is called %q, and its row is %q", p.Username(), ViewUser)
}
}
}
if views != 1 {
t.Fatalf("%d view users composed; there is one view", views)
}
// And without its hash it is named as missing, like any user — never written as a user anybody is.
_, missing := WithPasswords(with, map[string]string{})
found := false
for _, m := range missing {
found = found || m == ViewUser
}
if !found {
t.Error("a view with no password was not named as missing one")
}
}
@@ -5,6 +5,8 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// **The word does not come back through a manifest** (novox/hq ADR 0131). A module that wants // **The word does not come back through a manifest** (novox/hq ADR 0131). A module that wants
@@ -28,12 +30,12 @@ func TestAManifestRequiringAmqpIsRefused(t *testing.T) {
} }
} }
// And the catalogue as checked out beside this repository names it nowhere — the three modules that // And the catalogue (internal/beside) names it nowhere — the three modules that did are removed under
// did are removed under design 28 task 5.4, not converted. // design 28 task 5.4, not converted.
func TestNoCatalogueManifestNamesAmqp(t *testing.T) { func TestNoCatalogueManifestNamesAmqp(t *testing.T) {
modules, err := filepath.Glob("../../../mesh-catalog/modules/*/module.json") modules, err := filepath.Glob(filepath.Join(beside.Catalogue(t), "*", "module.json"))
if err != nil || len(modules) == 0 { if err != nil || len(modules) == 0 {
t.Skip("the catalogue is not checked out beside this repository") t.Fatalf("no manifests in the catalogue, so this proved nothing: %v", err)
} }
for _, path := range modules { for _, path := range modules {
raw, err := os.ReadFile(path) raw, err := os.ReadFile(path)
+5 -2
View File
@@ -2,8 +2,11 @@ package catalogue
import ( import (
"os" "os"
"path/filepath"
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// The module holding mesh-broker is the bus, and its account is granted the bus's snapshot API and // The module holding mesh-broker is the bus, and its account is granted the bus's snapshot API and
@@ -34,9 +37,9 @@ func TestTheBussAccountSaysNothingOnTheBus(t *testing.T) {
// dump into its snapshots, it has the account the dump runs as, and the dump runs the snapshot // dump into its snapshots, it has the account the dump runs as, and the dump runs the snapshot
// program in the bus's own container. // program in the bus's own container.
func TestTheCataloguesBusIsBackedUpBySnapshot(t *testing.T) { func TestTheCataloguesBusIsBackedUpBySnapshot(t *testing.T) {
raw, err := os.ReadFile("../../../mesh-catalog/modules/nats/module.json") raw, err := os.ReadFile(filepath.Join(beside.Catalogue(t), "nats", "module.json"))
if err != nil { if err != nil {
t.Skip("the catalogue is not checked out beside this repository") t.Fatal(err)
} }
m, err := ParseManifest(raw) m, err := ParseManifest(raw)
if err != nil { if err != nil {
+7 -12
View File
@@ -5,26 +5,21 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// TestEveryCatalogueManifestParses runs the real catalogue through the real gate. // TestEveryCatalogueManifestParses runs the real catalogue through the real gate.
// //
// Not a fixture: the point is whether the manifests as written are accepted by the control plane that // Not a fixture: the point is whether the manifests as written are accepted by the control plane that
// will read them, and a copy of one manifest proves nothing about the other seventy-one. // will read them, and a copy of one manifest proves nothing about the other seventy-one.
// catalogueRoot is the catalogue these checks run over: MESH_CATALOGUE when set, else the checkout // catalogueRoot is the catalogue these checks run over: in a merge check the clone the build seat put
// beside this one, the way the main layout has it. A check that only ran when somebody remembered a // beside this one, elsewhere the copy captured in testdata/beside (internal/beside). A check that only
// variable was a check nobody ran (novox/hq issue 134, 2026-09-30); it skips only when there is no // ran when somebody remembered a variable was a check nobody ran (novox/hq issue 134), and one that read
// catalogue to be found at all. // whatever checkout sat beside judged the machine, not the change (novox/hq issue 432): it never skips.
func catalogueRoot(t *testing.T) string { func catalogueRoot(t *testing.T) string {
t.Helper() t.Helper()
if root := os.Getenv("MESH_CATALOGUE"); root != "" { return beside.Dir(t, "mesh-catalog")
return root
}
sibling := filepath.Join("..", "..", "..", "mesh-catalog")
if _, err := os.Stat(filepath.Join(sibling, "modules")); err != nil {
t.Skip("no catalogue beside this checkout and MESH_CATALOGUE unset")
}
return sibling
} }
func TestEveryCatalogueManifestParses(t *testing.T) { func TestEveryCatalogueManifestParses(t *testing.T) {
+5 -2
View File
@@ -2,8 +2,11 @@ package catalogue
import ( import (
"os" "os"
"path/filepath"
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// **A machine trusts the mesh's authority because a module put its root there** (novox/hq ADR // **A machine trusts the mesh's authority because a module put its root there** (novox/hq ADR
@@ -16,9 +19,9 @@ import (
// itself — a plain client trusting an internal name on a machine holding this, and failing on one // itself — a plain client trusting an internal name on a machine holding this, and failing on one
// that does not — is the lab's, and cannot be had here. // that does not — is the lab's, and cannot be had here.
func TestCaTrustRendersTheAuthorityItWasBoundTo(t *testing.T) { func TestCaTrustRendersTheAuthorityItWasBoundTo(t *testing.T) {
raw, err := os.ReadFile("../../../mesh-catalog/modules/ca-trust/module.json") raw, err := os.ReadFile(filepath.Join(beside.Catalogue(t), "ca-trust", "module.json"))
if err != nil { if err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err) t.Fatal(err)
} }
m, err := ParseManifest(raw) m, err := ParseManifest(raw)
if err != nil { if err != nil {
+39 -5
View File
@@ -606,7 +606,18 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
"mode": "0600", "mode": "0600",
}) })
} }
for _, name := range sortedKeys(m.OwnSecrets) { // **A secret family's members, as given** (novox/hq ADR 0283): one file each at the family's path, and
// nothing for a member not given — the mesh never makes one, and the module says it waits for it.
for _, name := range sortedKeys(with.Needed[m.Module]) {
own, family, ok := m.OwnSecrets.Lookup(name)
if !ok || family == "" || with.Needed[m.Module][name] == "" {
continue
}
first = append(first, ownedBy(m.SecretsOwner, map[string]any{
"id": NeedID(name), "type": "file", "path": own.Path, "sealed": with.Needed[m.Module][name],
}))
}
for _, name := range sortedKeys(m.OwnSecrets.Plain()) {
sealed := with.Needed[m.Module][name] sealed := with.Needed[m.Module][name]
if sealed == "" && with.Foreseen[m.Module][name] { if sealed == "" && with.Foreseen[m.Module][name] {
// Not made, and the next send makes it: composed with a stand-in so that whatever // Not made, and the next send makes it: composed with a stand-in so that whatever
@@ -900,6 +911,15 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
return nil, err return nil, err
} }
} }
// **And every placeholder no pass fills where it stands is refused** (novox/hq issue 231):
// judged here, over the definition as written and before any pass, by the function the
// catalogue check runs — so a manifest registered by an older binary is judged too, and
// what a setting or a binding later puts into a file is its software's text, never swept.
for _, own := range m.Resources {
if problems := unconsumedPlaceholders(m.Module, own); len(problems) > 0 {
return nil, fmt.Errorf("%s", problems[0])
}
}
// Which of this module's files carry a secret, for the rule that a container may not read // Which of this module's files carry a secret, for the rule that a container may not read
// one of them as its environment without saying so (ADR 0086, issue 041). // one of them as its environment without saying so (ADR 0086, issue 041).
@@ -1056,7 +1076,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// credential the mesh had replaced, because its manifest restarted it on its // credential the mesh had replaced, because its manifest restarted it on its
// environment file and nobody had thought to name the credential too. Composed here so // environment file and nobody had thought to name the credential too. Composed here so
// no manifest has to say it, for a container or a daemon that names the secret's path. // no manifest has to say it, for a container or a daemon that names the secret's path.
if reads := secretsReadBy(copied, m); len(reads) > 0 { if reads := secretsReadBy(copied, m, with.Needed[m.Module]); len(reads) > 0 {
copied["restart-on"] = withRestartOn(copied["restart-on"], reads) copied["restart-on"] = withRestartOn(copied["restart-on"], reads)
} }
// **A version prepares its state before it runs** (novox/hq ADR 0135). Derived from the // **A version prepares its state before it runs** (novox/hq ADR 0135). Derived from the
@@ -2372,7 +2392,12 @@ func portOfEndpoint(values map[string]any, ports map[string]int) {
// — named in its volumes, its environment or its env-files by the secret's placed path — as // — named in its volumes, its environment or its env-files by the secret's placed path — as
// restart-on ids. Nothing for other shapes, and nothing for a scheduled or run-once process, which // restart-on ids. Nothing for other shapes, and nothing for a scheduled or run-once process, which
// the host refuses a restart-on for (it runs again anyway, and reads the file afresh). // the host refuses a restart-on for (it runs again anyway, and reads the file afresh).
func secretsReadBy(resource map[string]any, m Manifest) []string { //
// **A member of a secret family given here is read the same way** (novox/hq issue 405, ADR 0283 decision 6):
// it is an own secret placed at its own path, and a container that mounted it would keep the value it
// started with when the operator gives it again. given is the module's own secrets sealed to this
// machine; a member not given is no file, so nothing restarts on it.
func secretsReadBy(resource map[string]any, m Manifest, given map[string]string) []string {
kind := fmt.Sprint(resource["type"]) kind := fmt.Sprint(resource["type"])
if kind != "container" && kind != "process" { if kind != "container" && kind != "process" {
return nil return nil
@@ -2384,9 +2409,18 @@ func secretsReadBy(resource map[string]any, m Manifest) []string {
for _, key := range []string{"volumes", "env", "env-file"} { for _, key := range []string{"volumes", "env", "env-file"} {
mentioned = append(mentioned, stringsIn(resource[key])...) mentioned = append(mentioned, stringsIn(resource[key])...)
} }
paths := map[string]string{}
for name, own := range m.OwnSecrets.Plain() {
paths[name] = own.Path
}
for name, sealed := range given {
if own, family, ok := m.OwnSecrets.Lookup(name); ok && family != "" && sealed != "" {
paths[name] = own.Path
}
}
var out []string var out []string
for _, name := range sortedKeys(m.OwnSecrets) { for _, name := range sortedKeys(paths) {
path := m.OwnSecrets[name].Path path := paths[name]
if path == "" { if path == "" {
continue continue
} }
+9
View File
@@ -18,6 +18,15 @@ func deliveryVerbs() []Verb {
Input: schema(map[string]string{"state": "one state, e.g. delivering or held", Input: schema(map[string]string{"state": "one state, e.g. delivering or held",
"repository": "owner/repository", "group": "a group's id (its branch name)", "repository": "owner/repository", "group": "a group's id (its branch name)",
"all": "\"true\": the final ones of the last thirty days too"}, nil, "all")}, "all": "\"true\": the final ones of the last thirty days too"}, nil, "all")},
// Delivery time against the delivery budgets (novox/hq ADR 0282): what probe D16 reads, optional until its
// holder serves it.
{Name: "times", Description: "Delivery time: from a merge to every machine of its walk running the build. " +
"Each class's delivery budget (a leaf module five minutes, a core module ten), the last days' median and " +
"worst, how many within and over, the newest delivery of each class, every delivery over its budget with its " +
"longest phase, and the delivered ones whose time is not known. Given a delivery's id, its time phase by phase.",
Input: schema(map[string]string{"days": "how many days back (default 7)", "id": "one delivery's id: its phases"}, nil),
// Optional until mesh-delivery serves it: its holder lives in the catalogue (design 33 §7).
Optional: true},
{Name: "show", Description: "One delivery or group whole: its delivery plan (what it builds, what each " + {Name: "show", Description: "One delivery or group whole: its delivery plan (what it builds, what each " +
"machine receives, what is not an ordinary send), every transition with when and why, the machine " + "machine receives, what is not an ordinary send), every transition with when and why, the machine " +
"steps of its walk, its group and its order.", "steps of its walk, its group and its order.",
+36
View File
@@ -97,3 +97,39 @@ func TestTheDeliverySeatPromisesRetireHistoryOptionally(t *testing.T) {
t.Fatalf("a holder serving retire-history: %v", err) t.Fatalf("a holder serving retire-history: %v", err)
} }
} }
// The seat says delivery times (novox/hq ADR 0282, issue 382): `times`, over some days or for one delivery. Added
// after the holder shipped, it is optional, so the holder that does not serve it yet still holds the seat, and one
// that does is not refused for a verb the seat lacks.
func TestTheDeliverySeatPromisesTimesOptionally(t *testing.T) {
seat, _ := SeatNamed(DeliverySeat)
var times *Verb
for i := range seat.Serves {
if seat.Serves[i].Name == "times" {
times = &seat.Serves[i]
}
}
if times == nil || !times.Optional {
t.Fatalf("the delivery seat promises %v, times optionally", VerbNames(seat.Serves))
}
props, _ := times.Input["properties"].(map[string]any)
for _, arg := range []string{"days", "id"} {
if _, has := props[arg]; !has {
t.Errorf("times takes no %q", arg)
}
}
var without []string
for _, v := range VerbNames(seat.Serves) {
if v != "times" {
without = append(without, v)
}
}
m := Manifest{Module: "mesh-delivery", Claims: []Claim{{Name: DeliverySeat, Scope: ScopeMesh, Serves: without}}}
if err := CanHold(m, seat); err != nil {
t.Fatalf("a holder without times yet: %v", err)
}
m.Claims[0].Serves = VerbNames(seat.Serves)
if err := CanHold(m, seat); err != nil {
t.Fatalf("a holder serving times: %v", err)
}
}
+3
View File
@@ -226,6 +226,9 @@ func (m Manifest) contributionPlaceholderProblems() []string {
for _, r := range m.Resources { for _, r := range m.Resources {
problems = append(problems, placeholderProblems(m, r)...) problems = append(problems, placeholderProblems(m, r)...)
problems = append(problems, seatPlaceholderProblems(m, r)...) problems = append(problems, seatPlaceholderProblems(m, r)...)
// And every placeholder no pass fills where it stands: a misspelt namespace, a key its
// namespace cannot take, or a field its pass does not read (novox/hq issue 231).
problems = append(problems, unconsumedPlaceholders(m.Module, r)...)
} }
return problems return problems
} }
+49 -15
View File
@@ -134,27 +134,61 @@ func TestThePOSIXEnvironmentSourcedTwiceLeavesPATHAsOnce(t *testing.T) {
// The environment.d rendering, read by the service manager's own generator where this machine has // The environment.d rendering, read by the service manager's own generator where this machine has
// one — the same reader an account's user manager runs, so the PATH it composes is the one asserted. // one — the same reader an account's user manager runs, so the PATH it composes is the one asserted.
//
// The generator also reads the machine's own environment.d (/etc, /run, /usr/lib, /usr/local/lib), and
// no option points it elsewhere: a desktop whose snapd appends its bin directory failed this, on main,
// for a file the mesh never wrote (novox/hq issue 432). So the generator is run twice, once without the
// mesh's file, and what the machine's own files make of PATH and set is held out of the verdict.
//
// A machine without the generator — the build seat's toolchain image holds no systemd — cannot judge
// this and says so: there the rendering is held byte for byte by
// TestTheEnvironmentRendersForTheServiceManagerByteForByte, and this reading only where systemd runs.
func TestTheServiceManagerReadsTheSystemdRenderingAsMeant(t *testing.T) { func TestTheServiceManagerReadsTheSystemdRenderingAsMeant(t *testing.T) {
generator := "/usr/lib/systemd/user-environment-generators/30-systemd-environment-d-generator" generator := "/usr/lib/systemd/user-environment-generators/30-systemd-environment-d-generator"
if _, err := os.Stat(generator); err != nil { if _, err := os.Stat(generator); err != nil {
t.Skip("no environment.d generator on this machine") t.Skip("NOT JUDGED: no environment.d generator on this machine, so the service manager's reading " +
"of the rendering is judged only where systemd runs; the rendering itself is held byte for byte " +
"by TestTheEnvironmentRendersForTheServiceManagerByteForByte")
} }
config := t.TempDir() const base = "/usr/bin:/bin"
if err := os.MkdirAll(filepath.Join(config, "environment.d"), 0o755); err != nil { read := func(conf string) map[string]string {
t.Fatal(err) t.Helper()
config := t.TempDir()
if err := os.MkdirAll(filepath.Join(config, "environment.d"), 0o755); err != nil {
t.Fatal(err)
}
if conf != "" {
if err := os.WriteFile(filepath.Join(config, "environment.d", "50-mesh.conf"), []byte(conf), 0o644); err != nil {
t.Fatal(err)
}
}
cmd := exec.Command(generator)
cmd.Env = []string{"PATH=" + base, "HOME=" + config, "XDG_CONFIG_HOME=" + config}
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("%v\n%s", err, out)
}
vars := map[string]string{}
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
if k, v, ok := strings.Cut(line, "="); ok {
vars[k] = v
}
}
return vars
} }
if err := os.WriteFile(filepath.Join(config, "environment.d", "50-mesh.conf"), []byte(composedSystemd), 0o644); err != nil { machine, read50 := read(""), read(composedSystemd)
t.Fatal(err)
// What the machine's own files do to PATH: nothing, or append to it. One that replaces or prepends
// leaves nothing this test can say about the mesh's file, and says so rather than guess.
added, ok := strings.CutPrefix(machine["PATH"], base)
if machine["PATH"] != "" && !ok {
t.Skipf("NOT JUDGED: this machine's own environment.d sets PATH to %s, not %s with something after it, so "+
"what the mesh's file adds cannot be told apart from it", machine["PATH"], base)
} }
cmd := exec.Command(generator) want := "/home/op/go/bin:/usr/local/go/bin:/home/op/.local/bin:/home/op/bin:/usr/bin:/bin:/opt/agent/bin:/opt/scripts" + added
cmd.Env = []string{"PATH=/usr/bin:/bin", "HOME=" + config, "XDG_CONFIG_HOME=" + config} if read50["PATH"] != want || read50["GOPATH"] != "/home/op/go" {
out, err := cmd.CombinedOutput() t.Fatalf("the service manager read PATH=%s GOPATH=%s, not PATH=%s GOPATH=/home/op/go (the machine's own "+
if err != nil { "files add %q to PATH)", read50["PATH"], read50["GOPATH"], want, added)
t.Fatalf("%v\n%s", err, out)
}
want := "PATH=/home/op/go/bin:/usr/local/go/bin:/home/op/.local/bin:/home/op/bin:/usr/bin:/bin:/opt/agent/bin:/opt/scripts"
if !strings.Contains(string(out), want+"\n") || !strings.Contains(string(out), "GOPATH=/home/op/go\n") {
t.Fatalf("the service manager read\n%s", out)
} }
} }
@@ -4,19 +4,24 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"os" "os"
"path/filepath"
"reflect" "reflect"
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// The catalogue's foundation modules as they are, parsed by the real parser (novox/hq ADR 0100): // The catalogue's foundation modules as they are — in a merge check the catalogue cloned beside it,
// elsewhere the copy captured in testdata/beside (internal/beside, novox/hq issue 432) — parsed by the
// real parser (novox/hq ADR 0100):
// the store and the broker say which of their ports the mesh guards on an adopted node, and the // the store and the broker say which of their ports the mesh guards on an adopted node, and the
// filter module loads its table through a unit of its own whose stop deletes only that table. // filter module loads its table through a unit of its own whose stop deletes only that table.
func catalogueManifest(t *testing.T, module string) Manifest { func catalogueManifest(t *testing.T, module string) Manifest {
t.Helper() t.Helper()
raw, err := os.ReadFile("../../../mesh-catalog/modules/" + module + "/module.json") raw, err := os.ReadFile(filepath.Join(beside.Catalogue(t), module, "module.json"))
if err != nil { if err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err) t.Fatal(err)
} }
m, err := ParseManifest(raw) m, err := ParseManifest(raw)
if err != nil { if err != nil {
@@ -125,8 +130,11 @@ func TestTheForgesPortIsGivenLikeAnyOtherProvidersPort(t *testing.T) {
// seat's holder the answer when more than one module provides it — so a carried copy would be a // seat's holder the answer when more than one module provides it — so a carried copy would be a
// second answer to the same question, free to drift from the first. Asserted gone, not merely // second answer to the same question, free to drift from the first. Asserted gone, not merely
// unused. // unused.
//
// The builder is the build-agent on every machine since novox/hq ADR 0190; this read the retired
// `builder` and, finding no manifest, skipped unseen from then until novox/hq issue 432.
func TestTheBuilderRequiresTheRegistryTheNpmSeatDelivers(t *testing.T) { func TestTheBuilderRequiresTheRegistryTheNpmSeatDelivers(t *testing.T) {
builder := catalogueManifest(t, "builder") builder := catalogueManifest(t, "build-agent")
seat, _ := SeatNamed("npm-package-registry") seat, _ := SeatNamed("npm-package-registry")
var requires bool var requires bool
for _, r := range builder.Requires { for _, r := range builder.Requires {
+94
View File
@@ -1942,6 +1942,37 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, whileStoppedProblems(m, r, hasSchedule(r))...) problems = append(problems, whileStoppedProblems(m, r, hasSchedule(r))...)
} }
for name, own := range m.OwnSecrets { for name, own := range m.OwnSecrets {
// **A family is issued outside the mesh, and lands one file per member** (novox/hq ADR 0283): the mesh
// never makes a member, so a family the mesh may make would be one nothing ever fills.
if IsFamily(name) {
if !memberRest.MatchString(strings.TrimSuffix(FamilyPrefix(name), "-")) {
problems = append(problems, fmt.Sprintf(
"%s declares the secret family %q, whose prefix is not a name", m.Module, name))
}
if own.IssuedBy != IssuedOutside {
problems = append(problems, fmt.Sprintf(
"%s declares the secret family %q without \"issued-by\": %q; the mesh never makes a "+
"member of a family, so only a party outside the mesh can fill one (novox/hq ADR 0283)",
m.Module, name, IssuedOutside))
}
if strings.Count(own.Path, "*") != 1 {
problems = append(problems, fmt.Sprintf(
"%s keeps the secret family %q at %q, which does not hold exactly one *: each member lands "+
"where the * is replaced by its name (novox/hq ADR 0283)", m.Module, name, own.Path))
}
for other := range m.OwnSecrets {
if other != name && !IsFamily(other) {
if rest := strings.TrimPrefix(other, FamilyPrefix(name)); rest != other && memberRest.MatchString(rest) {
problems = append(problems, fmt.Sprintf(
"%s declares the secret %q, which is also a member of its family %q — a name names one",
m.Module, other, name))
}
}
}
} else if strings.Contains(name, "*") {
problems = append(problems, fmt.Sprintf(
"%s declares the secret %q: a * only ends a family's name, as \"<prefix>-*\"", m.Module, name))
}
if !placedOrAbsolute(own.Path) { if !placedOrAbsolute(own.Path) {
problems = append(problems, fmt.Sprintf( problems = append(problems, fmt.Sprintf(
"%s needs %q at %q, which is neither an absolute path nor a placed one", m.Module, name, own.Path)) "%s needs %q at %q, which is neither an absolute path nor a placed one", m.Module, name, own.Path))
@@ -2549,6 +2580,69 @@ func (o OwnSecrets) MarshalJSON() ([]byte, error) {
return json.Marshal(entries) return json.Marshal(entries)
} }
// A secret family (novox/hq ADR 0283): an own secret declared under a name ending in FamilySuffix is one
// member per part the module's settings name — `smb-password-*` holds `smb-password-games`,
// `smb-password-library` — each given by its full name, placed at the family's path with its one `*` replaced
// by what follows the prefix. The mesh never makes a member: a family is issued outside the mesh, and a member
// not given is no file.
const FamilySuffix = "-*"
// memberRest is what may follow a family's prefix: a name's characters.
var memberRest = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// IsFamily says an own secret's declared name is a family's.
func IsFamily(name string) bool { return strings.HasSuffix(name, FamilySuffix) }
// FamilyPrefix is what every member of a family begins with: the declared name without its `*`.
func FamilyPrefix(family string) string { return strings.TrimSuffix(family, "*") }
// Lookup resolves an own secret by the name it is given under: declared by that name, or a member of a family
// (the longest prefix that fits), with the family's path filled for the member. family is the family's
// declared name, or "" for a secret declared by name.
func (o OwnSecrets) Lookup(name string) (s OwnSecret, family string, ok bool) {
if s, ok := o[name]; ok && !IsFamily(name) {
return s, "", true
}
for declared, f := range o {
if !IsFamily(declared) {
continue
}
prefix := FamilyPrefix(declared)
rest := strings.TrimPrefix(name, prefix)
if !strings.HasPrefix(name, prefix) || !memberRest.MatchString(rest) {
continue
}
if family == "" || len(declared) > len(family) {
s, family, ok = OwnSecret{Path: strings.Replace(f.Path, "*", rest, 1), Taken: f.Taken, IssuedBy: f.IssuedBy}, declared, true
}
}
return s, family, ok
}
// Plain is every own secret declared by its own name: what the mesh makes when not given, and places by
// name. A family is not one, and is never made.
func (o OwnSecrets) Plain() OwnSecrets {
out := make(OwnSecrets, len(o))
for name, s := range o {
if !IsFamily(name) {
out[name] = s
}
}
return out
}
// Families is every family's declared name, sorted.
func (o OwnSecrets) Families() []string {
var out []string
for name := range o {
if IsFamily(name) {
out = append(out, name)
}
}
sort.Strings(out)
return out
}
// Paths is each own secret's path by name — the shape every placement and file walk reads. // Paths is each own secret's path by name — the shape every placement and file walk reads.
func (o OwnSecrets) Paths() map[string]string { func (o OwnSecrets) Paths() map[string]string {
out := make(map[string]string, len(o)) out := make(map[string]string, len(o))
+6 -8
View File
@@ -5,6 +5,8 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// A bind mount the module never declared is refused where it is written (novox/hq 04-ISSUES/026, // A bind mount the module never declared is refused where it is written (novox/hq 04-ISSUES/026,
@@ -67,16 +69,12 @@ func TestTheRuntimeSocketIsGrantedByTheCapabilityAndNotOtherwise(t *testing.T) {
} }
} }
// **Every manifest in the catalogue beside this checkout passes**, so the rule is not one the // **Every manifest in the catalogue (internal/beside) passes**, so the rule is not one the catalogue
// catalogue is already breaking. Skipped, aloud, where the catalogue is not there. // is already breaking.
func TestEveryCatalogueManifestDeclaresWhatItMounts(t *testing.T) { func TestEveryCatalogueManifestDeclaresWhatItMounts(t *testing.T) {
root := os.Getenv("MESH_CATALOG") files, _ := filepath.Glob(filepath.Join(beside.Catalogue(t), "*", "module.json"))
if root == "" {
root = "../../../mesh-catalog"
}
files, _ := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if len(files) == 0 { if len(files) == 0 {
t.Skipf("no catalogue at %s (set MESH_CATALOG to a checkout)", root) t.Fatal("no manifests in the catalogue, so this proved nothing")
} }
for _, file := range files { for _, file := range files {
raw, err := os.ReadFile(file) raw, err := os.ReadFile(file)
+10 -7
View File
@@ -7,6 +7,8 @@ import (
"regexp" "regexp"
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// **A manifest holds no subject** (novox/hq design 29 §1). // **A manifest holds no subject** (novox/hq design 29 §1).
@@ -17,10 +19,10 @@ import (
// held by construction is one a later field breaks quietly, with the symptom appearing as a // held by construction is one a later field breaks quietly, with the symptom appearing as a
// permission that does not match a subject rather than as a manifest that was wrong. // permission that does not match a subject rather than as a manifest that was wrong.
func TestNoManifestContainsASubject(t *testing.T) { func TestNoManifestContainsASubject(t *testing.T) {
root := filepath.Join("..", "..", "..", "mesh-catalog", "modules") root := beside.Catalogue(t)
entries, err := os.ReadDir(root) entries, err := os.ReadDir(root)
if err != nil { if err != nil {
t.Skipf("catalogue sibling not present: %v", err) t.Fatal(err)
} }
// Anything in the mesh's own subject space, and anything shaped like a wire address. // Anything in the mesh's own subject space, and anything shaped like a wire address.
@@ -63,7 +65,7 @@ func TestNoManifestContainsASubject(t *testing.T) {
walk(e.Name(), "", m) walk(e.Name(), "", m)
} }
if checked == 0 { if checked == 0 {
t.Skip("no manifests read") t.Fatal("no manifests read, so this proved nothing")
} }
if len(found) > 0 { if len(found) > 0 {
t.Errorf("a manifest names a subject, so reorganising the subject space would mean "+ t.Errorf("a manifest names a subject, so reorganising the subject space would mean "+
@@ -91,13 +93,14 @@ func TestEveryManifestsEventNamesAreLocal(t *testing.T) {
} }
} }
// theCatalogue is every manifest beside this checkout, parsed the way registration parses one. // theCatalogue is every manifest of the catalogue internal/beside finds, parsed the way registration
// parses one.
func theCatalogue(t *testing.T) []Manifest { func theCatalogue(t *testing.T) []Manifest {
t.Helper() t.Helper()
root := filepath.Join("..", "..", "..", "mesh-catalog", "modules") root := beside.Catalogue(t)
entries, err := os.ReadDir(root) entries, err := os.ReadDir(root)
if err != nil { if err != nil {
t.Skipf("catalogue sibling not present: %v", err) t.Fatal(err)
} }
var out []Manifest var out []Manifest
for _, e := range entries { for _, e := range entries {
@@ -115,7 +118,7 @@ func theCatalogue(t *testing.T) []Manifest {
out = append(out, m) out = append(out, m)
} }
if len(out) == 0 { if len(out) == 0 {
t.Skip("no manifests found beside this checkout") t.Fatalf("no manifests under %s, so this proved nothing", root)
} }
return out return out
} }
+10 -7
View File
@@ -2,7 +2,10 @@ package catalogue
import ( import (
"os" "os"
"path/filepath"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// The public issuer is the proxy's own fact (novox/hq ADR 0226), and the folding of it must not // The public issuer is the proxy's own fact (novox/hq ADR 0226), and the folding of it must not
@@ -86,22 +89,22 @@ func TestRouteProxyKeepsItsPublicAccountDirectory(t *testing.T) {
} }
} }
// The modules ADR 0226 retired stay retired, and nothing asks for what they provided. // The modules ADR 0226 retired stay retired, and nothing asks for what they provided. A module is
// in the catalogue when its manifest is: a directory left behind with no manifest in it (a build's
// leftovers, untracked by git) is not a module, and failed this on a desktop (novox/hq issue 432).
func TestTheRetiredNetworkingModulesAreNotInTheCatalogue(t *testing.T) { func TestTheRetiredNetworkingModulesAreNotInTheCatalogue(t *testing.T) {
if _, err := os.Stat("../../../mesh-catalog/modules"); err != nil { modules := beside.Catalogue(t)
t.Skipf("the catalogue is not beside this checkout: %v", err)
}
for _, gone := range []string{"public-acme", "dhcpcd", "cloudflare-dns"} { for _, gone := range []string{"public-acme", "dhcpcd", "cloudflare-dns"} {
if _, err := os.Stat("../../../mesh-catalog/modules/" + gone); err == nil { if _, err := os.Stat(filepath.Join(modules, gone, "module.json")); err == nil {
t.Errorf("%s is in the catalogue again; ADR 0226 retired it", gone) t.Errorf("%s is in the catalogue again; ADR 0226 retired it", gone)
} }
} }
entries, err := os.ReadDir("../../../mesh-catalog/modules") entries, err := os.ReadDir(modules)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
for _, e := range entries { for _, e := range entries {
raw, err := os.ReadFile("../../../mesh-catalog/modules/" + e.Name() + "/module.json") raw, err := os.ReadFile(filepath.Join(modules, e.Name(), "module.json"))
if err != nil { if err != nil {
continue continue
} }
+3 -1
View File
@@ -161,7 +161,9 @@ func TestTheCataloguesBarRendersEveryExampleOfTheShape(t *testing.T) {
holder := catalogueManifest(t, "i3status-rust") holder := catalogueManifest(t, "i3status-rust")
s, _ := SeatNamed(BarSeat) s, _ := SeatNamed(BarSeat)
if !placesKind(holder, s, BarKindBlock) { if !placesKind(holder, s, BarKindBlock) {
t.Skip("the catalogue beside this checkout has a bar that places no blocks yet") // It places them since the catalogue's bar took the seat; a skip here hid a bar that stopped
// (novox/hq issue 432).
t.Fatal("the catalogue's bar places no blocks, so none of the shape's examples would render")
} }
tmpl, err := holderTemplate(holder, s, BarKindBlock) tmpl, err := holderTemplate(holder, s, BarKindBlock)
if err != nil { if err != nil {
+7 -5
View File
@@ -7,6 +7,8 @@ import (
"regexp" "regexp"
"strings" "strings"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// Defends novox/hq ADR 0110: a seat is a module assignment from a closed set. // Defends novox/hq ADR 0110: a seat is a module assignment from a closed set.
@@ -193,13 +195,13 @@ func TestAClaimThatIsMalformedIsRefusedOnceForThat(t *testing.T) {
// Every module in use claims a seat in the set, so closing it refuses nothing that runs. // Every module in use claims a seat in the set, so closing it refuses nothing that runs.
// //
// Read from the catalogue beside this checkout and from this repository's own manifest, the two // Read from the catalogue (internal/beside) and from this repository's own manifest, the two places a
// places a manifest lives (ADR 0069). The private-network module's manifest is composed in code, // manifest lives (ADR 0069). The private-network module's manifest is composed in code, and its claim
// and its claim is checked where it is composed. // is checked where it is composed.
func TestEveryManifestInUseClaimsASeatTheMeshDefines(t *testing.T) { func TestEveryManifestInUseClaimsASeatTheMeshDefines(t *testing.T) {
paths, _ := filepath.Glob("../../../mesh-catalog/modules/*/module.json") paths, _ := filepath.Glob(filepath.Join(beside.Catalogue(t), "*", "module.json"))
if len(paths) == 0 { if len(paths) == 0 {
t.Skip("the catalogue is not beside this checkout") t.Fatal("no manifests in the catalogue, so this proved nothing")
} }
paths = append(paths, "../../module.json") paths = append(paths, "../../module.json")
var checked int var checked int
+100
View File
@@ -0,0 +1,100 @@
package catalogue
import (
"strings"
"testing"
)
// A secret family (novox/hq ADR 0283): one own secret per part the settings name, issued outside the mesh, each
// given by its full name, never made by the mesh.
func familyManifest(t *testing.T, ownSecrets string) (Manifest, error) {
t.Helper()
return ParseManifest([]byte(`{"module":"mounts","version":"1","own-secrets":{` + ownSecrets + `}}`))
}
func TestAFamilyIsDeclaredIssuedOutsideWithOneStar(t *testing.T) {
m, err := familyManifest(t, `"smb-password-*":{"path":"/var/lib/mounts/smb-password-*.secret","issued-by":"outside"}`)
if err != nil {
t.Fatalf("a well-formed family was refused: %v", err)
}
if got := m.OwnSecrets.Families(); len(got) != 1 || got[0] != "smb-password-*" {
t.Fatalf("families: %v", got)
}
if len(m.OwnSecrets.Plain()) != 0 {
t.Fatalf("a family counted as a secret declared by name: %v", m.OwnSecrets.Plain())
}
}
func TestAMalformedFamilyIsRefused(t *testing.T) {
for name, c := range map[string]struct{ own, says string }{
"made by the mesh": {`"smb-password-*":{"path":"/s/smb-password-*.secret","taken":"at-start"}`, "issued-by"},
"no star in its path": {`"smb-password-*":{"path":"/s/smb-password.secret","issued-by":"outside"}`,
"exactly one *"},
"two stars in its path": {`"smb-password-*":{"path":"/s/*/smb-password-*.secret","issued-by":"outside"}`,
"exactly one *"},
"a star inside a name": {`"smb*password":{"path":"/s/x","issued-by":"outside"}`, "only ends a family"},
"a name that is also a member": {`"smb-password-*":{"path":"/s/p-*","issued-by":"outside"},
"smb-password-games":{"path":"/s/games","issued-by":"outside"}`, "also a member"},
} {
if _, err := familyManifest(t, c.own); err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: %v; want a refusal saying %q", name, err, c.says)
}
}
}
func TestAMemberIsFoundByItsFullNameAndLandsWhereTheStarIs(t *testing.T) {
o := OwnSecrets{
"smb-password-*": {Path: "/s/smb-password-*.secret", IssuedBy: IssuedOutside},
"smb-password-big-*": {Path: "/big/*.secret", IssuedBy: IssuedOutside},
"token": {Path: "/s/token", IssuedBy: IssuedOutside},
}
s, family, ok := o.Lookup("smb-password-games")
if !ok || family != "smb-password-*" || s.Path != "/s/smb-password-games.secret" || s.IssuedBy != IssuedOutside {
t.Fatalf("a member: %+v %q %v", s, family, ok)
}
if s, family, ok := o.Lookup("smb-password-big-one"); !ok || family != "smb-password-big-*" || s.Path != "/big/one.secret" {
t.Fatalf("the longest family that fits: %+v %q %v", s, family, ok)
}
if s, family, ok := o.Lookup("token"); !ok || family != "" || s.Path != "/s/token" {
t.Fatalf("a secret declared by name: %+v %q %v", s, family, ok)
}
for _, name := range []string{"smb-password-*", "smb-password-", "smb-password-../etc", "smb-password-a/b",
"smb-password-a.b", "smb-password-Games", "smb-password--x", "other"} {
if _, _, ok := o.Lookup(name); ok {
t.Errorf("%q was found as a secret", name)
}
}
}
// A member given is one file at its path; one not given is nothing, and the machine still composes — the mesh never
// makes a member.
func TestAMemberGivenIsPlacedAndOneNotGivenIsNothing(t *testing.T) {
m, err := familyManifest(t, `"smb-password-*":{"path":"/var/lib/mounts/smb-password-*.secret","issued-by":"outside"}`)
if err != nil {
t.Fatal(err)
}
r := Resolution{Node: "workstation", Modules: []Manifest{m}}
out, err := r.Declaration(Rendering{Needed: map[string]map[string]string{"mounts": {"smb-password-games": "sealed"}}})
if err != nil {
t.Fatalf("a module with one member given did not compose: %v", err)
}
var paths []string
for _, res := range out {
if res["sealed"] != nil {
paths = append(paths, res["path"].(string))
}
}
if len(paths) != 1 || paths[0] != "/var/lib/mounts/smb-password-games.secret" {
t.Fatalf("placed: %v", paths)
}
out, err = r.Declaration(Rendering{})
if err != nil {
t.Fatalf("a module with no member given did not compose: %v", err)
}
for _, res := range out {
if res["sealed"] != nil {
t.Fatalf("a member nobody gave was placed: %v", res)
}
}
}
+38
View File
@@ -50,3 +50,41 @@ func TestAContainerReadingAnOwnSecretIsRestartedWhenItChanges(t *testing.T) {
t.Fatalf("a container that reads no secret was given one to restart on: %v", by["agent.other"]["restart-on"]) t.Fatalf("a container that reads no secret was given one to restart on: %v", by["agent.other"]["restart-on"])
} }
} }
// A member of a secret family is an own secret too (novox/hq issue 405, ADR 0283 decision 6): a container that
// reads a member given is restarted when it changes, as for a secret declared by name. A member not given is no
// file, so nothing is restarted on it.
func TestAContainerReadingAFamilyMemberIsRestartedWhenItChanges(t *testing.T) {
m := Manifest{Module: "mounts", Version: "1",
OwnSecrets: OwnSecrets{"smb-password-*": {Path: "/var/lib/mesh/mounts/smb-password-*.secret", IssuedBy: IssuedOutside}},
Resources: []map[string]any{
{"id": "games", "type": "container", "name": "mounts-games", "network": "host",
"image": "registry.example/mounts@sha256:" + strings.Repeat("a", 64),
"volumes": []any{"/var/lib/mesh/mounts/smb-password-games.secret:/run/password:ro"}},
{"id": "library", "type": "container", "name": "mounts-library", "network": "host",
"image": "registry.example/mounts@sha256:" + strings.Repeat("a", 64),
"volumes": []any{"/var/lib/mesh/mounts/smb-password-library.secret:/run/password:ro"}},
}}
got, err := Resolve(shelf(m), []string{m.Module},
Node{Name: "anchor", At: "10.0.0.1", Capabilities: map[string]bool{"container-runtime": true}}, World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(Rendering{Needed: map[string]map[string]string{"mounts": {"smb-password-games": "SEALED"}}})
if err != nil {
t.Fatal(err)
}
by := map[string]map[string]any{}
for _, r := range out {
by[r["id"].(string)] = r
}
if want := []any{"mounts.needs-smb-password-games"}; !reflect.DeepEqual(by["mounts.games"]["restart-on"], want) {
t.Fatalf("a container reading a member given is not restarted on it: %v", by["mounts.games"]["restart-on"])
}
if _, placed := by["mounts.needs-smb-password-games"]; !placed {
t.Fatalf("the member given is not placed, so a restart-on names nothing: %v", out)
}
if _, has := by["mounts.library"]["restart-on"]; has {
t.Fatalf("a container reading a member not given was given a restart-on naming nothing: %v", by["mounts.library"]["restart-on"])
}
}
+1 -1
View File
@@ -56,7 +56,7 @@ func secretsUsed(content string) []string {
// name would end that, to save writing a file. // name would end that, to save writing a file.
func sealedFor(m Manifest, needs []Needed, with Rendering) (map[string]string, error) { func sealedFor(m Manifest, needs []Needed, with Rendering) (map[string]string, error) {
sealed := map[string]string{} sealed := map[string]string{}
for name := range m.OwnSecrets { for name := range m.OwnSecrets.Plain() {
if value := with.Needed[m.Module][name]; value != "" { if value := with.Needed[m.Module][name]; value != "" {
sealed[name] = value sealed[name] = value
} }
+5 -2
View File
@@ -2,7 +2,10 @@ package catalogue
import ( import (
"os" "os"
"path/filepath"
"testing" "testing"
"github.com/novox/mesh-controller/internal/beside"
) )
// **The showcase module is parsed by the real parser, in the real test suite.** // **The showcase module is parsed by the real parser, in the real test suite.**
@@ -11,9 +14,9 @@ import (
// Written as a test rather than a script so it runs whenever anything about manifests changes — // Written as a test rather than a script so it runs whenever anything about manifests changes —
// which is exactly when a module using all of it would quietly stop being valid. // which is exactly when a module using all of it would quietly stop being valid.
func TestTheShowcaseModuleIsAValidManifest(t *testing.T) { func TestTheShowcaseModuleIsAValidManifest(t *testing.T) {
raw, err := os.ReadFile("../../../mesh-catalog/modules/showcase/module.json") raw, err := os.ReadFile(filepath.Join(beside.Catalogue(t), "showcase", "module.json"))
if err != nil { if err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err) t.Fatal(err)
} }
m, err := ParseManifest(raw) m, err := ParseManifest(raw)
if err != nil { if err != nil {
@@ -0,0 +1,172 @@
package catalogue
import (
"fmt"
"regexp"
"strings"
)
// A placeholder no pass consumes is refused, never written out as text (novox/hq issue 231).
//
// Each namespace is filled by its own pass with its own pattern, in the fields that pass reads. A
// word in that shape that no pattern matched — `${machnie:address}`, `${shel:zsh:first}`, a setting
// key no definition could declare such as `${setting:Undeclared}` — was left in the file as it was
// written and reached a machine as a value: the predecessor's failure the namespaced placeholders
// were meant to end (novox/hq ADR 0164). So the definition is swept, field by field, against the
// same table of what each pass fills where.
//
// **The definition, never the values.** Swept as the manifest wrote it, at the catalogue check and
// again at composition before any pass has run: what an operator's setting, a binding or a merged
// JSON setting puts into a file is its own software's text — `${env:HOME}` for Log4j, `${timeout:30}`
// for Spring — and refusing it there would refuse something its author never wrote, on a machine the
// check had passed (novox/hq issue 231, review of mesh-controller #211).
//
// What is refused:
// - a placeholder of a namespace the mesh knows, in a field that namespace's pass does not read: it
// would reach the machine as the same text;
// - any other `${<known namespace>:`, case-insensitively, unless a shell's parameter operator follows
// its colon — `${Machine:address}`, `${machine:.address}`, `${machine: address}`, an unclosed
// `${machine:address` — because no pass's pattern takes it;
// - a namespace-shaped placeholder of a word the mesh does not know: a lower-case word, a colon, and
// a key that begins with a letter or a digit and holds no space, brace or `$`.
//
// **The shell's own syntax is not that shape, and passes.** `${NAME:-…}` has an upper-case word,
// `${(%):-…}` and `${1:-.}` begin with no letter, and a lower-case variable with an operator after its
// colon — `${count:-}`, `${trial:+…}`, `${state:=…}` — has no key that begins with a letter or a digit.
// And an operator — `:-`, `:=`, `:+`, `:?` — after a word the mesh also uses is the shell's too:
// `${PORT:-8080}`, `${SHELL:-/bin/sh}`, `${SECRET:?unset}` and `${dir:-/tmp}` are among the commonest
// lines of a script or an env file, and pass whatever the word's case.
// What the shape does catch is a zsh modifier (`${path:t}`) or a substring (`${where:0:12}`) in a
// resource's own text: shell code of that kind belongs in the module's contributed shell code, which
// is not a resource and is never swept (novox/hq ADR 0204).
// filler is one namespace's pass: the patterns it fills with, and the fields it reads them in, by
// resource type ("*" for any type).
type filler struct {
namespace string
patterns []*regexp.Regexp
fields map[string][]string
// why, when set, is the rule that keeps the namespace out of every other field, said with a
// refusal of one written there.
why string
}
// fillers is the table of what each pass fills where — read from the passes themselves: settingInto,
// dirInto, accessInto, intoFile (whose ${secret:…} the node-engine fills), boundInto, portInto,
// seatInto, machineInto and contributionsInto. A pass that comes to read another field adds it here,
// or the sweep refuses the placeholder it would have filled.
var fillers = []filler{
{namespace: "setting", patterns: []*regexp.Regexp{settingRef}, fields: map[string][]string{"file": {"content"}, "service": {"unit"}}},
{namespace: "dir", patterns: []*regexp.Regexp{dirRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
{namespace: "access", patterns: []*regexp.Regexp{accessRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
{namespace: "secret", patterns: []*regexp.Regexp{placeholder}, fields: map[string][]string{"file": {"content"}},
why: "a secret is never filled into an environment variable or any other field: put it in a file the " +
"module declares and mount that (novox/hq ADR 0086)"},
{namespace: "bound", patterns: []*regexp.Regexp{bound}, fields: map[string][]string{"file": {"content"}}},
{namespace: "port", patterns: []*regexp.Regexp{ofPort}, fields: map[string][]string{"file": {"content"}, "container": {"env"}, "process": {"env"}}},
{namespace: "seat", patterns: []*regexp.Regexp{ofSeat, ofSeatReach}, fields: map[string][]string{"file": {"content"}, "container": {"env"}, "process": {"env"}}},
{namespace: "machine", patterns: []*regexp.Regexp{ofMachine}, fields: map[string][]string{"*": {"path", "owner", "content", "name", "user", "root", "home"}}},
// Placed in a file's content by their holders, and judged there by their own rules
// (placeholderProblems, seatPlaceholderProblems).
{namespace: "environment", patterns: []*regexp.Regexp{ofEnvironment}, fields: map[string][]string{"*": {"content"}}},
{namespace: "shell", patterns: []*regexp.Regexp{ofShell}, fields: map[string][]string{"*": {"content"}}},
{namespace: "contribution", patterns: []*regexp.Regexp{ofContribution}, fields: map[string][]string{"*": {"content"}}},
// Filled in what a provider serves (consumer_into_serves.go), never in a resource.
{namespace: "consumer", patterns: []*regexp.Regexp{consumerFact}},
}
// reads is whether this pass fills a field of a resource of this type.
func (f filler) reads(kind, field string) bool {
return oneOf(f.fields[kind], field) || oneOf(f.fields["*"], field)
}
// ofKnownNamespace is any `${<known namespace>:` and what follows it up to its brace or the end of
// its line, whatever its case, unless what follows the colon is a shell's parameter operator (`-`,
// `=`, `+`, `?`): what remains of one once every pass's pattern is set aside is a misspelling.
var ofKnownNamespace = regexp.MustCompile(`(?im)\$\{(?:` + strings.Join(namespacesOf(fillers), "|") +
`):(?:[^-=+?}\n][^}\n]*\}?|\}|$)`)
// namespaceShaped is a placeholder in the mesh's shape: a lower-case word, a colon, and a key that
// begins with a letter or a digit and holds no space, brace or `$`.
var namespaceShaped = regexp.MustCompile(`\$\{[a-z][a-z0-9_-]*:[A-Za-z0-9][^\s{}$]*\}`)
func namespacesOf(fs []filler) []string {
out := make([]string, len(fs))
for i, f := range fs {
out[i] = f.namespace
}
return out
}
// unconsumedPlaceholders is every placeholder in one resource of a definition that no pass fills
// where it stands, each named with the module, the resource, the field and the token. The same
// function at the catalogue check and at composition, over the resource as the manifest wrote it.
func unconsumedPlaceholders(module string, r map[string]any) []string {
kind := fmt.Sprint(r["type"])
var problems []string
var sweep func(top, field string, v any)
sweep = func(top, field string, v any) {
switch v := v.(type) {
case string:
rest := v
for _, f := range fillers {
for _, p := range f.patterns {
if !f.reads(kind, top) {
for _, token := range p.FindAllString(rest, -1) {
problems = append(problems, fmt.Sprintf(
"%s's resource %v holds %s in its %s, and ${%s:…} is not filled in this field: "+
"it would reach the machine as that text (novox/hq issue 231)%s",
module, r["id"], token, field, f.namespace, whereFilled(f)))
}
}
rest = p.ReplaceAllString(rest, "")
}
}
for _, token := range ofKnownNamespace.FindAllString(rest, -1) {
problems = append(problems, fmt.Sprintf(
"%s's resource %v holds %s in its %s, which is no placeholder the mesh fills: a "+
"misspelt key, or a namespace in the wrong case, would reach the machine as that "+
"text (novox/hq issue 231)", module, r["id"], token, field))
}
rest = ofKnownNamespace.ReplaceAllString(rest, "")
for _, token := range namespaceShaped.FindAllString(rest, -1) {
problems = append(problems, fmt.Sprintf(
"%s's resource %v holds %s in its %s, and no pass of the mesh fills it: a misspelt "+
"placeholder would reach the machine as that text (novox/hq issue 231). The mesh "+
"fills ${%s:…}; the shell's own syntax belongs in the module's shell code (ADR 0204)",
module, r["id"], token, field, strings.Join(namespacesOf(fillers), ":…}, ${")))
}
case []any:
for i, e := range v {
sweep(top, fmt.Sprintf("%s[%d]", field, i), e)
}
case map[string]any:
for _, k := range sortedKeys(v) {
sweep(top, field+"."+k, v[k])
}
}
}
for _, k := range sortedKeys(r) {
sweep(k, k, r[k])
}
return problems
}
// whereFilled says where a namespace's pass does fill, for a refusal of one written elsewhere.
func whereFilled(f filler) string {
if f.why != "" {
return ". " + strings.ToUpper(f.why[:1]) + f.why[1:]
}
if len(f.fields) == 0 {
return "; it is filled only in what a provider serves"
}
var where []string
for _, kind := range sortedKeys(f.fields) {
of := "a " + kind + "'s"
if kind == "*" {
of = "any resource's"
}
where = append(where, of+" "+strings.Join(f.fields[kind], ", "))
}
return "; it is filled in " + strings.Join(where, "; ")
}
@@ -0,0 +1,159 @@
package catalogue
import (
"strings"
"testing"
)
// novox/hq issue 231 — a misspelled placeholder is written out as text.
//
// The four placeholders of the issue, in one file: two misspelled namespaces, a setting key no
// definition could declare, and the shell's own syntax. The first three are refused by name, at the
// catalogue check and at composition; the fourth reaches the file as written.
const (
misspelledShell = "${shel:zsh:first}"
undeclaredSetting = "${setting:Undeclared}"
misspelledMachine = "${machnie:address}"
shellsOwn = "${XDG_CACHE_HOME:-x}"
// The shell's operators after a word the mesh also uses, in any case: the shell's, and passed.
shellsOperators = "${PORT:-8080} ${SHELL:-/bin/sh} ${SECRET:?unset} ${dir:-/tmp}"
)
// The catalogue check — the strict parse registration runs too — refuses each by name, with the
// module and the field it stands in, and says nothing about the shell's own syntax.
func TestAMisspelledPlaceholderIsRefusedAtTheCheck(t *testing.T) {
raw := `{"module":"speller","version":"1","resources":[{"id":"rc","type":"file","path":"/etc/speller.rc",` +
`"content":"a=` + misspelledShell + `\nb=` + undeclaredSetting + `\nc=` + misspelledMachine +
`\nd=` + shellsOwn + `\n"}]}`
_, err := ParseManifest([]byte(raw))
if err == nil {
t.Fatal("a file holding three placeholders no pass consumes was accepted")
}
for _, token := range []string{misspelledShell, undeclaredSetting, misspelledMachine} {
if !strings.Contains(err.Error(), "speller's resource rc holds "+token+" in its content") {
t.Errorf("the refusal does not name %s with its module and field: %v", token, err)
}
}
if strings.Contains(err.Error(), "XDG_CACHE_HOME") {
t.Errorf("the shell's own syntax was refused: %v", err)
}
// A namespace the mesh knows, misspelt in any way its own pattern does not take, and the same
// namespace in a field its pass does not read: refused at the check as at composition.
for _, c := range []struct{ resource, token, field string }{
{`{"id":"rc","type":"file","path":"/etc/rc","content":"${Machine:address}"}`, "${Machine:address}", "content"},
{`{"id":"rc","type":"file","path":"/etc/rc","content":"${machine:.address}"}`, "${machine:.address}", "content"},
{`{"id":"rc","type":"file","path":"/etc/rc","content":"${machine: address}"}`, "${machine: address}", "content"},
{`{"id":"rc","type":"file","path":"/etc/rc","content":"${Dir:x}"}`, "${Dir:x}", "content"},
{`{"id":"rc","type":"file","path":"/etc/rc","content":"a=${machine:address\nb=1"}`, "${machine:address", "content"},
{`{"id":"rc","type":"process","name":"speller","env":{"NAME":"${machine:name}"}}`, "${machine:name}", "env.NAME"},
} {
raw := `{"module":"speller","version":"1","resources":[` + c.resource + `]}`
_, err := ParseManifest([]byte(raw))
if err == nil || !strings.Contains(err.Error(), "speller's resource rc holds "+c.token+" in its "+c.field) {
t.Errorf("%s in its %s was accepted at the check, or not named: %v", c.token, c.field, err)
}
}
// And the shell's syntax alone, beside placeholders every pass knows, is accepted — as is the
// shape a lower-case shell variable takes with an operator after its colon.
raw = `{"module":"speller","version":"1","resources":[{"id":"rc","type":"file","path":"${machine:account-home}/.rc",` +
`"content":"` + shellsOwn + ` ${(%):-%n} ${1:-.} ${count:-} ${trial:+ on trial} ${machine:address} ` +
shellsOperators + `\n"},` +
`{"id":"server","type":"container","name":"server","env":{"PORT":"${PORT:-80}"}}]}`
if _, err := ParseManifest([]byte(raw)); err != nil {
t.Fatalf("the shell's own syntax was refused: %v", err)
}
}
// Composition refuses the same placeholders in the same words — a manifest the store already holds
// was never parsed by this binary — and a namespace the mesh knows, written in a field its pass does
// not read, is refused there too, because it would reach the machine as the same literal text.
func TestAMisspelledPlaceholderIsRefusedAtComposition(t *testing.T) {
for _, c := range []struct {
field string
r map[string]any
token string
}{
{"content", map[string]any{"id": "rc", "type": "file", "path": "/etc/speller.rc", "content": "a=" + misspelledShell + "\n"}, misspelledShell},
{"content", map[string]any{"id": "rc", "type": "file", "path": "/etc/speller.rc", "content": "b=" + undeclaredSetting + "\n"}, undeclaredSetting},
{"content", map[string]any{"id": "rc", "type": "file", "path": "/etc/speller.rc", "content": "c=" + misspelledMachine + "\n"}, misspelledMachine},
{"env.NAME", map[string]any{"id": "rc", "type": "process", "name": "speller", "env": map[string]any{"NAME": "${machine:name}"}}, "${machine:name}"},
{"content", map[string]any{"id": "rc", "type": "file", "path": "/etc/speller.rc", "content": "${Machine:address}"}, "${Machine:address}"},
{"content", map[string]any{"id": "rc", "type": "file", "path": "/etc/speller.rc", "content": "${machine:.address}"}, "${machine:.address}"},
{"content", map[string]any{"id": "rc", "type": "file", "path": "/etc/speller.rc", "content": "${machine: address}"}, "${machine: address}"},
{"content", map[string]any{"id": "rc", "type": "file", "path": "/etc/speller.rc", "content": "${Dir:x}"}, "${Dir:x}"},
{"content", map[string]any{"id": "rc", "type": "file", "path": "/etc/speller.rc", "content": "a=${machine:address\nb=1"}, "${machine:address"},
} {
m := Manifest{Module: "speller", Version: "1", Resources: []map[string]any{c.r}}
r := Resolution{Node: "workstation", Account: "op", Modules: []Manifest{m}}
_, err := r.Declaration(Rendering{})
if err == nil || !strings.Contains(err.Error(), "speller's resource rc holds "+c.token+" in its "+c.field) {
t.Errorf("%s in its %s was composed rather than refused by name: %v", c.token, c.field, err)
}
}
m := Manifest{Module: "speller", Version: "1", Resources: []map[string]any{
{"id": "rc", "type": "file", "path": "/etc/speller.rc", "content": "d=" + shellsOwn + " " + shellsOperators + "\n"},
{"id": "server", "type": "process", "name": "server", "env": map[string]any{"PORT": "${PORT:-80}"}},
}}
out, err := Resolution{Node: "workstation", Account: "op", Modules: []Manifest{m}}.Declaration(Rendering{})
if err != nil {
t.Fatalf("the shell's own syntax was refused: %v", err)
}
if got := contentOf(t, out, "speller.rc"); got != "d="+shellsOwn+" "+shellsOperators+"\n" {
t.Fatalf("the shell's own syntax did not pass through as written: %q", got)
}
}
// contentOf is the content of the composed resource with this id, failing when it was not composed.
func contentOf(t *testing.T, out []map[string]any, id string) string {
t.Helper()
for _, res := range out {
if res["id"] == id {
return plainly(res["content"])
}
}
t.Fatalf("%s was not composed: %v", id, out)
return ""
}
// What a value puts into a file is its software's text, not the definition's, and is never swept
// (review of mesh-controller #211): an operator's setting holding `${labels:instance}` filled through
// ${setting:…}, and a JSON setting `${level:upper}` merged into a mergeable file, reach the machine as
// set — software that templates its own configuration (Log4j's `${env:…}`, Spring's `${timeout:30}`)
// is configured exactly this way.
func TestAValueHoldingAPlaceholderShapeComposesUnchanged(t *testing.T) {
m := Manifest{Module: "templater", Version: "1", Resources: []map[string]any{
{"id": "conf", "type": "file", "path": "/etc/templater.conf", "content": "template=${setting:template}\n"},
{"id": "json", "type": "file", "path": "/etc/templater.json", "merge": MergeJSON, "content": `{"fmt":"plain"}`},
}}
settings := SettingsBy{"templater": {{From: "the operator", Values: map[string]any{
"template": "${labels:instance}", "fmt": "${level:upper}",
}}}}
out, err := Resolution{Node: "workstation", Account: "op", Modules: []Manifest{m}}.Declaration(Rendering{Settings: settings})
if err != nil {
t.Fatalf("a value holding a placeholder's shape was refused as the definition's: %v", err)
}
if got := contentOf(t, out, "templater.conf"); got != "template=${labels:instance}\n" {
t.Errorf("the setting did not reach the file as set: %q", got)
}
if got := contentOf(t, out, "templater.json"); !strings.Contains(got, `${level:upper}`) {
t.Errorf("the merged setting did not reach the file as set: %q", got)
}
}
// Contributed shell code is the shell's, and no pass reads it (novox/hq ADR 0204): zsh's own
// `${path:t}` is namespace-shaped, and reaches the holder's file untouched.
func TestContributedShellCodeIsNotSwept(t *testing.T) {
modules := []Manifest{zshHolder(), {Module: "modifier", Shell: []ShellCode{
{For: "zsh", Slot: "normal", Code: "echo ${path:t} ${shel:zsh:first}"},
}}}
out, err := Resolution{Node: "workstation", Account: "op", Modules: modules}.Declaration(Rendering{})
if err != nil {
t.Fatalf("contributed shell code was swept: %v", err)
}
if got := contentOf(t, out, "zsh.zshrc"); !strings.Contains(got, "echo ${path:t} ${shel:zsh:first}") {
t.Fatalf("the contributed code did not reach the holder's file as written: %q", got)
}
}
+7 -5
View File
@@ -164,8 +164,9 @@ var ControllerVerbs = []Verb{
Input: schema(map[string]string{"plan": "the walk's id", "why": "why", "by": "who stopped the delivery"}, Input: schema(map[string]string{"plan": "the walk's id", "why": "why", "by": "who stopped the delivery"},
[]string{"plan", "why"})}, []string{"plan", "why"})},
{Name: "delivery-walks", Description: "The walks the controller keeps (novox/hq ADR 0239): every open one and the " + {Name: "delivery-walks", Description: "The walks the controller keeps (novox/hq ADR 0239): every open one and the " +
"last ended ones, each whole — its tiers, each module's state, first machines and gate — and whether the " + "last ended ones, each whole — its tiers, each module's state, first machines and first-node gate with its readings, and its " +
"delivery seat has a holder on record. Given a plan, that one.", "phases from its merge to every machine running it (novox/hq ADR 0282) — and whether the delivery seat has a " +
"holder on record. Given a plan, that one.",
Input: schema(map[string]string{"plan": "one walk's id", "limit": "how many ended walks beside the open ones (default 50)"}, Input: schema(map[string]string{"plan": "one walk's id", "limit": "how many ended walks beside the open ones (default 50)"},
nil)}, nil)},
{Name: "plan", Description: "What one machine would run, and why: the declaration the mesh would send it — " + {Name: "plan", Description: "What one machine would run, and why: the declaration the mesh would send it — " +
@@ -375,10 +376,11 @@ var ControllerVerbs = []Verb{
"recorded — who, why and the cause of each, and which causes repeat: each repeat is a healer the mesh lacks.", "recorded — who, why and the cause of each, and which causes repeat: each repeat is a healer the mesh lacks.",
Input: schema(map[string]string{"days": "how many days back (default 14)"}, nil)}, Input: schema(map[string]string{"days": "how many days back (default 14)"}, nil)},
{Name: "durations", Description: "How long things take, as the controller measured them: a send to its machine's " + {Name: "durations", Description: "How long things take, as the controller measured them: a send to its machine's " +
"report (apply), a machine's silence between words (heartbeat-gap), a plan's tier, a build — per machine, " + "report (apply), a machine's silence between words (heartbeat-gap), a plan's tier, a build, and each phase of an " +
"repository or module, with median, p90 and max. What the core's bounds are set from (novox/hq to-be 45 Phase 0).", "ended walk per class (walk-phase, novox/hq ADR 0282) — per machine, repository, module or class/phase, with " +
"median, p90 and max. What the core's bounds are set from (novox/hq to-be 45 Phase 0).",
Input: schema(map[string]string{ Input: schema(map[string]string{
"kind": "one kind: apply, heartbeat-gap, plan-tier or build; every kind when absent", "kind": "one kind: apply, heartbeat-gap, plan-tier, build or walk-phase; every kind when absent",
"days": "how many days back (default 14)", "days": "how many days back (default 14)",
}, nil)}, }, nil)},
// What is wrong, and the self-check (novox/hq to-be 45 §2, §4). // What is wrong, and the self-check (novox/hq to-be 45 §2, §4).
+18 -1
View File
@@ -152,7 +152,11 @@ type Condition struct {
// reopened after a send was there at the send unless the send fell in one of its gaps. // reopened after a send was there at the send unless the send fell in one of its gaps.
First time.Time `json:"first,omitzero"` First time.Time `json:"first,omitzero"`
Gaps []Gap `json:"gaps,omitempty"` Gaps []Gap `json:"gaps,omitempty"`
// Observations is how many times it was observed since raised. // Observations is how many times it was observed since raised: a look that sees it, for a source
// that looks at the present; for one that reads a record of what happened (Observation.Happened),
// how many times it happened, and LastObserved when it last did (novox/hq issue 402). That count is a
// running total since raised, never lowered: a consumer's dead letters count each message given up on
// while the condition is open, not the number held now (novox/hq issue 440).
Observations int `json:"observations"` Observations int `json:"observations"`
// Count is how many times it has been raised, a reopening within ReopenWithin counted. // Count is how many times it has been raised, a reopening within ReopenWithin counted.
Count int `json:"count"` Count int `json:"count"`
@@ -208,6 +212,15 @@ type Observation struct {
Said string Said string
Source string Source string
Resolver string Resolver string
// Happened is set by a source that reads a record of what happened rather than looking at the
// present — a bus advisory, remembered for an hour and read again on every look — to when it last
// happened; Times is how many times the record counts it. The condition then counts what happened,
// not the looks (novox/hq issue 402): one refusal looked at every half minute read as "observed 15
// time(s)", with a line of evidence every 30 seconds, and was taken for a refusal going on. A look
// that brings nothing newer than the condition's last observation adds no observation and no
// evidence. Zero for a source that looks at the present, whose every look is an observation.
Happened time.Time
Times int
// Confirm says a single look can be wrong about this finding — a question over the network that // Confirm says a single look can be wrong about this finding — a question over the network that
// went unanswered, a time measured once on a loaded machine. The keeper does not read it: the // went unanswered, a time measured once on a loaded machine. The keeper does not read it: the
// source that looks again does, and raises it only when the next look sees it too, or while it is // source that looks again does, and raises it only when the next look sees it too, or while it is
@@ -265,6 +278,10 @@ func (o Observation) check() error {
return fmt.Errorf("the condition %s says nothing", o.Key()) return fmt.Errorf("the condition %s says nothing", o.Key())
case strings.TrimSpace(o.Source) == "": case strings.TrimSpace(o.Source) == "":
return fmt.Errorf("the condition %s does not say what raised it", o.Key()) return fmt.Errorf("the condition %s does not say what raised it", o.Key())
case o.Times != 0 && o.Happened.IsZero():
// Times is what a record counts beside when it last happened: alone, the raise would ignore it
// and an update count it, so the count would mean two things (novox/hq issue 440).
return fmt.Errorf("the condition %s says how often it happened (%d) but not when", o.Key(), o.Times)
} }
return nil return nil
} }
+20 -7
View File
@@ -191,11 +191,16 @@ func (k *Keeper) Observe(ctx context.Context, o Observation) (Condition, error)
if err != nil { if err != nil {
return Condition{}, fmt.Errorf("reading the condition %s: %w", key, err) return Condition{}, fmt.Errorf("reading the condition %s: %w", key, err)
} }
// What was observed and when: the look, or what the record says happened (novox/hq issue 402).
at, observations := now, 1
if !o.Happened.IsZero() {
at, observations = o.Happened.UTC(), max(1, o.Times)
}
if !found { if !found {
c := Condition{Key: key, Kind: o.Kind, Subject: Subject{Scope: o.Scope, ID: o.ID, Machine: o.Machine, Also: o.Also}, c := Condition{Key: key, Kind: o.Kind, Subject: Subject{Scope: o.Scope, ID: o.ID, Machine: o.Machine, Also: o.Also},
Severity: o.Severity, Summary: o.Summary, Headline: o.Headline, Explanation: o.Explanation, Severity: o.Severity, Summary: o.Summary, Headline: o.Headline, Explanation: o.Explanation,
Resolved: o.Resolved, Needs: o.Needs, Actions: o.Actions, Evidence: []Evidence{{At: now, Said: said}}, Resolved: o.Resolved, Needs: o.Needs, Actions: o.Actions, Evidence: []Evidence{{At: at, Said: said}},
Source: o.Source, Raised: now, LastObserved: now, Observations: 1, Count: 1, Source: o.Source, Raised: now, LastObserved: at, Observations: observations, Count: 1,
Resolver: orSelf(o.Resolver)} Resolver: orSelf(o.Resolver)}
change := ChangeRaised change := ChangeRaised
k.mu.Lock() k.mu.Lock()
@@ -244,7 +249,10 @@ func (k *Keeper) Observe(ctx context.Context, o Observation) (Condition, error)
} }
// The kind as the source says it now: a source that gave the same key a kind of its own since // The kind as the source says it now: a source that gave the same key a kind of its own since
// (a probe's finding split out for a healer) is read by that kind from its next observation. // (a probe's finding split out for a healer) is read by that kind from its next observation.
c.Kind, c.Summary, c.Source, c.LastObserved = o.Kind, o.Summary, o.Source, now // A record read again with nothing newer in it is not observed again: its words are refreshed,
// its count, evidence and last observation stay (novox/hq issue 402).
fresh := o.Happened.IsZero() || at.After(c.LastObserved)
c.Kind, c.Summary, c.Source = o.Kind, o.Summary, o.Source
wasNeeds, wasActions := c.Needs, c.Actions wasNeeds, wasActions := c.Needs, c.Actions
c.Headline, c.Explanation, c.Resolved, c.Needs, c.Actions = o.Headline, o.Explanation, o.Resolved, o.Needs, o.Actions c.Headline, c.Explanation, c.Resolved, c.Needs, c.Actions = o.Headline, o.Explanation, o.Resolved, o.Needs, o.Actions
escalatedWords(&c) escalatedWords(&c)
@@ -257,10 +265,15 @@ func (k *Keeper) Observe(ctx context.Context, o Observation) (Condition, error)
if len(o.Also) > 0 { if len(o.Also) > 0 {
c.Subject.Also = o.Also c.Subject.Also = o.Also
} }
c.Observations++ if fresh {
c.Evidence = append([]Evidence{{At: now, Said: said}}, c.Evidence...) // A record counts what happened since it began; the condition never counts down, so a
if len(c.Evidence) > KeptEvidence { // record begun again (a controller restarted) still adds the one it shows.
c.Evidence = c.Evidence[:KeptEvidence] c.Observations = max(c.Observations+1, o.Times)
c.LastObserved = at
c.Evidence = append([]Evidence{{At: at, Said: said}}, c.Evidence...)
if len(c.Evidence) > KeptEvidence {
c.Evidence = c.Evidence[:KeptEvidence]
}
} }
if err := k.stamp(&c); err != nil { if err := k.stamp(&c); err != nil {
return Condition{}, err return Condition{}, err
+16
View File
@@ -494,3 +494,19 @@ func TestASecondReopeningKeepsBothGaps(t *testing.T) {
t.Fatalf("open at the wrong moments: %+v", g) t.Fatalf("open at the wrong moments: %+v", g)
} }
} }
// **Times is how often a record says it happened, never alone** (novox/hq issue 440). The raise read
// Times only beside Happened while an update read it always, so a source setting Times alone would have
// jumped the count on its second look and not its first. The keeper refuses it, saying why.
func TestTimesWithoutWhenItHappenedIsRefused(t *testing.T) {
k, store, _, _ := keeper(t)
o := Observation{Scope: ScopeMachine, ID: "ace", Kind: "silent", Machine: "ace", Severity: Warning,
Summary: "ace has not been heard from", Source: "S1", Times: 5}
_, err := k.Observe(t.Context(), o)
if err == nil || !strings.Contains(err.Error(), "when") {
t.Fatalf("Times without Happened was taken: %v", err)
}
if _, found, _ := ReadOne(t.Context(), store, o.Key()); found {
t.Fatal("a refused observation raised its condition")
}
}
-9
View File
@@ -108,15 +108,6 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
for _, p := range people { for _, p := range people {
out.People[p.Name] = p.Invokes out.People[p.Name] = p.Invokes
} }
// The view is composed once its credential is minted and until it is forgotten: its row is the
// record of it, nothing else being declared about it (broker.KindView).
kept, err := i.BusUsers(ctx)
if err != nil {
return broker.Records{}, err
}
if u, minted := kept[broker.ViewUser]; minted && u.Kind == BusView {
out.View = true
}
return out, nil return out, nil
} }
-3
View File
@@ -45,9 +45,6 @@ const (
// BusNodeTools is a machine's tool runtime (novox/hq ADR 0175): named like the module it // BusNodeTools is a machine's tool runtime (novox/hq ADR 0175): named like the module it
// stands for, recorded as what it is. // stands for, recorded as what it is.
BusNodeTools = "node-tools" BusNodeTools = "node-tools"
// BusView is the one read-only view onto the bus (broker.KindView): its row is the whole record of
// it, minted by `bus view-credential` and forgotten by `bus view-revoke`.
BusView = "view"
) )
// MintBusPassword makes a bus password and records its hash under a username, replacing whatever was // MintBusPassword makes a bus password and records its hash under a username, replacing whatever was
+18 -8
View File
@@ -94,7 +94,9 @@ type DataChange struct {
} }
// readingEvery is how often a measurement is kept as a reading: the shrink is read over days, and a // readingEvery is how often a measurement is kept as a reading: the shrink is read over days, and a
// row every five minutes would say the same thing sixty times an hour. // row every five minutes would say the same thing sixty times an hour. A reading keeps the item's path
// as it stands after the measurement — the holder's, or the last one known when it named none — and an
// item at a path with no recent reading is read at once (novox/hq issue 368).
const readingEvery = 55 * time.Minute const readingEvery = 55 * time.Minute
// readingsKept is how long readings are kept. // readingsKept is how long readings are kept.
@@ -187,10 +189,14 @@ func (i *Inventory) RecordData(ctx context.Context, machine string, declared []D
} }
if m.Size != nil && m.MeasuredAt != nil && Comparable(m.Precision) { if m.Size != nil && m.MeasuredAt != nil && Comparable(m.Precision) {
if _, err := tx.Exec(ctx, ` if _, err := tx.Exec(ctx, `
insert into data_reading (machine, module, item, at, size_bytes, last_write) insert into data_reading (machine, module, item, at, size_bytes, last_write, path)
select $1, $2, $3, $4, $5, $6 select $1, $2, $3, $4, $5, $6, d.path
where not exists (select 1 from data_reading from data_item d
where machine = $1 and module = $2 and item = $3 and at > $4::timestamptz - $7::interval)`, where d.machine = $1 and d.module = $2 and d.item = $3
and not exists (select 1 from data_reading
where machine = $1 and module = $2 and item = $3 and path = d.path
and at > $4::timestamptz - $7::interval)
on conflict (machine, module, item, at) do nothing`,
machine, d.Module, d.Item, *m.MeasuredAt, *m.Size, m.LastWrite, machine, d.Module, d.Item, *m.MeasuredAt, *m.Size, m.LastWrite,
fmt.Sprintf("%d seconds", int(readingEvery.Seconds()))); err != nil { fmt.Sprintf("%d seconds", int(readingEvery.Seconds()))); err != nil {
return change, err return change, err
@@ -281,10 +287,14 @@ func (i *Inventory) DataOf(ctx context.Context, machine, module, item string) (D
return r, err return r, err
} }
// DataPeaks is each item's largest reading since a moment, keyed by DataRecord.Key. // DataPeaks is each item's largest reading since a moment, keyed by DataRecord.Key — read only from
// readings at the item's path now (novox/hq issue 368): a directory is never compared with another one
// that once held the same item, so a moved item starts its size history again at its new path.
func (i *Inventory) DataPeaks(ctx context.Context, since time.Time) (map[string]int64, error) { func (i *Inventory) DataPeaks(ctx context.Context, since time.Time) (map[string]int64, error) {
rows, err := i.store.Pool().Query(ctx, `select machine, module, item, max(size_bytes) from data_reading rows, err := i.store.Pool().Query(ctx, `select r.machine, r.module, r.item, max(r.size_bytes)
where at >= $1 group by machine, module, item`, since) from data_reading r
join data_item d on d.machine = r.machine and d.module = r.module and d.item = r.item and d.path = r.path
where r.at >= $1 group by r.machine, r.module, r.item`, since)
if err != nil { if err != nil {
return nil, err return nil, err
} }
+63
View File
@@ -128,3 +128,66 @@ func TestAPartialMeasurementIsNeverAReading(t *testing.T) {
t.Fatalf("%+v, %v", r, err) t.Fatalf("%+v, %v", r, err)
} }
} }
// A shrink is read against what the same directory held (novox/hq issue 368): an item whose path moved
// — an agent's home moved to its own account — starts its size history again at the new path, and a
// genuine shrink at the new path is still read against what that path held.
func TestThePeakIsReadAtTheItemsPathOnly(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
now := time.Date(2026, 10, 10, 0, 0, 0, 0, time.UTC)
declared := []DeclaredData{{Module: "claude-code", Item: "agent-home", Class: "valuable", Owned: true}}
measure := func(when time.Time, path string, s int64) {
t.Helper()
if _, err := inv.RecordData(ctx, "novox", declared, map[string]map[string]Measurement{"claude-code": {
"agent-home": {Path: path, Size: size(s), MeasuredAt: at(when)}}}, "", when); err != nil {
t.Fatal(err)
}
}
measure(now, "/home/operator/.claude", 94_700_000)
// The path moves ten minutes later: the new directory is measured at once, not an hour on.
measure(now.Add(10*time.Minute), "/home/agent/.claude", 490)
peaks, err := inv.DataPeaks(ctx, now.Add(-time.Hour))
if err != nil || peaks["novox/claude-code/agent-home"] != 490 {
t.Fatalf("after the path moved the peak is %v (%v), want 490: the old directory's size is no shrink "+
"of the new one", peaks, err)
}
// The new directory grows, then genuinely loses most of it: that is read against the new path's peak.
measure(now.Add(2*time.Hour), "/home/agent/.claude", 80_000_000)
measure(now.Add(4*time.Hour), "/home/agent/.claude", 1_000)
peaks, err = inv.DataPeaks(ctx, now.Add(-time.Hour))
if err != nil || peaks["novox/claude-code/agent-home"] != 80_000_000 {
t.Fatalf("a shrink at the new path is read against %v (%v), want 80000000", peaks, err)
}
// A measurement that names no path is the item's last known path's, not a new history.
measure(now.Add(6*time.Hour), "", 2_000)
peaks, err = inv.DataPeaks(ctx, now.Add(-time.Hour))
if err != nil || peaks["novox/claude-code/agent-home"] != 80_000_000 {
t.Fatalf("a measurement with no path started a new history: peak %v (%v)", peaks, err)
}
// Moving back to a directory measured before reads it against what it held then.
measure(now.Add(8*time.Hour), "/home/operator/.claude", 94_000_000)
peaks, err = inv.DataPeaks(ctx, now.Add(-time.Hour))
if err != nil || peaks["novox/claude-code/agent-home"] != 94_700_000 {
t.Fatalf("back at the first path the peak is %v (%v), want 94700000", peaks, err)
}
}
// Two measurements at the same moment at two paths keep one reading and fail nothing: the second
// would otherwise break the reading's key and lose the machine's whole record (issue 368 review).
func TestTwoPathsAtOneMomentFailNothing(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
now := time.Date(2026, 10, 10, 0, 0, 0, 0, time.UTC)
declared := []DeclaredData{{Module: "claude-code", Item: "agent-home", Class: "valuable", Owned: true}}
for _, path := range []string{"/home/operator/.claude", "/home/agent/.claude"} {
if _, err := inv.RecordData(ctx, "novox", declared, map[string]map[string]Measurement{"claude-code": {
"agent-home": {Path: path, Size: size(100), MeasuredAt: at(now)}}}, "", now); err != nil {
t.Fatalf("a measurement at %s failed: %v", path, err)
}
}
r, err := inv.DataOf(ctx, "novox", "claude-code", "agent-home")
if err != nil || r.Path != "/home/agent/.claude" {
t.Fatalf("%+v, %v", r, err)
}
}
+12 -1
View File
@@ -17,10 +17,13 @@ const (
DurationHeartbeatGap = "heartbeat-gap" DurationHeartbeatGap = "heartbeat-gap"
DurationPlanTier = "plan-tier" DurationPlanTier = "plan-tier"
DurationBuild = "build" DurationBuild = "build"
// DurationWalkPhase is one phase of an ended walk, per class (novox/hq ADR 0282 decision 6): subject
// "<class>/<phase>", and "<class>/total" for its merge to every machine running it.
DurationWalkPhase = "walk-phase"
) )
// DurationKinds are every kind, in the order `durations` shows them. // DurationKinds are every kind, in the order `durations` shows them.
var DurationKinds = []string{DurationApply, DurationHeartbeatGap, DurationPlanTier, DurationBuild} var DurationKinds = []string{DurationApply, DurationHeartbeatGap, DurationPlanTier, DurationBuild, DurationWalkPhase}
// DurationsKeptFor is how long a duration is kept: long enough to set a bound from, and to correct it // DurationsKeptFor is how long a duration is kept: long enough to set a bound from, and to correct it
// in Phase 1's first live week. // in Phase 1's first live week.
@@ -100,6 +103,14 @@ func (i *Inventory) Durations(ctx context.Context, kind string, since time.Time)
return out, rows.Err() return out, rows.Err()
} }
// WalkPhasesKept says whether any phase of a walk is kept as a walk-phase duration, and whether its total is.
func (i *Inventory) WalkPhasesKept(ctx context.Context, plan string) (anyKept, totalKept bool, err error) {
err = i.store.Pool().QueryRow(ctx,
`select count(*) > 0, count(*) filter (where ref = $2) > 0 from duration where kind = $1 and ref like $3`,
DurationWalkPhase, plan+"/total", plan+"/%").Scan(&anyKept, &totalKept)
return anyKept, totalKept, err
}
// ForgetOldDurations removes what is older than DurationsKeptFor, and says how many. // ForgetOldDurations removes what is older than DurationsKeptFor, and says how many.
func (i *Inventory) ForgetOldDurations(ctx context.Context) (int64, error) { func (i *Inventory) ForgetOldDurations(ctx context.Context) (int64, error) {
tag, err := i.store.Pool().Exec(ctx, `delete from duration where recorded < $1`, tag, err := i.store.Pool().Exec(ctx, `delete from duration where recorded < $1`,
+1 -1
View File
@@ -155,7 +155,7 @@ func (i *Inventory) ReplaceGivenAfterStart(ctx context.Context, node, declared s
} }
// The definition is asked again now, not only when the value was given: one that has since // The definition is asked again now, not only when the value was given: one that has since
// said the value is an outside party's, or applied, keeps it as given, and the mark stays gone. // said the value is an outside party's, or applied, keeps it as given, and the mark stays gone.
if own, ok := m.OwnSecrets[d.name]; !ok || !own.MeshMayMake() { if own, _, ok := m.OwnSecrets.Lookup(d.name); !ok || !own.MeshMayMake() {
continue continue
} }
if err := i.remakeOwn(ctx, d.nodeID, key, m, d.module, d.name); err != nil { if err := i.remakeOwn(ctx, d.nodeID, key, m, d.module, d.name); err != nil {
+11
View File
@@ -34,6 +34,17 @@ type ResourceHealth struct {
// Root is "never" on an account verdict that judged whether the account can become root without a // Root is "never" on an account verdict that judged whether the account can become root without a
// person (novox/hq ADR 0266). // person (novox/hq ADR 0266).
Root string `json:"root,omitempty"` Root string `json:"root,omitempty"`
// Waits is what a waiting resource waits for the operator to give (novox/hq ADR 0283).
Waits []Wait `json:"waits,omitempty"`
}
// Wait is one thing a waiting resource waits for the operator to give: exactly one of Secret and Setting
// (novox/hq ADR 0283), as link.Wait carries it.
type Wait struct {
Part string `json:"part"`
Secret string `json:"secret,omitempty"`
Setting string `json:"setting,omitempty"`
What string `json:"what"`
} }
// NodeHealth is a machine's newest statement, as kept. // NodeHealth is a machine's newest statement, as kept.
@@ -0,0 +1,21 @@
-- A reading says the path it was measured at (novox/hq issue 368).
--
-- A shrink is read against the largest reading of the last seven days. Readings were kept by machine,
-- module and item only, so when an item's path moved — on 2026-10-10 an agent's home moved from the
-- operator's own home to the agent account's (ADR 0266) — the new, fresh directory was read against
-- the old one's size, and `data-shrank` was raised for data that was never lost. A reading now keeps
-- its path, and the peak is read only from readings at the item's path now: a moved item starts its
-- size history again, and moving back to a path reads it against what that path held.
--
-- **Every reading kept so far is taken to be at its item's path now.** Where it was really measured
-- is not on record; taking the path now keeps every item's history, so a shrink that is real stays
-- raised. An item whose path moved before this migration stays compared with its old directory until
-- those readings leave the seven-day window. Readings of an item no longer kept keep no path, and are
-- read for nothing.
--
-- Numbered 0092, past 0091, the highest on main or any open branch when this was written.
alter table data_reading add column path text;
update data_reading r set path = d.path
from data_item d
where d.machine = r.machine and d.module = r.module and d.item = r.item;
@@ -0,0 +1,10 @@
-- A walk keeps its phases (novox/hq ADR 0282 decision 6, issue 382).
--
-- A small fix took 45 minutes to reach a node on 2026-10-10, and where the time went was read back from the
-- walks by hand: the walk kept when its modules were asked, built, sent first, judged and sent to the rest,
-- but not when its batch's window closed or when it was cut, so the window and the wait behind another walk
-- could not be told apart. A walk now keeps those moments and its class (core or leaf, read at its cut).
-- Its readings, each module's send to the rest and the times of the merges it answers are kept in the plan's
-- own records (modules, delivery). Null for a walk kept before this: its phases before the build are said
-- unknown.
alter table release_plan add column times jsonb;
+73 -8
View File
@@ -54,6 +54,25 @@ type Plan struct {
// walk can carry several. Repository and Commit above keep one of them, for a reader that knows one. Empty // walk can carry several. Repository and Commit above keep one of them, for a reader that knows one. Empty
// for a walk kept before it was: Repository and Commit are then the whole of it. // for a walk kept before it was: Repository and Commit are then the whole of it.
Commits []PlanCommit `json:"commits,omitempty"` Commits []PlanCommit `json:"commits,omitempty"`
// Times are the moments of a walk no other field keeps (novox/hq ADR 0282 decision 6): when its batch's
// window closed, when it was cut, and its class. Nil for a walk kept before they were, whose phases before
// its build are said unknown.
Times *PlanTimes `json:"times,omitempty"`
// Phases is the walk's time from its merge, phase by phase (ADR 0282 decision 6): never kept, worked out
// from the walk's own moments by whoever says the walk (`delivery walks`, `plan-moved`).
Phases *WalkPhases `json:"phases,omitempty"`
}
// PlanTimes are a walk's own moments beside its modules' (novox/hq ADR 0282 decision 6).
type PlanTimes struct {
// WindowClosed is when its batch's merge window closed: no merge for the window's length, or its maximum.
// Nil for a walk no window assembled (one merge walked alone).
WindowClosed *time.Time `json:"window_closed,omitempty"`
// Cut is when the batch became the walk.
Cut *time.Time `json:"cut,omitempty"`
// Class is the walk's module class, read at its cut: core when it moves a module on the controller's own
// path or the mesh's resolver, leaf otherwise (ADR 0282 decision 1).
Class string `json:"class,omitempty"`
} }
// PlanCommit is one repository's commit a walk carries: the latest merge of its branch in the batch. // PlanCommit is one repository's commit a walk carries: the latest merge of its branch in the batch.
@@ -76,6 +95,10 @@ type PlanMerge struct {
Number int `json:"number,omitempty"` Number int `json:"number,omitempty"`
Title string `json:"title,omitempty"` Title string `json:"title,omitempty"`
Moves []string `json:"moves,omitempty"` Moves []string `json:"moves,omitempty"`
// Merged is when the forge made the merge and Heard when the controller heard it (novox/hq ADR 0282): where
// its delivery time starts. Zero for a merge named before they were kept.
Merged time.Time `json:"merged,omitzero"`
Heard time.Time `json:"heard,omitzero"`
} }
// PlanBatch is a batch's window while it is one (novox/hq ADR 0276): when it closes unless another merge // PlanBatch is a batch's window while it is one (novox/hq ADR 0276): when it closes unless another merge
@@ -220,6 +243,9 @@ type PlanModule struct {
// went there in one send (novox/hq issue 281), and one gate judges what one send moved. Empty for // went there in one send (novox/hq issue 281), and one gate judges what one send moved. Empty for
// the module the gate is kept on, and for a plan from before tiers were sent whole. // the module the gate is kept on, and for a plan from before tiers were sent whole.
GatedBy string `json:"gated_by,omitempty"` GatedBy string `json:"gated_by,omitempty"`
// Rest is, per machine of the rest, the declaration the send after the first-node gate carried there (novox/hq ADR
// 0282 decision 6): the machine's first report of it, applied, is when the build runs there.
Rest map[string]SentDeclaration `json:"rest,omitempty"`
} }
// PlanGate is one module's rollout record at its gate (to-be 45 §8): the component, the first machine, // PlanGate is one module's rollout record at its gate (to-be 45 §8): the component, the first machine,
@@ -276,6 +302,35 @@ type PlanGate struct {
BrokenWhy string `json:"broken_why,omitempty"` BrokenWhy string `json:"broken_why,omitempty"`
// Returned names the broken modules already put back, at once, while the rest of the send is judged. // Returned names the broken modules already put back, at once, while the rest of the send is judged.
Returned []string `json:"returned,omitempty"` Returned []string `json:"returned,omitempty"`
// Readings are the judging's readings with their times (novox/hq ADR 0282 decision 6): every reading that
// counted a pass, and the first that did not after one that did. At most maxReadings, the newest kept.
Readings []GateReading `json:"readings,omitempty"`
}
// GateReading is one reading of a first-node gate.
type GateReading struct {
At time.Time `json:"at"`
Healthy bool `json:"healthy"`
// Said is what a reading that did not pass found wanting.
Said string `json:"said,omitempty"`
}
// maxReadings bounds a first-node gate's readings: a judging that never passes reads every few seconds for ten minutes.
const maxReadings = 24
// Read keeps one reading: a pass always, and a reading that did not pass only when the one before passed or
// there is none, so a judging waiting for a module to start keeps one line of it, not hundreds.
func (g *PlanGate) Read(at time.Time, healthy bool, said string) {
if !healthy && len(g.Readings) > 0 && !g.Readings[len(g.Readings)-1].Healthy {
return
}
if r := []rune(said); len(r) > 200 {
said = string(r[:200])
}
g.Readings = append(g.Readings, GateReading{At: at, Healthy: healthy, Said: said})
if len(g.Readings) > maxReadings {
g.Readings = g.Readings[len(g.Readings)-maxReadings:]
}
} }
// CarriedMove is one module's build moving on a machine with a gated send. // CarriedMove is one module's build moving on a machine with a gated send.
@@ -342,7 +397,12 @@ func (i *Inventory) SavePlan(ctx context.Context, p *Plan) error {
if err != nil { if err != nil {
return err return err
} }
var release, delivery, commits []byte var release, delivery, commits, times []byte
if p.Times != nil {
if times, err = json.Marshal(p.Times); err != nil {
return err
}
}
if len(p.Commits) > 0 { if len(p.Commits) > 0 {
if commits, err = json.Marshal(p.Commits); err != nil { if commits, err = json.Marshal(p.Commits); err != nil {
return err return err
@@ -373,18 +433,18 @@ func (i *Inventory) SavePlan(ctx context.Context, p *Plan) error {
var revision int64 var revision int64
err = tx.QueryRow(ctx, err = tx.QueryRow(ctx,
`insert into release_plan (id, repository, commit_hash, created, updated, state, tier, tiers, modules, note, `insert into release_plan (id, repository, commit_hash, created, updated, state, tier, tiers, modules, note,
branch, tier_entered, revision, epoch, release, delivery, merged_at, commits) branch, tier_entered, revision, epoch, release, delivery, merged_at, commits, times)
values ($1, $2, $3, $4, now(), $5, $6, $7, $8, $9, $10, $11, 1, $13, $14, $15, $16, $17) values ($1, $2, $3, $4, now(), $5, $6, $7, $8, $9, $10, $11, 1, $13, $14, $15, $16, $17, $18)
on conflict (id) do update set updated = now(), state = excluded.state, tier = excluded.tier, on conflict (id) do update set updated = now(), state = excluded.state, tier = excluded.tier,
tiers = excluded.tiers, modules = excluded.modules, note = excluded.note, branch = excluded.branch, tiers = excluded.tiers, modules = excluded.modules, note = excluded.note, branch = excluded.branch,
tier_entered = excluded.tier_entered, revision = release_plan.revision + 1, epoch = excluded.epoch, tier_entered = excluded.tier_entered, revision = release_plan.revision + 1, epoch = excluded.epoch,
release = excluded.release, delivery = excluded.delivery, repository = excluded.repository, release = excluded.release, delivery = excluded.delivery, repository = excluded.repository,
commit_hash = excluded.commit_hash, merged_at = excluded.merged_at, commits = excluded.commits, commit_hash = excluded.commit_hash, merged_at = excluded.merged_at, commits = excluded.commits,
created = excluded.created created = excluded.created, times = excluded.times
where release_plan.revision = $12 where release_plan.revision = $12
returning revision`, returning revision`,
p.ID, p.Repository, p.Commit, p.Created, p.State, p.Tier, tiers, modules, p.Note, p.Branch, entered, p.ID, p.Repository, p.Commit, p.Created, p.State, p.Tier, tiers, modules, p.Note, p.Branch, entered,
p.Revision, epoch, release, delivery, mergedAt(p.Merged), commits).Scan(&revision) p.Revision, epoch, release, delivery, mergedAt(p.Merged), commits, times).Scan(&revision)
if errors.Is(err, pgx.ErrNoRows) { if errors.Is(err, pgx.ErrNoRows) {
// The row is there and at another revision — moved since this was read, or there already // The row is there and at another revision — moved since this was read, or there already
// when this one is new: either way not this writer's to overwrite. (A plan saved before plans // when this one is new: either way not this writer's to overwrite. (A plan saved before plans
@@ -467,7 +527,7 @@ func (i *Inventory) PlanByID(ctx context.Context, id string) (Plan, error) {
func (i *Inventory) plans(ctx context.Context, tail string, args ...any) ([]Plan, error) { func (i *Inventory) plans(ctx context.Context, tail string, args ...any) ([]Plan, error) {
rows, err := i.store.Pool().Query(ctx, rows, err := i.store.Pool().Query(ctx,
`select id, repository, commit_hash, created, updated, state, tier, tiers, modules, note, branch, `select id, repository, commit_hash, created, updated, state, tier, tiers, modules, note, branch,
coalesce(tier_entered, created), revision, coalesce(epoch, 0), release, delivery, merged_at, commits coalesce(tier_entered, created), revision, coalesce(epoch, 0), release, delivery, merged_at, commits, times
from release_plan `+tail, args...) from release_plan `+tail, args...)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -476,14 +536,19 @@ func (i *Inventory) plans(ctx context.Context, tail string, args ...any) ([]Plan
var out []Plan var out []Plan
for rows.Next() { for rows.Next() {
var p Plan var p Plan
var tiers, modules, release, delivery, commits []byte var tiers, modules, release, delivery, commits, times []byte
var epoch int64 var epoch int64
var merged *time.Time var merged *time.Time
if err := rows.Scan(&p.ID, &p.Repository, &p.Commit, &p.Created, &p.Updated, &p.State, if err := rows.Scan(&p.ID, &p.Repository, &p.Commit, &p.Created, &p.Updated, &p.State,
&p.Tier, &tiers, &modules, &p.Note, &p.Branch, &p.TierEntered, &p.Revision, &epoch, &release, &p.Tier, &tiers, &modules, &p.Note, &p.Branch, &p.TierEntered, &p.Revision, &epoch, &release,
&delivery, &merged, &commits); err != nil { &delivery, &merged, &commits, &times); err != nil {
return nil, err return nil, err
} }
if len(times) > 0 {
if err := json.Unmarshal(times, &p.Times); err != nil {
return nil, err
}
}
if len(commits) > 0 { if len(commits) > 0 {
if err := json.Unmarshal(commits, &p.Commits); err != nil { if err := json.Unmarshal(commits, &p.Commits); err != nil {
return nil, err return nil, err
+76
View File
@@ -0,0 +1,76 @@
package inventory
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A member of a secret family is given by its full name, at the desk too, and a name outside the family is refused
// (novox/hq ADR 0283).
func TestAMemberIsGivableAtTheDeskAndANameOutsideTheFamilyIsNot(t *testing.T) {
m := catalogue.Manifest{Module: "mounts", OwnSecrets: catalogue.OwnSecrets{
"smb-password-*": {Path: "/s/smb-password-*.secret", IssuedBy: catalogue.IssuedOutside},
}}
if err := GivableAtDesk(m, "smb-password-games"); err != nil {
t.Fatalf("a member was refused: %v", err)
}
for _, name := range []string{"smb-password-*", "smb-password-", "smb-password-a/b", "smb-password-A", "smb-credentials"} {
err := GivableAtDesk(m, name)
if err == nil {
t.Errorf("%q was givable", name)
} else if !strings.Contains(err.Error(), "smb-password-<name>") {
t.Errorf("%q: the refusal does not say the family's form: %v", name, err)
}
}
}
// A member given is read back as given, sealed to the machine's key; the mesh makes none, so nothing else is.
func TestAMemberGivenIsReadAsGivenAndNothingElseIs(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
node, err := inv.AddNode(ctx, "workstation")
if err != nil {
t.Fatal(err)
}
key, _ := aSealingKey(t)
if err := inv.RecordSealingKey(ctx, node.ID, key); err != nil {
t.Fatal(err)
}
m := catalogue.Manifest{Module: "mounts", Version: "1", OwnSecrets: catalogue.OwnSecrets{
"smb-password-*": {Path: "/s/smb-password-*.secret", IssuedBy: catalogue.IssuedOutside},
"token": {Path: "/s/token", IssuedBy: catalogue.IssuedOutside},
}}
if err := inv.RegisterModule(ctx, m, Source{}); err != nil {
t.Fatal(err)
}
if err := inv.AcceptSecretForModule(ctx, "workstation", "mounts", "smb-password-games", "hunter2"); err != nil {
t.Fatalf("a member was refused: %v", err)
}
if err := inv.AcceptSecretForModule(ctx, "workstation", "mounts", "smb-pass", "x"); err == nil {
t.Fatal("a name of no family was accepted")
}
members, err := inv.GivenMembers(ctx, "workstation", "mounts", "smb-password-*")
if err != nil {
t.Fatal(err)
}
if len(members) != 1 || members[0].Name != "smb-password-games" || !members[0].Current || members[0].Sealed == "" ||
strings.Contains(members[0].Sealed, "hunter2") {
t.Fatalf("members: %+v", members)
}
given, err := inv.GivenOwnSecrets(ctx, "workstation", "mounts")
if err != nil {
t.Fatal(err)
}
if _, was := given["smb-password-games"]; !was || len(given) != 1 {
t.Fatalf("given: %v", given)
}
// A value the mesh made is not one a person gave.
if _, err := inv.SecretForModule(ctx, "workstation", "mounts", "token"); err != nil {
t.Fatal(err)
}
if given, _ := inv.GivenOwnSecrets(ctx, "workstation", "mounts"); len(given) != 1 {
t.Fatalf("a value the mesh made counted as given: %v", given)
}
}
+77 -4
View File
@@ -7,6 +7,7 @@ import (
"slices" "slices"
"sort" "sort"
"strings" "strings"
"time"
"github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5"
@@ -446,7 +447,7 @@ func (i *Inventory) acceptOwn(ctx context.Context, node, module, name, value str
if err != nil { if err != nil {
return false, err return false, err
} }
own, declared := m.OwnSecrets[name] own, _, declared := m.OwnSecrets.Lookup(name)
if !declared { if !declared {
return false, fmt.Errorf("%s does not declare %q as an own secret; %s — a secret it requires from a provider is accepted with `--provider <node> [--local <name>]`, the value the running service already uses (novox/hq ADR 0163)", module, name, declaresOwn(m)) return false, fmt.Errorf("%s does not declare %q as an own secret; %s — a secret it requires from a provider is accepted with `--provider <node> [--local <name>]`, the value the running service already uses (novox/hq ADR 0163)", module, name, declaresOwn(m))
} }
@@ -583,7 +584,7 @@ const BrokerSecret = "broker"
// in place of a value given (catalogue.OwnSecret.MeshMayMake). The desk takes only what a person holds and // in place of a value given (catalogue.OwnSecret.MeshMayMake). The desk takes only what a person holds and
// the mesh cannot make — a bot's token — so nobody is asked to type the mesh's own credential into a prompt. // the mesh cannot make — a bot's token — so nobody is asked to type the mesh's own credential into a prompt.
func GivableAtDesk(m catalogue.Manifest, name string) error { func GivableAtDesk(m catalogue.Manifest, name string) error {
own, ok := m.OwnSecrets[name] own, _, ok := m.OwnSecrets.Lookup(name)
if !ok { if !ok {
return fmt.Errorf("%s does not declare %q as an own secret; %s", m.Module, name, declaresOwn(m)) return fmt.Errorf("%s does not declare %q as an own secret; %s", m.Module, name, declaresOwn(m))
} }
@@ -602,7 +603,79 @@ func declaresOwn(m catalogue.Manifest) string {
if len(m.OwnSecrets) == 0 { if len(m.OwnSecrets) == 0 {
return "it declares no own secrets" return "it declares no own secrets"
} }
return "it declares: " + strings.Join(sortedNames(m.OwnSecrets.Paths()), ", ") names := sortedNames(m.OwnSecrets.Plain().Paths())
// A family is said as its members are given (novox/hq ADR 0283): one per part, by its full name.
for _, f := range m.OwnSecrets.Families() {
names = append(names, catalogue.FamilyPrefix(f)+"<name> (one per part, issued outside the mesh)")
}
return "it declares: " + strings.Join(names, ", ")
}
// GivenMember is one member of a secret family given on a machine (novox/hq ADR 0283): its full name, its value
// sealed to the machine, and whether it is sealed to the key the machine holds now.
type GivenMember struct {
Name string
Sealed string
Current bool
}
// GivenMembers is every member of a module's secret family given on a machine, by name. The mesh never makes a
// member, so only a value a person gave is one; a machine with no sealing key holds none.
func (i *Inventory) GivenMembers(ctx context.Context, node, module, family string) ([]GivenMember, error) {
key, err := i.SealingKeyOf(ctx, node)
if err != nil || key == "" {
return nil, err
}
record, err := i.NodeByName(ctx, node)
if err != nil {
return nil, err
}
prefix := catalogue.FamilyPrefix(family)
rows, err := i.store.Pool().Query(ctx,
`select name, sealed, node_key from module_secret
where node = $1 and module = $2 and origin = 'accepted' and left(name, length($3)) = $3
order by name`, record.ID, module, prefix)
if err != nil {
return nil, err
}
defer rows.Close()
var out []GivenMember
for rows.Next() {
var g GivenMember
var against string
if err := rows.Scan(&g.Name, &g.Sealed, &against); err != nil {
return nil, err
}
g.Current = against == key
out = append(out, g)
}
return out, rows.Err()
}
// GivenOwnSecrets is every own secret of a module a person gave on a machine, by name, with when (novox/hq ADR
// 0283): what the controller checks a module's wait for a secret against. A value the mesh made is not one.
func (i *Inventory) GivenOwnSecrets(ctx context.Context, node, module string) (map[string]time.Time, error) {
record, err := i.NodeByName(ctx, node)
if err != nil {
return nil, err
}
rows, err := i.store.Pool().Query(ctx,
`select name, coalesce(made_at, now()) from module_secret where node = $1 and module = $2 and origin = 'accepted'`,
record.ID, module)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]time.Time{}
for rows.Next() {
var name string
var at time.Time
if err := rows.Scan(&name, &at); err != nil {
return nil, err
}
out[name] = at
}
return out, rows.Err()
} }
func sortedNames(of map[string]string) []string { func sortedNames(of map[string]string) []string {
@@ -645,7 +718,7 @@ func (i *Inventory) RotateModuleSecret(ctx context.Context, node, module, name s
if err != nil { if err != nil {
return err return err
} }
own, declared := m.OwnSecrets[name] own, _, declared := m.OwnSecrets.Lookup(name)
if !declared { if !declared {
return fmt.Errorf("%s does not declare %q as an own secret; %s — a secret it requires from a provider is accepted with `--provider <node> [--local <name>]`, the value the running service already uses (novox/hq ADR 0163)", module, name, declaresOwn(m)) return fmt.Errorf("%s does not declare %q as an own secret; %s — a secret it requires from a provider is accepted with `--provider <node> [--local <name>]`, the value the running service already uses (novox/hq ADR 0163)", module, name, declaresOwn(m))
} }
+478
View File
@@ -0,0 +1,478 @@
package inventory
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/jackc/pgx/v5"
)
// Where a walk's time went (novox/hq ADR 0282 decision 6, issue 382): from its merge to every machine of its
// rest running its builds, phase by phase, worked out from the walk's own moments. Measurement only: nothing
// here decides anything about the walk.
//
// **A phase that cannot be measured is said unknown, never zero.** Where a moment is missing (a walk kept
// before it was recorded, a machine not yet reported), the phases around it are unknown, and the span between
// the moments on either side is counted as unknown time: the measured phases and the unknown time always add
// up to the total. A phase that did not happen (no window for a merge walked alone, no first machine for a
// module nobody runs) is said none, with no time.
// The phases, in the order a walk passes them (ADR 0282's table).
const (
PhaseWindow = "window" // the merge to its batch's window closing
PhaseQueued = "queued" // the window closed to the walk being cut: waiting behind another walk
PhaseWord = "word" // the cut to the delivery's word, for a walk that waits for one
PhaseBetween = "between-tiers" // one tier's end to the next tier's ask
PhaseBuild = "build" // a tier asked to its last module built
PhaseSend = "send-first" // built to sent to the first machines
PhaseJudge = "judgement" // sent first to the first-node gate's last verdict: its readings
PhaseRest = "send-rest" // judged to sent to the rest
PhaseApply = "apply" // the last send to every machine of the rest reporting the build applied
PhaseOpen = "open" // a walk still running: since its last moment
PhaseMeasured = "measured"
PhaseUnknown = "unknown"
PhaseNone = "none"
)
// The classes of a walk (ADR 0282 decision 1) and their delivery budgets.
const (
ClassCore = "core"
ClassLeaf = "leaf"
)
// ApplySilentAfter is how long after a send to the rest a machine that has said nothing is left out of the
// walk's end, as a machine not heard from (ADR 0282 decision 1: a sleeping laptop is listed, not waited for).
const ApplySilentAfter = 15 * time.Minute
// WalkPhases is a walk's time, phase by phase.
type WalkPhases struct {
Class string `json:"class,omitempty"`
// From is the walk's earliest merge; End when its last phase ended: every machine of its rest reported the
// build applied. Nil End while that is not known.
From *time.Time `json:"from,omitempty"`
End *time.Time `json:"end,omitempty"`
// TotalMS is End − From; zero while either is unknown, Total its words.
TotalMS int64 `json:"total_ms,omitempty"`
Total string `json:"total,omitempty"`
// UnknownMS is the time within the walk no phase could be measured over.
UnknownMS int64 `json:"unknown_ms,omitempty"`
Phases []WalkPhase `json:"phases"`
Silent []string `json:"silent,omitempty"`
Said string `json:"said"`
Merges []MergeStart `json:"merges,omitempty"`
}
// MergeStart is one merge the walk answers and when it was made: where that merge's delivery time starts.
type MergeStart struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
Merged time.Time `json:"merged"`
}
// WalkPhase is one phase of a walk.
type WalkPhase struct {
Name string `json:"name"`
// Tier is the tier a tier's phase belongs to; -1 for a phase of the walk.
Tier int `json:"tier"`
State string `json:"state"`
Start *time.Time `json:"start,omitempty"`
End *time.Time `json:"end,omitempty"`
TookMS int64 `json:"took_ms,omitempty"`
Took string `json:"took,omitempty"`
Said string `json:"said,omitempty"`
}
// AppliedReport is a machine's first report, after a send, that it applied what it was sent.
type AppliedReport struct {
At time.Time
Outcome string
}
// AppliedLookup answers a machine's first report after a send to it; false when it has not reported.
type AppliedLookup func(node string, sent time.Time) (AppliedReport, bool)
// point is one moment of the walk, ending the phase named.
type point struct {
phase string
tier int
at *time.Time
none bool // the phase did not happen
said string // why it is none, or unknown
}
// Phases is the walk's time phase by phase, at now; applied answers the rest's reports (nil: none read).
func (p Plan) WalkPhases(now time.Time, applied AppliedLookup) *WalkPhases {
if p.Release != nil || p.Batch() {
return nil
}
w := &WalkPhases{}
if p.Times != nil {
w.Class = p.Times.Class
}
from := p.firstMerge(w)
if from != nil {
f := from.Truncate(time.Millisecond)
from = &f
}
if from == nil {
w.Said = "when its merge was made is not kept: its delivery time is unknown"
} else {
w.From = from
}
points := p.points(now, applied, w)
// Walk the moments: each known moment after a known one is a measured phase; a missing moment makes the
// phases up to the next known one unknown, and their span unknown time.
last := from
var pending []int
for _, pt := range points {
if pt.none {
w.Phases = append(w.Phases, WalkPhase{Name: pt.phase, Tier: pt.tier, State: PhaseNone, Said: pt.said})
continue
}
ph := WalkPhase{Name: pt.phase, Tier: pt.tier, Said: pt.said}
if pt.at == nil {
ph.State = PhaseUnknown
w.Phases = append(w.Phases, ph)
pending = append(pending, len(w.Phases)-1)
continue
}
at := pt.at.Truncate(time.Millisecond)
ph.End = &at
if last != nil && at.Before(*last) {
// **Out of order** (issue 382): this moment was recorded before the one before it — a module's first
// send kept before its build was, two clocks, a late record. Neither phase can be measured honestly:
// the one before it is said unknown and its time joins the unknown time, and this one is said unknown
// with no time. The walk goes on from the later moment, so nothing is negative and the sum holds.
for i := len(w.Phases) - 1; i >= 0; i-- {
prev := &w.Phases[i]
if prev.End == nil {
continue // a phase that did not happen, or is unknown with no moment: the one before carries last
}
if prev.State == PhaseMeasured {
w.UnknownMS += prev.TookMS
}
prev.State, prev.Start, prev.TookMS, prev.Took = PhaseUnknown, nil, 0, ""
prev.Said = "out of order: the phase after it ended first"
break
}
ph.State, ph.Said = PhaseUnknown, "out of order: it ended before the phase before it"
w.Phases = append(w.Phases, ph)
pending = nil
continue
}
if last != nil && len(pending) == 0 {
start := *last
ph.State, ph.Start = PhaseMeasured, &start
ph.TookMS = at.Sub(start).Milliseconds()
ph.Took = words(at.Sub(start))
} else {
ph.State = PhaseUnknown
if last != nil {
w.UnknownMS += at.Sub(*last).Milliseconds()
}
}
w.Phases = append(w.Phases, ph)
pending = nil
last = &at
}
if len(pending) > 0 {
// The walk's last moments are unknown: it has no end.
if w.Said == "" {
w.Said = "its end is unknown: " + w.Phases[pending[0]].Name + " " + orNot(w.Phases[pending[0]].Said)
}
return w
}
if last == nil || from == nil {
return w
}
if p.Open() {
since := *last
w.Phases = append(w.Phases, WalkPhase{Name: PhaseOpen, Tier: -1, State: PhaseOpen, Start: &since,
TookMS: now.Sub(since).Milliseconds(), Took: words(now.Sub(since)), Said: "the walk is " + p.State})
w.Said = "open: " + p.State + ", " + words(now.Sub(*from)) + " since its merge"
return w
}
if p.State != PlanDone {
w.Said = "ended " + p.State + ": no delivery time"
return w
}
end := *last
w.End = &end
w.TotalMS = end.Sub(*from).Milliseconds()
w.Total = words(end.Sub(*from))
if w.Said == "" {
if w.UnknownMS > 0 {
w.Said = fmt.Sprintf("%s from its merge to running everywhere, %s of it unknown", w.Total,
words(time.Duration(w.UnknownMS)*time.Millisecond))
} else {
w.Said = w.Total + " from its merge to running everywhere"
}
}
return w
}
// firstMerge is when the walk's earliest merge was made, and every merge's start kept on w.
func (p Plan) firstMerge(w *WalkPhases) *time.Time {
var first *time.Time
if p.Delivery != nil {
for _, m := range p.Delivery.Merges {
if m.Merged.IsZero() {
continue
}
w.Merges = append(w.Merges, MergeStart{Repository: m.Repository, Commit: m.Commit, Merged: m.Merged})
if first == nil || m.Merged.Before(*first) {
at := m.Merged
first = &at
}
}
}
if first == nil {
for _, c := range p.Carried() {
if c.Merged.IsZero() {
continue
}
w.Merges = append(w.Merges, MergeStart{Repository: c.Repository, Commit: c.Commit, Merged: c.Merged})
if first == nil || c.Merged.Before(*first) {
at := c.Merged
first = &at
}
}
}
return first
}
// points are the walk's moments in order.
func (p Plan) points(now time.Time, applied AppliedLookup, w *WalkPhases) []point {
var out []point
// The window and the wait behind another walk.
cut := p.Created
if p.Times != nil && p.Times.Cut != nil {
cut = *p.Times.Cut
}
switch {
case p.Times == nil:
out = append(out, point{phase: PhaseWindow, tier: -1, said: "not kept for a walk made before ADR 0282"},
point{phase: PhaseQueued, tier: -1, at: &cut, said: "the window and the wait behind another walk together"})
case p.Times.WindowClosed == nil:
out = append(out, point{phase: PhaseWindow, tier: -1, none: true, said: "no window: walked on its own"},
point{phase: PhaseQueued, tier: -1, at: &cut})
default:
closed := *p.Times.WindowClosed
out = append(out, point{phase: PhaseWindow, tier: -1, at: &closed}, point{phase: PhaseQueued, tier: -1, at: &cut})
}
if p.Delivery != nil && p.Delivery.Awaits != "" {
out = append(out, point{phase: PhaseWord, tier: -1, at: p.Delivery.Go, said: "waiting for " + p.Delivery.Awaits + "'s word"})
} else {
out = append(out, point{phase: PhaseWord, tier: -1, none: true, said: "waits for no word"})
}
var lastSends []restSend
for t, tier := range p.Tiers {
if t > p.Tier || (t == p.Tier && p.Tier < len(p.Tiers) && !askedAnyOf(p, tier)) {
break
}
var asked, built, first, judged, rest *time.Time
allBuilt, anyFirst, allJudged, allRest := true, false, true, true
for _, m := range tier {
s := p.Modules[m]
if s == nil {
allBuilt, allRest = false, false
continue
}
asked = earliest(asked, s.AskedAt)
if s.BuiltAt == nil {
if s.State != "deleted" {
allBuilt = false
}
} else {
built = latest(built, s.BuiltAt)
}
if s.FirstAt != nil {
anyFirst = true
first = earliest(first, s.FirstAt)
if s.Gate != nil {
if s.Gate.JudgedAt == nil {
allJudged = false
} else {
judged = latest(judged, s.Gate.JudgedAt)
}
}
}
if s.SentAt == nil {
if s.State != "deleted" {
allRest = false
}
} else {
rest = latest(rest, s.SentAt)
for node := range s.Rest {
lastSends = append(lastSends, restSend{module: m, node: node, at: *s.SentAt})
}
}
}
if t > 0 {
out = append(out, point{phase: PhaseBetween, tier: t, at: asked})
}
if !allBuilt {
built = nil
}
out = append(out, point{phase: PhaseBuild, tier: t, at: built})
if anyFirst {
if !allJudged {
judged = nil
}
out = append(out, point{phase: PhaseSend, tier: t, at: first}, point{phase: PhaseJudge, tier: t, at: judged})
} else {
out = append(out, point{phase: PhaseSend, tier: t, none: true, said: "no first machine: nothing to judge"},
point{phase: PhaseJudge, tier: t, none: true, said: "no first machine: nothing to judge"})
}
if !allRest {
rest = nil
}
out = append(out, point{phase: PhaseRest, tier: t, at: rest})
}
if p.State != PlanDone {
return out
}
// Every machine of the rest running the build: its first report after the send, applied.
if p.Times == nil {
return append(out, point{phase: PhaseApply, tier: -1, said: "the rest's sends are not kept for a walk made before ADR 0282"})
}
if len(lastSends) == 0 {
return append(out, point{phase: PhaseApply, tier: -1, none: true,
said: "no machine beyond the first: the first-node gate's readings were its run"})
}
if applied == nil {
return append(out, point{phase: PhaseApply, tier: -1, said: "the rest's reports were not read"})
}
var end *time.Time
var waiting, failed []string
for _, s := range lastSends {
r, ok := applied(s.node, s.at)
switch {
case !ok && now.Sub(s.at) > ApplySilentAfter, ok && r.At.Sub(s.at) > ApplySilentAfter:
w.Silent = appendOnce(w.Silent, s.node)
case !ok:
waiting = appendOnce(waiting, s.node)
case r.Outcome != OutcomeApplied:
failed = appendOnce(failed, s.node+" ("+r.Outcome+")")
default:
at := r.At
end = latest(end, &at)
}
}
switch {
case len(failed) > 0:
return append(out, point{phase: PhaseApply, tier: -1, said: "not applied on " + strings.Join(failed, ", ")})
case len(waiting) > 0:
return append(out, point{phase: PhaseApply, tier: -1, said: "waiting for " + strings.Join(waiting, ", ") +
" to report it applied"})
case end == nil:
return append(out, point{phase: PhaseApply, tier: -1, said: "no machine of the rest heard from: " +
strings.Join(w.Silent, ", ")})
}
// A machine may have reported before the last tier ended: the walk ends at whichever is later.
for i := len(out) - 1; i >= 0; i-- {
if out[i].at != nil {
if out[i].at.After(*end) {
e := *out[i].at
end = &e
}
break
}
}
said := ""
if len(w.Silent) > 0 {
sort.Strings(w.Silent)
said = "not waited for, not heard from: " + strings.Join(w.Silent, ", ")
}
return append(out, point{phase: PhaseApply, tier: -1, at: end, said: said})
}
type restSend struct {
module, node string
at time.Time
}
func askedAnyOf(p Plan, tier []string) bool {
for _, m := range tier {
if s := p.Modules[m]; s != nil && s.AskedAt != nil {
return true
}
}
return false
}
func earliest(a, b *time.Time) *time.Time {
if b == nil {
return a
}
if a == nil || b.Before(*a) {
t := *b
return &t
}
return a
}
func latest(a, b *time.Time) *time.Time {
if b == nil {
return a
}
if a == nil || b.After(*a) {
t := *b
return &t
}
return a
}
func appendOnce(to []string, s string) []string {
for _, x := range to {
if x == s {
return to
}
}
return append(to, s)
}
func orNot(s string) string {
if s == "" {
return "is not known"
}
return "(" + s + ")"
}
// words is a duration as a person reads it.
func words(d time.Duration) string {
if d < 0 {
return "-" + words(-d)
}
return d.Round(100 * time.Millisecond).String()
}
// FirstAppliedAfter is a machine's first report of a send made at or after a walk's send to it, within
// ApplySilentAfter of it — the controller's `apply` durations, which measure every send to its first report (to-be
// 45 Phase 0). A declaration sent later carries the build too, so its report counts; one sent past the bound is a
// machine that slept, listed as silent and never moving the walk's end (ADR 0282 decision 1).
func (i *Inventory) FirstAppliedAfter(ctx context.Context, node string, sent time.Time) (AppliedReport, bool, error) {
var started time.Time
var ms int64
var outcome string
err := i.store.Pool().QueryRow(ctx,
`select started, took_ms, detail from duration
where kind = $1 and subject = $2 and started >= $3 and started < $4
order by started limit 1`,
DurationApply, node, sent.Add(-appliedSlack), sent.Add(ApplySilentAfter)).Scan(&started, &ms, &outcome)
if errors.Is(err, pgx.ErrNoRows) {
return AppliedReport{}, false, nil
}
if err != nil {
return AppliedReport{}, false, err
}
return AppliedReport{At: started.Add(time.Duration(ms) * time.Millisecond).UTC(), Outcome: outcome}, true, nil
}
// appliedSlack is how much earlier than a walk's record of its send the machine's own record of it may be:
// the send is recorded on the machine first, then on the walk.
const appliedSlack = 2 * time.Second
+330
View File
@@ -0,0 +1,330 @@
package inventory
import (
"encoding/json"
"strings"
"testing"
"time"
)
// A walk recorded on the live mesh on 2026-10-10 (mesh-delivery, one tier, one machine), as `delivery walks`
// gave it, with the moments ADR 0282 adds: its window closed ninety seconds after its merge was heard, and it was
// cut eleven seconds later.
const recordedWalk = `{
"id": "plan-1791654663505629616", "repository": "novox/mesh-catalog", "branch": "main",
"commit": "a14fa306f262d88bdcca83432a70df353d2eceb0", "merged": "2026-10-10T17:50:53Z",
"created": "2026-10-10T19:52:34.037755+02:00", "updated": "2026-10-10T19:55:37.974069+02:00",
"state": "done", "tier": 1, "tiers": [["mesh-delivery"]],
"modules": {"mesh-delivery": {"state": "built",
"asked_at": "2026-10-10T17:52:34.209423145Z", "built_at": "2026-10-10T17:52:50.818011314Z",
"sent_at": "2026-10-10T17:55:35.894985053Z", "first": ["novox"], "first_at": "2026-10-10T17:53:07.287136827Z",
"gate": {"machines": ["novox"], "since": "2026-10-10T17:53:07.287136827Z", "passes": 3,
"verdict": "passed", "judged_at": "2026-10-10T17:55:35.894985053Z"}}},
"delivery": {"awaits": "", "merges": [{"repository": "novox/mesh-catalog",
"commit": "a14fa306f262d88bdcca83432a70df353d2eceb0", "number": 187, "merged": "2026-10-10T17:50:53Z"}]},
"times": {"window_closed": "2026-10-10T17:52:23Z", "cut": "2026-10-10T17:52:34.037755Z", "class": "core"}
}`
func walkOf(t *testing.T, raw string) Plan {
t.Helper()
var p Plan
if err := json.Unmarshal([]byte(raw), &p); err != nil {
t.Fatal(err)
}
return p
}
// sumsUp checks the phases' measured time and the unknown time add up to the total, to the millisecond.
func sumsUp(t *testing.T, w *WalkPhases) {
t.Helper()
if w.End == nil || w.From == nil {
t.Fatalf("the walk has no end: %+v", w)
}
var sum int64
for _, ph := range w.Phases {
if ph.State == PhaseMeasured {
if ph.Start == nil || ph.End == nil || ph.End.Sub(*ph.Start).Milliseconds() != ph.TookMS {
t.Errorf("phase %s %d says %dms between %v and %v", ph.Name, ph.Tier, ph.TookMS, ph.Start, ph.End)
}
sum += ph.TookMS
}
if ph.State == PhaseUnknown && ph.TookMS != 0 {
t.Errorf("an unknown phase %s says a time: %dms", ph.Name, ph.TookMS)
}
}
if sum+w.UnknownMS != w.TotalMS || w.End.Sub(*w.From).Milliseconds() != w.TotalMS {
t.Fatalf("the phases add up to %dms and %dms unknown, the total is %dms (%s): %+v", sum, w.UnknownMS,
w.TotalMS, w.End.Sub(*w.From), w.Phases)
}
}
func phase(w *WalkPhases, name string, tier int) WalkPhase {
for _, ph := range w.Phases {
if ph.Name == name && ph.Tier == tier {
return ph
}
}
return WalkPhase{}
}
// The phases of a recorded walk add up to its measured total: its merge at 17:50:53 to its verdict on its only
// machine at 17:55:35.894, 4m42.894s.
func TestARecordedWalksPhasesAddUpToItsTotal(t *testing.T) {
w := walkOf(t, recordedWalk).WalkPhases(time.Date(2026, 10, 10, 18, 0, 0, 0, time.UTC), nil)
sumsUp(t, w)
if w.TotalMS != 282894 || w.Class != ClassCore || w.UnknownMS != 0 {
t.Fatalf("total %dms (want 282894), class %q, unknown %d", w.TotalMS, w.Class, w.UnknownMS)
}
for name, want := range map[string]int64{PhaseWindow: 90000, PhaseQueued: 11037, PhaseBuild: 16781,
PhaseSend: 16469, PhaseJudge: 148607, PhaseRest: 0} {
tier := 0
if name == PhaseWindow || name == PhaseQueued {
tier = -1
}
if got := phase(w, name, tier); got.State != PhaseMeasured || got.TookMS != want {
t.Errorf("%s: %s %dms, want measured %dms", name, got.State, got.TookMS, want)
}
}
// The cut to the first ask (171ms) is a time no phase of ADR 0282's table names: it is counted in the build.
if got := phase(w, PhaseWord, -1); got.State != PhaseNone {
t.Errorf("a walk that waits for no word says its word phase %s", got.State)
}
if got := phase(w, PhaseApply, -1); got.State != PhaseNone || !strings.Contains(got.Said, "no machine beyond the first") {
t.Errorf("one machine only: apply %s (%s)", got.State, got.Said)
}
}
// Two tiers, a machine of the rest each, both reports read: every phase measured, the walk ends at the last
// report applied, and the phases add up.
func TestAWalkOfTwoTiersEndsAtItsRestsLastReport(t *testing.T) {
at := func(s string) *time.Time {
v, err := time.Parse(time.RFC3339Nano, "2026-10-10T18:"+s+"Z")
if err != nil {
t.Fatal(err)
}
return &v
}
p := Plan{ID: "plan-2", State: PlanDone, Tier: 2, Tiers: [][]string{{"a"}, {"b"}}, Created: *at("01:40"),
Delivery: &PlanDelivery{Merges: []PlanMerge{{Repository: "novox/x", Commit: "c1", Merged: *at("00:00")},
{Repository: "novox/x", Commit: "c0", Merged: *at("00:30")}}},
Times: &PlanTimes{WindowClosed: at("01:30"), Cut: at("01:40"), Class: ClassLeaf},
Modules: map[string]*PlanModule{
"a": {State: "built", AskedAt: at("01:41"), BuiltAt: at("02:05"), FirstAt: at("02:20"), SentAt: at("05:00"),
Gate: &PlanGate{JudgedAt: at("04:50")}, Rest: map[string]SentDeclaration{"ace": {Digest: "d1"}}},
"b": {State: "built", AskedAt: at("05:10"), BuiltAt: at("05:40"), FirstAt: at("05:50"), SentAt: at("08:00.5"),
Gate: &PlanGate{JudgedAt: at("07:59")}, Rest: map[string]SentDeclaration{"shanks": {Digest: "d2"}}},
}}
reports := map[string]AppliedReport{"ace": {At: *at("05:12"), Outcome: OutcomeApplied},
"shanks": {At: *at("08:15.25"), Outcome: OutcomeApplied}}
w := p.WalkPhases(*at("30:00"), func(node string, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
sumsUp(t, w)
if w.TotalMS != (8*time.Minute + 15250*time.Millisecond).Milliseconds() {
t.Fatalf("the walk's delivery time runs from its earliest merge to the last report: %s", w.Total)
}
if got := phase(w, PhaseBetween, 1); got.TookMS != 10000 {
t.Errorf("between the tiers: %dms", got.TookMS)
}
if got := phase(w, PhaseApply, -1); got.State != PhaseMeasured || got.TookMS != 14750 {
t.Errorf("apply: %s %dms", got.State, got.TookMS)
}
if len(w.Merges) != 2 {
t.Errorf("each merge's start is said, for its own delivery time: %+v", w.Merges)
}
// A report not yet in: the walk has no end, and its apply is unknown — never zero.
delete(reports, "shanks")
w = p.WalkPhases(*at("10:00"), func(node string, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
if w.End != nil || w.TotalMS != 0 {
t.Fatalf("a walk whose rest has not reported has no end: %+v", w)
}
if got := phase(w, PhaseApply, -1); got.State != PhaseUnknown || got.TookMS != 0 || !strings.Contains(got.Said, "shanks") {
t.Errorf("apply while shanks has not reported: %+v", got)
}
// Silent past the bound: listed, not waited for.
w = p.WalkPhases(at("08:00.5").Add(ApplySilentAfter+time.Second), func(node string, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
sumsUp(t, w)
if len(w.Silent) != 1 || w.Silent[0] != "shanks" {
t.Errorf("a machine silent past the bound is said, not waited for: %+v", w.Silent)
}
// A failed report: the walk has no end, said.
reports["shanks"] = AppliedReport{At: *at("08:10"), Outcome: OutcomeFailed}
w = p.WalkPhases(*at("30:00"), func(node string, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
if w.End != nil || !strings.Contains(phase(w, PhaseApply, -1).Said, "shanks (failed)") {
t.Errorf("a failed apply ends nothing: %+v", phase(w, PhaseApply, -1))
}
}
// A moment that is missing makes the phases around it unknown, and their span unknown time: never a zero.
func TestAMissingMomentIsUnknownNeverZero(t *testing.T) {
p := walkOf(t, recordedWalk)
p.Modules["mesh-delivery"].BuiltAt = nil
w := p.WalkPhases(time.Date(2026, 10, 10, 18, 0, 0, 0, time.UTC), nil)
sumsUp(t, w)
if got := phase(w, PhaseBuild, 0); got.State != PhaseUnknown || got.TookMS != 0 {
t.Errorf("a build without its moment: %+v", got)
}
if got := phase(w, PhaseSend, 0); got.State != PhaseUnknown {
t.Errorf("the send after a build without its moment: %+v", got)
}
if w.UnknownMS != 16781+16469 {
t.Errorf("the unknown time is the span between the moments around it: %dms", w.UnknownMS)
}
// A walk kept before ADR 0282: its window and its wait together, and its apply unknown.
p = walkOf(t, recordedWalk)
p.Times = nil
w = p.WalkPhases(time.Date(2026, 10, 10, 18, 0, 0, 0, time.UTC), nil)
if got := phase(w, PhaseWindow, -1); got.State != PhaseUnknown {
t.Errorf("a window not kept: %+v", got)
}
if got := phase(w, PhaseApply, -1); got.State != PhaseUnknown || w.End != nil {
t.Errorf("a rest not kept: %+v, end %v", got, w.End)
}
}
// An open walk is said open since its last moment, with no end.
func TestAnOpenWalkHasNoEnd(t *testing.T) {
p := walkOf(t, recordedWalk)
p.State, p.Tier = PlanRolling, 0
p.Modules["mesh-delivery"].SentAt = nil
p.Modules["mesh-delivery"].Gate.JudgedAt = nil
w := p.WalkPhases(time.Date(2026, 10, 10, 17, 54, 0, 0, time.UTC), nil)
if w.End != nil || !strings.HasPrefix(w.Said, "its end is unknown") && !strings.HasPrefix(w.Said, "open") {
t.Fatalf("an open walk: %+v", w)
}
if got := phase(w, PhaseBuild, 0); got.State != PhaseMeasured {
t.Errorf("an open walk's phases so far are measured: %+v", got)
}
}
// A gate keeps every pass and the first reading after one that did not pass, at most maxReadings.
func TestAGateKeepsItsReadings(t *testing.T) {
var g PlanGate
t0 := time.Date(2026, 10, 10, 18, 0, 0, 0, time.UTC)
g.Read(t0, false, "starting")
g.Read(t0.Add(time.Second), false, "starting")
g.Read(t0.Add(40*time.Second), true, "")
g.Read(t0.Add(80*time.Second), true, "")
g.Read(t0.Add(81*time.Second), false, "gone")
g.Read(t0.Add(82*time.Second), false, "gone")
if len(g.Readings) != 4 || g.Readings[0].Said != "starting" || !g.Readings[2].Healthy || g.Readings[3].Said != "gone" {
t.Fatalf("readings: %+v", g.Readings)
}
for i := 0; i < 50; i++ {
g.Read(t0.Add(time.Duration(100+i)*time.Second), true, "")
}
if len(g.Readings) != maxReadings {
t.Fatalf("readings are bounded: %d", len(g.Readings))
}
}
// A walk's moments are kept and read back with it; a machine's first report after a send is read from the
// controller's apply durations.
func TestAWalksMomentsAreKeptAndItsRestsReportRead(t *testing.T) {
inv := ForTest(t)
ctx := t.Context()
p := walkOf(t, recordedWalk)
p.Revision, p.Epoch = 0, 0
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
back, err := inv.PlanByID(ctx, p.ID)
if err != nil || back.Times == nil || back.Times.Class != ClassCore || back.Times.WindowClosed == nil ||
!back.Times.WindowClosed.Equal(*p.Times.WindowClosed) {
t.Fatalf("the walk's moments were not kept: %v %+v", err, back.Times)
}
if back.Delivery.Merges[0].Merged.IsZero() {
t.Fatal("a merge's time was not kept")
}
sent := time.Now().UTC().Add(-time.Minute).Truncate(time.Millisecond)
if _, ok, err := inv.FirstAppliedAfter(ctx, "ace", sent); err != nil || ok {
t.Fatalf("no report yet: %v %v", ok, err)
}
for _, d := range []Duration{
{Kind: DurationApply, Subject: "ace", Node: "ace", Ref: "old@1", Started: sent.Add(-time.Hour), Took: time.Second, Detail: OutcomeApplied},
{Kind: DurationApply, Subject: "ace", Node: "ace", Ref: "d1@2", Started: sent.Add(-time.Second), Took: 12 * time.Second, Detail: OutcomeApplied},
} {
if err := inv.RecordDuration(ctx, d); err != nil {
t.Fatal(err)
}
}
r, ok, err := inv.FirstAppliedAfter(ctx, "ace", sent)
if err != nil || !ok || r.Outcome != OutcomeApplied || !r.At.Equal(sent.Add(11*time.Second)) {
t.Fatalf("the report after the send: %+v %v %v", r, ok, err)
}
if anyKept, total, err := inv.WalkPhasesKept(ctx, p.ID); err != nil || anyKept || total {
t.Fatalf("nothing kept yet: %v %v %v", anyKept, total, err)
}
if err := inv.RecordDuration(ctx, Duration{Kind: DurationWalkPhase, Subject: "core/total", Ref: p.ID + "/total",
Started: sent, Took: time.Minute}); err != nil {
t.Fatal(err)
}
if anyKept, total, err := inv.WalkPhasesKept(ctx, p.ID); err != nil || !anyKept || !total {
t.Fatalf("the total kept: %v %v %v", anyKept, total, err)
}
}
// A machine that slept past the bound and reported later is listed silent: its late report never moves the
// walk's end; and a report sent past the bound is not read as the walk's.
func TestALateReportIsSilentNotTheEnd(t *testing.T) {
sent := time.Date(2026, 10, 10, 18, 0, 0, 0, time.UTC)
p := walkOf(t, recordedWalk)
p.Modules["mesh-delivery"].SentAt = &sent
p.Modules["mesh-delivery"].Rest = map[string]SentDeclaration{"laptop": {Digest: "d"}, "server": {Digest: "e"}}
w := p.WalkPhases(sent.Add(3*time.Hour), func(node string, _ time.Time) (AppliedReport, bool) {
if node == "server" {
return AppliedReport{At: sent.Add(20 * time.Second), Outcome: OutcomeApplied}, true
}
return AppliedReport{At: sent.Add(2 * time.Hour), Outcome: OutcomeApplied}, true
})
if len(w.Silent) != 1 || w.Silent[0] != "laptop" || w.End == nil || !w.End.Equal(sent.Add(20*time.Second)) {
t.Fatalf("a late report: silent %v, end %v", w.Silent, w.End)
}
inv := ForTest(t)
ctx := t.Context()
if err := inv.RecordDuration(ctx, Duration{Kind: DurationApply, Subject: "laptop", Node: "laptop", Ref: "late@1",
Started: sent.Add(time.Hour), Took: time.Second, Detail: OutcomeApplied}); err != nil {
t.Fatal(err)
}
if _, ok, err := inv.FirstAppliedAfter(ctx, "laptop", sent); err != nil || ok {
t.Fatalf("a send past the bound read as the walk's: %v %v", ok, err)
}
}
// Moments out of order — #206's own walk on 2026-10-10 kept its first send 783 ms before its build — are said
// unknown, never a negative phase: the phase before is unknown and its time joins the unknown time, the phase
// out of order has no time, and the phases still add up to the total.
func TestMomentsOutOfOrderAreUnknownNeverNegative(t *testing.T) {
p := walkOf(t, recordedWalk)
built := time.Date(2026, 10, 10, 17, 53, 8, 70000000, time.UTC) // 783 ms after its first send
p.Modules["mesh-delivery"].BuiltAt = &built
w := p.WalkPhases(time.Date(2026, 10, 10, 18, 0, 0, 0, time.UTC), nil)
sumsUp(t, w)
for _, ph := range w.Phases {
if ph.TookMS < 0 {
t.Fatalf("a negative phase: %+v", ph)
}
}
build, send := phase(w, PhaseBuild, 0), phase(w, PhaseSend, 0)
if build.State != PhaseUnknown || send.State != PhaseUnknown || send.TookMS != 0 ||
!strings.Contains(send.Said, "out of order") || !strings.Contains(build.Said, "out of order") {
t.Fatalf("build %+v, send-first %+v", build, send)
}
// The build's span (cut to its later moment) is unknown time; the judgement is measured from the later moment.
if w.UnknownMS != built.Sub(time.Date(2026, 10, 10, 17, 52, 34, 37000000, time.UTC)).Milliseconds() {
t.Fatalf("unknown %dms", w.UnknownMS)
}
if j := phase(w, PhaseJudge, 0); j.State != PhaseMeasured || j.Start == nil || !j.Start.Equal(built) {
t.Fatalf("the judgement is measured from the later moment: %+v", j)
}
}
+25
View File
@@ -468,6 +468,31 @@ func kept(args json.RawMessage) json.RawMessage {
if words, ok := v.([]any); ok && len(words) > 0 { if words, ok := v.([]any); ok && len(words) > 0 {
out[k] = []any{words[0], "(the rest given, not kept)"} out[k] = []any{words[0], "(the rest given, not kept)"}
} }
case k == "command":
// The controller's own command line: its first word, never the rest, as a mesh-cli line's
// first word is. `settings set <module> '<value>' --node <node>` through the `command` verb
// put the value itself in this record, which answers anyone who may call the seat (novox/hq
// issue 397). Its words are parted by any whitespace, because the verb's own splitter parts
// them on a space, a tab or a newline, and a line written with tabs is the same line. The
// first word is capped as every other string here is: the record is written before the verb
// judges the line, so one word may be anything a caller sent, a token included.
if !isString {
out[k] = "(given, not kept)"
break
}
words := strings.Fields(s)
if len(words) == 0 {
out[k] = ""
break
}
first := words[0]
if len(first) > 120 {
first = first[:120] + "…"
}
if len(words) > 1 {
first += " (the rest given, not kept)"
}
out[k] = first
case isString && len(s) <= 120: case isString && len(s) <= 120:
out[k] = s out[k] = s
case isString: case isString:
+50
View File
@@ -11,6 +11,7 @@ import (
"sync" "sync"
"testing" "testing"
"time" "time"
"unicode"
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
@@ -166,6 +167,55 @@ func TestACallKeepsNoSettingsOrSecrets(t *testing.T) {
} }
} }
// A command line's own words may carry a setting's value or a secret, so only its first word is kept
// — as a mesh-cli line's is (novox/hq issue 397). Each case says the whole of what is kept, because
// what matters is as much what is left out as what is there.
func TestACallKeepsNoCommandLine(t *testing.T) {
long := strings.Repeat("s3cret", 50) // one word, 300 bytes: a token, as far as this can tell
for line, want := range map[string]string{
`settings set notes 'the operator's own passphrase' --node laptop`: "settings (the rest given, not kept)",
`settings set notes --values {"token":"s3cret"}`: "settings (the rest given, not kept)",
// Parted by a tab or a newline, which the verb's splitter reads as this line's words too.
"settings\tset\tnotes\tthe-operators-passphrase": "settings (the rest given, not kept)",
"builds\n--limit 5": "builds (the rest given, not kept)",
"builds\t--limit 5": "builds (the rest given, not kept)",
// Its first word is kept, and the spaces before it are not part of it.
` node show laptop`: "node (the rest given, not kept)",
// A command of one word is kept whole: there is nothing after it to withhold.
`builds`: "builds",
// One word is still capped, as every other string in a kept record is.
long: long[:120] + "…",
} {
raw, err := json.Marshal(map[string]any{"command": line})
if err != nil {
t.Fatal(err)
}
var got struct{ Command string }
if err := json.Unmarshal(kept(raw), &got); err != nil {
t.Fatal(err)
}
if got.Command != want {
t.Errorf("%q is kept as %q; want %q", line, got.Command, want)
}
// Whole words, so that "set" can be looked for although "settings" is kept, and "show" cannot
// be found in a word such as "shown".
for _, never := range []string{"set", "notes", "laptop", "show", "passphrase", "operators"} {
for _, word := range strings.FieldsFunc(got.Command, func(r rune) bool { return !unicode.IsLetter(r) }) {
if word == never {
t.Errorf("%q is kept as %q: it carries the word %q", line, got.Command, never)
}
}
}
}
// Not a string, so its first word cannot be taken: none of it is kept. The verb refuses such a
// call, but the record is written before it judges it.
raw, _ := json.Marshal(map[string]any{"command": []string{"settings", "set", "notes", "s3cret"}})
if got := string(kept(raw)); strings.Contains(got, "s3cret") {
t.Errorf("kept %s", got)
}
}
// Only the newest KeptCalls are kept. // Only the newest KeptCalls are kept.
func TestTheLogKeepsTheNewest(t *testing.T) { func TestTheLogKeepsTheNewest(t *testing.T) {
l := NewCallLog() l := NewCallLog()
+86 -7
View File
@@ -62,6 +62,9 @@ type DeadLetter struct {
// taken. The record of it is kept all the same, so it is said and dropped, never silently missing. // taken. The record of it is kept all the same, so it is said and dropped, never silently missing.
Lost string `json:"lost,omitempty"` Lost string `json:"lost,omitempty"`
Size int `json:"size"` Size int `json:"size"`
// Original says what became of the ask it was kept from, in its seat's work queue, when it was
// delivered again (novox/hq issue 334).
Original string `json:"original,omitempty"`
// Body and Headers are the message itself, given only for one dead letter asked by its id; a header // Body and Headers are the message itself, given only for one dead letter asked by its id; a header
// with several values keeps them all. // with several values keeps them all.
Body string `json:"body,omitempty"` Body string `json:"body,omitempty"`
@@ -167,6 +170,29 @@ func HeldDeadLetters(js nats.JetStreamContext) (map[string]int, error) {
return held, nil return held, nil
} }
// NewestDeadLetters is when DEAD_LETTERS kept the newest message it holds for each consumer of held, by
// `<stream>.<consumer>`: the stored time of its last message, which rises with the stream's sequence. A
// consumer whose letters were all delivered again or dropped since held was read is left out. The
// consumer's condition counts the messages given up on by it, never how often the controller looked at
// them (novox/hq issue 440), so it needs a time that is newer for every message newly kept. The time the
// server said it gave up is not one: a notice that could not be kept is offered again a minute later
// (noticeRetry), so a later give-up can be kept first, and the one kept after it carries an older time.
func NewestDeadLetters(js nats.JetStreamContext, held map[string]int) (map[string]time.Time, error) {
newest := map[string]time.Time{}
for key := range held {
stream, consumer, _ := strings.Cut(key, ".")
raw, err := js.GetLastMsg(broker.DeadLettersStream, broker.DeadLetterSubject(stream, consumer))
if errors.Is(err, nats.ErrMsgNotFound) {
continue // delivered again or dropped since it was counted
}
if err != nil {
return nil, fmt.Errorf("the newest dead letter of %s cannot be read: %w", key, err)
}
newest[key] = raw.Time.UTC()
}
return newest, nil
}
// DeadLetters lists what DEAD_LETTERS holds, newest first, at most most of them (all when most is not // DeadLetters lists what DEAD_LETTERS holds, newest first, at most most of them (all when most is not
// positive); for one consumer when consumer names one (its name, or `<stream>.<consumer>`). The total is // positive); for one consumer when consumer names one (its name, or `<stream>.<consumer>`). The total is
// the stream's own count per consumer, so it is right however few are read. // the stream's own count per consumer, so it is right however few are read.
@@ -227,9 +253,15 @@ func DeadLetterNamed(js nats.JetStreamContext, id uint64) (DeadLetter, error) {
// AgainTo is where a kept message is delivered again so that only the consumer that gave it up gets // AgainTo is where a kept message is delivered again so that only the consumer that gave it up gets
// it: an event under that consumer's own again subject on EVENTS — and only when the consumer exists and // it: an event under that consumer's own again subject on EVENTS — and only when the consumer exists and
// filters that subject, so a message is never let go as delivered while nobody receives it. Any other // filters that subject, so a message is never let go as delivered while nobody receives it.
// stream's message is refused, with why: an ask given up on by a seat's worker is not delivered again yet //
// (novox/hq issue 330's follow-up), since publishing it again leaves the original stuck in the queue. // An ask the controller itself made (broker.TheControllersAsk) goes back on its own subject: a seat's
// work queue has one worker per subject, so only the worker that gave it up takes it, and DeliverAgain
// removes the original from the queue first, so there are never two (novox/hq issue 334); only while the
// worker that gave it up is on the bus. Any other stream's
// message is refused, with why: an ask to a seat the controller does not ask would need a publish it is
// not granted, in its asker's name (ADR 0264's consequences, ADR 0259 §3), and a stream that is neither
// would reach every consumer of its subject.
func AgainTo(js nats.JetStreamContext, d DeadLetter) (string, error) { func AgainTo(js nats.JetStreamContext, d DeadLetter) (string, error) {
switch { switch {
case d.Lost != "": case d.Lost != "":
@@ -237,10 +269,22 @@ func AgainTo(js nats.JetStreamContext, d DeadLetter) (string, error) {
case d.Subject == "": case d.Subject == "":
return "", fmt.Errorf("dead letter %d does not say the subject it was published on, so it cannot be "+ return "", fmt.Errorf("dead letter %d does not say the subject it was published on, so it cannot be "+
"delivered again. Drop it", d.ID) "delivered again. Drop it", d.ID)
case broker.TheControllersAsk(d.Stream, d.Subject):
// The seat's worker takes it, and only while it is on the bus: without one the queue would keep
// the ask for a holder that may never come, and it would be let go as delivered meanwhile.
if _, err := js.ConsumerInfo(d.Stream, d.Consumer); errors.Is(err, nats.ErrConsumerNotFound) {
return "", fmt.Errorf("dead letter %d was given up by %s, which is no longer on the bus, so the ask "+
"would only wait in %s for a holder. Nothing was done, and it is still kept: deliver it again once "+
"the seat has a holder, or drop it", d.ID, d.Who, d.Stream)
} else if err != nil {
return "", fmt.Errorf("whether %s can receive dead letter %d cannot be read: %w", d.Who, d.ID, err)
}
return d.Subject, nil
case d.Stream != broker.EventsStream: case d.Stream != broker.EventsStream:
return "", fmt.Errorf("dead letter %d is from %s, and only an event is delivered again: publishing it "+ return "", fmt.Errorf("dead letter %d is from %s, and only an event or an ask the controller made is "+
"again would reach every consumer of its subject, or leave the original in its queue. Drop it, and "+ "delivered again: publishing it again would reach every consumer of its subject, or need a grant the "+
"have its sender say it again", d.ID, d.Stream) "controller does not hold to speak for its asker. Drop it, and have its sender say it again",
d.ID, d.Stream)
} }
info, err := js.ConsumerInfo(d.Stream, d.Consumer) info, err := js.ConsumerInfo(d.Stream, d.Consumer)
if errors.Is(err, nats.ErrConsumerNotFound) { if errors.Is(err, nats.ErrConsumerNotFound) {
@@ -261,7 +305,7 @@ func AgainTo(js nats.JetStreamContext, d DeadLetter) (string, error) {
} }
// DeliverAgain hands a kept message to the consumer that gave it up, and nobody else, then removes it // DeliverAgain hands a kept message to the consumer that gave it up, and nobody else, then removes it
// from DEAD_LETTERS. The message carries its own headers and AgainHeader; its de-duplication id is the // from DEAD_LETTERS; an ask's original is removed from its work queue before. The message carries its own headers and AgainHeader; its de-duplication id is the
// kept copy's, so asking twice delivers it once. // kept copy's, so asking twice delivers it once.
func DeliverAgain(js nats.JetStreamContext, id uint64) (DeadLetter, string, error) { func DeliverAgain(js nats.JetStreamContext, id uint64) (DeadLetter, string, error) {
d, err := DeadLetterNamed(js, id) d, err := DeadLetterNamed(js, id)
@@ -272,6 +316,9 @@ func DeliverAgain(js nats.JetStreamContext, id uint64) (DeadLetter, string, erro
if err != nil { if err != nil {
return d, "", err return d, "", err
} }
if d.Stream != broker.EventsStream {
d.Original = removeOriginal(js, d)
}
again := &nats.Msg{Subject: to, Header: nats.Header{}, Data: []byte(d.Body)} again := &nats.Msg{Subject: to, Header: nats.Header{}, Data: []byte(d.Body)}
for k, v := range d.Headers { for k, v := range d.Headers {
again.Header[k] = append([]string(nil), v...) again.Header[k] = append([]string(nil), v...)
@@ -279,6 +326,10 @@ func DeliverAgain(js nats.JetStreamContext, id uint64) (DeadLetter, string, erro
again.Header.Set(AgainHeader, strconv.FormatUint(id, 10)) again.Header.Set(AgainHeader, strconv.FormatUint(id, 10))
again.Header.Set(nats.MsgIdHdr, "again."+broker.DeadLettersStream+"."+strconv.FormatUint(id, 10)) again.Header.Set(nats.MsgIdHdr, "again."+broker.DeadLettersStream+"."+strconv.FormatUint(id, 10))
if _, err := js.PublishMsg(again); err != nil { if _, err := js.PublishMsg(again); err != nil {
if d.Original != "" {
return d, to, fmt.Errorf("dead letter %d could not be delivered again on %s: %w; it is still kept, so "+
"delivering it again tries once more. Its original: %s", id, to, err, d.Original)
}
return d, to, fmt.Errorf("dead letter %d could not be delivered again on %s: %w; it is still kept", id, to, err) return d, to, fmt.Errorf("dead letter %d could not be delivered again on %s: %w; it is still kept", id, to, err)
} }
if err := js.DeleteMsg(broker.DeadLettersStream, id); err != nil && !errors.Is(err, nats.ErrMsgNotFound) { if err := js.DeleteMsg(broker.DeadLettersStream, id); err != nil && !errors.Is(err, nats.ErrMsgNotFound) {
@@ -288,6 +339,34 @@ func DeliverAgain(js nats.JetStreamContext, id uint64) (DeadLetter, string, erro
return d, to, nil return d, to, nil
} }
// removeOriginal takes the ask a dead letter was kept from out of its seat's work queue, and says what
// became of it. Given up on, the original is never acknowledged and would stay beside its copy until the
// stream's age drops it (novox/hq issue 334). It is removed before the copy is published, so a copy that
// cannot be published leaves the kept one to try again, and never two.
//
// **Only when the queue still holds that same message**: its subject and the time it was stored are the
// dead letter's. A seat's stream deleted and made again — the build handover deletes one (builds.go) —
// numbers from one again, and an old dead letter's sequence may then name another, live ask; deleting by
// the number alone would drop that one silently. Anything else is said, never an error: the original is
// gone or is not this one, and the copy is the only one there will be.
func removeOriginal(js nats.JetStreamContext, d DeadLetter) string {
held, err := js.GetMsg(d.Stream, d.Sequence)
switch {
case errors.Is(err, nats.ErrMsgNotFound):
return fmt.Sprintf("%s no longer held message %d, so there was nothing to remove", d.Stream, d.Sequence)
case err != nil:
return fmt.Sprintf("message %d of %s could not be read, so it was left as it is: %v", d.Sequence, d.Stream, err)
case d.Published.IsZero() || held.Subject != d.Subject || !held.Time.Equal(d.Published):
return fmt.Sprintf("message %d of %s is another message now (%s, stored %s), so it was left as it is; "+
"the one given up on is gone", d.Sequence, d.Stream, held.Subject, held.Time.UTC().Format(time.RFC3339))
}
if err := js.DeleteMsg(d.Stream, d.Sequence); err != nil && !errors.Is(err, nats.ErrMsgNotFound) {
return fmt.Sprintf("message %d of %s could not be removed, so it stays beside its copy until the "+
"stream's age drops it; nothing delivers it again: %v", d.Sequence, d.Stream, err)
}
return fmt.Sprintf("message %d of %s, the one given up on, was removed from the queue", d.Sequence, d.Stream)
}
// DropDeadLetter removes a kept message for good. // DropDeadLetter removes a kept message for good.
func DropDeadLetter(js nats.JetStreamContext, id uint64) (DeadLetter, error) { func DropDeadLetter(js nats.JetStreamContext, id uint64) (DeadLetter, error) {
d, err := DeadLetterNamed(js, id) d, err := DeadLetterNamed(js, id)
+217
View File
@@ -0,0 +1,217 @@
package link
import (
"errors"
"strings"
"testing"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
)
// What a seat's worker gave up on, when the ask is one the controller itself makes (novox/hq issue 334).
//
// The controller already publishes on the accept subjects of the seats it asks (broker's
// seatsTheControllerAsks) and reaches the stream API, so delivering its own ask again needs no grant it
// does not hold: the original is removed from the work queue by its sequence, and the kept copy is
// published on the ask's own subject, where the seat's one worker takes it. An ask to any other seat is
// still refused, and stays kept: delivering it would need a publish the controller is not granted
// (ADR 0264's consequences), in the asker's name (ADR 0259 §3).
// theBuildWorker is the build seat's worker as a holder pulls from it.
func theBuildWorker(t *testing.T, js *broker.JetStream) jetstream.Consumer {
t.Helper()
api, err := jetstream.New(js.Conn())
if err != nil {
t.Fatal(err)
}
stream := "SEAT_NODE_BUILD_AGENT"
worker, err := api.Consumer(t.Context(), stream, stream+"_worker")
if err != nil {
t.Fatal(err)
}
return worker
}
func TestAnAskTheControllerMadeIsDeliveredAgainToTheSeatsWorker(t *testing.T) {
js := aBusWithTheBuildRole(t)
keeping(t, js)
worker := theBuildWorker(t, js)
subject := BuildWorkOf(TheBuildMachine)
ask := &nats.Msg{Subject: subject, Data: []byte(`{"module":"x"}`), Header: nats.Header{}}
ask.Header.Set(nats.MsgIdHdr, "build-1")
if _, err := js.Context().PublishMsg(ask); err != nil {
t.Fatal(err)
}
d := givenUp(t, js, worker)
if d.Stream != "SEAT_NODE_BUILD_AGENT" || d.Subject != subject || d.Lost != "" {
t.Fatalf("kept as %+v", d)
}
if _, err := js.Context().GetMsg(d.Stream, d.Sequence); err != nil {
t.Fatalf("the original is not in the work queue before it is delivered again: %v", err)
}
_, to, err := DeliverAgain(js.Context(), d.ID)
if err != nil {
t.Fatal(err)
}
if to != subject {
t.Fatalf("delivered again on %s, not the ask's own subject %s", to, subject)
}
if _, err := js.Context().GetMsg(d.Stream, d.Sequence); !errors.Is(err, nats.ErrMsgNotFound) {
t.Fatalf("the original is still in the work queue beside its copy: %v", err)
}
again := next(t, worker)
if again == nil {
t.Fatal("the seat's worker was not handed the ask again")
}
if string(again.Data()) != `{"module":"x"}` || again.Headers().Get(AgainHeader) == "" {
t.Fatalf("handed again as %s %v", again.Data(), again.Headers())
}
_ = again.Ack()
if m := next(t, worker); m != nil {
t.Fatalf("the worker was handed it twice: %s", m.Subject())
}
if _, err := DeadLetterNamed(js.Context(), d.ID); !errors.Is(err, ErrNoDeadLetter) {
t.Fatalf("still kept after it was delivered again: %v", err)
}
if _, _, err := DeliverAgain(js.Context(), d.ID); !errors.Is(err, ErrNoDeadLetter) {
t.Fatalf("a second delivery answered %v", err)
}
}
// An ask to a seat the controller does not ask is refused, says why, and nothing is done.
func TestAnAskTheControllerDidNotMakeIsStillOnlyDropped(t *testing.T) {
js := aBus(t)
for _, d := range []DeadLetter{
{ID: 2, Stream: "SEAT_TELEGRAM_SENDER", Consumer: "SEAT_TELEGRAM_SENDER_worker",
Subject: "mesh.seat.telegram-sender.accept.send"},
// The subject of a seat the controller asks, on a stream that is not that seat's queue.
{ID: 3, Stream: "SEAT_TELEGRAM_SENDER", Consumer: "SEAT_TELEGRAM_SENDER_worker",
Subject: "mesh.seat.node-build-agent.accept.build"},
} {
if to, err := AgainTo(js.Context(), d); err == nil {
t.Errorf("%s on %s was given %s to be delivered again on", d.Subject, d.Stream, to)
} else if !strings.Contains(err.Error(), "Drop it") {
t.Errorf("refused without saying what to do: %v", err)
}
}
}
// aBuildAskGivenUp publishes one build ask and lets the worker give it up.
func aBuildAskGivenUp(t *testing.T, js *broker.JetStream, worker jetstream.Consumer, body string) DeadLetter {
t.Helper()
if _, err := js.Context().Publish(BuildWorkOf(TheBuildMachine), []byte(body)); err != nil {
t.Fatal(err)
}
return givenUp(t, js, worker)
}
// A copy that cannot be published: the original is already out of the queue, the dead letter is kept and
// the answer says both; asked again once it can be published, it is delivered once.
func TestAnAskWhoseCopyIsRefusedStaysKeptAndIsDeliveredOnTheNextTry(t *testing.T) {
js := aBusWithTheBuildRole(t)
keeping(t, js)
worker := theBuildWorker(t, js)
d := aBuildAskGivenUp(t, js, worker, `{"module":"x"}`)
// The queue stops taking the ask's subject, so the copy's publish is refused by the server.
info, err := js.Context().StreamInfo(d.Stream)
if err != nil {
t.Fatal(err)
}
cfg := info.Config
taking := cfg.Subjects
cfg.Subjects = []string{"mesh.seat." + TheBuildMachine + ".accept.nothing"}
if _, err := js.Context().UpdateStream(&cfg); err != nil {
t.Fatal(err)
}
_, _, err = DeliverAgain(js.Context(), d.ID)
if err == nil || !strings.Contains(err.Error(), "still kept") || !strings.Contains(err.Error(), "was removed") {
t.Fatalf("a refused copy answered %v", err)
}
if _, err := js.Context().GetMsg(d.Stream, d.Sequence); !errors.Is(err, nats.ErrMsgNotFound) {
t.Fatalf("the original is still in the queue: %v", err)
}
if _, err := DeadLetterNamed(js.Context(), d.ID); err != nil {
t.Fatalf("a refused copy let the dead letter go: %v", err)
}
cfg.Subjects = taking
if _, err := js.Context().UpdateStream(&cfg); err != nil {
t.Fatal(err)
}
retried, _, err := DeliverAgain(js.Context(), d.ID)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(retried.Original, "no longer held") {
t.Fatalf("the retry said of the original: %q", retried.Original)
}
if again := next(t, worker); again == nil || string(again.Data()) != `{"module":"x"}` {
t.Fatal("the retry did not hand the ask to the worker")
} else {
_ = again.Ack()
}
if m := next(t, worker); m != nil {
t.Fatalf("handed twice: %s", m.Data())
}
}
// A queue made again numbers from one: the old dead letter's sequence then names a live ask, which is left
// alone.
func TestAnAskWhoseSequenceNamesAnotherMessageLeavesThatOneAlone(t *testing.T) {
js := aBusWithTheBuildRole(t)
keeping(t, js)
d := aBuildAskGivenUp(t, js, theBuildWorker(t, js), `{"module":"old"}`)
// The build handover's way: the seat's queue deleted and made again, with its worker.
if err := js.Context().DeleteStream(d.Stream); err != nil {
t.Fatal(err)
}
if err := broker.RaiseSeats(js, []broker.DeclaredSeat{{Name: TheBuildMachine, Accepts: []string{"build"},
Emits: []string{"built"}}}, map[string]broker.Holder{TheBuildMachine: {Node: "anchor", Module: "builder"}}); err != nil {
t.Fatal(err)
}
live, err := js.Context().Publish(BuildWorkOf(TheBuildMachine), []byte(`{"module":"live"}`))
if err != nil {
t.Fatal(err)
}
if live.Sequence != d.Sequence {
t.Fatalf("the live ask is message %d, the dead letter names %d: the test does not set up the collision",
live.Sequence, d.Sequence)
}
delivered, _, err := DeliverAgain(js.Context(), d.ID)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(delivered.Original, "another message") {
t.Fatalf("the answer said of the original: %q", delivered.Original)
}
if held, err := js.Context().GetMsg(d.Stream, d.Sequence); err != nil || string(held.Data) != `{"module":"live"}` {
t.Fatalf("the live ask at that sequence was touched: %v", err)
}
}
// No worker on the seat's queue: refused, kept, and nothing published.
func TestAnAskIsNotDeliveredAgainWhileTheSeatHasNoWorker(t *testing.T) {
js := aBusWithTheBuildRole(t)
keeping(t, js)
d := aBuildAskGivenUp(t, js, theBuildWorker(t, js), `{"module":"x"}`)
if err := js.Context().DeleteConsumer(d.Stream, d.Consumer); err != nil {
t.Fatal(err)
}
if _, _, err := DeliverAgain(js.Context(), d.ID); err == nil || !strings.Contains(err.Error(), "wait in") {
t.Fatalf("delivered with no worker: %v", err)
}
if _, err := js.Context().GetMsg(d.Stream, d.Sequence); err != nil {
t.Fatalf("a refusal removed the original: %v", err)
}
if _, err := DeadLetterNamed(js.Context(), d.ID); err != nil {
t.Fatalf("a refusal let the dead letter go: %v", err)
}
}
+47
View File
@@ -334,3 +334,50 @@ func TestNoticesThatCannotBeTakenAreSaidAndServingGoesOn(t *testing.T) {
} }
eventually(t, "a report heard while the notices cannot be taken", func() bool { return held.count() == 1 }) eventually(t, "a report heard while the notices cannot be taken", func() bool { return held.count() == 1 })
} }
// When DEAD_LETTERS kept each consumer's newest message, for the condition that counts what was given up
// on rather than how often it was looked at (novox/hq issue 440). It rises with every message kept, even
// one the consumer gave up on before the last: a notice that could not be kept is offered again a minute
// later, so a later give-up can be kept first, and the one kept after it must still count.
func TestTheNewestHeldDeadLetterIsWhenItWasKept(t *testing.T) {
js := aBus(t)
gaveUp := time.Date(2026, 10, 10, 21, 4, 0, 0, time.UTC)
keep := func(consumer string, seq int, at time.Time) {
t.Helper()
if _, err := KeepDeadLetter(js.Context(), []byte(fmt.Sprintf(`{"stream":"EVENTS","consumer":%q,`+
`"stream_seq":%d,"deliveries":5,"timestamp":%q}`, consumer, seq, at.Format(time.RFC3339Nano)))); err != nil {
t.Fatal(err)
}
}
look := func() map[string]time.Time {
t.Helper()
held, err := HeldDeadLetters(js.Context())
if err != nil {
t.Fatal(err)
}
newest, err := NewestDeadLetters(js.Context(), held)
if err != nil {
t.Fatal(err)
}
return newest
}
// Sonarr gave up at 21:05 and that is kept first; radarr's letter is kept after it.
keep("media_sonarr", 2, gaveUp.Add(time.Minute))
keep("media_radarr", 3, gaveUp)
before := look()
if len(before) != 2 || before["EVENTS.media_sonarr"].IsZero() ||
!before["EVENTS.media_radarr"].After(before["EVENTS.media_sonarr"]) {
t.Fatalf("%v", before)
}
// Sonarr's give-up of 21:04, kept late, is newer than anything the condition has seen.
keep("media_sonarr", 1, gaveUp)
after := look()
if !after["EVENTS.media_sonarr"].After(before["EVENTS.media_sonarr"]) ||
!after["EVENTS.media_sonarr"].After(before["EVENTS.media_radarr"]) {
t.Fatalf("a letter kept late does not read as newer: before %v, after %v", before, after)
}
if !after["EVENTS.media_radarr"].Equal(before["EVENTS.media_radarr"]) {
t.Fatalf("radarr's newest moved: before %v, after %v", before, after)
}
}
+16
View File
@@ -556,8 +556,21 @@ const (
StateStarting = "starting" StateStarting = "starting"
StateHeld = "held" StateHeld = "held"
StateUnknown = "unknown" StateUnknown = "unknown"
// StateWaiting is a resource whose module's tool check says it waits for the operator: a named own secret
// or setting not given yet (novox/hq ADR 0283). Not a fault; the controller checks the wait before it
// excuses it.
StateWaiting = "waiting"
) )
// Wait is one thing a waiting resource waits for the operator to give: exactly one of Secret and Setting, the
// part that waits and what the operator gives, in words (novox/hq ADR 0283; mesh-sdk go/health's shape).
type Wait struct {
Part string `json:"part"`
Secret string `json:"secret,omitempty"`
Setting string `json:"setting,omitempty"`
What string `json:"what"`
}
// ResourceHealth is one long-running resource's state. // ResourceHealth is one long-running resource's state.
type ResourceHealth struct { type ResourceHealth struct {
Module string `json:"module"` Module string `json:"module"`
@@ -581,6 +594,9 @@ type ResourceHealth struct {
// without a person (novox/hq ADR 0266): the engine judged that too, and a healthy verdict says it cannot. // without a person (novox/hq ADR 0266): the engine judged that too, and a healthy verdict says it cannot.
// Empty from an engine older than RootContract, and on every other verdict. // Empty from an engine older than RootContract, and on every other verdict.
Root string `json:"root,omitempty"` Root string `json:"root,omitempty"`
// Waits is what a resource in StateWaiting waits for (novox/hq ADR 0283). Empty otherwise, and from an
// engine older than that.
Waits []Wait `json:"waits,omitempty"`
} }
// HealthSaid is the health event's body: the machine and its statement. The machine is read from the // HealthSaid is the health event's body: the machine and its statement. The machine is read from the
+17 -18
View File
@@ -10,31 +10,30 @@ import (
"encoding/json" "encoding/json"
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/beside"
) )
// **The bus the tests run is the bus the mesh runs** (novox/hq ADR 0227 rule 9, to-be 45 §9). The server // **The bus the tests run is the bus the mesh runs** (novox/hq ADR 0227 rule 9, to-be 45 §9). The server
// linked into the tests is held to the release the catalogue's bus image is — read from beside this // linked into the tests is held to the release the catalogue's bus image is — the catalogue a merge check
// checkout — and, in a merge check that has the facts snapshot, to the release the mesh's bus server says // clones beside this one, or the copy captured in testdata/beside (internal/beside, novox/hq issue 432) —
// it runs. A bus upgrade moves the catalogue's pin; this then fails until the tests' pin moves with it, // and, in a merge check that has the facts snapshot, to the release the mesh's bus server says it runs. A
// so the controller is never tested against a bus the mesh no longer runs, or not yet. // bus upgrade moves the catalogue's pin; this then fails in the merge check until the tests' pin moves
// with it, so the controller is never tested against a bus the mesh no longer runs, or not yet.
func TestTheBusTheTestsRunIsTheBusTheMeshRuns(t *testing.T) { func TestTheBusTheTestsRunIsTheBusTheMeshRuns(t *testing.T) {
dockerfile := filepath.Join("..", "..", "..", "mesh-catalog", "modules", "nats", "Dockerfile") dockerfile := filepath.Join(beside.Catalogue(t), "nats", "Dockerfile")
raw, err := os.ReadFile(dockerfile) raw, err := os.ReadFile(dockerfile)
switch { if err != nil {
case err == nil:
pinned := regexp.MustCompile(`upstream: nats ([0-9.]+)-alpine`).FindSubmatch(raw)
if pinned == nil {
t.Fatalf("%s says no release its digest is", dockerfile)
}
if string(pinned[1]) != Version {
t.Errorf("the tests run bus %s and the catalogue's bus image is %s: move go.mod's nats-server pin "+
"(and `go mod vendor`) with the image", Version, pinned[1])
}
case os.IsNotExist(err):
t.Logf("the catalogue is not beside this checkout, so its bus image is not compared")
default:
t.Fatal(err) t.Fatal(err)
} }
pinned := regexp.MustCompile(`upstream: nats ([0-9.]+)-alpine`).FindSubmatch(raw)
if pinned == nil {
t.Fatalf("%s says no release its digest is", dockerfile)
}
if string(pinned[1]) != Version {
t.Errorf("the tests run bus %s and the catalogue's bus image is %s: move go.mod's nats-server pin "+
"(and `go mod vendor`) with the image", Version, pinned[1])
}
path := os.Getenv("MESH_FACTS") path := os.Getenv("MESH_FACTS")
if path == "" { if path == "" {
t.Logf("no MESH_FACTS: the bus the mesh runs is compared in a merge check, which has it") t.Logf("no MESH_FACTS: the bus the mesh runs is compared in a merge check, which has it")
+22
View File
@@ -0,0 +1,22 @@
What a test reads of another repository when no merge check has cloned it beside this one (internal/beside,
novox/hq issue 432). Copied from the repositories at the commits below, never written by hand. Each
module.json is kept as module.json.captured: a module.json in this repository is a module of it to the
forge and the planner, and a merge would build and register it.
The copy carries the catalogue's private details: domains, first names in module names, a node name and
private addresses. That is acceptable while mesh-controller is private; if it ever goes public, this copy is
a second place to clean besides the catalogue itself.
mesh-catalog b9de001833b1b61b297182b8e3bcdae1cbdeace9 modules/*/module.json, modules/nats/Dockerfile
mesh-host bd5cc6980419c1bd4824be6d58dfebd2381a9de3 examples/foundation-first-node-nats.lock
To move them, from this repository's root, with the two repositories checked out beside it:
rm -rf testdata/beside/mesh-catalog testdata/beside/mesh-host
mkdir -p testdata/beside/mesh-catalog testdata/beside/mesh-host
git -C ../mesh-catalog archive <commit> modules | tar -x -C testdata/beside/mesh-catalog --wildcards \
'modules/*/module.json' 'modules/nats/Dockerfile'
find testdata/beside -name module.json -exec mv {} {}.captured \;
git -C ../mesh-host archive <commit> examples/foundation-first-node-nats.lock | tar -x -C testdata/beside/mesh-host
and write the commits here.
@@ -0,0 +1,118 @@
{
"module": "adwaita",
"version": "1",
"capabilities": [
"package-manager"
],
"tools": [
"adwaita_appearance",
"adwaita_cursor",
"adwaita_icons",
"adwaita_portal_check"
],
"environment": {
"variables": {
"GTK_THEME": "Adwaita:dark",
"GTK2_RC_FILES": "/usr/share/themes/Adwaita-dark/gtk-2.0/gtkrc",
"QT_QPA_PLATFORMTHEME": "qt6ct",
"QT_STYLE_OVERRIDE": "Fusion",
"QT_SELECT": "6",
"XCURSOR_THEME": "Adwaita",
"XCURSOR_SIZE": "24"
}
},
"shell": [
{
"for": "xresources",
"slot": "normal",
"code": "! adwaita: the cursor, for X programs that take it from the resources.\nXcursor.theme: Adwaita\nXcursor.size: 24\n"
},
{
"for": "xinitrc",
"slot": "normal",
"code": "# The appearance (module adwaita): GSettings is where the portal reads dark or light, and the portal is\n# the only way it reaches Electron, Chromium, Firefox and flatpaks. Set at every session start to the\n# module's default; adwaita_appearance switches it for a session.\ngsettings set org.gnome.desktop.interface color-scheme 'prefer-dark' || true\ngsettings set org.gnome.desktop.interface gtk-theme 'Adwaita' || true\ngsettings set org.gnome.desktop.interface icon-theme 'Adwaita' || true\ngsettings set org.gnome.desktop.interface cursor-theme 'Adwaita' || true\ngsettings set org.gnome.desktop.interface cursor-size 24 || true\ngsettings set org.gnome.desktop.interface font-name 'Inter 11' || true\ngsettings set org.gnome.desktop.interface monospace-font-name 'JetBrainsMono Nerd Font 11' || true\n"
}
],
"resources": [
{
"id": "package-gnome-themes-extra",
"type": "package",
"package": "gnome-themes-extra"
},
{
"id": "package-adwaita-icon-theme",
"type": "package",
"package": "adwaita-icon-theme"
},
{
"id": "package-adwaita-cursors",
"type": "package",
"package": "adwaita-cursors"
},
{
"id": "package-qt6ct",
"type": "package",
"package": "qt6ct"
},
{
"id": "package-xdg-desktop-portal-gtk",
"type": "package",
"package": "xdg-desktop-portal-gtk"
},
{
"id": "gtk3",
"type": "file",
"path": "${machine:account-home}/.config/gtk-3.0/settings.ini",
"owner": "${machine:account}",
"mode": "0644",
"content": "# Written by the mesh (module adwaita, novox/hq ADR 0208), for GTK 3 and GTK 4 alike. Replaced at\n# every push; adwaita_appearance switches dark and light for the running session.\n[Settings]\ngtk-theme-name=Adwaita\ngtk-icon-theme-name=Adwaita\ngtk-cursor-theme-name=Adwaita\ngtk-cursor-theme-size=24\ngtk-font-name=Inter 11\ngtk-application-prefer-dark-theme=1\n"
},
{
"id": "gtk4",
"type": "file",
"path": "${machine:account-home}/.config/gtk-4.0/settings.ini",
"owner": "${machine:account}",
"mode": "0644",
"content": "# Written by the mesh (module adwaita, novox/hq ADR 0208), for GTK 3 and GTK 4 alike. Replaced at\n# every push; adwaita_appearance switches dark and light for the running session.\n[Settings]\ngtk-theme-name=Adwaita\ngtk-icon-theme-name=Adwaita\ngtk-cursor-theme-name=Adwaita\ngtk-cursor-theme-size=24\ngtk-font-name=Inter 11\ngtk-application-prefer-dark-theme=1\n"
},
{
"id": "qt6ct",
"type": "file",
"path": "${machine:account-home}/.config/qt6ct/qt6ct.conf",
"owner": "${machine:account}",
"mode": "0644",
"content": "[Appearance]\ncolor_scheme_path=/usr/share/qt6ct/colors/darker.conf\ncustom_palette=true\nicon_theme=Adwaita\nstandard_dialogs=default\nstyle=Fusion\n\n[Fonts]\nfixed=\"JetBrainsMono Nerd Font,11,-1,5,50,0,0,0,0,0\"\ngeneral=\"Inter,11,-1,5,50,0,0,0,0,0\"\n\n[Interface]\nactivate_item_on_single_click=1\nbuttonbox_layout=0\ncursor_flash_time=1000\ndialog_buttons_have_icons=1\ndouble_click_interval=400\ngui_effects=@Invalid()\nkeyboard_scheme=2\nmenus_have_icons=true\nshow_shortcuts_in_context_menus=true\nstylesheets=@Invalid()\ntoolbutton_style=4\nunderline_shortcut=1\nwheel_scroll_lines=3\n\n[Troubleshooting]\nforce_raster_widgets=1\nignored_applications=@Invalid()\n"
},
{
"id": "portals",
"type": "file",
"path": "${machine:account-home}/.config/xdg-desktop-portal/portals.conf",
"owner": "${machine:account}",
"mode": "0644",
"content": "# Written by the mesh (module adwaita, novox/hq ADR 0208). Replaced at every push.\n#\n# Which portal backend answers each interface. i3 is not a desktop xdg-desktop-portal knows, so with\n# no preference it uses whichever backend happens to be installed: fine while gtk is the only one,\n# wrong the day another arrives as somebody else's dependency. Named instead. gtk also serves\n# org.freedesktop.appearance (dark or light) from GSettings, which the session's start sets.\n[preferred]\ndefault=gtk\n# Secrets for sandboxed programs come from the keyring's backend. Without this line no backend answers\n# the interface: gtk does not implement it, and gnome-keyring's names only GNOME as its desktop.\norg.freedesktop.impl.portal.Secret=gnome-keyring\n"
},
{
"id": "cursor",
"type": "file",
"path": "${machine:account-home}/.icons/default/index.theme",
"owner": "${machine:account}",
"mode": "0644",
"content": "# Written by the mesh (module adwaita, novox/hq ADR 0208): the default cursor theme, for programs that\n# read neither XCURSOR_THEME nor the X resources.\n[Icon Theme]\nName=Default\nInherits=Adwaita\n"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/adwaita-tools",
"binary": "adwaita-tools",
"loads": [
"adwaita-tools"
]
}
]
}
}
@@ -0,0 +1,32 @@
{
"module": "artifact-store-tools",
"version": "1",
"invokes": [
"seat:mesh-controller.artifacts",
"seat:mesh-controller.collect"
],
"tools": [
"artifact_store_repositories",
"artifact_store_usage",
"artifact_store_references",
"artifact_store_collect"
],
"build": {
"artifacts": [
{
"name": "tools-go",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/artifact-store-tools",
"binary": "artifact-store-tools",
"loads": [
"artifact-store-tools"
],
"env": {
"MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry"
}
}
]
}
}
@@ -0,0 +1,224 @@
{
"module": "asus-zephyrus-g14",
"version": "1",
"capabilities": [
"package-manager",
"service-manager"
],
"requires": [
"x11-display"
],
"emits": [
"profile.switched"
],
"tools": [
"zephyrus_brightness",
"zephyrus_battery",
"zephyrus_charge_limit",
"zephyrus_gpu_mode",
"zephyrus_profile",
"zephyrus_thermals",
"zephyrus_power_draw",
"zephyrus_profile_policy",
"zephyrus_fan_curves",
"zephyrus_keys",
"zephyrus_check"
],
"resources": [
{
"id": "asusctl",
"type": "package",
"package": "asusctl"
},
{
"id": "playerctl",
"type": "package",
"package": "playerctl"
},
{
"id": "asusd",
"type": "service",
"unit": "asusd.service",
"state": "running"
},
{
"id": "supergfxd",
"type": "service",
"unit": "supergfxd.service",
"state": "running",
"boot": "enabled"
},
{
"id": "scripts",
"type": "archive",
"path": "/usr/local/lib/asus-zephyrus-g14",
"artifact": "scripts"
},
{
"id": "nvidia-options",
"type": "file",
"path": "/etc/modprobe.d/g14-nvidia-power.conf",
"mode": "0644",
"content": "# Managed by the mesh (module asus-zephyrus-g14). Replaced on every push; edit the catalogue instead.\n#\n# The discrete GPU's driver options on the ROG Zephyrus G14 (GA403, RTX 40 series, hybrid graphics).\n#\n# NVreg_DynamicPowerManagement=0x00 turns runtime D3 off. With it on, a change of power source sends\n# the driver an ACPI notification it fails to handle on this model (\"RmHandleDNotifierEvent: Failed to\n# handle ACPI D-Notifier event, status=0x62\"), and the GPU stops making progress until the machine is\n# powered off. Off costs a few idle watts in hybrid mode and keeps the machine up.\n#\n# NVreg_PreserveVideoMemoryAllocations=1 saves video memory across suspend, so what used the GPU still\n# works after waking. It needs nvidia-suspend, -hibernate and -resume to run around a sleep, which this\n# module's drop-ins on the sleep services ask for.\n#\n# A change here applies when the driver next loads: at the next boot.\noptions nvidia NVreg_PreserveVideoMemoryAllocations=1\noptions nvidia NVreg_DynamicPowerManagement=0x00\n"
},
{
"id": "video-options",
"type": "file",
"path": "/etc/modprobe.d/video-brightness-switch.conf",
"mode": "0644",
"content": "# Managed by the mesh (module asus-zephyrus-g14). Replaced on every push; edit the catalogue instead.\n#\n# The ACPI video driver does not change the backlight itself on the brightness keys: on this model it\n# moves the wrong one. The keys are triggerhappy's (see /etc/triggerhappy/triggers.d/asus-g14.conf).\n# Applies when the module next loads: at the next boot.\noptions video brightness_switch_enabled=0\n"
},
{
"id": "suspend-drop-ins",
"type": "directory",
"path": "/etc/systemd/system/systemd-suspend.service.d",
"mode": "0755"
},
{
"id": "nvidia-on-suspend",
"type": "file",
"path": "/etc/systemd/system/systemd-suspend.service.d/asus-zephyrus-g14-nvidia.conf",
"mode": "0644",
"content": "# Managed by the mesh (module asus-zephyrus-g14). Replaced on every push; edit the catalogue instead.\n#\n# The NVIDIA driver's own sleep actions, asked for by the sleep itself rather than enabled as\n# links: the mesh declares files and never makes links (novox/hq ADR 0012), and the host's service\n# shape must not start these units by hand, which would put the GPU to sleep with the machine awake.\n[Unit]\nWants=nvidia-suspend.service nvidia-resume.service\n"
},
{
"id": "hibernate-drop-ins",
"type": "directory",
"path": "/etc/systemd/system/systemd-hibernate.service.d",
"mode": "0755"
},
{
"id": "nvidia-on-hibernate",
"type": "file",
"path": "/etc/systemd/system/systemd-hibernate.service.d/asus-zephyrus-g14-nvidia.conf",
"mode": "0644",
"content": "# Managed by the mesh (module asus-zephyrus-g14). Replaced on every push; edit the catalogue instead.\n#\n# The NVIDIA driver's own sleep actions, asked for by the sleep itself rather than enabled as\n# links: the mesh declares files and never makes links (novox/hq ADR 0012), and the host's service\n# shape must not start these units by hand, which would put the GPU to sleep with the machine awake.\n[Unit]\nWants=nvidia-hibernate.service nvidia-resume.service\n"
},
{
"id": "suspend-then-hibernate-drop-ins",
"type": "directory",
"path": "/etc/systemd/system/systemd-suspend-then-hibernate.service.d",
"mode": "0755"
},
{
"id": "nvidia-on-suspend-then-hibernate",
"type": "file",
"path": "/etc/systemd/system/systemd-suspend-then-hibernate.service.d/asus-zephyrus-g14-nvidia.conf",
"mode": "0644",
"content": "# Managed by the mesh (module asus-zephyrus-g14). Replaced on every push; edit the catalogue instead.\n#\n# The NVIDIA driver's own sleep actions, asked for by the sleep itself rather than enabled as\n# links: the mesh declares files and never makes links (novox/hq ADR 0012), and the host's service\n# shape must not start these units by hand, which would put the GPU to sleep with the machine awake.\n[Unit]\nWants=nvidia-suspend-then-hibernate.service nvidia-resume.service\n"
},
{
"id": "powerd-drop-ins",
"type": "directory",
"path": "/etc/systemd/system/nvidia-powerd.service.d",
"mode": "0755"
},
{
"id": "powerd-opt-in",
"type": "file",
"path": "/etc/systemd/system/nvidia-powerd.service.d/asus-zephyrus-g14.conf",
"mode": "0644",
"content": "# Managed by the mesh (module asus-zephyrus-g14). Replaced on every push; edit the catalogue instead.\n#\n# nvidia-powerd (Dynamic Boost) was the first error in the chain that hung this model's GPU on a change\n# of power source, and asusd starts it on mains. It runs only when the kernel command line says\n# zephyrus.nvidia-powerd — an explicit opt-in, at boot.\n[Unit]\nConditionKernelCommandLine=zephyrus.nvidia-powerd\n"
},
{
"id": "backlight-rule",
"type": "file",
"path": "/etc/udev/rules.d/90-backlight.rules",
"mode": "0644",
"content": "# Managed by the mesh (module asus-zephyrus-g14). Replaced on every push; edit the catalogue instead.\n#\n# The backlights are writable by the video group, so the brightness keys and the module's brightness tool\n# move the panel without root.\nACTION==\"add\", SUBSYSTEM==\"backlight\", RUN+=\"/usr/bin/chgrp video /sys/class/backlight/%k/brightness\", RUN+=\"/usr/bin/chmod g+w /sys/class/backlight/%k/brightness\"\n"
},
{
"id": "udev",
"type": "service",
"unit": "systemd-udevd.service",
"reload-on": [
"backlight-rule"
]
},
{
"id": "upower-package",
"type": "package",
"package": "upower"
},
{
"id": "upower-drop-ins",
"type": "directory",
"path": "/etc/UPower/UPower.conf.d",
"mode": "0755"
},
{
"id": "low-battery",
"type": "file",
"path": "/etc/UPower/UPower.conf.d/50-asus-zephyrus-g14.conf",
"mode": "0644",
"content": "# Managed by the mesh (module asus-zephyrus-g14). Replaced on every push; edit the catalogue instead.\n#\n# On low battery the machine suspends rather than powering off, at 7 % — s2idle still draws a little,\n# so it leaves headroom. A drop-in over the package's own UPower.conf, which stays the package's.\n[UPower]\nUsePercentageForPolicy=true\nPercentageLow=15.0\nPercentageCritical=10.0\nPercentageAction=7.0\nCriticalPowerAction=Suspend\nAllowRiskyCriticalPowerAction=true\n"
},
{
"id": "upower",
"type": "service",
"unit": "upower.service",
"state": "running",
"boot": "enabled",
"restart-on": [
"low-battery"
]
},
{
"id": "xorg-drop-ins",
"type": "directory",
"path": "/etc/X11/xorg.conf.d",
"mode": "0755"
},
{
"id": "touchpad",
"type": "file",
"path": "/etc/X11/xorg.conf.d/30-asus-zephyrus-g14-touchpad.conf",
"mode": "0644",
"content": "# Managed by the mesh (module asus-zephyrus-g14). Replaced on every push; edit the catalogue instead.\n#\n# The touchpad's settings, applied by X every time the device appears — at login and after every\n# resume, when the device is initialised again. This replaces the predecessor's sleep hook, which ran\n# xinput after a resume as a named person on a guessed display.\nSection \"InputClass\"\n Identifier \"asus-zephyrus-g14 touchpad\"\n MatchIsTouchpad \"on\"\n Option \"Tapping\" \"on\"\n Option \"NaturalScrolling\" \"true\"\n Option \"AccelSpeed\" \"0.15\"\nEndSection\n"
}
],
"build": {
"artifacts": [
{
"name": "tools-go",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/zephyrus",
"binary": "zephyrus",
"loads": [
"zephyrus"
]
},
{
"name": "scripts",
"kind": "archive",
"from": "files"
}
]
},
"contributions": [
{
"seat": "node-hotkeys",
"kind": "trigger",
"content": "# The ROG Zephyrus G14's vendor keys, which reach no X client: media (the M-keys), panel brightness,\n# and the touchpad key. Each runs this module's own script, as the operator's account.\nKEY_PROG1\t1\t/usr/local/lib/asus-zephyrus-g14/bin/zephyrus-session /usr/local/lib/asus-zephyrus-g14/bin/zephyrus-media play-pause\nKEY_PROG3\t1\t/usr/local/lib/asus-zephyrus-g14/bin/zephyrus-session /usr/local/lib/asus-zephyrus-g14/bin/zephyrus-media previous\nKEY_PROG4\t1\t/usr/local/lib/asus-zephyrus-g14/bin/zephyrus-session /usr/local/lib/asus-zephyrus-g14/bin/zephyrus-media next\nKEY_BRIGHTNESSDOWN\t1\t/usr/local/lib/asus-zephyrus-g14/bin/zephyrus-backlight -\nKEY_BRIGHTNESSDOWN\t2\t/usr/local/lib/asus-zephyrus-g14/bin/zephyrus-backlight -\nKEY_BRIGHTNESSUP\t1\t/usr/local/lib/asus-zephyrus-g14/bin/zephyrus-backlight +\nKEY_BRIGHTNESSUP\t2\t/usr/local/lib/asus-zephyrus-g14/bin/zephyrus-backlight +\nKEY_F21\t1\t/usr/local/lib/asus-zephyrus-g14/bin/zephyrus-touchpad reset\n"
},
{
"seat": "node-display-session",
"kind": "config",
"content": "# The laptop's own lines in i3 (module asus-zephyrus-g14, novox/hq ADR 0208, ADR 0210). Owned by the\n# mesh: replaced at every push. Once the controller places contributions to node-display-session\n# (ADR 0210), these become the module's contribution instead of a file in i3's directory.\n#\n# The keys the firmware turns into ordinary key presses. The vendor keys that reach no X client are\n# triggerhappy's (/etc/triggerhappy/triggers.d/asus-g14.conf): M4 and Fn+F4/F5 for media, Fn+F7/F8 for\n# the panel, Fn+F10 for the touchpad. The keyboard backlight (Fn+F2/F3) is the firmware's and asusd's.\n\n# Fn+F6, the screenshot key: the firmware sends Super+Shift+S. Released before it runs, because the\n# screenshot grabs the pointer to select a region, which fails while the key is still held.\nbindsym --release $mod+Shift+s exec --no-startup-id $XDG_CONFIG_HOME/i3/scripts/screenshot.sh\n\n# Fn+F9, the display key: the firmware sends Super+P. Odd workspaces to the panel, even ones to the\n# external output.\nbindsym $mod+p exec --no-startup-id /usr/local/lib/asus-zephyrus-g14/bin/zephyrus-display order\n\n# The keyboard backlight's level, shown when it changes.\nexec --no-startup-id /usr/local/lib/asus-zephyrus-g14/bin/zephyrus-kbd-notify\n"
},
{
"seat": "node-display-session",
"kind": "config",
"content": "# The laptop's own lines in i3 (module asus-zephyrus-g14, novox/hq ADR 0208, ADR 0210). Owned by the\n# mesh: replaced at every push. Once the controller places contributions to node-display-session\n# (ADR 0210), these become the module's contribution instead of a file in i3's directory.\n#\n# The model's panel and touchpad.\n\n# The internal panel is the primary output, where the bars' tray goes. Which monitors are on and where\n# is the display server's (autorandr, run at every session start).\nexec --no-startup-id /usr/local/lib/asus-zephyrus-g14/bin/zephyrus-display primary\n\n# The touchpad's settings again, by hand. X applies them itself whenever the device appears.\nbindsym $mod+Shift+x exec --no-startup-id /usr/local/lib/asus-zephyrus-g14/bin/zephyrus-touchpad reset\n"
}
],
"shell": [
{
"for": "after-wake",
"slot": "normal",
"code": "# The touchpad's settings again after waking, in the operator's session: X applies the module's\n# input class when the device appears, and this covers a wake that does not initialise it again.\n# Runs as root from the power module; the reset itself runs as the session's owner.\nsleep 2\nowner=$(ps -o user= -C i3 | head -n 1)\n[ -n \"$owner\" ] && runuser -u \"$owner\" -- /usr/local/lib/asus-zephyrus-g14/bin/zephyrus-touchpad reset\ntrue\n"
}
]
}
@@ -0,0 +1,66 @@
{
"module": "audit-logger",
"version": "1",
"slug": "audit",
"consumes": [
"**"
],
"build": {
"artifacts": [
{
"name": "code",
"kind": "bundle",
"language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js"
],
"loads": [
"index.js",
"tools/index.js"
],
"env": {
"AUDIT_LOG": "${dir:trail}/audit.log",
"AUDIT_SPOOL": "${dir:spool}"
}
}
]
},
"data": {
"own": [
{
"id": "trail",
"path": "${dir:trail}",
"class": "valuable",
"why": "the audit trail, written nowhere else"
},
{
"id": "spool",
"path": "${dir:spool}",
"class": "valuable",
"why": "events the trail could not take yet; after the bus gives one up, the only copy"
}
]
},
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "trail",
"type": "directory",
"mode": "0700"
},
{
"id": "spool",
"type": "directory",
"mode": "0700"
}
],
"capabilities": [
"container-runtime"
]
}
@@ -0,0 +1,43 @@
{
"module": "avahi",
"version": "1",
"capabilities": [
"package-manager",
"service-manager"
],
"tools": [
"avahi_status",
"avahi_browse",
"avahi_resolve",
"avahi_services"
],
"resources": [
{
"id": "package",
"type": "package",
"package": "avahi"
},
{
"id": "daemon",
"type": "service",
"unit": "avahi-daemon.service",
"state": "running",
"boot": "enabled"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/avahi-tools",
"binary": "avahi-tools",
"loads": [
"avahi-tools"
]
}
]
}
}
@@ -0,0 +1,128 @@
{
"module": "baserow",
"version": "1",
"capabilities": [
"container-runtime"
],
"requires": [
"postgres-database",
"route"
],
"contributes": {
"postgres-database": {
"name": "baserow"
},
"route": {
"label": "baserow",
"endpoint": "web"
}
},
"binds": {
"postgres-database": "${dir:state}/database.json",
"route": "${dir:state}/route.json"
},
"secrets": {
"postgres-database": "${dir:state}/database.secret"
},
"own-secrets": {
"admin": "${dir:state}/admin.secret"
},
"listens": [
{
"name": "web",
"port": 80,
"protocol": "tcp",
"from": "mesh",
"why": "the Baserow web UI and REST API, served by the image's own Caddy; a public name is the route's"
}
],
"data": {
"own": [
{
"id": "data",
"path": "${dir:data}",
"class": "valuable",
"why": "uploaded files and media; the tables are in the database"
}
]
},
"resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "data",
"type": "directory",
"mode": "0755",
"owner": "9999:9999"
},
{
"id": "server-env",
"type": "file",
"path": "${dir:state}/server.env",
"mode": "0600",
"content": "DATABASE_HOST=${bound:postgres-database:at}\nDATABASE_PORT=${bound:postgres-database:port}\nDATABASE_NAME=${bound:postgres-database:as}\nDATABASE_USER=${bound:postgres-database:as}\nDATABASE_PASSWORD_FILE=/run/secrets/database\nDISABLE_EMBEDDED_PSQL=true\nBASEROW_PUBLIC_URL=https://${bound:route:name}\n"
},
{
"id": "net",
"type": "network",
"name": "baserow"
},
{
"id": "server",
"type": "container",
"name": "baserow",
"image": "baserow/baserow@sha256:263ea6c4b72c9eccabcd975ffe9fdebf23913a293a514bec6a3897a5e0a5a080",
"health": {
"kind": "runtime"
},
"network": "baserow",
"env-file": [
"${dir:state}/server.env"
],
"ports": [
"80"
],
"volumes": [
"${dir:data}:/baserow/data",
"${dir:state}/database.secret:/run/secrets/database:ro"
]
},
{
"id": "runtime-config",
"type": "file",
"path": "${dir:mesh-state}/config.json",
"mode": "0600",
"content": "{\n \"password\": \"${secret:admin}\",\n \"host\": \"${bound:route:name}\"\n}\n",
"merge": "json"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "typescript",
"entrypoints": [
"tools/index.js"
],
"loads": [
"tools/index.js"
],
"env": {
"MESH_BASEROW_URL": "http://127.0.0.1:${port:80}",
"MESH_BASEROW_CONFIG_FILE": "${dir:mesh-state}/config.json"
}
}
]
}
}
@@ -0,0 +1,55 @@
{
"module": "betterbird",
"version": "1",
"requires": [
"x11-display"
],
"settings": {
"prefs": {
"kind": "preference",
"default": "// none",
"why": "Betterbird runs with its own defaults until a machine's assignment names a preference: one line of user_pref(\"name\", value); statements, which the module's user.js holds and Betterbird reads at its start (novox/hq issue 345)"
}
},
"tools": [
"betterbird_status",
"betterbird_prefs",
"betterbird_user_js",
"betterbird_log",
"betterbird_restart",
"betterbird_check",
"betterbird_reminder_test"
],
"resources": [
{
"id": "configuration-dir",
"type": "directory",
"path": "${machine:account-home}/.config/betterbird",
"owner": "${machine:account}",
"mode": "0755"
},
{
"id": "user-js",
"type": "file",
"path": "${machine:account-home}/.config/betterbird/user.js",
"owner": "${machine:account}",
"mode": "0644",
"content": "// Betterbird's user.js (module betterbird, novox/hq issue 345). Owned by the mesh: the node-engine\n// writes it here at every push, and the module's tools copy it whole into Betterbird's profile\n// (betterbird_user_js with apply, betterbird_restart), where Betterbird reads it at its start and lets it\n// win over the preferences Betterbird saves itself. Change the module's setting prefs, never this file or the profile's copy.\n//\n// Only comments and user_pref(\"name\", value); statements; the tools refuse anything else, and any\n// preference whose name can hold a credential.\n${setting:prefs}\n"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/betterbird-tools",
"binary": "betterbird-tools",
"loads": [
"betterbird-tools"
]
}
]
}
}
@@ -0,0 +1,37 @@
{
"module": "blueman",
"version": "1",
"capabilities": [
"package-manager"
],
"requires": [
"x11-display"
],
"tools": [
"blueman_status",
"blueman_restart",
"blueman_check"
],
"resources": [
{
"id": "package",
"type": "package",
"package": "blueman"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/blueman-tools",
"binary": "blueman-tools",
"loads": [
"blueman-tools"
]
}
]
}
}
@@ -0,0 +1,53 @@
{
"module": "bluetooth",
"version": "1",
"capabilities": [
"package-manager",
"service-manager"
],
"tools": [
"bluetooth_controller",
"bluetooth_power",
"bluetooth_devices",
"bluetooth_scan",
"bluetooth_connect",
"bluetooth_disconnect",
"bluetooth_trust",
"bluetooth_pair",
"bluetooth_remove"
],
"resources": [
{
"id": "stack",
"type": "package",
"package": "bluez"
},
{
"id": "utilities",
"type": "package",
"package": "bluez-utils"
},
{
"id": "daemon",
"type": "service",
"unit": "bluetooth.service",
"state": "running",
"boot": "enabled"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/bluetooth-tools",
"binary": "bluetooth-tools",
"loads": [
"bluetooth-tools"
]
}
]
}
}
@@ -0,0 +1,101 @@
{
"module": "build-agent",
"version": "1",
"slug": "agent",
"capabilities": [
"container-runtime"
],
"claims": [
{
"name": "node-build-agent",
"scope": "node",
"serves": ["current", "kill", "pause", "resume"]
}
],
"requires": [
"artifact-store",
"npm-package-registry"
],
"binds": {
"npm-package-registry": "${dir:mesh-state}/package-registry.json"
},
"secrets": {
"npm-package-registry": "${dir:mesh-state}/package-registry.secret"
},
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"data": {
"own": [
{
"id": "workspace",
"path": "${dir:workspace}",
"class": "cache",
"why": "a build's working copy, cloned again for every build"
}
]
},
"resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{
"id": "workspace",
"type": "directory",
"mode": "0700"
},
{
"id": "agent-env",
"type": "file",
"path": "${dir:mesh-state}/build-agent.env",
"mode": "0600",
"content": "MESH_BROKER_FILE=/run/mesh/broker\nMESH_NODE=${machine:name}\nMESH_REGISTRY=${bound:artifact-store:at}:${bound:artifact-store:port}\nMESH_PACKAGE_BINDING=/run/mesh/package-registry.json\nMESH_NPM_TOKEN_FILE=/run/mesh/package-registry.secret\nMESH_WORKSPACE=${dir:workspace}\n"
},
{
"id": "server",
"type": "container",
"name": "mesh-build-agent",
"artifact": "server",
"env-file": [
"${dir:mesh-state}/build-agent.env"
],
"volumes": [
"${dir:mesh-state}:/run/mesh:ro",
"${dir:workspace}:${dir:workspace}",
"/var/run/docker.sock:/var/run/docker.sock"
],
"restart-on": [
"agent-env"
],
"network": "host"
}
],
"build": {
"artifacts": [
{
"name": "server",
"kind": "image",
"from": "Dockerfile",
"compiles": "cmd/mesh-builder",
"context": {
"seat": "git",
"repository": "novox/mesh-controller",
"ref": "main"
}
}
],
"on": [
{
"arg": "GO_BASE",
"image": "golang@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c"
},
{
"arg": "ALPINE_BASE",
"image": "alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"
}
]
}
}
@@ -0,0 +1,56 @@
{
"module": "ca-trust",
"version": "1",
"slug": "catrust",
"capabilities": [
"service-manager"
],
"requires": [
"internal-acme-ca"
],
"seats": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"claims": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "anchor",
"type": "file",
"path": "${dir:state}/anchor",
"mode": "0755",
"content": "#!/bin/sh\n# The mesh's internal certificate authority, trusted by this machine.\n#\n# Written by the mesh from the ca-trust module's manifest (novox/hq ADR 0147).\n# Editing it here lasts until the next apply.\n#\n# There is no prior trust to verify the fetch against \u2014 this is the thing that\n# establishes it \u2014 so it is made over the mesh's own private network, which is\n# what authenticates it (novox/hq ADR 0098, the same reasoning that lets the\n# route proxy fetch this root for itself). What comes back is checked here: a\n# body that is not a certificate is refused now, rather than believed and then\n# failed by whatever reads the trust store next.\nset -eu\n\nROOTS='https://${bound:internal-acme-ca:at}:${bound:internal-acme-ca:port}${bound:internal-acme-ca:roots}'\nANCHORS=/etc/ca-certificates/trust-source/anchors\nANCHOR=\"$ANCHORS/mesh-internal-ca.crt\"\n\n# Arch's layout, said out loud rather than assumed: a machine that keeps its\n# anchors elsewhere fails here, visibly, instead of writing a file nothing\n# reads. That failure is the signal that this belongs in the host, where one\n# operating system's difference lives (novox/hq ADR 0147, option 2).\n[ -d \"$ANCHORS\" ] || {\n\techo \"this machine keeps no trust anchors in $ANCHORS; ca-trust is written for that layout\" >&2\n\texit 1\n}\n\ncase \"${1:-}\" in\ninstall)\n\ttmp=$(mktemp)\n\ttrap 'rm -f \"$tmp\"' EXIT\n\t# The authority may still be starting, or this machine may have come up\n\t# before it: two minutes of asking, then an honest failure.\n\tn=0\n\twhile [ \"$n\" -lt 60 ]; do\n\t\tif curl --fail --silent --show-error --insecure --max-time 10 \\\n\t\t\t--output \"$tmp\" \"$ROOTS\" &&\n\t\t\tgrep -q 'BEGIN CERTIFICATE' \"$tmp\"; then\n\t\t\tinstall -m 0644 \"$tmp\" \"$ANCHOR\"\n\t\t\tupdate-ca-trust\n\t\t\texit 0\n\t\tfi\n\t\tn=$((n + 1))\n\t\tsleep 2\n\tdone\n\techo \"the authority at $ROOTS did not serve a certificate within two minutes\" >&2\n\texit 1\n\t;;\nremove)\n\t# What stopping the unit does, and therefore what being unassigned does.\n\trm -f \"$ANCHOR\"\n\tupdate-ca-trust\n\t;;\n*)\n\techo \"usage: $(basename \"$0\") install|remove\" >&2\n\texit 2\n\t;;\nesac\n"
},
{
"id": "unit",
"type": "file",
"path": "/etc/systemd/system/mesh-ca-trust.service",
"mode": "0644",
"content": "[Unit]\nDescription=The mesh's internal certificate authority, trusted by this machine\n# novox/hq ADR 0147. Starting this unit places the mesh's root among this\n# machine's trust anchors; stopping it takes the root away again, which is what\n# the host does when the module is no longer assigned here.\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=${dir:state}/anchor install\nExecStop=${dir:state}/anchor remove\n\n[Install]\nWantedBy=multi-user.target\n"
},
{
"id": "trust",
"type": "service",
"unit": "mesh-ca-trust.service",
"state": "running",
"boot": "enabled",
"restart-on": [
"anchor",
"unit"
]
}
]
}
@@ -0,0 +1,174 @@
{
"module": "claude-code",
"version": "1",
"slug": "agent",
"capabilities": [
"package-manager"
],
"requires": [
"mcp-endpoint"
],
"binds": {
"mcp-endpoint": "${dir:state}/mcp-endpoint.json"
},
"uses": [
"operator-channel"
],
"state": [
"servers",
"holdings",
"config",
"proposals"
],
"reads": [
"claude-licence-manager.bindings"
],
"tools": [
"claude_code_status",
"claude_code_render",
"claude_code_guard",
"claude_code_pull",
"claude_code_grant",
"claude_code_add_api_key",
"claude_code_registrations",
"claude_code_mcp_list",
"claude_code_mcp_register",
"claude_code_mcp_unregister",
"claude_code_skill_list",
"claude_code_skill_register",
"claude_code_skill_unregister",
"claude_code_agent_list",
"claude_code_agent_register",
"claude_code_agent_unregister",
"claude_code_command_list",
"claude_code_command_register",
"claude_code_command_unregister",
"claude_code_hook_list",
"claude_code_hook_register",
"claude_code_hook_unregister",
"claude_code_output_style_list",
"claude_code_output_style_register",
"claude_code_output_style_unregister",
"claude_code_instructions_list",
"claude_code_instructions_register",
"claude_code_instructions_unregister",
"claude_code_instructions_propose",
"claude_code_proposals",
"claude_code_settings_get",
"claude_code_settings_set",
"claude_code_settings_clear",
"claude_code_permission_add",
"claude_code_permission_remove",
"claude_code_config_list",
"claude_code_config_show",
"claude_code_config_status",
"claude_code_config_import",
"claude_code_home_show",
"claude_code_home_remove",
"claude_code_home_removed",
"claude_code_home_restore",
"claude_code_judge"
],
"data": {
"own": [
{
"id": "agent-home",
"path": "${dir:agent-home}",
"class": "valuable",
"why": "the operator's agent's own files: its memory, history and projects"
}
]
},
"resources": [
{
"id": "package",
"type": "package",
"package": "claude-code"
},
{
"id": "managed",
"type": "directory",
"path": "/etc/claude-code",
"mode": "0755"
},
{
"id": "start",
"type": "file",
"path": "/usr/local/bin/claude-agent",
"mode": "0755",
"content": "#!/bin/sh\n# The mesh's (module claude-code, novox/hq ADR 0266): start the agent as the account agents run as on this\n# machine. Written whole at every push: an edit here is overwritten.\nagent='${machine:agent-account}'\nif [ \"$(id -un)\" = \"$agent\" ]; then\n\texec claude \"$@\"\nfi\n# Another account, the operator's, becomes the agent's through its own sudo: the person is the authority. The\n# operator's override of the guard travels only when it is set in this shell, as it would reach claude.\nif [ -n \"$MESH_GUARD_OVERRIDE\" ]; then\n\texec sudo -iu \"$agent\" env MESH_GUARD_OVERRIDE=\"$MESH_GUARD_OVERRIDE\" claude \"$@\"\nfi\nexec sudo -iu \"$agent\" claude \"$@\"\n"
},
{
"id": "agent-account-name",
"type": "file",
"path": "/etc/claude-code/agent-account",
"mode": "0644",
"content": "${machine:agent-account}\n"
},
{
"id": "agent-account",
"type": "user",
"name": "${machine:agent-account}",
"home": "${machine:agent-home}",
"root": "${machine:agent-root}"
},
{
"id": "agent-home",
"type": "directory",
"path": "${machine:agent-home}/.claude",
"mode": "0700",
"owner": "${machine:agent-account}"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "${machine:account}",
"place": "."
},
{
"id": "facts",
"type": "file",
"path": "${dir:state}/facts.json",
"mode": "0600",
"owner": "${machine:account}",
"content": "{\n \"node\": \"${machine:name}\",\n \"console\": \"http://127.0.0.1:${bound:mcp-endpoint:port}/mcp\"\n}\n"
},
{
"id": "settings",
"type": "file",
"path": "${dir:state}/settings.json",
"mode": "0600",
"owner": "${machine:account}",
"merge": "json",
"content": "{\n \"role\": \"\",\n \"mcp_servers\": {},\n \"managed_settings\": {},\n \"instructions\": {},\n \"output_styles\": {},\n \"skills\": {},\n \"commands\": {},\n \"agents\": {}\n}\n"
}
],
"shell": [
{
"for": "zsh",
"slot": "normal",
"code": "# The agent's session (module claude-code, novox/hq ADR 0266): where this machine names an account of the\n# agents' own, claude in an interactive zsh is claude-agent, which starts the session as that account. Where\n# agents run as the operator's account the file names that account, and nothing is added. claude-agent runs\n# the real claude: exec, sudo and its sh see no alias.\nif [[ -r /etc/claude-code/agent-account ]]; then\n\t() {\n\t\tlocal agent=$(</etc/claude-code/agent-account)\n\t\tif [[ -n $agent && $agent != $USERNAME ]]; then\n\t\t\talias claude=claude-agent\n\t\tfi\n\t}\nfi\n"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/claude-code",
"binary": "claude-code",
"loads": [
"claude-code"
],
"env": {
"MESH_CLAUDE_CODE_STATE": "${dir:state}",
"MESH_CLAUDE_CODE_FACTS": "${dir:state}/facts.json",
"MESH_CLAUDE_CODE_SETTINGS": "${dir:state}/settings.json"
}
}
]
}
}

Some files were not shown because too many files have changed in this diff Show More