Compare commits

...
Author SHA1 Message Date
jochen dc5a8208a2 The view reads directly and makes no consumer: a consumer's deliver subject publishes anywhere
mesh/delivery ready: it delivers once merged
mesh/merge-gate out of date: judged against a trunk it does not name, the trunk is now f8daec01d3f0; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against a trunk it does not name, the trunk is now f8daec01d3f0; judged again on a recheck or when it is merged
Review of the view (research 036): granted CONSUMER.CREATE on the bucket's stream, the view made a
push consumer delivering to mesh.mod.mesh-issues.event.opened, and a module subscribed there
received the bucket's entry as the tracker's event — measured on 2.11.17. The server does not hold a
deliver subject to its creator's permissions, so a consumer grant is a publish to any subject.

The view now binds and reads directly, nothing else: STREAM.INFO, DIRECT.GET by key, and the batch
DIRECT.GET (multi_last) that lists every key's newest value into its inbox. No watch: the page
re-reads the key a tracker event names (every event carries the number). The live test lists with
the batch, follows a moved event to the re-read, and is refused the consumer and the watch with
nothing reaching the event subject; the mutation (CONSUMER.CREATE back) fails three tests.

The credential says how to read, and that the step making it work is the next `bus upgrade` while
a new bus build waits, not a push.
2026-10-10 16:15:16 +02:00
jochen a5a355aeec The view: one read-only bus user for a page in a browser, composed like every other (hq research 036)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Research 036 names the gap (G1): no way for a browser to reach the bus. The bus module now listens
over WebSocket (mesh-catalog, nats); this is who connects there. The view is a fixed principal
(broker.KindView, user `view`) composed into the user list like every user once its credential is
minted, and left out once it is forgotten: it subscribes the issue tracker's events (opened, moved,
noted, linked), the controller's plan-moved and condition-raised/changed/cleared, and the delivery
owner's transition and group; it publishes only the JetStream API requests a read-only watcher of
the tracker's bucket (mesh-issues_issues) makes — STREAM.INFO, DIRECT.GET, CONSUMER.CREATE/INFO/
DELETE, flow control — answered in its own inbox; no reply, no tool, no event, no `$KV` write.

`bus view-credential` mints and prints it once (hash kept, like a person's); `bus view-revoke`
forgets it, real at the next composition. Tests: the composed grants are exactly these and a write
grant of any shape fails; the view is composed only once minted; and against a real server read from
the composed file over WebSocket, the view binds the bucket, reads a key, watches a put land, and is
refused a put, a delete and an event, the bucket unchanged.
2026-10-10 16:01:32 +02:00
mesh-admin 4d30b5de13 Merge pull request 'A module's own secret is asked for by a verb and typed at the desk, bounded, the prompt naming who asked (hq ADR 0277)' (#201) from feat/0277-secret-ask into main 2026-10-10 13:34:20 +00:00
jochen c97942d591 A module's own secret is asked for by a verb and typed at the desk, bounded, the prompt naming who asked (hq ADR 0277)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The give verb opened the desk's hidden prompt for a module's own secret, with the desk named. Now the
secret-ask verb (secret ask <node> <module> <name> [--at <desk>]) opens the same prompt from anywhere on
the mesh, with the desk the module's machine unless named, and give composes the same line. Every ask
is recorded in the store before the prompt opens (migration 0091): one open ask per secret, three an
hour, so an agent cannot keep a prompt in front of the operator. The prompt names who asked, from the
bus's word on the caller cut to a name's characters, never an argument of the call. The value stays
typed at the desk, sealed to the one call and then to the module's machine, never in an argument, a
log or an event; a secret the mesh makes itself and a trusted party's secret are refused as before.
2026-10-10 15:24:08 +02:00
mesh-admin 9c69b9da17 Merge pull request 'An own-path merge never shares a batch with a catalogue merge, and plans names a walk by what it moves (tracker issues 377, 378)' (#200) from fix/377-378-own-path-batches-and-named-plan-lines into main 2026-10-10 13:14:37 +00:00
mesh-admin 00065dbdaf Merge pull request 'A trusted setting is proposed through the settings verb and set only on the operator's warrant (hq ADR 0277)' (#199) from feat/0277-settings-propose into main 2026-10-10 12:59:13 +00:00
jochen 16362e1bbd A deferred walk is one whose note says so: a failed first ask is watched as before (review of #200)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 14:51:27 +02:00
jochen 3496b58f66 A walk let go behind another reads as a wait: its note on the line, never late, and no tier of it watched (hq ADR 0276 review of #200)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
2026-10-10 14:47:11 +02:00
mesh-admin d1cc9b4e20 Merge pull request 'Ask again after a refusal, with a growing wait (issues 369 and 373)' (#198) from fix/373-one-message-per-condition into main 2026-10-10 12:45:07 +00:00
jochen cc11de2513 A trusted setting is proposed through the settings verb and set only on the operator's warrant (hq ADR 0277)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Issue 339 made every trusted setting the controller's terminal's alone, so an agent could only hand the
operator a line to type at the control node. Now anyone the bus admits may PROPOSE a layer: settings
propose keeps the proposal in the controller's own asks (the asked bucket, which the controller alone
writes), judged as settings set judges, and asks the operator on the operator channel at the level
approve with every key, its exact new value (in its shape where a path or an address may not leave the
mesh), the was of a changed key, the removed keys and the layer's fingerprint. The serving controller
sets the layer on the warrant alone: once, for the ask it holds, only when the record's values still
digest to what the option bound and the layer is still the one shown, with the terminal's judgement and
history, and keeps who approved it beside the layer, which settings says back (migration 0090). Decline,
expiry, a cancel, a replacement or the router's refusal discard it; nothing is asked when no router, no
grant or no channel can carry it. settings proposals lists them. The push afterwards is a separate act.
2026-10-10 14:42:40 +02:00
jochen 04fcce2fcc An own-path batch is cut first and folds no waiting walk; a walk let go beside a started one starts once it ended; a late catalogue merge is not answered by a walk that waited for nobody (hq ADR 0276 review, tracker issue 377)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-10 14:33:10 +02:00
mesh-admin b9ceeace41 Merge pull request 'Merges are assembled in a rolling window, and each batch is walked once (hq ADR 0276, issue 362)' (#197) from fix/362-a-walk-answers-every-merge-it-contains into main 2026-10-10 12:26:20 +00:00
jochen 2e0a7d1cbd Review: an answered retry never brings its old refusal back (issue 369)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/373-one-message-per-condition stopped: superseded, not carried by a later delivery: novox/mesh-catalog@8dfc19c9f7c8
A refusal stood for its part whenever no ask was open, so after the retry
was taken and answered, "Questions for you not delivered" came back with
the old words for an hour, then again after each answer. A refusal is now
current only while no ask was made after it that the router did not refuse;
the verdict wait applies only to that newer ask. Tests reconcile inside the
wait and after an answer.
2026-10-10 14:14:07 +02:00
jochen 3ced96fc6f A merge on the controller's own path never shares a batch with one that waits for the delivery's word (hq ADR 0276, decided during the build)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 14:12:18 +02:00
jochen d3b4549611 Ask again after a refusal, with a growing wait (issues 369 and 373)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group fix/373-one-message-per-condition ready: every member ready, and composed together they pass
The router refused an ask while its channels had not yet said they could
send; both could twenty minutes later, and the controller repeated that
refusal every minute for eleven hours, escalating "Questions for you not
delivered" on it, because a refused ask was asked again only when the
channels' claims changed.

A refused ask is now asked again after 1 min, doubling with each refusal
in a row, at most 30 min, and at once when the channels change. While the
router's word on an ask made again is awaited (2 min), the condition
stands as it was, so it neither flaps nor clears early; once the ask is
taken it clears; a new refusal is said in its own words. Its explanation
no longer says its verdict twice.
2026-10-10 14:02:59 +02:00
jochen 065cfa0d1d Owe a merge to the record from its branch's first kept merge (hq ADR 0276 review)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
2026-10-10 13:54:40 +02:00
jochen 0e0e143055 Keep a merge a cut made history, and build a batch's walk at the branch (hq ADR 0276 review)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
2026-10-10 13:48:31 +02:00
jochen 2887691414 Walk the earlier merges of a failed walk as news (hq ADR 0276 review)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: FAIL github.com/novox/mesh-controller/cmd/mesh-controller 256.824s
mesh/delivery superseded: a newer head of the same pull request
2026-10-10 13:41:42 +02:00
jochen e1499d4196 A late merge is named however its modules read since the cut; a walk builds the commits it carries (hq ADR 0276 review) 2026-10-10 13:39:21 +02:00
jochen 96fc4209d3 Assemble merges in a rolling window and walk each batch once (hq ADR 0276, issue 362)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Every merge opened a walk and the next merge of the branch superseded it: two
catalogue merges 18 s apart left a walk no delivery held, and the operator
started it by hand 58 minutes later. A merge now joins the open batch, kept in
the store (migration 0089), which is cut into one walk when no merge came for
merge-window (90 s) or at merge-window-at-most (10 min): one commit per
repository, the latest of its branch, with every file the batch's merges
changed. One walk at a time; a started walk is never superseded, a waiting one
is folded into the next. The walk names every merge it answers on the wire
(delivery.merges, taken_over_by, batch). A failed walk walks its earlier merges
alone, newest first, until one is delivered. A delivery group's order becomes
tier edges inside the walk. plans shows the batch assembling; S18 and S19
bound its waits; S16 names the merges a waiting walk answers.
2026-10-10 13:20:04 +02:00
mesh-admin 8a53532d89 Merge pull request 'A plan says why each module is in it (hq ADR 0267, issue 363)' (#196) from fix/0267-a-plan-says-why-each-module-is-in-it into main 2026-10-10 11:04:07 +00:00
jochen d0161fac52 Say a deleted module's reason, read a context's reason at the merged branch, and keep the snapshot's bytes (hq ADR 0267, review)
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
2026-10-10 12:56:22 +02:00
jochen 887de9b5f2 Say why each module is in a plan: its build source's changed files, or why it is read whole (hq ADR 0267, issue 363)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
A what-if named the build seat's holder as packaging the controller's
source, "rebuilt without their own source moving", while it was in the
plan because an open plan had not built it yet. The what-if and the merge
log now say, per module, which changed files of its build source moved it,
or that it is read whole and why: no build source recorded, a newer build
failed, or a plan has not built it yet.
2026-10-10 12:49:44 +02:00
jochen 1560c498b6 Ask the rollback test's failed build after the registered one, whenever it runs
Its id named 2026-10-10 02:40 UTC; once the clock passed that, the
registered build read as newer and the test failed on main.
2026-10-10 12:49:44 +02:00
mesh-admin df6d72aec2 Merge pull request 'Record a module's act on the operator's warrant from the router's own record (hq ADR 0274)' (#194) from feat/warranted-hand-act into main 2026-10-10 10:39:34 +00:00
mesh-admin a6bc0936e1 Merge pull request 'The gate's module check notes a seat another module declares, which it was not given (hq issue 364)' (#195) from fix/364-a-touched-manifest-uses-a-seat-another-module-declares into main 2026-10-10 10:06:27 +00:00
mesh-admin 513ebd0577 Merge pull request 'A merge moves a module only when its build source holds a changed file (hq ADR 0267, issue 363)' (#193) from fix/0267-a-merge-moves-what-its-build-source-holds into main 2026-10-10 02:00:18 +00:00
jochen 88e84a4dd7 warranted says what the operator chose, never that the module acted (hq ADR 0274 review)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
2026-10-10 03:55:44 +02:00
jochen 7b5c063cd3 The gate's module check notes a seat another module declares, which it was not given (hq issue 364)
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
The gate passes only the manifests a change touches, so a module that starts using the operator channel
was refused for a declaration it could not see. Over every manifest it stays a refusal.
2026-10-10 03:53:07 +02:00
jochen abd3078491 warranted takes no word of the caller's: the record is the router's alone (hq ADR 0274 review)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
A caller's own line, recorded first under the one id the ask decides, would stand for every node's.
2026-10-10 03:52:04 +02:00
jochen 0e5aed1253 Record a module's act on the operator's warrant from the router's own record (hq ADR 0274)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A module that asks the operator acts with its own grants, and the hand-act log is where a person's
decisions are read back. The new verb warranted records who chose, how and with which proofs from the
router's record, never the caller's word, once per ask however many instances ask.
2026-10-10 03:40:49 +02:00
mesh-admin 6d3523ff10 Merge pull request 'A build says its build source; an image compiling Go is handed only that (hq ADR 0267, issue 363)' (#192) from fix/0267-a-build-says-its-build-source into main 2026-10-10 01:34:45 +00:00
jochen bd35b1c06c Judge a packaging module's news by plans that built it, over every plan since its build (hq ADR 0267, review)
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
A plan that closed without building a module hid a missed merge from it;
a window of recent plans let an old closure back in once the plan that
overtook it slid out; a merge the forge gave no time was acted on again
every pass. A plan answering the merge's own commit is now a look at it,
and clocks a little apart do not make a merge history.
2026-10-10 03:23:50 +02:00
jochen 150f038ff8 Read a module whole while a plan has overtaken its build source, and plan every view alike (hq ADR 0267, review)
A failed build, or a plan closed before reaching a module, left the
closure its last good build said, and a fix-forward to a newly imported
package would have moved nothing. A missed merge moving only a module
that packages the repository was never caught up, and an older merge
read as history for it through a look that was not its own. The gate,
a pull request's check, the what-if and a delivery's order now read the
same view the merge handler does.
2026-10-10 03:20:36 +02:00
jochen 6c616838a5 Move a module on a merge only when its build source holds a changed file (hq ADR 0267, issue 363)
Every merge to the controller's repository planned the controller, the
build seat's holder and the route proxy in three gated tiers, whatever it
changed (issue 338). The planner now maps a merge's files onto the build
source each module's newest trunk build said: a README moves nothing, the
controller's command the controller alone, the proxy's program the proxy
alone. A module with none said, or one an open plan has yet to build, is
read whole as before. Sharing a repository draws no packages edge any more,
and one recorded before neither widens nor orders a plan.
2026-10-10 03:20:36 +02:00
jochen de55c63e12 Hold a cgo file's directory whole: its preamble may include from below it (hq ADR 0267, review)
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
2026-10-10 03:20:35 +02:00
jochen abe5f7dc17 Say a build source only for the trunk's head, and hold what C, assembly and a new go.mod reach (hq ADR 0267, review)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A hand build of an older trunk commit said a closure lacking what was
imported since, and the planner would have mapped the next merge onto it.
C and assembly beside Go may include files below their directory, and a
go.mod made above a package moves it out of its module: each is now held.
2026-10-10 03:11:06 +02:00
jochen 4d1b81b6cb Say what a build was made from, and hand an image compiling Go only that (hq ADR 0267, issue 363)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A merge to the controller's repository moved the route proxy and the build
seat's holder whatever it changed, because nothing said which files their
builds read. A build of a trunk commit now says its build source per
repository: a Go program's import closure, an archive's directory, an
image's recipe and the package it names in the new 'compiles' field. That
image is built from its build source alone, so a recipe reading past it
fails by name, and its fingerprint is over what it was handed.
2026-10-10 02:47:35 +02:00
mesh-admin 9517f590ac Merge pull request 'Ask a node's engine for a fresh setuid search at the terminal (hq issue 361)' (#191) from fix/361-node-setuid-search-at-the-terminal into main 2026-10-10 00:06:33 +00:00
jochen 9cef820117 Ask a node's engine for a fresh setuid search at the terminal (hq issue 361)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
After the operator removes by hand what the last search found, no apply says so and nothing searched again
until the next day. node setuid-search <node> signs the ask as a hand-over is, under its own context, on a
subject only the node's engine hears and only the controller may publish.
2026-10-10 01:56:56 +02:00
mesh-admin 272ca2a578 Merge pull request 'Keep quiet for the setuid search the engine now runs to completion (hq issue 361)' (#190) from fix/361-the-setuid-search-runs-to-completion into main 2026-10-09 23:28:36 +00:00
jochen 7160d95323 Pin the node-engine at its merge of the resumable setuid walk (hq issue 361)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 01:19:17 +02:00
jochen 9551bc2380 Say the setuid search's quiet in the tests' words, and pin the node-engine at its reviewed head (hq issue 361)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/361-the-setuid-search-runs-to-completion delivering: 0 of 2 delivered
mesh/delivery superseded: a newer head of the same pull request
2026-10-10 01:11:24 +02:00
jochen cdaba36eca Pin the node-engine at its pull request's resumable walk (hq issue 361)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/361-the-setuid-search-runs-to-completion checking: 1 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
2026-10-10 01:01:18 +02:00
jochen eaf5195998 Keep quiet for the setuid search the engine now runs to completion (hq issue 361)
The node-engine's search has no bound any more: it runs at idle priority
and judges from its last complete, fresh result. The controller's quiet
while an agent account waits is the engine's rootsearch.Quiet, not the
old fifteen-minute bound, and the node-engine is pinned at its pull
request.
2026-10-10 00:54:52 +02:00
mesh-admin a6f831a633 Merge pull request 'Say a secret given in plain words, and log words the keeper refuses (hq issue 359)' (#189) from fix/the-secret-given-words-pass-plain into main 2026-10-09 22:00:15 +00:00
jochen a138c045bb Log a refused wording without what it quotes, and keep the secret-given words within bounds
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The reviewer found that the logged reason quoted the refused fragment: a
hash-shaped secret would reach the journal, and a changing clock time
defeated the once-per-kind dedupe. The reason is now logged without its
quoted fragment, the test resets the dedupe so it repeats, and a module
name too long for the headline falls back to the machine.
2026-10-09 23:44:19 +02:00
jochen 988e501ccc Say a secret given in plain words, and log words the keeper refuses
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
The secret-given condition's explanation carried a clock time, which the
plain rule refuses, so the operator read the scope's fallback ("needs a
look") instead of what changed. Its words now carry no time and say the
secret's name as words; a test holds them to the rule through a keeper.
With no test hook set, the keeper logs a refused or missing wording once
per kind and reason, so a fallback is never silent again (hq issue 359).
2026-10-09 23:41:22 +02:00
mesh-admin 19eefb66c6 Merge pull request 'node hand-over: the terminal hands a directory used as found to the mesh, asked of the node's engine (hq issue 356)' (#187) from fix/356-node-hand-over-at-the-terminal into main 2026-10-09 17:57:17 +00:00
jochen 081d9244d6 Merge remote-tracking branch 'origin/main' into fix/356-node-hand-over-at-the-terminal
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-09 19:44:55 +02:00
jochen b55a38ca9f Sign the hand-over ask, and fail the line on the engine's refusal (hq issue 356, review)
The subject proved nothing: the bus lets any principal allowed to answer reply to a message it received on the reply subject that message named, so a tool server — the operator's account, every agent — could deliver a hand-over to an engine. The controller now signs the ask with the mesh's key over a fixed context (node, path, who asked, a minute's expiry, a fresh nonce), as declarations are signed, and the engine verifies it. The writers table gains the row for mesh.node.*.ask.hand-over; the subject's comment no longer claims who the engine hears. The line's known-node check and the refusal branch are tested; every check was removed in turn and a test failed.
2026-10-09 19:44:55 +02:00
mesh-admin 747734687e Merge pull request 'Ask the operator only once the bus holds the controller's grant to ask (hq issue 353)' (#186) from fix/353-the-controller-asks-only-once-the-bus-holds-its-grant into main 2026-10-09 17:34:08 +00:00
jochen 9006c82393 node hand-over: the terminal hands a directory used as found to the mesh, asked of the node's engine (hq issue 356)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A module's condition told the operator to run the node-engine's hand-over at the machine, as root (issue 339), and the mesh had no channel for it. Now node hand-over <node> <path> is the controller's terminal's — a node subcommand that is not a read, so every verb and mesh-cli outside the terminal refuse it — and asks that node's engine on mesh.node.<node>.ask.hand-over, a request only the controller may publish and only that node's engine may hear and answer (its grant gains the subject and the right to answer what it was asked). The line's node and path are judged before anything is asked; the engine's answer is printed, a refusal as a refusal. Every text addressed to the operator names the nox line (ADR 0272); the condition's words stay plain.
2026-10-09 18:38:09 +02:00
jschoubben 0c8c9ffae9 Ask the operator only once the bus holds the controller's grant to ask (hq issue 353)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The grant is composed from the router's assignment and reaches the bus when its machine is next
pushed. Between assign and push the record said a router was here and the bus refused every ask
(seven refusals on 2026-10-09, 17:54 to 17:56). The asker now judges, as a push does, whether the
bus's machine was last sent the user list composed now; while it was not, nothing is published, it
is said once, and the conditions that need the operator are raised as undelivered, naming the push
that carries the grant.
2026-10-09 18:13:58 +02:00
mesh-admin 1c7c385839 Merge pull request 'A gate judges its own send and the build it sent, and never puts the controller back behind its store (hq issue 352)' (#185) from fix/a-gate-outlives-the-controller-and-judges-the-build-it-sent into main 2026-10-09 16:10:13 +00:00
mesh-admin 65ff6159ad Merge pull request 'mesh-cli: give a terminal line the standard input mesh-cli carried, refuse it to any other, keep it nowhere (hq ADR 0259 §10, ADR 0272)' (#184) from feat/a-terminal-line-takes-standard-input into main 2026-10-09 16:10:11 +00:00
jschoubben e6e1e3bc89 A gate judges its own send and the build it sent, and never puts the controller back behind its store (hq issue 352)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
On 2026-10-09 a release's gate on the control node read the machine's
report against a newer send another plan had just made there, failed
three builds the machine had reported healthy, and put them back on
every machine to a controller older than the store's schema; that
controller then passed the newer plan's gate from its own health.

- A gate keeps what its send carried (digest, sequence) and reads the
  report against it; a report on the last send is on it too.
- A gate judges only the build the machine was last sent: another build
  there supersedes the judging — no verdict, nothing put back.
- A controller is told its build (MESH_CONTROLLER_VERSION, ${version}
  in a process's env) and records how far it reads the store's schema;
  a put-back to a build that reaches less, or never said, is refused
  and the current build kept, said as urgent.
- A release's open gate holds other sends of its modules there, and a
  plan's own first send waits on it.
2026-10-09 17:15:40 +02:00
jschoubben 07e59c535e Pin mesh-host at its main (d8ff154), where the installer's first user list carries the controller's ask grants
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer delivery to the same trunk took over its walk
mesh/delivery-group group feat/a-terminal-line-takes-standard-input stopped: a member was stopped
The repo-check reads the installer's user list at the pinned commit, and the old pin predated mesh-host
#59 and #68: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose failed on main's own grants.
2026-10-09 17:10:16 +02:00
jschoubben ba97297f66 mesh-cli: give a terminal line the standard input mesh-cli carried, refuse it to any other, keep it nowhere
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.62s)
mesh/delivery-group group feat/a-terminal-line-takes-standard-input rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
A secret given at the controller's terminal through mesh-cli (secret accept … --from -) never reached the
line: every line ran with no standard input. A line that runs as the terminal now reads what mesh-cli
carried (at most 64 KiB); an ordinary call carrying any is refused and nothing runs; the calls record keeps
only that some was given, the journal and the answer nothing of it.
2026-10-09 17:01:36 +02:00
mesh-admin e6b00e2e51 Merge pull request 'give: take a module's own secret through a hidden prompt at the operator's desk (hq ADR 0259 §10)' (#156) from feat/a-secret-given-at-the-desk into main 2026-10-09 14:30:47 +00:00
jschoubben fa19a2d718 give: test that a trusted party's secret given at the terminal is announced before it is kept, and refuse an unknown machine before anybody types
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The terminal path's order is one function, keepGiven, so the test fails when the announcement before a
trusted party's secret is removed, and when a failed announcement still keeps it. give also refuses a
machine the mesh does not know, as the secret's or as the desk, before the prompt (the final review).
2026-10-09 16:22:47 +02:00
jschoubben 3e5086a2f6 give: never take a trusted party's secret at a desk, and announce it before it is kept (hq ADR 0259 §10, the confirmation review's N1-give)
A desk's prompt is answered over the desk machine's bus, on a subject that machine's runtime answers;
on a desk machine agents run as the operator, who holds that credential, so an agent could answer first
with a bot token of its own sealed to the call's key. The secret of a module running as an account of
its own (the Telegram bot's token) is now refused at the desk, before anybody is asked, naming the line
to type at the controller's terminal; there it is announced on every channel, the old one among them,
before it is kept, and not kept when that announcement fails. What is typed at the terminal is not
echoed. Tests: the refusal with nobody asked, and only the desk machine's launcher may answer its
prompt (MaySubscribe).
2026-10-09 16:22:47 +02:00
jschoubben ed331eb972 Let the give verb's exact line past the terminal rule for secrets, and nothing else (hq ADR 0259 §10, ADR 0266)
Restacked on #157, which carries #164's rule that no verb runs secret accept. give's line carries no value:
the operator types it into the desk's hidden prompt, sealed to the call and then to the module's machine.
Only that exact line passes: a value, a file, a provider or any extra word stays the terminal's.
2026-10-09 16:22:47 +02:00
jschoubben be59f29f46 give: take only a value a person holds, ask the desk by name, let the controller alone ask it, and announce every value given
The review of 2026-10-09 (M4):
- give refuses broker (the bus account issue mints) and any own secret the mesh may make itself;
- the desk's prompt is asked by module, secret and machine, never with words of the caller's, and the
  bus denies the prompt's subjects to every principal but the controller, however wide its grant, so the
  prompt's 'the controller asks' is the bus's word (Permissions.PublishDeny, broker.ControllerOnly);
- secret accept with a value is refused through a verb: a value comes from the terminal or the desk;
- every value given for an own secret, at the terminal or the desk, raises the urgent condition
  secret-given on every channel, until the operator silences it.
2026-10-09 16:22:47 +02:00
jochen 5689553406 Take a module's own secret through a hidden prompt on the operator's desk, so a bot token never passes through an agent's session (hq ADR 0259 §10) 2026-10-09 16:22:47 +02:00
jschoubben 0559b80887 Move to mesh-sdk 16984aa, rebased on its main, which refuses a warrant with no time or for an ask with no expiry
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.56s)
mesh/delivery stopped: the pull request closed unmerged
2026-10-09 16:22:34 +02:00
jschoubben 74d35600a6 Keep an approval asked through a silence of its condition (hq ADR 0259, the confirmation review's M1)
Choosing Silence silenced the condition, the condition was no longer wanted, and the next reconcile
cancelled the Restart or Release ask beside it: an acknowledgement, which any desk click may give,
took an approval back. An open approval ask now stays until it is answered or expires while its
condition is open and silenced with the same answers. The test silences as the controller does; it
failed before (0 open) and passes, and the kept Restart is performed on its warrant.
2026-10-09 16:22:34 +02:00
jschoubben d19c9ed5b7 Start a rehearsal only at the controller's terminal as main now judges it (hq ADR 0259, ADR 0272)
rehearse refused only a verb's process. Since mesh-cli (ADR 0272 §4) the serving controller runs an
ordinary mesh-cli line without a verb, naming its caller: such a line, from an agent's account, read as
the terminal and could start a question the operator did not ask. rehearse now asks
startedAtTheTerminal. And main's mesh-cli test helper asked is cliAsked, beside the asker's asked.
2026-10-09 16:22:34 +02:00
jschoubben 799eec0a5a Ask an acknowledgement apart from an approval, change every kept ask by compare-and-set, and rehearse rather than drill (hq ADR 0259, review M1/L2/L3/L7)
- M1: a condition offering both kinds of answer is asked twice: its authorising answers about the
  condition, its acknowledging ones (Silence) apart, so an answer from a channel that only acknowledges
  never ends an approval.
- L2: the asked store creates once and changes only over the revision it read, deciding again on what it
  reads; a stale cancel no longer writes over an act.
- L3: every ask is kept before it is published, one whose publishing failed is marked unsent and asked
  again, and a cancel is kept before it is said. The terminal's test question is now `rehearse`, so it is
  not called what the glossary calls a drill; its two answers are both approve-level.
- L7: two deliveries of one warrant to two controllers at once act exactly once, on a real bus.
- Re-vendored onto mesh-sdk 76902998 (canonical digests): an option binds an asks.Act with each argument
  as arg.<name>.
- The lab's bus fixture composes verified-sender only where the lab says its machine is root-free
  (MESH_LAB_ASKS_ROOT_FREE=true).
2026-10-09 16:22:34 +02:00
jschoubben ad406e81b8 Say loudly when a condition that needs the operator could not be asked on any channel (hq ADR 0259)
With no router, or an ask the router refused and nothing changed since, the controller asked nothing
and said it only in its own log. It now keeps a condition of its own, asks-undelivered, naming the
conditions not asked and why, cleared once each can be asked again.
2026-10-09 16:22:34 +02:00
jschoubben 646c5e53db Add drill: an ask the operator starts at the controller's terminal, whose approval performs nothing and is recorded (hq ADR 0259)
The live acceptance needs an approval the operator can ask for at will and that changes nothing. A
drill is asked like any condition's ask, bound to its own act, claimed once on its warrant and recorded
as a warrant hand-act with who answered, through which channel and the proofs. A verb's process may not
start one, so no agent asks the operator a question they did not start.
2026-10-09 16:22:34 +02:00
jschoubben 2190e2c664 Compose and raise the bus of the lab's proof of the operator's answers, as this controller would (hq ADR 0259)
mesh-lab's asks proof runs the router, the Telegram channel and an asker on a real bus. Its accounts,
streams, workers, buckets and memberships come from this test at the controller's commit, so the lab
proves the composition and not a copy of it. Skipped unless the lab asks.
2026-10-09 16:22:34 +02:00
jschoubben 0909d7b125 Bind each asked option to the exact act, and perform only that act on its warrant (hq ADR 0259 §6)
Every option the controller asks with carries the digest of its verb, machine, arguments and level
(the SDK's Option.Binds). A warrant must name the digest of the ask the controller keeps, and before
acting the controller checks that the act it is about to perform is the one the option bound: a
record changed after the ask is refused, never performed. mesh-sdk moves to d4077b4.
2026-10-09 16:22:34 +02:00
jochen 744b0b9162 Ask at most three at a time, wait out a refusal, need a router, and act only on a claimed open ask, as the review asked (hq ADR 0259) 2026-10-09 16:22:34 +02:00
jochen 8de4dc7951 Ask the operator for a condition's answers and act on the warrant, so release, stop, start and restart can be answered from any channel that proves who answered (hq ADR 0259) 2026-10-09 16:22:34 +02:00
mesh-admin 2913c54c29 Merge pull request 'Kinded benches, verbs named by their caller, proofs and records (hq ADR 0259 §3)' (#154) from feat/asks-answered-on-any-channel into main 2026-10-09 14:17:43 +00:00
mesh-admin ef26d4cb0f Merge pull request 'Test the newest merge's order, a second reopening's gaps and sub-second merge times (hq issues 348, 349)' (#183) from fix/348-349-test-gaps into main 2026-10-09 13:55:14 +00:00
jschoubben d9a730307c Test the newest merge's order, a second reopening's gaps and sub-second merge times (hq issues 348, 349)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The review of PR 179 found four paths no test held: which of two earlier
plans NewestMergeOf takes, a tie between them, gaps kept across a second
reopening in one keeper, and a merge time's fraction of a second.
2026-10-09 15:29:31 +02:00
mesh-admin 9ca7952d5e Merge pull request 'Judge a send by when a fault began, not when it was last raised (hq issue 348)' (#179) from fix/a-fault-from-before-a-send-fails-no-gate into main 2026-10-09 13:25:21 +00:00
jschoubben 58cb586c37 Answer the review of hq issues 348 and 349: gaps, parts, the newest merge in any state
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
- A reopened fault keeps its gaps: it was there at a send unless the send
  fell in one, so a send that breaks a machine recovered before it still
  fails its gate (A2).
- An undecided part holds only the conditions that name it (A4).
- D2 holds a silent resolver for the next run again, refused or not: a
  burst of refusals is also a restart (A3).
- A late merge is planned at the newest planned merge of its branch in any
  state, not only an open one (A1); merge times to the nanosecond (A5).
2026-10-09 15:00:56 +02:00
jschoubben c3c56a69e2 Take either binding until the catalogue's main binds once (hq issue 348)
The test reads the catalogue beside it, which the build seat checks out at
main; mesh-catalog PR 161 changes the binding, so the two land in either
order.
2026-10-09 15:00:56 +02:00
jschoubben 63b87b6f7b Hold the resolver to bind-interfaces, as mesh-catalog PR 161 makes it (hq issue 348) 2026-10-09 15:00:56 +02:00
jschoubben 997a4925b0 Order a branch's plans by its merges, and build the newest commit (hq issue 349)
A merge acted on late by the catch-up made its plan after the plan of the
merge that followed it, superseded it by creation time, and folded its
unbuilt modules into a plan at the older commit: on 2026-10-09 the
forge's security fix (a082615b) was superseded by 8ff8197a. A plan now
keeps its merge time (migration 0086), supersession follows it, and a
merge older than an open plan of its branch is planned at that plan's
commit, which contains it.
2026-10-09 15:00:56 +02:00
jschoubben 077ddf0eb8 Judge a send by when a fault began, not when it was last raised (hq issue 348)
On 2026-10-09 the control node's resolver refused from 10:57:57 UTC. A
node-engine restarted by the 10:59:34 send said its names undecided, that
statement cleared the network condition, the next look raised it again
after the send, and the gate put back two builds for a fault older than
them.

- A condition keeps First across a reopening; the gate reads Began.
- An undecided network statement (unknown, starting) clears nothing.
- D10 counts what a release's tier names as rolling, so a walked
  node-engine is not core-behind on its own first machine.
- D2 raises a resolver that refuses every try at once: a refusal is an
  answer, not a loaded resolver (issue 277).
2026-10-09 15:00:56 +02:00
mesh-admin c58516f819 Merge pull request 'plan --diff says which modules a push would leave out, and why' (#181) from fix/plan-diff-says-what-it-leaves-out into main 2026-10-09 12:41:15 +00:00
jochen 54b04a7604 plan --diff says which modules a push would leave out and why, so a module just assigned whose settings cannot compose is not shown as "nothing would change"
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-09 14:27:31 +02:00
mesh-admin af025562c7 Merge pull request 'Say the terminal explicitly, only from a login session, and name the unannounced pull request (hq ADR 0272)' (#180) from feat/272-the-terminal-said-explicitly into main 2026-10-09 12:02:52 +00:00
jschoubben c544c2a17b Key root-not-free apart from DA, keep ADR 0266's quiet window there, and judge at one clock (hq ADR 0259 §8)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer delivery to the same trunk took over its walk
mesh/delivery-group group feat/asks-answered-on-any-channel stopped: a member was stopped
The confirmation review of 2026-10-09 found D-root and DA writing one key, machine.<m>.agent-root, from
two probes with different words, so it flapped every run; D-root is now root-not-free. D-root raised the
urgent condition after every node-engine restart while the first setuid search ran; it now keeps the
same quiet window as DA, and the root-free verb still answers that machine not free. agentConfined
takes the judging clock.
2026-10-09 13:55:06 +02:00
jschoubben 5866b2db94 Hold a private kind's holder to an account of its own, as a verified one is (hq ADR 0259 §7, §8)
A private kind is where the router shows a link's code, and the code makes an account the operator's.
Registration refused a verified-sender holder on the machine's runtime and let a private one stand;
it now refuses both (the confirmation review of 2026-10-09, low).
2026-10-09 13:51:54 +02:00
jschoubben 983bf65141 Answer the controller's own verbs, root-free among them, from the serving controller alone (hq ADR 0259 §8)
The confirmation review asked who may answer root-free on the bus. Composed from the controller's own
manifest, the module principal of the machine running the controller and that machine's runtime were
granted the controller seat's tool subjects too: either could answer root-free, and the runtime's
credential is one an agent on that machine may hold. The controller's seat is now served by the
controller principal alone, in grants and memberships; TestOnlyTheServingControllerMayAnswerRootFree
failed before (3 answerers) and passes. And a machine waiting for its first setuid search is not
root-free, whatever ADR 0266's quiet window does to the self-check.
2026-10-09 13:51:18 +02:00
jschoubben 0e46b8302d Let root-free take its machines as a list, as the router names them
The router's contract names the machines as a JSON array. A verb's argument declared a list now takes
an array of names (or one text separated by commas), and refuses anything else in it.
2026-10-09 13:48:56 +02:00
jschoubben 4c375dafed Judge a machine root-free only on a positive, fresh measure, and believe a verified sender only there (hq ADR 0259 §8, review H2/H3)
The agent-root probe read the sudo module's answer, given in the machine's runtime as the very account
an agent could become, and took a missing account, a missing answer or no accounts as a pass. One
judgement now decides: the machine names an agent account its node-engine judged unable to become
root within 15 minutes (agentConfined, mesh-controller #164), and the login shell's execute is not
served there; anything not read is not free. The probe raises agent-root on it, the new root-free
verb answers it live for the router, and a push composes verified-sender for a kind only while its
machine and the router's pass it.
2026-10-09 13:48:56 +02:00
jschoubben e2a45b18ba Refuse a module of its own account running as the node's operator or agent account (hq ADR 0259 §8, review L4) 2026-10-09 13:48:56 +02:00
jschoubben 5fa8e40667 Raise agent-root where who can become root is not measured, so the router never reads a missing measure as a no (hq ADR 0259 §8)
A machine where the router or a verified channel runs and the sudo module is absent or does not answer
made the probe fail to run, which raises nothing the router reads, so it went on approving there. Each
such machine now raises the same urgent condition, saying it was not measured.
2026-10-09 13:48:56 +02:00
jschoubben b3fd360ddb Count the login shell where its execute is served, not where its seat is held (hq ADR 0268)
The control-node withholds execute through its holder's setting since ADR 0268, so probe D-root read a
closed path as open. It now counts the verb as served while the holder's setting for that machine is
serve, or the bus hears execute answered there, or the bus could not be asked: a withheld value not yet
pushed, or a holder answering against its setting, is never taken for closed.
2026-10-09 13:48:56 +02:00
jochen 9372e80cec Serve a trusted holder from a runtime of its own account, refuse it in the machine's runtime, and say while an agent can become root where it runs (hq ADR 0259 §8) 2026-10-09 13:48:56 +02:00
jochen c9be75b13e Carry a channel's capabilities on its claim and tell the router every kind, so an answer is judged by the controller's record and not the channel's word 2026-10-09 13:48:56 +02:00
jochen f5f315cc95 Grant a seat's traffic by caller and by kind, so an ask's asker and a channel's kind are facts the bus enforces (hq ADR 0259) 2026-10-09 13:48:56 +02:00
jochen 51c8c7ec52 Say the terminal explicitly, only from a login session, and name the unannounced pull request (hq ADR 0272)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
- commandEnvironment takes the terminal as a bool instead of reading an
  empty verb as one; every non-terminal line names its verb and is stripped
  of MESH_CLI_TERMINAL, and a test with the mark set in the serving
  environment fails when that strip is taken out.
- On the control-node the operator's account is the terminal only from a
  login session, as the node-engine reads it from the kernel's cgroup; the
  tool runner and the account's user units run as the operator too, and are
  ordinary calls. The request carries `session` (field-name tests on both
  sides).
- A pull request the forge never announced is named with its number in
  the condition's headline (hq issue 347), from the stalled line's
  `number`, which mesh-delivery sends.
2026-10-09 13:41:32 +02:00
mesh-admin 63e85b25e6 Merge pull request 'Say a pull request the forge never announced as one, with what to do (hq issue 347)' (#178) from fix/347-an-unannounced-pull-request-says-what-to-do into main 2026-10-09 11:38:47 +00:00
mesh-admin 93c6ca5432 Merge pull request 'Answer mesh-cli: the control-node's operator is the terminal, everyone else is not (hq ADR 0272)' (#176) from feat/272-answer-mesh-cli into main 2026-10-09 11:38:43 +00:00
jochen 14ab2ddd9b Announce this head: the pull request was opened after its push, which the announcer misses (hq issue 347)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-09 13:22:22 +02:00
jochen 510c91f144 Say a pull request the forge never announced as one, with what to do (hq issue 347)
mesh-delivery now says a pull request on a base that requires the merge
check, with none on its head for ten minutes, as a stalled line in state
unannounced, which D14 raises as delivery.<id>.stalled. No delivery exists
for it, so the generic words — stop it, release it — named an act that
does not apply. It now says the pull request has had no merge check, why,
and that a new commit on its branch is announced and checked.
2026-10-09 13:15:20 +02:00
mesh-admin 33dc85d850 Merge pull request 'Judge the one protection rule the forge applies, and keep a recorded repository id (#174 follow-up)' (#177) from fix/the-forges-first-rule-and-a-kept-identity into main 2026-10-09 11:14:28 +00:00
jochen ea1d3ed96d Merge main (hq ADR 0266) into the mesh-cli answer, and close what the confirmation review found
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@14ab2ddd9b49 (merged as 63e85b25 into main, walk plan-17915459…
- The generic command verb only reads now (commandReads) and terminal-only
  commands are refused through any verb (terminalOnly). mesh-cli's
  ordinary line made neither check: `node account`, `token issue` and
  `secret export` from another node would have run. It now meets both, in
  the one function the command verb shares.
- The serving controller marks itself and its children never the terminal
  (ADR 0266); a line mesh-cli runs as the terminal drops that mark and
  carries MESH_CLI_TERMINAL, so it reads as the terminal it is.
- Two withholding tests searched the answer's text while JSON writes bytes
  as base64, so they held nothing. They search both now, each proved by
  disabling what it guards (Shown, the bus withholding, `calls` via Get).
- The control-node refusal is tested through the assign and unassign acts.
2026-10-09 13:08:48 +02:00
jochen eca6390d6f Judge the one protection rule the forge applies, and keep a recorded repository id
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The forge applies the rule named for a branch, else the first glob covering
it; the judge passed a branch whose applied rule let pushes when a later rule
happened to guard it. And a registration whose id the forge could not give
cleared the id already recorded (the confirmation review of 2026-10-09).
2026-10-09 12:58:28 +02:00
mesh-admin 5438d7ff75 Merge pull request 'Do not raise agent-can-become-root while the first setuid search is within its bound (hq ADR 0266, note E)' (#175) from fix/a-search-not-finished-after-a-restart-is-not-urgent into main 2026-10-09 10:56:15 +00:00
mesh-admin eb7a158086 Merge pull request 'Register a module only from the repository the catalogue builds it from (hq ADR 0266 §7, route A)' (#174) from fix/a-build-registers-only-from-its-modules-source into main 2026-10-09 10:56:08 +00:00
mesh-admin 03dac87d3a Merge pull request 'Name the account agents run as on a node, and say whether it can become root (hq ADR 0266)' (#164) from feat/a-node-names-the-account-its-agents-run-as into main 2026-10-09 10:56:04 +00:00
jochen 0c47f48537 mesh-cli lines are calls, followed to their answer; the terminal does not follow assign (review of hq ADR 0272)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
- A line ran past the bus's one-minute window for an answer and its answer
  was refused; mesh-cli then said nothing ran of a line that had. Every
  line is now a call (calls.go): answered within AnswerWithin that it is
  running, with its call, and followed by the same account on the same
  node until it ends. Its record keeps the command word only, and its
  answer stays in the memory of the controller that ran it — never on the
  bus, never in `calls`, which answers anyone who may call the seat. Each
  line is said in the journal with its call, who asked where, and how.
- Lines running at once are bounded (8); one over is answered busy.
- An ordinary `settings` line is composed as the settings verb composes
  its own and meets that verb's refusals, the terminal-only keys among
  them, instead of the generic command's blanket refusal.
- The controller's module is assigned and unassigned at the terminal only:
  where it runs is the control-node, whose operator is the terminal.
- mesh.control.*.cli is in the writers table as the machine's own.
2026-10-09 12:48:12 +02:00
jochen 2e1a9abbf7 Count the quiet after a restart from when the controller first saw the search pending
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery failed: its walk failed: a gate on a first machine (what it carried put back), a build, a machine
The node-engine's own since starts again at every restart, so an agent that
restarted the engine in a loop kept agent-can-become-root quiet for ever (the
review of 2026-10-09). The controller now keeps when it first saw the verdict
waiting for the setuid search (migration 0085), forgets it at the next
complete verdict, and raises once the engine's own bound has passed since;
the bound and the pending reason are read from mesh-host's rootsearch.
2026-10-09 12:37:18 +02:00
jochen 95e7a7120a Register only from a protected trunk of the same repository, and mark the serving controller never the terminal
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@eca6390d6fe4 (merged as 33dc85d8 into main, walk plan-17915444…
A source repository's name is not its identity, and a trunk anyone may push
to makes the trunk rule mean nothing (the review of 2026-10-09). Through any
verb a module now registers only from a repository on the mesh's forge whose
trunk refuses direct pushes, requires a status and lets no administrator
merge past one, asked of the forge's own tools; and only from the repository
by the forge's id, recorded at registration (migration 0084), so one deleted
and made again under the name is refused. The serving controller marks its
environment, so nothing it runs or starts reads as the terminal, and a
terminal request covers only the repository and path it asked.
2026-10-09 12:37:18 +02:00
jochen d15eee61bb Do not raise agent-can-become-root while the first setuid search is still within its bound
After every node-engine restart the account verdict says not judged yet until
the engine's first search for setuid programs ends, and DA raised the urgent
condition each time. The account stays unconfined and node show still says
not judged; the condition is raised once the search fails, runs out its bound,
finds a way to root, or the statement goes stale.
2026-10-09 12:37:18 +02:00
jochen 1b780eae3a Put back on a failed gate only a build of the module's own repository
A build refused for its repository is still recorded, and a fork carries the
commit the module was registered at: the rollback's search for the previous
build would have found it and registered it by the back door.
2026-10-09 12:37:18 +02:00
jochen a5e8baf6da Register a module only from the repository the catalogue builds it from
An agent could make a repository of its own, or fork one the mesh builds
from, commit a module.json naming sudo or mesh-host, and ask the build verb
for it: the outcome was registered under that name, and the next push made
whoever wrote it root on every node (novox/hq ADR 0266 §7, the review of
2026-10-09). The trunk rule checked the trunk of the repository built, which
was the agent's.

The take-in, which every outcome reaches whichever verb asked it, now
registers a module only from its registered repository, and a new module only
from a repository the catalogue already builds from (a merge adding one);
anything else only when the build request was kept as asked at the
controller's terminal (migration 0084). Through a verb, a build of a
repository the catalogue builds nothing from is not asked at all.
2026-10-09 12:37:18 +02:00
jochen e168b60159 Pin the node-engine at #61's head, which reads the homes in one order
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer delivery to the same trunk took over its walk
Only the engine's own test changed; the pin follows the head so the two are
judged together (hq ADR 0266). Move it to the engine's merged commit before
this merges.
2026-10-09 12:37:12 +02:00
jochen d951f22c04 Pin the node-engine at the head whose setuid search runs to its own bound
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
mesh-host #61 dropped the two-minute limit inside the search's 15-minute
bound, asks pacman once, and calls a walk with errors incomplete; the pin
follows it so the two are judged together (hq ADR 0266). Move it to the
engine's merged commit before this merges.
2026-10-09 12:12:32 +02:00
jochen b1897a3498 Answer mesh-cli: the control-node's operator is the terminal, everyone else is not (hq ADR 0272)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The records send the operator to "the controller's terminal", and nothing
reached it (hq issue 343). The serving controller now answers each node's
mesh.control.<node>.cli, where only that node's engine may publish, with the
account the engine read from the kernel. A line from the control-node's
operator account runs as the terminal: a fresh process of this binary with
no MESH_VERB, whatever the serving process carries. The same account on any
other node, where agents may run as it, is an ordinary call: the generic
command verb's refusals (one function now, so the two routes cannot drift)
and MESH_VERB=mesh-cli, so a terminal-only change is refused with its
reason. Any other account, root included, runs nothing. Servers are never
run, and an answer over one bus message is cut and says so.

The terminal-only refusals and the usage now name mesh-cli on the
control-node as the way to the terminal.
2026-10-09 12:10:24 +02:00
jochen 926dbd7a97 Fill machine facts in a user's home, so an agent account named with a home is made there
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
node agent-account <node> <account> [home] stored a home that nothing used: the
account was created at /home/<account> while its files went to the named home
(hq ADR 0266). The engine reads a user's home only when it creates the account,
so an existing one is never moved.
2026-10-09 11:02:44 +02:00
jochen db702312af Pin the node-engine at its pull request rebased onto main
The engine's pull request now keeps main's placement guard (issue 339) and adds
only the agent-home rule; the pin follows it so the two are judged together
(hq ADR 0266). Move it to the engine's merged commit before this merges.
2026-10-09 11:02:44 +02:00
jochen d7fe5b609b Pin the node-engine at the head that refuses a placement at the machine's own
The validator and the wire are unchanged; the pin follows mesh-host's pull
request so the two are judged together (hq ADR 0266).
2026-10-09 10:12:41 +02:00
jochen b4dff64ae0 Refuse a plans line that acts wherever its subcommand stands
A flag before the subcommand (plans --json go <id>) still acts, so the
generic command verb judges every word of a plans line, retry included
(hq ADR 0266). The rest of this change, places and accesses at the
terminal and one line per setting, is issue 339's on main (#168, #170,
#172), which now does it for every process a verb runs; this branch's
--through-verb flag and its copy of those rules go.
2026-10-09 10:12:41 +02:00
jochen 2b8a28adab Pin the node-engine at the head that judges an opened file's kind and links
Keeps the controller judged together with mesh-host's pull request (hq ADR
0266); the validator and the wire are unchanged.
2026-10-09 10:12:05 +02:00
jochen 20d4f1ae71 Let the generic command verb only read, and keep keys and tokens at the terminal
Review found a chain through the command verb: set the operator's key to one
the caller holds, rotate secrets so they are sealed to it too, read the sealed
copies, open them. Whoever may call a verb includes agents (hq ADR 0266), so
command now runs an allow list of reading forms, and operator, identity,
token, broker, api, licence and every secret command but rotate are refused
through any verb.
2026-10-09 10:12:05 +02:00
jochen 528951319c Pin the node-engine at the commit that refuses a system account as the agents'
The validator is unchanged; the pin follows the mesh-host pull request's head
so the two are judged together (hq ADR 0266).
2026-10-09 10:11:21 +02:00
jochen dcbbf4487a Refuse a node's accounts through any verb, and a stale or service-account agent verdict
The generic command verb ran node account and node agent-account, so an agent
could name itself the operator account and have the next send grant it root
(hq ADR 0266 review). Refuse every node subcommand but list and show through
any verb; refuse the operator account as the agent account in both
directions and well-known service accounts as an agent account; and count a
verdict heard more than 15 minutes ago as not judged, so stopping the
node-engine cannot freeze a healthy one. Re-pin mesh-host to its review head.
2026-10-09 10:11:21 +02:00
jochen fcfbf7e69e Name the account agents run as on a node, and say whether it can become root
On the control node every agent ran as the operator's account, which has
passwordless sudo, so an agent could become root without a person (hq ADR
0266). A node now names an agent account at the controller's terminal only;
the agent's module declares it never to become root, the node-engine judges
that, and the self-check (DA) raises agent-can-become-root while it does not
hold, so ADR 0259's router can rest on it.
2026-10-09 10:11:21 +02:00
mesh-admin 4d60628f37 Merge pull request 'A mergeable file's own keys are the terminal's (hq issue 340)' (#172) from fix/340-a-mergeable-files-own-keys-are-the-terminals into main 2026-10-09 07:46:40 +00:00
mesh-admin b13a0f71a4 Merge pull request 'Tell the operator at once when sends wait for the bus's planned step, and say it before the merge (hq issue 336)' (#173) from fix/336-bus-step-waiting into main 2026-10-09 07:23:51 +00:00
jochen 822e52123c Say S17 clears once the new bus is sent, as it does
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The comment and the signals table's bound said it cleared once the bus's machine runs the new build; it
clears once that machine has been sent it, when no send is refused for the bus any more.
2026-10-09 03:23:47 +02:00
jochen 9a7ae131cb TestReplay336 reads the bus call in its mesh form
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The previous commit named the upgrade as a mesh call, and the replay still looked for the command-line
words; it now asserts the seat and the upgrade, as any wording of the call says them.
2026-10-09 03:08:21 +02:00
jochen 55d8b43f30 Refuse any change through a verb to a module with a trusted mergeable file
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A mergeable file takes any key, not only those its content names, so an
empty runtime configuration a provider reads took a url of the caller's
through the settings verb (hq issue 340 review).
2026-10-09 02:54:41 +02:00
jochen 4354d9d7c7 Name the bus upgrade as the mesh call a person makes, and say a same-source rebuild needs no step (review of #173)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestReplay336 (1.12s)
mesh/delivery superseded: a newer head of the same pull request
The condition and the plan named the verb in command-line form; the operator reaches it through the mesh
MCP server, so it is written as that call. A rebuild of the same source moves nothing (issue 280), which
the plan cannot know before the build, so its line says so.
2026-10-09 02:54:23 +02:00
jochen 1fdc68a8aa Tell the operator at once when sends wait for the bus's planned step, and say it before the merge (hq issue 336)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
For 28 minutes on 2026-10-08 every send to the control-node was refused for a new bus build that only a
person's bus upgrade moves, and no condition said so: the refusal lived only in each walk's note, and S3
would have called it lateness after half an hour, in words that named neither the bus nor the verb.

- Row S17, bus.<module>.step-waiting: raised on the first watchdog tick after a walk's send is refused for
  the bus, for the operator, naming the machines, what waits, the bus build from and to, since when and
  mesh-controller.bus upgrade. It clears once the bus's machine runs the build the mesh holds.
- S3 leaves out a walk held only by the bus's step.
- A change that builds the bus says in its delivery plan and summary (which mesh/merge-gate carries) that
  merging it needs a person's bus upgrade, and that nothing else reaches its machine until then.
- TestReplay336 fails on the commit before and passes on this one.
2026-10-09 02:33:57 +02:00
mesh-admin 36008e0642 Merge pull request 'Fill ${setting:} in a service's unit name, and refuse a value that makes no unit name' (#171) from feat/setting-in-a-unit-name into main 2026-10-09 00:32:51 +00:00
jochen f476494173 Make a mergeable file's own keys the terminal's, so no verb can set what every agent session obeys
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The claude-code module keeps the managed settings and tool servers every
Claude Code session on a node runs in a mergeable file, and TerminalKeys
only counted ${setting:} placeholders, so any caller of the settings verb,
an agent included, could plant a hook in the operator's sessions on every
node (hq issue 340).
2026-10-09 02:27:24 +02:00
mesh-admin 155819f307 Merge pull request 'Issue 339 follow-ups: only step-ca's root may hold lines, every line end refused, the runtime's data and any .ssh refused' (#170) from fix/339-review-follow-ups into main 2026-10-08 23:55:34 +00:00
jochen eb5f9d2f53 Allow a setting in a unit's name only as a template's whole instance, never leading with a dash, at most 64 characters
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Third review of mesh-catalog #147. A setting anywhere else in a unit's name could make the unit another
unit or another kind; it is now refused where the manifest is read. A value beginning with '-' would be
read by systemctl as an option, and a long one is no pool's name.
2026-10-09 01:51:37 +02:00
jochen 0f58602c74 Count a file that says nothing as trusted, and drop the refusal date
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The fourth review (hq issue 339): the safe reading of a file that asks for a
setting and does not say is that root or a consumer trusts it, so its
settings are the terminal's; `"trusted": false` is the opt-out. With that,
nothing unsafe is left to refuse: `module check` lists and counts the
unmarked files and never refuses them.
2026-10-09 01:44:50 +02:00
jochen c3ae3f3e09 Refuse a file that asks for a setting without saying whether it is trusted from 2026-10-10
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
The operator's date (hq issue 339). A test holds the warning before it and the
refusal from it.
2026-10-09 01:37:43 +02:00
mesh-admin 2a9697cf70 Merge pull request 'Switch the memory store's failure under its lock, so a test cannot race the keeper' (#148) from fix/conditions-store-race into main 2026-10-08 23:35:59 +00:00
jochen e4d2868679 Fill ${setting:} in a service's unit name, and refuse a value that makes no unit name
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A module that holds the distribution's scrub timer for the pool the operator names
(zfs-scrub-weekly@<pool>.timer) cannot write the pool into its definition (hq ADR 0112). Settings were
substituted only into file content, so the unit reached the machine as
"zfs-scrub-weekly@${setting:scrub-pool}.timer", a unit no machine has, and the apply failed far from its
cause (second review of mesh-catalog #147).

A service's unit now takes ${setting:} from the same layers a file does, and is refused by key when
nothing sets it. A value is also refused unless it is only letters, digits, ':', '_', '.' and '-': the
name ends up in unit files and systemctl arguments, and a space, a slash or a newline must never reach
them. A key a unit asks for is not called stray.
2026-10-09 01:32:32 +02:00
jochen fe857ba081 Switch the memory store's failure under its lock, so a test cannot race the keeper
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The keeper keeps each transition from its own goroutine, which reads the memory
store's Fail field under the store's lock; tests assigned the exported field
bare, so TestAnUnreadableStoreClearsNothing failed under -race whenever the
goroutine appended in that window. The field is now set only through SetFail
(and Told's likewise), and a test makes the race certain rather than rare.
Test-only: the controller runs the bus store, never InMemory.
2026-10-09 01:25:04 +02:00
jochen b9fc09c375 Derive the terminal's settings from what a module serves and which files it trusts; a found directory is its own condition
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The third review of #170 (hq issue 339): a setting overrides any key a
provider serves, so any caller of the settings verb could move a database's
port, a registry's port or an issuer to a listener of its own and collect
what consumers present. TerminalKeys now derives from the manifest: places,
accesses, every served key and every setting a served value asks for, and
every setting a file marked `trusted` asks for. `trusted` is the catalogue's
word, taken out before the declaration; `module check` warns of a file that
asks for a setting without saying, and refuses it from 2026-10-30. The hand
list is gone. A directory used as found is now its own condition kind, the
operator's, never urgent, and the gate exempts it where it exempts a relogin.
2026-10-09 01:23:44 +02:00
jochen ec7b8bcd58 Keep the mesh's trust anchors at the terminal, refuse containerd's tree, and read a found directory as a wait
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The review of #170: step-ca's root, roots and path and the identity
provider's issuer are what every consumer trusts, and any caller of the
settings verb could replace them; they are now terminal keys like places and
accesses (hq issue 339). /var/lib/containerd joins the runtimes' data. And a
directory the node-engine uses as found failed its module's gate and rolled
its builds back; found before the send, it is now a wait for a person the
gate passes with, as a relogin is (ADR 0254), and only one the send itself
found holds the module.
2026-10-09 00:57:10 +02:00
mesh-admin cec797e996 Merge pull request 'Make the login shell's execute optional, so a machine may withhold it (hq ADR 0268)' (#169) from withhold-login-shell-execute into main 2026-10-08 22:52:28 +00:00
jochen 0e0ba93f6c Take back the test store's lock fix: open #148 carries the fuller one
Merged beside #148 the two would not compile (SetFail declared twice, m.Fail undefined). #148 lands on its own.
2026-10-09 00:51:18 +02:00
jochen 0f1e1b09fd Change a test store's failure under its lock, which the keeper's goroutine reads
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
TestAnUnreadableStoreClearsNothing wrote InMemory.Fail and its values unguarded
while the keeper's teller appended to the same store, and the race detector
failed mesh/repo-check on #170 (a test race on main, not the change).
2026-10-09 00:38:07 +02:00
jochen 1b502a37e0 Let only the authority's root hold lines, refuse every line end, and keep places off the runtime's data and any .ssh
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestAnUnreadableStoreClearsNothing (0.01s)
The review of hq issue 339 found the PEM exception too wide (any module, any
key, any label, anything base64), \v, \f, NEL and the Unicode separators
still let a value end a line in some readers, and the spool, /opt, the
container runtimes' data and an account's .ssh still placeable. Lines are now
taken only in step-ca's root setting, as certificates encoding/pem decodes and
x509 parses; every line end is refused; and those paths are the machine's own.
The test certificate is a real one, made for the tests with its key thrown away.
2026-10-09 00:28:08 +02:00
mesh-admin 522f2253e7 Merge pull request 'Issue 339: places and accesses only at the terminal, never at the machine's own trees; a setting is one line' (#168) from fix/339-places-and-accesses-are-the-terminals into main 2026-10-08 22:21:02 +00:00
jochen 2073bfe2e6 Make the login shell's execute optional, so a machine may withhold it
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group withhold-login-shell-execute delivered: every member is delivered
execute runs any command as the operator account, which can become root
without a person. The operator withholds it on the control-node until a
call needs a person's approval (hq ADR 0268); the holder withholds it per
machine through its own setting. With execute required, that holder could
not hold the seat there and would be judged silent. ADR 0246's optional
mark lets it hold the seat without serving the verb.
2026-10-09 00:07:33 +02:00
jochen f5824b31c6 Keep places and accesses at the terminal, and refuse a line break in any setting
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
Through the settings verb, or a settings line run by the generic command
verb, any caller of the mesh's console could place a module's directory at
/etc with an owner of its own and have the node-engine, as root, hand it
over at the next push, or mount any of the machine's paths into a container
(hq issue 339). A change to either key is now refused in every process a
verb runs, the generic verb refuses settings writes outright, and neither key
may name the machine's own trees from anywhere, the terminal included. A
line break, carriage return or NUL in any setting, which a file it is
written into reads as a line of the caller's own, is refused where a layer
is kept and where it is composed; PEM blocks alone may hold lines.
2026-10-08 23:58:08 +02:00
mesh-admin d059311c0f Merge pull request 'Describe node-nfs-server's exports and test as per-node addresses (hq ADR 0263, review follow-up)' (#166) from fix/shares-review-followups into main 2026-10-08 21:13:05 +00:00
mesh-admin 1a28cb3357 Merge pull request 'Hold the delivery planner to its recorded rules (table + property); a false cycle report found' (#167) from test/planner-rules into main 2026-10-08 20:41:15 +00:00
jochen 758537dd4e Plan a controller merge in the worker-of order in the three-kinds test
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The test's fixture had no worker-of edge and expected the builder first,
then the controller and the proxy together: the order before hq issue 206.
Since then the build seat's holder follows the controller that defines its
worker, and every controller merge plans controller, builder, proxy in
three tiers. The fixture now carries the edge and the test that order.
2026-10-08 22:22:24 +02:00
jochen add807f034 Say no cycle for a packages edge in a plan's last tier
A packages edge orders nothing, so a module and what packages its source
share a tier by rule; hasCycle counted the edge and the merge handler said
"the last tier depends on itself" of plans with no cycle. Skip the kind as
tiersOf does. The planner tests' cycle rows and property now pass.
2026-10-08 22:22:02 +02:00
jochen 8ca4b04321 Hold the delivery planner to its recorded rules with a table and a property
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check fail: its merge-check.sh failed: FAIL github.com/novox/mesh-controller/cmd/mesh-controller 167.881s
mesh/delivery superseded: a newer head of the same pull request
A table of merges (two repositories, every edge kind, a diamond, a cycle,
files no build reads) and a seeded property over 500 random catalogues pin
what a merge moves and in which tiers, per ADR 0162 and ADR 0238 §3. The
shared-repository rows document today's behaviour that issue 338 would
change, once with hand edges and once with edges derived from the store.

The cycle check fails on main: hasCycle reads a packages edge between two
modules of the last tier as a cycle, so a plan with none is said to have
one. Left failing, marked BUG, for the planner's fix.
2026-10-08 22:15:20 +02:00
mesh-admin 8170fc58a3 Merge pull request 'Keep a passed gate's verdict when the first machine's later reports go quiet (hq issue 335)' (#165) from fix/335-a-passed-gate-is-not-judged-again into main 2026-10-08 19:48:03 +00:00
mesh-admin efcdd5dd7d Merge pull request 'Serve the read verbs on the serving controller's own connection, and name every connection (hq issue 327, ADR 0265)' (#161) from fix/327-a-verb-reads-on-the-serving-connection into main 2026-10-08 19:32:11 +00:00
jochen 5d7d8ee2d6 Describe nfs-server's exports and test as per-node addresses, as the server exports them since the review (hq ADR 0263 rule 5)
mesh/delivery delivered
mesh/delivery-group group fix/shares-review-followups delivered: every member is delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
2026-10-08 21:14:09 +02:00
jochen 5b7e6ff453 Answer dead-letters on the lent serving connection, and keep one clip helper
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
Two handles to the same serving connection, and two copies of one helper,
would drift (review of hq issues 327 and 330).
2026-10-08 21:09:08 +02:00
jochen cc7fb99f29 Answer a panicking verb with an error, say flag errors in the answer, and leave refused logins out of D15
Read verbs now run in the serving process, where a panic would end every
call; refused logins are nobody's reconnect loop (review of hq issue 327).
2026-10-08 21:08:34 +02:00
jochen 1e04670052 Serve the read verbs on the serving controller's own connection, and name every connection
Each verb ran as a process that dialled the bus, so hundreds of short
connections an hour, all named mesh-controller, hid any client reconnecting
in a loop (hq issue 327). D15 now says a user whose connections keep dropping.
2026-10-08 21:08:34 +02:00
jochen 81e5458cbf Test that a carried module takes its lead's pass before its step is read (hq issue 335 review)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Without the reorder the passed build's walk stopped on the first machine's
later silence; the guard that caught it is now said to be one.
2026-10-08 21:08:12 +02:00
jochen fb74e24c9e Keep a passed gate's verdict when the first machine's later reports go quiet (hq issue 335)
A build that passed on its first machine was judged again from that
machine's next reports while its send to the rest waited; another walk's
unreported send there then failed the passed build at the wait's bound
and put it back.
2026-10-08 21:08:12 +02:00
mesh-admin ca09a07fdf Merge pull request 'Keep what a consumer gives up on until a person delivers it again or drops it (hq issue 330, ADR 0264)' (#159) from fix/330-a-message-given-up-on-is-kept into main 2026-10-08 19:07:33 +00:00
mesh-admin 9b028b4212 Merge pull request 'Add the node-nfs-server and node-mounts seats (hq ADR 0263)' (#163) from feat/mounts-module into main 2026-10-08 18:39:52 +00:00
jochen d7fab82a89 Say the adopt switch is the string "true" and that reload only has the kernel reread its exports, as the holders do
mesh/delivery delivered
mesh/delivery-group group feat/mounts-module delivered: every member is delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
2026-10-08 20:11:33 +02:00
jochen 2b01f8786e Let node-nfs-server.test take a client's address: the server knows the range, not the mesh's node names
mesh/delivery-group group feat/mounts-module rejected: a member's own check failed
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-08 18:34:38 +02:00
jochen 909062e729 Deliver a dead letter again only where it is received, and never stop serving for the notices
mesh/delivery delivered
mesh/delivery-group group fix/330-a-message-given-up-on-is-kept delivered: every member is delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
A dead letter was let go as delivered even when its consumer did not filter
its again subject; seat asks needed a grant over every seat's queue and left
the original stuck; a notices bind failure stopped the controller (review).
2026-10-08 18:32:58 +02:00
jochen 826dcb91b1 Keep what a consumer gives up on until a person delivers it again or drops it
Design 25 promised a dead-letter stream that did not exist: a message a
consumer gave up on stayed only in its source, which drops it after a week,
and its condition cleared when the advisories stopped (hq issue 330, ADR 0264).
2026-10-08 18:32:58 +02:00
jochen 59fdffb979 Add the node-nfs-server and node-mounts seats, so a share and a mount have a role the mesh defines (hq ADR 0263)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group feat/mounts-module ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
A machine sharing folders and a machine mounting them each need one holder
per machine, with verbs an agent calls instead of exportfs, fstab edits or
zfs set. Both seats deliver nothing: nfs-share is provided at the mesh's
scope. The two adopt verbs are dry runs unless confirmed.
2026-10-08 18:24:38 +02:00
237 changed files with 26792 additions and 879 deletions
+8 -1
View File
@@ -27,6 +27,8 @@ import (
"syscall"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
@@ -175,7 +177,9 @@ func dialFor(credential Credential) (*broker.JetStream, string, error) {
if !credential.onTheNewBus() {
return nil, "", fmt.Errorf("the credential at hand names %q, which is not the mesh's bus", credential.URL)
}
js, err := broker.DialPinned(credential.natsURL(), credential.Fingerprint)
// Named for what it is, not the controller whose code dials it (novox/hq issue 327).
host, _ := os.Hostname()
js, err := broker.DialPinned(credential.natsURL(), credential.Fingerprint, nats.Name("build agent on "+host))
if err != nil {
return nil, "", err
}
@@ -302,6 +306,9 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
for _, s := range built.Sources {
result.Sources = append(result.Sources, link.BuildSource{Repository: s.Repository, Ref: s.Ref, Paths: s.Paths})
}
say("built", built.Manifest.Module+" from "+short(built.Commit))
}
}
+19 -2
View File
@@ -5,9 +5,11 @@ import (
"errors"
"fmt"
"os"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
@@ -178,7 +180,21 @@ func (a *actor) release() {
// holderOf is this process as the lease's holder.
func holderOf(instance string) lease.Holder {
host, _ := os.Hostname()
return lease.Holder{Instance: instance, Host: host, Build: version}
build := runningBuild()
if build == "" {
build = version
}
return lease.Holder{Instance: instance, Host: host, Build: build}
}
// RunningBuildVar is where the declaration tells this process which build it is (module.json, the
// controller process's env): the version its bundle is delivered as, `${version}` composed by the
// catalogue from the bundle's digest. Empty for a process placed by hand.
const RunningBuildVar = "MESH_CONTROLLER_VERSION"
// runningBuild is the version of the build this process is, or empty when the declaration did not say.
func runningBuild() string {
return strings.TrimSpace(os.Getenv(RunningBuildVar))
}
// serveUnderTheLease takes the lease for the serving controller, waiting while another holds it, and
@@ -187,7 +203,8 @@ func (a *actor) serveUnderTheLease(ctx context.Context, inv *inventory.Inventory
a.mu.Lock()
a.serving = true
a.mu.Unlock()
js, err := broker.Dial(address)
// Its own connection, held as long as the lease, and named so (novox/hq issue 327).
js, err := broker.Dial(address, nats.Name(broker.ConnectionName+" lease"))
if err != nil {
return nil, fmt.Errorf("the mesh is on the bus at %s and this control plane cannot reach it to take the "+
"lease: %w", broker.BareAddress(address), err)
+20
View File
@@ -51,6 +51,9 @@ func assignWith(ctx context.Context, open *stores, node string, opts assignOptio
if len(modules) == 0 {
return "", fmt.Errorf("assign %s names no module", node)
}
if err := refusedMovingTheController(modules); err != nil {
return "", err
}
// Held while it is recorded, so it cannot land between a converge's preview and its flip and
// be taken without ever having been previewed (novox/hq ADR 0100).
ctx, release, err := holdNodes(ctx, open, []string{node})
@@ -209,6 +212,9 @@ func seatDependenciesOnAssign(ctx context.Context, open *stores, node string, mo
// (novox/hq ADR 0207) — the other side of refusing that module's assignment without one. Several
// modules in one act are judged together, so a holder and its dependents come off in one command.
func unassign(ctx context.Context, open *stores, node string, modules ...string) (string, error) {
if err := refusedMovingTheController(modules); err != nil {
return "", err
}
if len(modules) == 0 {
return "", fmt.Errorf("unassign %s names no module", node)
}
@@ -360,3 +366,17 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
}
return fmt.Sprintf("its bus credential is issued and sealed to %s, and arrives with the push", node)
}
// refusedMovingTheController refuses assigning or unassigning the controller's module through a verb (review of
// novox/hq ADR 0272): the node it is assigned to is the control-node, and the control-node's operator account is
// the controller's terminal, so whoever moves the module chooses the terminal. At the terminal alone, as every
// change that says who may change what.
func refusedMovingTheController(modules []string) error {
verb, through := throughAVerb()
if !through || !slices.Contains(modules, controllerModule) {
return nil
}
return fmt.Errorf("%s is assigned and unassigned at the controller's terminal only (mesh-cli on the control-node), "+
"never through a verb (this came through %q): the node it runs on is the control-node, whose operator is the "+
"terminal (novox/hq ADR 0272). Nothing was assigned", controllerModule, verb)
}
+253
View File
@@ -0,0 +1,253 @@
package main
// The account agents run as (novox/hq ADR 0266).
//
// On the control node every agent session ran as the operator's account, which may become root without a
// password — so any agent there could become root without a person, and ADR 0259 §8 (an answer from the
// operator's phone authorises an act) rests on that being false where the router and its channels run. The
// decision: a node may name an account its agents run as, of their own and without sudo; the operator's
// account keeps its sudo.
//
// - **Named at the controller's terminal only** (`node agent-account`): not a verb, not a setting, so no
// agent can name itself another account. Empty is a real state: agents run as the operator there.
// - **Composed** as `${machine:agent-account}`, `${machine:agent-home}` and `${machine:agent-root}` for the
// agent's module, which declares the account with `root: never`, and as MESH_AGENT_ACCOUNT and
// MESH_AGENT_HOME for its tools (catalogue/machine_into_files.go, runtime.go).
// - **Judged by the machine itself.** The node-engine reads, on every look, whether an account declared
// `root: never` can become root without a person — uid 0, a group that grants root, a sudo rule, a
// secret of the mesh it may read — and says it as the declaring module's account verdict, marked
// Root "never". agentConfined reads that verdict; the self-check (probe DA) raises
// `agent-can-become-root` while it does not hold, and `node show` says it.
//
// A verdict not given is never a pass: an engine older than the judging, an account not yet declared, a
// statement that says nothing of it — each is "not judged", and fails.
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// kindAgentCanBecomeRoot is the condition raised while a machine's agent account can become root without
// a person, or is not judged (ADR 0266).
const kindAgentCanBecomeRoot = "agent-can-become-root"
// agentAccountProbe is the self-check's probe of it.
const agentAccountProbe = "DA"
// agentConfined says whether the agents of a machine that names an agent account are confined: the
// machine's newest statement holds a healthy account verdict, judged for root, on that account. named is
// false for a machine that names none — agents run as the operator account there, which this does not
// judge. why is said either way, in the mesh's words; err is a store that could not be read.
//
// The one judgement: `node show`, the self-check, and ADR 0259's router honouring a verified sender read
// it here.
// now is the judging clock, threaded so a caller judging several things at one instant judges them all at it.
func agentConfined(ctx context.Context, inv *inventory.Inventory, node string, now time.Time) (named, confined bool, why string, err error) {
n, err := inv.NodeByName(ctx, node)
if err != nil {
return false, false, "", err
}
if n.AgentAccount == "" {
return false, false, fmt.Sprintf("%s names no agent account: agents run as the operator account (%s)",
node, orNoneKnown(n.Account)), nil
}
h, had, err := inv.HealthOf(ctx, node)
if err != nil {
return true, false, "", err
}
confined, why = judgedConfined(n.AgentAccount, h, had, now)
return true, confined, why, nil
}
// verdictFreshFor is how old the statement holding the verdict may be, by this controller's clock. A
// node-engine states its health on every change and at least every five minutes (mesh-host's sayAnyway), so
// three statements missed is a node-engine stopped, or a machine away. **A stale verdict is not a pass**: an
// agent that stopped the node-engine must not leave "cannot become root" standing from before.
const verdictFreshFor = 15 * time.Minute
// judgedConfined is the judgement over one statement, without the store, at now.
func judgedConfined(agent string, h inventory.NodeHealth, had bool, now time.Time) (bool, string) {
if !had {
return false, fmt.Sprintf("the agent account %s is not judged: the machine's node-engine has stated "+
"nothing of what it runs", agent)
}
if age := now.Sub(h.HeardAt); age > verdictFreshFor {
return false, fmt.Sprintf("the agent account %s is not judged: the machine's newest statement was heard at "+
"%s, more than %d minutes ago, and a verdict that old is not a verdict on now", agent,
h.HeardAt.Local().Format("2006-01-02 15:04"), int(verdictFreshFor.Minutes()))
}
if h.Contract < link.RootContract {
return false, fmt.Sprintf("the agent account %s is not judged: the machine's node-engine is older than "+
"the judging of an account's root (its statement's contract is %d, the judging is %d)",
agent, h.Contract, link.RootContract)
}
var verdicts []inventory.ResourceHealth
for _, r := range h.Resources {
if r.Kind == link.KindAccount && r.Target == agent && r.Root == link.RootNever {
verdicts = append(verdicts, r)
}
}
if len(verdicts) == 0 {
return false, fmt.Sprintf("the agent account %s is not judged: the machine's newest statement holds no "+
"verdict on it — no module there declares it never to become root, or the declaration naming it "+
"has not been applied", agent)
}
sort.Slice(verdicts, func(i, j int) bool {
return verdicts[i].Module+verdicts[i].Resource < verdicts[j].Module+verdicts[j].Resource
})
for _, v := range verdicts {
switch v.State {
case link.StateHealthy:
case link.StateUnhealthy:
// The engine's own words, which start with link.ReasonRoot when it found a way to root.
return false, fmt.Sprintf("the agent account %s %s (said by %s's %s)", agent,
orNoneKnown(v.Reason), v.Module, v.Resource)
default:
return false, fmt.Sprintf("the agent account %s is not judged: %s (%s's %s, %s)", agent,
orNoneKnown(v.Reason), v.Module, v.Resource, v.State)
}
}
return true, fmt.Sprintf("the agent account %s cannot become root without a person (judged %s)", agent,
h.SaidAt.Local().Format("2006-01-02 15:04"))
}
// searchQuietFor is how long the controller lets an agent account's verdict wait for the node-engine's setuid
// search before that is itself the urgent condition: the longest a search is expected to take (link.RootSearchQuiet,
// the engine's own value; novox/hq issue 361 — the search runs to completion, with no bound of its own), counted
// from when this controller first saw it waiting, never from the engine's start. Quiet raises nothing; it never
// makes the agent account confined, which only a healthy verdict from a complete, fresh search does.
const searchQuietFor = link.RootSearchQuiet
// The kinds of an agent account's verdict, for the quiet a search earns.
const (
verdictOther = iota // anything else: a stale statement, an older engine, no verdict, another unknown
verdictPending // waiting for the search, and otherwise healthy
verdictComplete // judged: healthy, or a way to root found
)
// rootVerdictKind reads a statement for the agent account (novox/hq ADR 0266): pending when every verdict on it
// is healthy or not judged yet because the engine's setuid search runs, at least one of them that; complete when
// every verdict is healthy or one found a way to root. The engine's own "since" is not read: it starts again at
// every restart of the engine.
func rootVerdictKind(agent string, h inventory.NodeHealth, had bool, now time.Time) int {
if !had || now.Sub(h.HeardAt) > verdictFreshFor || h.Contract < link.RootContract {
return verdictOther
}
pending, any := false, false
for _, r := range h.Resources {
if r.Kind != link.KindAccount || r.Target != agent || r.Root != link.RootNever {
continue
}
any = true
switch {
case r.State == link.StateHealthy:
case r.State == link.StateUnhealthy:
return verdictComplete
case r.State == link.StateUnknown && strings.HasPrefix(r.Reason, link.ReasonRootPending):
pending = true
default:
return verdictOther
}
}
switch {
case !any:
return verdictOther
case pending:
return verdictPending
}
return verdictComplete
}
// searchStillRunning says the one thing keeping an agent account from being judged is the node-engine's setuid
// search, and that this controller first saw it waiting less than searchQuietFor ago — kept in the store, so a
// node-engine restarted in a loop does not keep it quiet. A complete verdict forgets when it began.
func searchStillRunning(ctx context.Context, inv *inventory.Inventory, node, agent string, h inventory.NodeHealth,
had bool, now time.Time) (bool, error) {
switch rootVerdictKind(agent, h, had, now) {
case verdictComplete:
return false, inv.RootSearchJudged(ctx, node)
case verdictPending:
since, err := inv.RootSearchPending(ctx, node, now)
if err != nil {
return false, err
}
return now.Sub(since) <= searchQuietFor, nil
}
return false, nil
}
// probeAgentAccounts is DA: every machine that names an agent account has it judged, on its node-engine's
// newest statement, unable to become root without a person (ADR 0266).
func probeAgentAccounts(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
inv := d.open.inventory
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, err
}
var out []conditions.Observation
for _, n := range nodes {
if n.AgentAccount == "" {
continue
}
h, had, err := inv.HealthOf(ctx, n.Name)
if err != nil {
return nil, err
}
now := time.Now()
quiet, err := searchStillRunning(ctx, inv, n.Name, n.AgentAccount, h, had, now)
if err != nil {
return nil, err
}
confined, why := judgedConfined(n.AgentAccount, h, had, now)
if confined {
continue
}
// Not judged yet only because the node-engine's search runs and no complete, fresh one judges: not the
// urgent condition after every restart. The agent is still not confined — ADR 0259's router reads
// agentConfined, not this — and `node show` still says not judged. Loud again once the search fails,
// waits past searchQuietFor, or the statement goes stale.
if quiet {
continue
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: n.Name, Token: "agent-root",
Machine: n.Name, Severity: conditions.Urgent,
Summary: fmt.Sprintf("on %s, %s (ADR 0266): an agent there may become root without a person, and "+
"no answer from a channel authorises an act there (ADR 0259 §8)", n.Name, why),
Said: why})
}
return sortedFound(out), nil
}
// agentAccountLines is what `node show` says of the account agents run as.
func agentAccountLines(ctx context.Context, inv *inventory.Inventory, n inventory.Node) []string {
if n.AgentAccount == "" {
return []string{fmt.Sprintf(" agents run as the operator account (%s); no agent account is named",
orNoneKnown(n.Account))}
}
_, confined, why, err := agentConfined(ctx, inv, n.Name, time.Now())
if err != nil {
return []string{fmt.Sprintf(" agents run as %s (home %s); whether it can become root could NOT be read: %v",
n.AgentAccount, n.AgentHome(), err)}
}
verdict := "CAN become root, or is not judged: " + why
if confined {
verdict = why
}
return []string{fmt.Sprintf(" agents run as %s (home %s)", n.AgentAccount, n.AgentHome()),
" " + verdict}
}
// orNoneKnown is a value, or that none is known.
func orNoneKnown(s string) string {
if strings.TrimSpace(s) == "" {
return "none known"
}
return s
}
+279
View File
@@ -0,0 +1,279 @@
package main
import (
"github.com/novox/mesh-host/rootsearch"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The agent account's judgement (novox/hq ADR 0266): confined only on a healthy account verdict judged for
// root, on the very account; every verdict not given — no statement, an older engine, no verdict on it, a
// verdict of unknown — is "not judged" and fails, never a pass.
func TestAnAgentAccountIsConfinedOnlyOnAHealthyVerdictJudgedForRoot(t *testing.T) {
at := time.Date(2026, 10, 8, 19, 21, 0, 0, time.UTC)
verdict := func(target, root, state, reason string) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: target, State: state, Reason: reason, Root: root, Account: target}
}
statement := func(contract int, rs ...inventory.ResourceHealth) inventory.NodeHealth {
return inventory.NodeHealth{Node: "anchor", Contract: contract, SaidAt: at, HeardAt: at, Resources: rs}
}
for _, c := range []struct {
name string
h inventory.NodeHealth
had bool
confined bool
says string
}{
{"judged and unable", statement(link.RootContract, verdict("agent", link.RootNever, link.StateHealthy, "")),
true, true, "cannot become root without a person"},
{"judged and able", statement(link.RootContract, verdict("agent", link.RootNever, link.StateUnhealthy,
link.ReasonRoot+": in the group docker, which grants root")), true, false, "in the group docker"},
{"a verdict of unknown", statement(link.RootContract, verdict("agent", link.RootNever, link.StateUnknown,
"sudo could not be read")), true, false, "not judged"},
{"no statement", inventory.NodeHealth{}, false, false, "not judged"},
{"an older engine", statement(link.ReadinessContract, verdict("agent", "", link.StateHealthy, "")),
true, false, "older than the judging"},
{"a verdict on groups only", statement(link.RootContract, verdict("agent", "", link.StateHealthy, "")),
true, false, "no verdict on it"},
{"a verdict on another account", statement(link.RootContract, verdict("ops", link.RootNever, link.StateHealthy, "")),
true, false, "no verdict on it"},
} {
confined, why := judgedConfined("agent", c.h, c.had, at.Add(time.Minute))
if confined != c.confined || !strings.Contains(why, c.says) {
t.Errorf("%s: confined %v, %q; want %v saying %q", c.name, confined, why, c.confined, c.says)
}
}
// A verdict heard longer ago than the bound is no verdict: an agent that stopped the node-engine must not
// leave "healthy" standing.
fresh := statement(link.RootContract, verdict("agent", link.RootNever, link.StateHealthy, ""))
if ok, _ := judgedConfined("agent", fresh, true, at.Add(verdictFreshFor)); !ok {
t.Error("a verdict exactly at the bound is still one")
}
if ok, why := judgedConfined("agent", fresh, true, at.Add(verdictFreshFor+time.Second)); ok ||
!strings.Contains(why, "not judged") {
t.Errorf("a stale healthy verdict passed: %q", why)
}
}
// DA raises an urgent condition, with plain words, on a machine whose agent account is not judged unable to
// become root; a machine that names none is not its to judge.
func TestTheSelfCheckSaysAnAgentAccountThatCanBecomeRoot(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.NodeByName(ctx, n); err != nil {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAccount(ctx, n, "ops", ""); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
found = onlyMachine(found, "anchor")
if len(found) != 1 || found[0].Machine != "anchor" || found[0].Severity != conditions.Urgent ||
!strings.Contains(found[0].Said, "not judged") {
t.Fatalf("a named agent account with no verdict: %+v", found)
}
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if w.Headline == "" || w.Needs == "" || w.Resolved == "" {
t.Errorf("the condition has no plain words: %+v", w)
}
healthy := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: link.StateHealthy, Root: link.RootNever, Account: "agent"}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{healthy}}); err != nil {
t.Fatal(err)
}
if found, err = probeAgentAccounts(ctx, d); err != nil || len(onlyMachine(found, "anchor")) != 0 {
t.Fatalf("a judged agent account still fails: %+v %v", found, err)
}
if named, confined, why, err := agentConfined(ctx, inv, "anchor", time.Now()); err != nil || !named || !confined {
t.Fatalf("agentConfined on anchor: %v %v %q %v", named, confined, why, err)
}
if named, _, why, err := agentConfined(ctx, inv, "laptop", time.Now()); err != nil || named ||
!strings.Contains(why, "operator account") {
t.Fatalf("agentConfined on a machine naming none: %v %q %v", named, why, err)
}
}
func TestTheAgentRootWordsArePlain(t *testing.T) {
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if why, ok := conditions.PlainWords(w, "anchor"); !ok {
t.Fatalf("not plain: %s: %+v", why, w)
}
}
func onlyMachine(obs []conditions.Observation, machine string) []conditions.Observation {
var out []conditions.Observation
for _, o := range obs {
if o.Machine == machine {
out = append(out, o)
}
}
return out
}
// The snapshot a merge check composes from carries the agent account as a pseudonym (novox/hq ADR 0266),
// so a change is judged against machines that name one, and the name never leaves.
func TestTheFactsCarryTheAgentAccountAsAPseudonym(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
if err := open.inventory.SetAccount(ctx, "anchor", "keeper", ""); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetAgentAccount(ctx, "anchor", "warden", ""); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
body, _ := f.Encode()
if strings.Contains(string(body), "warden") {
t.Error("the agent account's name is in the snapshot")
}
m, ok := f.Machine(snapshot.Pseudonym("machine", "anchor"))
if !ok || m.AgentAccount != snapshot.Pseudonym("account", "warden") || m.Account == m.AgentAccount {
t.Fatalf("the anchor's agent account reads as %q (operator %q)", m.AgentAccount, m.Account)
}
}
// No verb runs a `node` command that sets something: through the generic `command` verb, `node account`,
// `node agent-account` and every other `node` subcommand but list and show are refused, naming the terminal.
func TestNoVerbSetsANodesAccounts(t *testing.T) {
for _, line := range []string{
"node agent-account novox --clear",
"node agent-account novox ops",
"node account novox agent",
"node account novox",
"node add intruder",
"node public-domain novox --clear",
"node",
"node frobnicate",
} {
argv, err := argvFor("command", map[string]any{"command": line})
if err == nil || !strings.Contains(err.Error(), "controller's terminal") ||
!strings.Contains(err.Error(), "ADR 0266") {
t.Errorf("%q ran as %v (%v); want a refusal naming the terminal", line, argv, err)
}
}
for _, line := range []string{"node show novox", "node list --json", "status --json"} {
if _, err := argvFor("command", map[string]any{"command": line}); err != nil {
t.Errorf("%q, a read, was refused: %v", line, err)
}
}
if err := terminalOnly([]string{"node", "account", "a", "b"}); err == nil {
t.Error("the refusal is not only the command verb's")
}
}
// Naming the agent account is the controller's terminal's alone: the `node` verb only shows.
func TestTheNodeVerbOnlyShows(t *testing.T) {
argv, err := argvFor("node", map[string]any{"node": "anchor"})
if err != nil || strings.Join(argv, " ") != "node show anchor" {
t.Fatalf("the node verb runs %v (%v)", argv, err)
}
for _, v := range catalogue.ControllerVerbs {
if strings.Contains(v.Name, "agent") {
t.Errorf("a verb %q may name the agent account", v.Name)
}
}
}
// Until a complete search for setuid programs judges — none since the node-engine's state was kept, or the last
// older than the engine lets one judge — the agent account is not judged, and the search runs to its end with no
// bound (novox/hq issue 361). DA does not raise that as urgent within searchQuietFor, counted from when this
// controller first saw it waiting — never from the engine's own "since", which a restart resets, so an engine
// restarted in a loop does not keep it quiet. The account is still not confined, and `node show` still says not
// judged; a search that failed, or a way to root found, is urgent at once.
func TestASearchStillRunningAfterARestartIsNotUrgent(t *testing.T) {
if searchQuietFor != rootsearch.Quiet {
t.Fatalf("the quiet is %s and the node-engine's %s: they are one value", searchQuietFor, rootsearch.Quiet)
}
// A daily search that finishes within the quiet never leaves the account unjudged between two of them.
if rootsearch.FreshFor < rootsearch.Every+rootsearch.Quiet {
t.Fatalf("a complete search judges for %s, less than a day's search (%s) and the quiet (%s)",
rootsearch.FreshFor, rootsearch.Every, rootsearch.Quiet)
}
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.NodeByName(ctx, "anchor"); err != nil {
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
say := func(state, reason string) []conditions.Observation {
t.Helper()
// The engine's since is always now: it was just restarted.
v := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: state, Reason: reason, Root: link.RootNever,
Account: "agent", Since: time.Now()}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{v}}); err != nil {
t.Fatal(err)
}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
return onlyMachine(found, "anchor")
}
running := link.ReasonRootPending + ": the search for setuid programs, started at 19:21, has not finished yet"
healthy := func() {
t.Helper()
if found := say(link.StateHealthy, ""); len(found) != 0 {
t.Fatalf("a healthy verdict raised: %+v", found)
}
}
if found := say(link.StateUnknown, running); len(found) != 0 {
t.Fatalf("a search first seen now was raised: %+v", found)
}
if _, confined, why, _ := agentConfined(ctx, inv, "anchor", time.Now()); confined || !strings.Contains(why, "not judged") {
t.Fatalf("an account whose search runs was read as confined: %q", why)
}
// The engine restarted again and again, each statement's own since fresh: the controller's clock runs on.
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err)
}
healthy() // a complete verdict forgets when the waiting began …
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err) // … and this restart loop began past the bound
}
if found := say(link.StateUnknown, running); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search pending past the bound, by the controller's clock, was not urgent: %+v", found)
}
healthy()
incomplete := rootsearch.ReasonIncomplete + ": the search for setuid programs did not finish (last tried at " +
"19:23: timeout); it is tried again later"
if found := say(link.StateUnknown, incomplete); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search that did not finish was not urgent: %+v", found)
}
if found := say(link.StateUnhealthy, link.ReasonRoot+": in the group docker; "+running); len(found) != 1 ||
found[0].Severity != conditions.Urgent {
t.Fatalf("a way to root found while the search runs was not urgent: %+v", found)
}
}
+937
View File
@@ -0,0 +1,937 @@
package main
// The controller asks, and acts on the operator's warrant (novox/hq ADR 0259 §6). It holds no channel, no
// identity and no factor: it asks the router like any other module, and performs the answer chosen with its
// own grant.
//
// - **For every open, unsilenced condition that needs the operator and names its answers**, one ask is
// published on the `operator-channel` seat under the controller's own name: the condition's words, its
// actions as options at their levels (Silence acknowledges; Release, Stop, Start and Restart approve),
// answered by the operator, expiring after a day (a week when every option only acknowledges). A
// condition that clears, is silenced, or changes its answers has its ask cancelled; an ask that expired
// unanswered is asked again while the condition lasts; one the router refused is asked again after a wait
// that grows with each refusal in a row, at most half an hour, so a refusal is never believed longer. Each ask is kept in the controller's bucket
// `asked`, so a restart neither asks twice nor forgets.
// - **On a warrant**, heard on the seat's event under the controller's own name (which only the router may
// say), the controller acts once per ask: only for an ask it holds, only for the option it offered at
// that option's level, and only while the condition is still open. It performs the action as itself —
// a silence through its own conditions, any other through the verb the action names — with the warrant's
// words as its why, and records it in the hand-act log as the operator's decision, naming the channel,
// the ask and the proofs. An ask that ended without a choice is recorded and nothing is done.
// - **A warrant it missed** while away is read from the router's record of its asks, under its own name.
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// The asker's name on the seat: the controller's module.
const askerName = broker.ControllerSeat
// How long an ask lasts: a day when an answer approves, a week when every answer only acknowledges.
const (
// askApproveFor is a day less a margin, so an ask is never refused at the router for lasting a day and
// a moment (the SDK's bound is a day).
askApproveFor = 24*time.Hour - 10*time.Minute
askAcknowledgeFor = 7 * 24 * time.Hour
// askEvery is how often what is open is asked about again, beside every change.
askEvery = time.Minute
// askCatchUpAfter is how old an open ask is before the router's record of it is read: a warrant heard
// on the event needs no reading.
askCatchUpAfter = 2 * time.Minute
// askAgainAfterAnswer is how long a condition the operator answered is not asked about again with the
// same answers: what was chosen takes a while to clear it, and asking again at once would ask twice.
askAgainAfterAnswer = time.Hour
// askMostOpen is how many asks the controller holds open at once (the router refuses a fourth): the
// most urgent conditions first, then the oldest.
askMostOpen = asks.MostOpen
// askRefusedRetryMost is the longest a refusal is believed without asking again (novox/hq issue 369): the
// router refused while its channels had not yet said they could send, the channels could send twenty
// minutes later, and the controller repeated that refusal for eleven hours. A refused ask is asked again
// after askEvery, then twice as long after each refusal in a row, never longer than this — and at once when
// the channels change.
askRefusedRetryMost = 30 * time.Minute
// askRefusedCounted is how far back refusals in a row are counted for that wait.
askRefusedCounted = 6 * time.Hour
// askVerdictWait is how long an ask made again after a refusal waits for the router's word before it is
// taken as taken: the router refuses an ask as it reads it, so a refusal comes within seconds. Meanwhile
// the condition saying it was not delivered stands as it was, neither cleared nor raised again.
askVerdictWait = 2 * time.Minute
)
// refusedRetryAfter is how long a part refused n times in a row waits before it is asked again.
func refusedRetryAfter(n int) time.Duration {
wait := askEvery
for i := 1; i < n && wait < askRefusedRetryMost; i++ {
wait *= 2
}
return min(wait, askRefusedRetryMost)
}
// What became of an ask, as the controller keeps it.
const (
askOpen = "open"
askCancelled = "cancelled"
)
// asked is one ask the controller made, as it keeps it.
type asked struct {
ID string `json:"id"`
Condition string `json:"condition"`
// Channels is what the channels were when it was asked (asker.channels): an ask the router refused is asked
// again at once when the condition's answers or the channels change, and otherwise after a wait that grows
// with each refusal in a row (refusedRetryAfter, novox/hq issue 369).
Channels string `json:"channels,omitempty"`
Ask asks.Ask `json:"ask"`
Actions []conditions.Action `json:"actions"`
// Options are the actions by option id.
Options map[string]int `json:"options"`
State string `json:"state"`
Opened time.Time `json:"opened"`
Ended time.Time `json:"ended,omitempty"`
Warrant *asks.Warrant `json:"warrant,omitempty"`
// Acted is what the controller did on the warrant: empty before it did anything, "acting" while it acts,
// then "done", "failed: …" or "nothing: …". Anything but empty is never acted on again.
Acted string `json:"acted,omitempty"`
// Part is which ask of its condition this is (askPart): empty for the one that carries the condition's
// answers, or the authorising ones where it has both; "acknowledge" for its acknowledging answers asked
// apart (the review of 2026-10-09, M1).
Part string `json:"part,omitempty"`
// Rehearsal is an ask started at the controller's terminal (rehearse.go): about no condition, its answers
// perform nothing, and the reconciling of conditions leaves it alone.
Rehearsal bool `json:"rehearsal,omitempty"`
// Proposal is a settings layer proposed through a verb (proposals.go, novox/hq ADR 0277): about no
// condition, set on Approve by the serving controller itself, and left alone by the reconciling of conditions.
Proposal *settingsProposal `json:"proposal,omitempty"`
}
// ofACondition says an ask is one of a condition's: not a rehearsal, not a proposal.
func (r asked) ofACondition() bool { return !r.Rehearsal && r.Proposal == nil }
// partKey is an ask's place among what is asked: its condition and its part.
func partKey(condition, part string) string { return condition + "#" + part }
// partAcknowledge is the part of a condition asked apart for its acknowledging answers.
const partAcknowledge = "acknowledge"
// askPart is one ask a condition is asked with: its part, what it is about, and its answers.
type askPart struct {
name string
about string
actions []conditions.Action
}
// levelOf is an action's level as an option offers it: one that says none is never taken for less than
// approve.
func levelOf(act conditions.Action) asks.Level {
if act.Level == "" {
return asks.Approve
}
return asks.Level(act.Level)
}
// partsOf is the asks a condition is asked with (the review of 2026-10-09, M1): one, when its answers are all
// of one kind; else its authorising answers (Release, Stop, Restart) in one ask, about the condition, and its
// acknowledging ones (Silence) in another. **An acknowledgement never shares an ask with an approval**: a
// channel that only acknowledges would otherwise answer the ask, and end the approval with it.
func partsOf(c conditions.Condition) []askPart {
var ack, auth []conditions.Action
for _, act := range c.Actions {
if levelOf(act) == asks.Acknowledge {
ack = append(ack, act)
} else {
auth = append(auth, act)
}
}
if len(ack) == 0 || len(auth) == 0 {
return []askPart{{about: c.Key, actions: c.Actions}}
}
return []askPart{{about: c.Key, actions: auth},
{name: partAcknowledge, about: c.Key + "." + partAcknowledge, actions: ack}}
}
// askedStore keeps the asks (broker.AskedBucket). **Every write after the first is a compare-and-set** (the
// review of 2026-10-09, L2): an ask is created once, and changed only over the revision it was read at, the
// change decided again on what is read — so two controllers, or two deliveries of one warrant, never write
// over each other, and of two that would act only the one whose write stands does.
type askedStore interface {
Get(ctx context.Context, id string) (*asked, error)
// Create keeps a new ask, and refuses one already kept under its id.
Create(ctx context.Context, a asked) error
// Change applies change to the ask kept under id, by compare-and-set, and says whether its write stood.
// change says whether to write at all; on a write that came between, it is asked again on what is read.
Change(ctx context.Context, id string, change func(*asked) bool) (bool, error)
All(ctx context.Context) ([]asked, error)
}
// askChangeTries is how often a change is read and tried again when another write came between.
const askChangeTries = 5
// asker is the controller asking the operator and acting on the answer.
type asker struct {
open func(ctx context.Context) ([]conditions.Condition, error)
silence func(ctx context.Context, key string, d time.Duration, by, why string) error
store askedStore
// publish puts a message on a subject's stream, de-duplicated by id.
publish func(ctx context.Context, subject string, body []byte, id string) error
// call performs an action's verb with its arguments, as the controller.
call func(ctx context.Context, a conditions.Action, args map[string]string) error
// setLayer sets a proposed settings layer on the operator's warrant (novox/hq ADR 0277); nil cannot.
setLayer setOnWarrant
// record writes the hand-act log.
record func(ctx context.Context, act link.HandAct) error
// routerRecord reads the router's record of an ask for a warrant missed; nil reads nothing.
routerRecord func(ctx context.Context, id string) (*asks.Warrant, error)
// routerHere says whether a router holds the seat and takes asks under the asker's name; nil is yes.
routerHere func(ctx context.Context) (bool, error)
// grantHeld says whether the bus holds the controller's grant to ask: the user list the bus's machine was
// last sent is the one the mesh composes now (novox/hq issue 353). The grant is composed from the router's
// assignment and reaches the bus only when that machine is next pushed, so between `assign` and `push`
// the record says a router is here and the bus refuses every ask. why says what to do; nil is yes.
grantHeld func(ctx context.Context) (held bool, why string, err error)
// channels is what the channels are now, as a fingerprint: who holds which kind, promising what.
channels func(ctx context.Context) string
// raise keeps the asker's own condition (sourceAsker): which conditions needing the operator could not be
// asked, and why. Nil raises nothing (a test that does not look).
raise func(ctx context.Context, obs []conditions.Observation) error
now func() time.Time
logf func(string, ...any)
saidNoRouter bool
saidNoGrant bool
mu sync.Mutex
nudged chan struct{}
}
func (a *asker) nudge() {
if a == nil {
return
}
a.mu.Lock()
if a.nudged == nil {
a.nudged = make(chan struct{}, 1)
}
ch := a.nudged
a.mu.Unlock()
select {
case ch <- struct{}{}:
default:
}
}
// keep asks until ctx ends: now, on every change of a condition, and every askEvery.
func (a *asker) keep(ctx context.Context) {
a.nudge()
tick := time.NewTicker(askEvery)
defer tick.Stop()
a.mu.Lock()
nudged := a.nudged
a.mu.Unlock()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
case <-nudged:
}
if err := a.reconcile(ctx); err != nil {
a.logf("what the operator is asked could not be brought up to date: %v", err)
}
}
}
// wants says whether a condition is one to ask about now.
func wants(c conditions.Condition, now time.Time) bool {
return len(c.Actions) > 0 && c.Needs != "" && !c.SilencedAt(now)
}
func sameAsked(a []conditions.Action, b []conditions.Action) bool {
x, _ := json.Marshal(a)
y, _ := json.Marshal(b)
return string(x) == string(y)
}
// reconcile brings what is asked in line with what is open.
func (a *asker) reconcile(ctx context.Context) error {
now := a.now()
if a.routerHere != nil {
here, err := a.routerHere(ctx)
if err != nil {
return err
}
if !here {
if !a.saidNoRouter {
a.logf("no router takes asks under the controller's name (a module declaring %s with its ask "+
"named by its caller, assigned): the operator is asked nothing until one is", broker.AsksSeat)
a.saidNoRouter = true
}
open, err := a.open(ctx)
if err != nil {
return err
}
var unasked []conditions.Condition
for _, c := range open {
if wants(c, now) {
unasked = append(unasked, c)
}
}
return a.sayUnasked(ctx, unasked, "no router takes the controller's asks: no module holding "+
broker.AsksSeat+" that takes an ask under its asker's name is assigned")
}
a.saidNoRouter = false
}
if a.grantHeld != nil {
held, why, err := a.grantHeld(ctx)
if err != nil {
return err
}
if !held {
if !a.saidNoGrant {
a.logf("the bus does not hold the controller's grant to ask yet: %s; the operator is asked nothing until it does", why)
a.saidNoGrant = true
}
open, err := a.open(ctx)
if err != nil {
return err
}
var unasked []conditions.Condition
for _, c := range open {
if wants(c, now) {
unasked = append(unasked, c)
}
}
return a.sayUnasked(ctx, unasked, "the bus does not hold the controller's grant to ask yet: "+why)
}
a.saidNoGrant = false
}
channels := ""
if a.channels != nil {
channels = a.channels(ctx)
}
open, err := a.open(ctx)
if err != nil {
return err
}
all, err := a.store.All(ctx)
if err != nil {
return err
}
byCondition := map[string]asked{} // by partKey
// What is open and not a condition's — a rehearsal, a proposal — still counts toward what the router holds
// open for the controller (askMostOpen); expired unanswered, it is kept so, as the router says it too.
otherOpen := 0
for _, r := range all {
if r.State == askOpen && !r.ofACondition() && !now.Before(r.Ask.Expires) {
if _, err := a.store.Change(ctx, r.ID, func(x *asked) bool {
if x.State != askOpen || x.Acted != "" {
return false
}
x.State, x.Ended, x.Acted = string(asks.OutcomeExpired), now, "nothing: the ask expired unanswered"
return true
}); err != nil {
return err
}
continue
}
if r.State == askOpen && !r.ofACondition() {
otherOpen++
}
if r.State == askOpen && r.ofACondition() {
k := partKey(r.Condition, r.Part)
if prior, held := byCondition[k]; !held || r.Opened.After(prior.Opened) {
byCondition[k] = r
}
}
}
// A warrant missed while away, read from the router's record: for a condition's ask, and for a proposal's or a
// rehearsal's alike.
if a.routerRecord != nil {
var lookedUp []asked
for _, r := range byCondition {
lookedUp = append(lookedUp, r)
}
for _, r := range all {
if r.State == askOpen && !r.ofACondition() {
lookedUp = append(lookedUp, r)
}
}
for _, r := range lookedUp {
if now.Sub(r.Opened) < askCatchUpAfter {
continue
}
if w, err := a.routerRecord(ctx, r.ID); err == nil && w != nil {
body, _ := json.Marshal(w)
if err := a.Decided(ctx, body); err != nil {
return err
}
}
}
if all, err = a.store.All(ctx); err != nil {
return err
}
byCondition = map[string]asked{}
for _, r := range all {
if r.State == askOpen && r.ofACondition() {
byCondition[partKey(r.Condition, r.Part)] = r
}
}
}
// What the operator answered lately, by condition: not asked again at once; and what the router refused,
// newest first: not asked again until the answers or the channels change.
answered, refused := map[string]asked{}, map[string]asked{}
for _, r := range all {
k := partKey(r.Condition, r.Part)
if r.State == string(asks.OutcomeChosen) && now.Sub(r.Ended) < askAgainAfterAnswer {
answered[k] = r
}
if r.State == string(asks.OutcomeRefused) {
if prior, has := refused[k]; !has || r.Opened.After(prior.Opened) {
refused[k] = r
}
}
}
// Refusals in a row, by part: those since the last ask that was not refused, within askRefusedCounted.
// superseded: a part asked since its last refusal, by an ask the router did not refuse — open, answered,
// expired or cancelled. Its refusal is history then, never said again (the review of PR 198: an answered
// retry brought the refusal back).
inRow, superseded := map[string]int{}, map[string]asked{}
for k, last := range refused {
for _, r := range all {
if partKey(r.Condition, r.Part) == k && r.State != string(asks.OutcomeRefused) && r.State != askUnsent &&
r.Opened.After(last.Opened) && r.Opened.After(superseded[k].Opened) {
superseded[k] = r
}
}
var since time.Time
for _, r := range all {
if partKey(r.Condition, r.Part) == k && r.State != string(asks.OutcomeRefused) && r.State != askUnsent &&
!r.Opened.After(last.Opened) && r.Opened.After(since) {
since = r.Opened
}
}
for _, r := range all {
if partKey(r.Condition, r.Part) == k && r.State == string(asks.OutcomeRefused) && r.Opened.After(since) &&
now.Sub(r.Opened) < askRefusedCounted {
inRow[k]++
}
}
}
wanted := map[string]bool{}
var unasked []conditions.Condition // refused by the router, and nothing it was refused for changed
var refusedWords []string
// The most urgent first, then the oldest: those are asked when no more than askMostOpen may be.
sort.SliceStable(open, func(i, j int) bool {
ui, uj := open[i].Severity == conditions.Urgent, open[j].Severity == conditions.Urgent
if ui != uj {
return ui
}
if !open[i].Raised.Equal(open[j].Raised) {
return open[i].Raised.Before(open[j].Raised)
}
return open[i].Key < open[j].Key
})
openNow := otherOpen
for _, c := range open {
if !wants(c, now) {
continue
}
for _, p := range partsOf(c) {
if r, held := byCondition[partKey(c.Key, p.name)]; held && sameAsked(r.Actions, p.actions) && now.Before(r.Ask.Expires) {
openNow++
}
}
}
for _, c := range open {
if !wants(c, now) {
continue
}
saidUnasked := false
for _, p := range partsOf(c) {
key := partKey(c.Key, p.name)
wanted[key] = true
if r, was := refused[key]; was && sameAsked(r.Actions, p.actions) {
cur, held := byCondition[key]
ended := r.Ended
if ended.IsZero() {
ended = r.Opened
}
later, asked := superseded[key]
waiting := !held && !asked && r.Channels == channels && now.Sub(ended) < refusedRetryAfter(inRow[key])
// Asked again now (the wait over), or lately and the router's word not in yet: said as it was until
// that word, so the condition neither clears nor is raised again at each try.
retrying := !held && !asked && !waiting
verdictDue := held && asked && later.ID == cur.ID && now.Sub(cur.Opened) < askVerdictWait
if waiting || retrying || verdictDue {
if !saidUnasked {
unasked, saidUnasked = append(unasked, c), true
}
if r.Warrant != nil && r.Warrant.Words != "" {
refusedWords = append(refusedWords, r.Warrant.Words)
}
}
if waiting {
continue // refused lately, and nothing it was refused for has changed: asked again after the wait
}
}
if r, done := answered[key]; done && sameAsked(r.Actions, p.actions) {
if _, held := byCondition[key]; !held {
continue
}
}
if r, held := byCondition[key]; held {
switch {
case !sameAsked(r.Actions, p.actions):
if err := a.cancel(ctx, r, "its answers changed"); err != nil {
return err
}
case !now.Before(r.Ask.Expires):
// Expired unanswered: the router says so too; asked again below while it lasts.
if _, err := a.store.Change(ctx, r.ID, func(x *asked) bool {
if x.State != askOpen {
return false
}
x.State, x.Ended = string(asks.OutcomeExpired), now
return true
}); err != nil {
return err
}
openNow--
default:
continue
}
}
if openNow >= askMostOpen {
continue // asked when one of the open ones ends, most urgent first
}
if err := a.ask(ctx, c, p, channels); err != nil {
a.logf("the operator could not be asked about %s: %v", c.Key, err)
continue
}
openNow++
}
}
stillOpen := map[string]conditions.Condition{}
for _, c := range open {
stillOpen[c.Key] = c
}
for key, r := range byCondition {
if wanted[key] {
continue
}
// **A silence never takes an approval back** (the confirmation review of 2026-10-09, M1). Silence is an
// acknowledgement — anyone at the desk may give it — so a condition silenced while its approval is asked
// keeps that ask open, unchanged, until it is answered on a channel that proves who answered, or expires.
// It is not asked again once it ends, while the silence lasts.
if c, open := stillOpen[r.Condition]; open && c.SilencedAt(now) && r.Ask.Highest() != asks.Acknowledge &&
now.Before(r.Ask.Expires) && keepsItsAnswers(c, r) {
continue
}
if err := a.cancel(ctx, r, "the condition ended, was silenced or needs nothing now"); err != nil {
return err
}
}
why := "the router refused the ask"
if len(refusedWords) > 0 {
why += ": " + refusedWords[0]
}
return a.sayUnasked(ctx, unasked, why)
}
// keepsItsAnswers says a condition still offers the answers an ask kept was asked with.
func keepsItsAnswers(c conditions.Condition, r asked) bool {
for _, p := range partsOf(c) {
if partKey(c.Key, p.name) == partKey(r.Condition, r.Part) {
return sameAsked(r.Actions, p.actions)
}
}
return false
}
// sourceAsker raises the asker's own condition.
const sourceAsker = "asker"
// sayUnasked keeps the asker's one condition: while a condition that needs the operator could not be asked
// on any channel, said loudly (failure must be loud), cleared when every one could be.
func (a *asker) sayUnasked(ctx context.Context, unasked []conditions.Condition, why string) error {
if a.raise == nil {
return nil
}
var obs []conditions.Observation
if len(unasked) > 0 {
keys := make([]string, 0, len(unasked))
severity := conditions.Warning
for _, c := range unasked {
keys = append(keys, c.Key)
if c.Severity == conditions.Urgent {
severity = conditions.Urgent
}
}
sort.Strings(keys)
obs = append(obs, conditions.Observation{Scope: conditions.ScopeSeat, ID: broker.AsksSeat, Token: "unasked",
Kind: "asks-undelivered", Severity: severity, Source: sourceAsker,
Summary: fmt.Sprintf("%d condition(s) that need the operator could not be asked on any channel: %s; %s",
len(keys), strings.Join(keys, ", "), why),
Headline: "Questions for you not delivered",
Explanation: "The mesh could not send you its questions on any channel.",
Needs: "answer them from the mesh MCP server, and check why no channel carries them.",
Resolved: "The mesh can ask you again"})
}
if err := a.raise(ctx, obs); err != nil {
a.logf("whether the operator could be asked could not be kept as a condition: %v", err)
}
return nil
}
// optionID is an action's label as an option's id: "Silence for a week" is silence-for-a-week.
func optionID(label string) string {
var b strings.Builder
dash := false
for _, r := range strings.ToLower(label) {
switch {
case r >= 'a' && r <= 'z', r >= '0' && r <= '9':
b.WriteRune(r)
dash = false
case !dash && b.Len() > 0:
b.WriteByte('-')
dash = true
}
}
return strings.TrimSuffix(b.String(), "-")
}
// doesWords is what an action does, in the words an option says it with.
func doesWords(act conditions.Action) string {
switch {
case act.Arguments["silence"] != "":
return "nothing more is said of it for a week"
case act.Verb == "mesh-delivery.release":
return "the delivery goes on"
case act.Verb == "mesh-delivery.stop":
return "the delivery ends"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["go"] != "":
return "the delivery starts"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["stop"] != "":
return "the delivery is stopped"
case strings.HasSuffix(act.Verb, ".restart"):
return "its service is restarted on " + act.Machine
}
return strings.ToLower(act.Label)
}
// askText is a condition's words as an ask says them: without where an answer is given when no channel can
// give it (FromMeshMCPServer), since the ask is answered on a channel and the router says where else.
func askText(s string) string {
for _, with := range []string{", " + FromMeshMCPServer, " " + FromMeshMCPServer} {
s = strings.ReplaceAll(s, with, ".")
}
return strings.ReplaceAll(s, "..", ".")
}
// askOf is the ask one part of a condition is asked with.
func askOf(id string, c conditions.Condition, p askPart, now time.Time) (asks.Ask, map[string]int) {
q := asks.Ask{ID: id, Headline: c.Headline, Explanation: askText(c.Explanation), Who: asks.Operator,
OnExpiry: "nothing is done, and you are asked again while it lasts", About: p.about,
Urgent: c.Severity == conditions.Urgent}
options := map[string]int{}
approves := false
for i, act := range p.actions {
level := levelOf(act) // an action that says nothing of its level is never taken for less than approve
approves = approves || level != asks.Acknowledge
oid := optionID(act.Label)
options[oid] = i
// Every option binds the exact act it stands for (novox/hq ADR 0259 §6): the verb, the machine and
// every argument. The warrant then authorises that act and no other.
binds, _ := asks.ActDigest(boundAct(act))
q.Options = append(q.Options, asks.Option{ID: oid, Label: act.Label, Does: doesWords(act), Level: level,
Binds: binds})
}
q.Expires = now.Add(askAcknowledgeFor)
if approves {
q.Expires = now.Add(askApproveFor)
}
return q, options
}
// boundAct is what an option's Binds digests: the act exactly as the controller will perform it — its verb,
// machine, level, and each argument as "arg.<name>" — and never its label or words.
func boundAct(act conditions.Action) asks.Act {
out := asks.Act{"verb": act.Verb, "machine": act.Machine, "level": act.Level}
for k, v := range act.Arguments {
out["arg."+k] = v
}
return out
}
func newAskID() string {
var b [8]byte
_, _ = rand.Read(b[:])
return "c" + hex.EncodeToString(b[:])
}
// askUnsent is an ask kept and never published: asked again at the next look.
const askUnsent = "unsent"
// ask publishes one ask about a part of a condition, kept before it is published (the review of 2026-10-09,
// L3): a warrant for it then always finds it, and one whose publishing failed is marked so and asked again.
func (a *asker) ask(ctx context.Context, c conditions.Condition, p askPart, channels string) error {
now := a.now()
id := newAskID()
q, options := askOf(id, c, p, now)
if err := q.Check(now); err != nil {
return err
}
body, err := json.Marshal(q)
if err != nil {
return err
}
if err := a.store.Create(ctx, asked{ID: id, Condition: c.Key, Part: p.name, Ask: q, Actions: p.actions,
Options: options, State: askOpen, Opened: now, Channels: channels}); err != nil {
return fmt.Errorf("the ask could not be kept, so it was not asked: %w", err)
}
if err := a.publish(ctx, asks.AskSubject(askerName), body, "ask."+id); err != nil {
if _, cerr := a.store.Change(ctx, id, func(x *asked) bool {
if x.State != askOpen || x.Acted != "" {
return false
}
x.State, x.Ended, x.Acted = askUnsent, a.now(), "nothing: it could not be published: "+err.Error()
return true
}); cerr != nil {
a.logf("the ask %s could not be published, and could not be marked so: %v", id, cerr)
}
return err
}
a.logf("asked the operator about %s (%s): %d answer(s)", c.Key, id, len(q.Options))
return nil
}
// cancel takes an ask back: kept cancelled first, so a warrant that comes after is refused, then said to the
// router; a cancel the router did not hear leaves the ask to expire there, and nothing is done on it here.
func (a *asker) cancel(ctx context.Context, r asked, why string) error {
stood, err := a.store.Change(ctx, r.ID, func(x *asked) bool {
if x.State != askOpen {
return false
}
x.State, x.Ended = askCancelled, a.now()
return true
})
if err != nil || !stood {
return err
}
body, _ := json.Marshal(map[string]string{"id": r.ID})
if err := a.publish(ctx, asks.CancelSubject(askerName), body, "cancel."+r.ID); err != nil {
a.logf("the ask %s about %s is taken back here, and the router could not be told (%v): it expires there, "+
"and no answer to it is acted on", r.ID, r.Condition, err)
return nil
}
a.logf("took back the ask %s about %s: %s", r.ID, r.Condition, why)
return nil
}
// Decided takes the router's word on one of the controller's asks (link.Decider). An error is returned only
// when what was decided could not be kept, so the word is held and heard again.
func (a *asker) Decided(ctx context.Context, body []byte) error {
var w asks.Warrant
if err := json.Unmarshal(body, &w); err != nil {
a.logf("the router's word on an ask could not be read; ignored: %v", err)
return nil
}
if w.Asker != askerName {
a.logf("REFUSED a warrant for %s's ask %s: the controller acts only on its own", w.Asker, w.Ask)
return nil
}
r, err := a.store.Get(ctx, w.Ask)
if err != nil {
return err
}
if r == nil {
a.logf("REFUSED a warrant for the ask %s, which the controller does not hold", w.Ask)
return nil
}
if r.Acted != "" {
return nil // heard again: acted on once
}
now := a.now()
if w.Outcome != asks.OutcomeChosen {
acted := "nothing: the ask " + string(w.Outcome)
if w.Words != "" {
acted += ": " + w.Words
}
if _, err := a.store.Change(ctx, r.ID, func(x *asked) bool {
if x.Acted != "" {
return false
}
x.State, x.Ended, x.Warrant, x.Acted = string(w.Outcome), now, &w, acted
return true
}); err != nil {
return err
}
a.logf("the ask %s about %s ended %s; nothing is done", r.ID, r.Condition, w.Outcome)
return nil
}
if r.State != askOpen {
// Cancelled, replaced or expired in the controller's own record: no answer to it is acted on.
a.logf("REFUSED a warrant for the ask %s, which is %s in the controller's own record", r.ID, r.State)
return nil
}
option, err := w.For(askerName, r.Ask)
if err != nil {
a.logf("REFUSED a warrant for the ask %s: %v", r.ID, err)
return nil
}
index, offered := r.Options[option.ID]
if !offered || index >= len(r.Actions) {
a.logf("REFUSED a warrant for the ask %s: it chose %s, which no action stands for", r.ID, option.ID)
return nil
}
act := r.Actions[index]
// The act about to be performed is the one the option bound when the controller asked: a record changed
// since is refused, never performed.
if err := option.Performs(boundAct(act)); err != nil {
a.logf("REFUSED a warrant for the ask %s: %v", r.ID, err)
return nil
}
open, err := a.open(ctx)
if err != nil {
return err
}
stillOpen := r.Rehearsal || r.Proposal != nil // a rehearsal and a proposal are about no condition
for _, c := range open {
stillOpen = stillOpen || c.Key == r.Condition
}
if !stillOpen {
// The asker checks the state is still what it asked about before it acts (to-be 46 §10, step 7).
if _, err := a.store.Change(ctx, r.ID, func(x *asked) bool {
if x.State != askOpen || x.Acted != "" {
return false
}
x.State, x.Warrant, x.Ended, x.Acted = string(asks.OutcomeChosen), &w, now,
"nothing: the condition ended before the answer"
return true
}); err != nil {
return err
}
a.logf("%s, for %s, which ended meanwhile: nothing is done", w.Says(), r.Condition)
return nil
}
// Claimed before acting, by compare-and-set: only the delivery whose write stands acts (security review
// of 2026-10-08, finding 9). Not by the warrant's message id, which another publisher could take first:
// the controller's own record decides.
claimed, err := a.store.Change(ctx, r.ID, func(x *asked) bool {
if x.State != askOpen || x.Acted != "" {
return false
}
x.State, x.Warrant, x.Acted = string(asks.OutcomeChosen), &w, "acting"
return true
})
if err != nil {
return err
}
if !claimed {
a.logf("the warrant for the ask %s was already taken by another delivery; nothing more is done", r.ID)
return nil
}
r.Acted = "acting"
why := fmt.Sprintf("%s (ask %s)", w.Says(), r.ID)
args := map[string]string{}
for k, v := range act.Arguments {
args[k] = v
}
if v, takes := args["why"]; takes && v == "" {
args["why"] = why
}
var acted error
outcome := "done"
switch {
case r.Rehearsal && act.Verb == rehearsalVerb:
// A rehearsal's answer performs nothing: it is recorded below as the operator's decision.
case r.Proposal != nil && act.Verb == proposalVerb:
// A proposed settings layer, set by this controller itself on Approve (novox/hq ADR 0277): the act's
// digest of the values is held to the record's own values before anything is set.
outcome, acted = a.decideProposal(ctx, *r, act, w)
if acted == nil && outcome != "" && !strings.HasPrefix(outcome, "nothing") {
outcome = "done: " + outcome
}
case act.Arguments["silence"] != "":
acted = a.silence(ctx, act.Arguments["silence"], conditions.MaxSilence, byWords(w), why)
default:
acted = a.call(ctx, act, args)
}
ended := a.now()
if acted != nil {
outcome = "failed: " + acted.Error()
}
r.Ended, r.Acted = ended, outcome
if _, err := a.store.Change(ctx, r.ID, func(x *asked) bool {
if x.Acted != "acting" {
return false
}
x.Ended, x.Acted = ended, outcome
return true
}); err != nil {
a.logf("%s was acted on (%s), and how it ended could NOT be kept: %v", r.ID, outcome, err)
}
verbArgs := []string{act.Verb}
if act.Machine != "" {
verbArgs = append(verbArgs, "on "+act.Machine)
}
keys := make([]string, 0, len(args))
for k := range args {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if k != "why" {
verbArgs = append(verbArgs, k+"="+args[k])
}
}
if err := a.record(ctx, link.HandAct{Verb: handActWarrant, Args: verbArgs, Why: why, By: byWords(w),
Cause: conditions.CauseOperatorAnswer, Condition: r.Condition, Via: viaWords(w), Ask: r.ID,
Proofs: w.Proofs, RequestedBy: r.Condition, Outcome: r.Acted}); err != nil {
a.logf("%s was done, and could NOT be recorded in the hand-act log: %v", why, err)
}
a.logf("%s: %s", why, r.Acted)
return nil
}
// handActWarrant is the verb an act the operator chose on a warrant is recorded under: a person's decision,
// never a repair (handActVerbs).
const handActWarrant = "warrant"
// byWords is who chose, as the hand-act log says it: "the operator, as telegram identity 42".
func byWords(w asks.Warrant) string {
if w.By == nil {
return "the operator"
}
return fmt.Sprintf("the %s, as %s identity %s", w.By.Who, w.By.Kind, w.By.Identity)
}
// viaWords is the channel an answer came through: its module and kind, and how the sender was known.
func viaWords(w asks.Warrant) string {
if w.By == nil {
return w.Channel
}
via := w.Channel + " (" + w.By.Kind + ")"
if w.By.Verified != "" {
via += ", " + w.By.Verified
}
return via
}
// errNotGranted is an action whose verb the controller's grant does not name.
var errNotGranted = errors.New("the controller's grant does not name this verb")
+151
View File
@@ -0,0 +1,151 @@
package main
import (
"context"
"encoding/json"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// busAsker is an asker on a real bus's `asked` bucket, counting what it performs: two of them are two
// controllers sharing one record.
type busAskerRig struct {
mu sync.Mutex
called int
acts int
open []conditions.Condition
sent [][]byte
}
func (rig *busAskerRig) asker(t *testing.T, conn *nats.Conn, now time.Time) *asker {
return &asker{
open: func(context.Context) ([]conditions.Condition, error) {
rig.mu.Lock()
defer rig.mu.Unlock()
return rig.open, nil
},
silence: func(context.Context, string, time.Duration, string, string) error { return nil },
store: busAsked{conn: conn},
publish: func(_ context.Context, subject string, body []byte, _ string) error {
rig.mu.Lock()
defer rig.mu.Unlock()
if subject == asks.AskSubject(askerName) {
rig.sent = append(rig.sent, body)
}
return nil
},
call: func(context.Context, conditions.Action, map[string]string) error {
time.Sleep(20 * time.Millisecond) // long enough for the other delivery to arrive meanwhile
rig.mu.Lock()
defer rig.mu.Unlock()
rig.called++
return nil
},
record: func(context.Context, link.HandAct) error {
rig.mu.Lock()
defer rig.mu.Unlock()
rig.acts++
return nil
},
now: func() time.Time { return now },
logf: t.Logf,
}
}
func askedBus(t *testing.T) *nats.Conn {
t.Helper()
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
js, err := jetstream.New(conn)
if err != nil {
t.Fatal(err)
}
if _, err := js.CreateKeyValue(context.Background(), jetstream.KeyValueConfig{Bucket: broker.AskedBucket}); err != nil {
t.Fatal(err)
}
return conn
}
// The review of 2026-10-09 (L7): two deliveries of one warrant, to two controllers at once, perform its act
// exactly once and record it once — the record's compare-and-set decides, never the warrant's message id.
func TestTwoAnswersAtOnceActOnce(t *testing.T) {
conn := askedBus(t)
now := time.Date(2026, 10, 9, 14, 0, 0, 0, time.UTC)
rig := &busAskerRig{open: []conditions.Condition{heldCondition()}}
first, second := rig.asker(t, conn, now), rig.asker(t, conn, now)
if err := first.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
if len(rig.sent) != 1 {
t.Fatalf("asked %d times", len(rig.sent))
}
var q asks.Ask
_ = json.Unmarshal(rig.sent[0], &q)
release, _ := q.Option("release")
w := asks.Warrant{Ask: q.ID, Asker: askerName, About: q.About, Outcome: asks.OutcomeChosen, Option: release.ID,
Label: release.Label, Level: release.Level, Channel: "telegram", Proofs: []string{"P1"}, At: now,
AskDigest: q.Digest(), By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
body, _ := json.Marshal(w)
var wg sync.WaitGroup
for _, a := range []*asker{first, second, first, second} {
wg.Add(1)
go func(a *asker) {
defer wg.Done()
if err := a.Decided(context.Background(), body); err != nil {
t.Error(err)
}
}(a)
}
wg.Wait()
if rig.called != 1 || rig.acts != 1 {
t.Fatalf("performed %d time(s), recorded %d time(s)", rig.called, rig.acts)
}
got, err := busAsked{conn: conn}.Get(context.Background(), q.ID)
if err != nil || got == nil || got.Acted != "done" {
t.Fatalf("kept as %+v (%v)", got, err)
}
}
// The review of 2026-10-09 (L2): a write decided on a record read earlier never lands over one made since. A
// cancel read before the answer was acted on leaves the act's record as it is.
func TestAStaleCancelDoesNotWriteOverAnAct(t *testing.T) {
conn := askedBus(t)
now := time.Date(2026, 10, 9, 14, 0, 0, 0, time.UTC)
rig := &busAskerRig{open: []conditions.Condition{heldCondition()}}
a := rig.asker(t, conn, now)
if err := a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
var q asks.Ask
_ = json.Unmarshal(rig.sent[0], &q)
stale, _ := busAsked{conn: conn}.Get(context.Background(), q.ID)
release, _ := q.Option("release")
w := asks.Warrant{Ask: q.ID, Asker: askerName, About: q.About, Outcome: asks.OutcomeChosen, Option: release.ID,
Label: release.Label, Level: release.Level, Channel: "telegram", At: now, AskDigest: q.Digest(),
By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
body, _ := json.Marshal(w)
if err := a.Decided(context.Background(), body); err != nil {
t.Fatal(err)
}
if err := a.cancel(context.Background(), *stale, "the condition ended"); err != nil {
t.Fatal(err)
}
got, _ := busAsked{conn: conn}.Get(context.Background(), q.ID)
if got.State != string(asks.OutcomeChosen) || got.Acted != "done" {
t.Errorf("a stale cancel wrote over the act: %+v", got)
}
}
+810
View File
@@ -0,0 +1,810 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq ADR 0259 §6: the controller asks the operator for the answers its conditions name, and performs
// the one chosen on the router's warrant — once, for its own ask, the option offered, at its level.
type memAskedStore map[string]asked
// memAskedMu guards every memAskedStore: Change is a compare-and-set as the bus's is.
var memAskedMu sync.Mutex
func (m memAskedStore) Get(_ context.Context, id string) (*asked, error) {
memAskedMu.Lock()
defer memAskedMu.Unlock()
r, ok := m[id]
if !ok {
return nil, nil
}
return &r, nil
}
func (m memAskedStore) Create(_ context.Context, r asked) error {
memAskedMu.Lock()
defer memAskedMu.Unlock()
if _, kept := m[r.ID]; kept {
return errors.New("an ask is kept under that id")
}
m[r.ID] = r
return nil
}
func (m memAskedStore) Change(_ context.Context, id string, change func(*asked) bool) (bool, error) {
memAskedMu.Lock()
defer memAskedMu.Unlock()
r, ok := m[id]
if !ok || !change(&r) {
return false, nil
}
m[id] = r
return true, nil
}
func (m memAskedStore) All(context.Context) ([]asked, error) {
memAskedMu.Lock()
defer memAskedMu.Unlock()
var out []asked
for _, r := range m {
out = append(out, r)
}
return out, nil
}
type published struct {
subject, id string
body []byte
}
type askerRig struct {
a *asker
open []conditions.Condition
store memAskedStore
sent []published
called []string
silenced []string
acts []link.HandAct
now time.Time
}
func newAskerRig(t *testing.T) *askerRig {
r := &askerRig{store: memAskedStore{}, now: time.Date(2026, 10, 8, 14, 0, 0, 0, time.UTC)}
r.a = &asker{
open: func(context.Context) ([]conditions.Condition, error) { return r.open, nil },
silence: func(_ context.Context, key string, d time.Duration, by, why string) error {
r.silenced = append(r.silenced, key+" for "+d.String()+" by "+by+" because "+why)
// As the controller's conditions do (the confirmation review of 2026-10-09, M1): the condition is
// silenced from now on, so what is asked next sees it silenced.
for i := range r.open {
if r.open[i].Key == key {
r.open[i].Silenced = &conditions.Silence{Until: r.now.Add(d), By: by, Why: why, Since: r.now}
}
}
return nil
},
store: r.store,
publish: func(_ context.Context, subject string, body []byte, id string) error {
r.sent = append(r.sent, published{subject, id, body})
return nil
},
call: func(_ context.Context, a conditions.Action, args map[string]string) error {
raw, _ := json.Marshal(args)
r.called = append(r.called, a.Verb+"@"+a.Machine+" "+string(raw))
return nil
},
record: func(_ context.Context, act link.HandAct) error { r.acts = append(r.acts, act); return nil },
now: func() time.Time { return r.now },
logf: t.Logf,
}
return r
}
func heldCondition() conditions.Condition {
o := stalledObservations([]stalledLine{{ID: "novox/hq@055550802096", State: "held", For: "36h2m6s",
Bound: "24h0m0s", H2: "none: the state is the operator's"}})[0]
return conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: conditions.Warning, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
}
func unitsCondition() conditions.Condition {
key := "machine.shanks.units"
return conditions.Condition{Key: key, Kind: "machine-units", Severity: conditions.Warning,
Headline: "3 failed services on shanks", Explanation: "Needs you: mend or remove them on shanks, or silence this.",
Needs: "mend or remove them on shanks, or silence this.", Actions: []conditions.Action{conditions.SilenceAction(key)}}
}
func (r *askerRig) asksSent(t *testing.T) []asks.Ask {
t.Helper()
var out []asks.Ask
for _, p := range r.sent {
if p.subject != asks.AskSubject("mesh-controller") {
continue
}
var q asks.Ask
if err := json.Unmarshal(p.body, &q); err != nil {
t.Fatal(err)
}
out = append(out, q)
}
return out
}
func TestAnAskIsMadeForEachConditionThatNamesItsAnswers(t *testing.T) {
r := newAskerRig(t)
quiet := conditions.Condition{Key: "machine.ace.silent", Headline: "ace silent", Explanation: "Nothing for you to do. x"}
r.open = []conditions.Condition{heldCondition(), unitsCondition(), quiet}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 2 {
t.Fatalf("asked %d times: %+v", len(sent), sent)
}
byAbout := map[string]asks.Ask{}
for _, q := range sent {
byAbout[q.About] = q
if err := q.Check(r.now); err != nil {
t.Errorf("%s: %v", q.About, err)
}
}
held := byAbout[heldCondition().Key]
if len(held.Options) != 2 || held.Options[0].Label != "Release" || held.Options[0].Level != asks.Approve ||
held.Options[1].ID != "stop" || held.Expires != r.now.Add(askApproveFor) || held.Who != asks.Operator ||
held.OnExpiry == "" {
t.Errorf("the held delivery is asked %+v", held)
}
units := byAbout["machine.shanks.units"]
if len(units.Options) != 1 || units.Options[0].Level != asks.Acknowledge || units.Expires != r.now.Add(askAcknowledgeFor) {
t.Errorf("the failed units are asked %+v", units)
}
// No second ask while one is open.
r.now = r.now.Add(time.Minute)
_ = r.a.reconcile(context.Background())
if n := len(r.asksSent(t)); n != 2 {
t.Errorf("asked again while open: %d", n)
}
}
func TestAnAskIsTakenBackWhenItsConditionEndsAndAskedAgainAfterItExpires(t *testing.T) {
r := newAskerRig(t)
r.open = []conditions.Condition{heldCondition(), unitsCondition()}
_ = r.a.reconcile(context.Background())
// The units are silenced, the held delivery lasts past its ask's day.
units := unitsCondition()
units.Silenced = &conditions.Silence{Until: r.now.Add(48 * time.Hour)}
r.open = []conditions.Condition{heldCondition(), units}
r.now = r.now.Add(askApproveFor)
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
var cancels int
for _, p := range r.sent {
if p.subject == asks.CancelSubject("mesh-controller") {
cancels++
}
}
if cancels != 1 {
t.Errorf("cancels %d, want the silenced one's", cancels)
}
if sent := r.asksSent(t); len(sent) != 3 || sent[2].About != heldCondition().Key {
t.Errorf("the expired ask was not asked again: %+v", sent)
}
}
// warrantFor is the router's warrant for the open ask about a condition, choosing an option by label.
func (r *askerRig) warrantFor(t *testing.T, condition, label string) asks.Warrant {
t.Helper()
for _, a := range r.store {
if a.Condition != condition || a.State != askOpen {
continue
}
for _, o := range a.Ask.Options {
if o.Label == label {
return asks.Warrant{Ask: a.ID, Asker: "mesh-controller", About: condition, Outcome: asks.OutcomeChosen,
Option: o.ID, Label: o.Label, Level: o.Level, Channel: "telegram", Proofs: []string{"P1"}, At: r.now,
AskDigest: a.Ask.Digest(),
By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
}
}
}
t.Fatalf("no open ask about %s offers %s", condition, label)
return asks.Warrant{}
}
func answerWith(t *testing.T, r *askerRig, w asks.Warrant) {
t.Helper()
body, _ := json.Marshal(w)
if err := r.a.Decided(context.Background(), body); err != nil {
t.Fatal(err)
}
}
func TestAWarrantIsActedOnOnce(t *testing.T) {
r := newAskerRig(t)
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, heldCondition().Key, "Release")
answerWith(t, r, w)
answerWith(t, r, w) // heard again
if len(r.called) != 1 {
t.Fatalf("called %v", r.called)
}
want := `mesh-delivery.release@ {"id":"novox/hq@055550802096","why":"the operator, via telegram (user id verified), chose Release (ask ` + w.Ask + `)"}`
if r.called[0] != want {
t.Errorf("called\n %s\nwant\n %s", r.called[0], want)
}
if len(r.acts) != 1 {
t.Fatalf("hand-acts %+v", r.acts)
}
act := r.acts[0]
if act.Verb != handActWarrant || act.By != "the operator, as telegram identity 42" ||
act.Via != "telegram (telegram), user id verified" || act.Ask != w.Ask || strings.Join(act.Proofs, ",") != "P1" ||
act.Cause != conditions.CauseOperatorAnswer || act.Condition != heldCondition().Key || act.Outcome != "done" {
t.Errorf("the hand-act %+v", act)
}
if !personsDecision(act) {
t.Error("an act on a warrant counts as a repair")
}
if got := r.store[w.Ask]; got.State != string(asks.OutcomeChosen) || got.Acted != "done" {
t.Errorf("kept %+v", got)
}
}
func TestAWarrantThatIsNotForItsOwnAskIsRefused(t *testing.T) {
for name, change := range map[string]func(*asks.Warrant){
"another asker": func(w *asks.Warrant) { w.Asker = "mesh-delivery" },
"an ask not held": func(w *asks.Warrant) { w.Ask = "c0000000000000000" },
"an option not offered": func(w *asks.Warrant) { w.Option = "delete" },
"another level": func(w *asks.Warrant) { w.Level = asks.Acknowledge },
"no person": func(w *asks.Warrant) { w.By = nil },
"another ask's digest": func(w *asks.Warrant) { w.AskDigest = "sha256:0000" },
"no ask's digest": func(w *asks.Warrant) { w.AskDigest = "" },
} {
t.Run(name, func(t *testing.T) {
r := newAskerRig(t)
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, heldCondition().Key, "Stop")
change(&w)
answerWith(t, r, w)
if len(r.called)+len(r.acts)+len(r.silenced) != 0 {
t.Errorf("acted on it: %v %v %v", r.called, r.acts, r.silenced)
}
})
}
}
func TestEachAnswerCallsExactlyItsVerb(t *testing.T) {
plan := "plan-1791454185265004861"
waiting := conditions.Condition{Key: "plan." + plan + ".waiting", Severity: conditions.Urgent,
Headline: "openrazer delivery waiting to start", Needs: "start it, or stop it.",
Explanation: "Needs you: start it, or stop it.", Actions: waitingActions(plan, conditions.Urgent)}
module := conditions.Condition{Key: "module.openrazer.g14.unhealthy", Severity: conditions.Warning,
Headline: "openrazer not working on g14", Needs: "restart its service openrazer-daemon on g14.",
Explanation: "Needs you: restart it.", Actions: []conditions.Action{{Label: "Restart",
Verb: "node-service-manager.restart", Machine: "g14", Level: conditions.LevelApprove,
Arguments: map[string]string{"unit": "openrazer-daemon.service", "scope": "user"}}}}
for _, tc := range []struct {
c conditions.Condition
label string
want string
}{
{waiting, "Start", `mesh-controller.plans@ {"cause":"operator-answer","go":"` + plan + `","why":"`},
{waiting, "Stop", `mesh-controller.plans@ {"cause":"operator-answer","stop":"` + plan + `","why":"`},
{module, "Restart", `node-service-manager.restart@g14 {"scope":"user","unit":"openrazer-daemon.service"}`},
} {
r := newAskerRig(t)
r.open = []conditions.Condition{tc.c}
_ = r.a.reconcile(context.Background())
answerWith(t, r, r.warrantFor(t, tc.c.Key, tc.label))
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], tc.want) {
t.Errorf("%s: called %v, want %s…", tc.label, r.called, tc.want)
}
}
}
func TestASilenceChosenIsTheControllersOwnAndAnAnswerToAnAsk(t *testing.T) {
r := newAskerRig(t)
r.open = []conditions.Condition{unitsCondition()}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, "machine.shanks.units", "Silence for a week")
w.Level, w.Proofs = asks.Acknowledge, nil
w.By = &asks.Person{Who: asks.Operator, Kind: "desktop", Identity: "g14",
Verified: "a desk click: whoever was at the operator's session on g14"}
w.Channel = "desk-channel"
answerWith(t, r, w)
if len(r.called) != 0 || len(r.silenced) != 1 || !strings.HasPrefix(r.silenced[0], "machine.shanks.units for 168h0m0s by the operator, as desktop identity g14") {
t.Fatalf("silenced %v, called %v", r.silenced, r.called)
}
if len(r.acts) != 1 || r.acts[0].Cause != conditions.CauseOperatorAnswer || len(r.acts[0].Proofs) != 0 {
t.Errorf("%+v", r.acts)
}
}
func TestAnAskThatEndedWithoutAChoiceDoesNothing(t *testing.T) {
r := newAskerRig(t)
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, heldCondition().Key, "Release")
w.Outcome, w.Option, w.Label, w.Level, w.By, w.Words = asks.OutcomeExpired, "", "", "", nil, "nobody answered in time"
answerWith(t, r, w)
if len(r.called)+len(r.acts) != 0 || r.store[w.Ask].State != string(asks.OutcomeExpired) ||
!strings.HasPrefix(r.store[w.Ask].Acted, "nothing") {
t.Errorf("called %v acts %v kept %+v", r.called, r.acts, r.store[w.Ask])
}
// And a choice for a condition that ended meanwhile does nothing either.
r2 := newAskerRig(t)
r2.open = []conditions.Condition{heldCondition()}
_ = r2.a.reconcile(context.Background())
w2 := r2.warrantFor(t, heldCondition().Key, "Release")
r2.open = nil
answerWith(t, r2, w2)
if len(r2.called) != 0 || r2.store[w2.Ask].Acted != "nothing: the condition ended before the answer" {
t.Errorf("%v %+v", r2.called, r2.store[w2.Ask])
}
}
func TestAWarrantMissedWhileAwayIsReadFromTheRoutersRecord(t *testing.T) {
r := newAskerRig(t)
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, heldCondition().Key, "Stop")
r.a.routerRecord = func(_ context.Context, id string) (*asks.Warrant, error) {
if id != w.Ask {
return nil, errors.New("another ask")
}
return &w, nil
}
r.now = r.now.Add(askCatchUpAfter)
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "mesh-delivery.stop@") {
t.Errorf("called %v", r.called)
}
if n := len(r.asksSent(t)); n != 1 {
t.Errorf("asked again after the answer: %d", n)
}
}
// After review (2026-10-08): a refused ask is not asked again at once; issue 369: nor is the refusal believed
// for ever — it is asked again after a wait that doubles with each refusal in a row, and at once when the
// channels change.
func TestAnAskTheRouterRefusedIsAskedAgainAfterAGrowingWait(t *testing.T) {
r := newAskerRig(t)
channels := "channel/telegram=telegram@anchor[choice]own:true"
r.a.channels = func(context.Context) string { return channels }
r.open = []conditions.Condition{heldCondition()}
refuse := func() {
t.Helper()
sent := r.asksSent(t)
refusal, _ := json.Marshal(asks.Warrant{Ask: sent[len(sent)-1].ID, Asker: "mesh-controller",
Outcome: asks.OutcomeRefused, Words: "no channel can carry any of its answers now", At: r.now})
if err := r.a.Decided(context.Background(), refusal); err != nil {
t.Fatal(err)
}
}
_ = r.a.reconcile(context.Background())
first := r.asksSent(t)[0]
refuse()
if got := r.store[first.ID]; got.State != string(asks.OutcomeRefused) || !strings.Contains(got.Acted, "nothing") {
t.Fatalf("the refusal was kept as %+v", got)
}
// Each wait: nothing asked before it ends, asked once when it has.
for i, wait := range []time.Duration{time.Minute, 2 * time.Minute, 4 * time.Minute, 8 * time.Minute,
16 * time.Minute, 30 * time.Minute, 30 * time.Minute} {
before := len(r.asksSent(t))
r.now = r.now.Add(wait - 10*time.Second)
_ = r.a.reconcile(context.Background())
if n := len(r.asksSent(t)); n != before {
t.Fatalf("refusal %d: asked again before %s", i+1, wait)
}
r.now = r.now.Add(10 * time.Second)
_ = r.a.reconcile(context.Background())
if n := len(r.asksSent(t)); n != before+1 {
t.Fatalf("refusal %d: not asked again after %s (%d asks)", i+1, wait, n)
}
refuse()
}
before := len(r.asksSent(t))
channels = "channel/telegram=telegram@anchor[choice,verified-sender]own:true"
_ = r.a.reconcile(context.Background())
if n := len(r.asksSent(t)); n != before+1 {
t.Errorf("not asked again once the channels changed: %d", n-before)
}
}
// After review: at most three asks open at once, the most urgent first, then the oldest.
func TestAtMostThreeAsksAreOpenTheMostUrgentFirst(t *testing.T) {
r := newAskerRig(t)
var open []conditions.Condition
for i := 0; i < 4; i++ {
c := unitsCondition()
c.Key = "machine.m" + string(rune('a'+i)) + ".units"
c.Actions = []conditions.Action{conditions.SilenceAction(c.Key)}
c.Raised = r.now.Add(-time.Duration(10-i) * time.Hour)
open = append(open, c)
}
urgent := heldCondition()
urgent.Severity, urgent.Raised = conditions.Urgent, r.now.Add(-time.Minute)
r.open = append(open, urgent)
_ = r.a.reconcile(context.Background())
sent := r.asksSent(t)
if len(sent) != askMostOpen || sent[0].About != urgent.Key || sent[1].About != "machine.ma.units" || sent[2].About != "machine.mb.units" {
var about []string
for _, q := range sent {
about = append(about, q.About)
}
t.Fatalf("asked %v", about)
}
}
// After review: nothing is asked while no router takes asks under the controller's name, and that is said once.
func TestNothingIsAskedWithoutARouter(t *testing.T) {
r := newAskerRig(t)
var said []string
r.a.logf = func(f string, a ...any) { said = append(said, f) }
r.a.routerHere = func(context.Context) (bool, error) { return false, nil }
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
_ = r.a.reconcile(context.Background())
if len(r.asksSent(t)) != 0 {
t.Error("asked with no router")
}
n := 0
for _, s := range said {
if strings.Contains(s, "no router takes asks") {
n++
}
}
if n != 1 {
t.Errorf("said %d times", n)
}
}
// After review: the condition's words keep where an answer is given without a channel; the ask's text does not.
func TestTheAskDropsWhereItIsAnsweredAndTheConditionKeepsIt(t *testing.T) {
c := heldCondition()
if !strings.Contains(c.Explanation, FromMeshMCPServer) {
t.Fatalf("the condition lost where it is answered: %q", c.Explanation)
}
q, _ := askOf("x", c, partsOf(c)[0], time.Now())
if strings.Contains(q.Explanation, "mesh MCP server") || !strings.HasPrefix(q.Explanation, "Needs you: release it, or stop it.") {
t.Errorf("the ask says %q", q.Explanation)
}
if askApproveFor >= 24*time.Hour {
t.Errorf("an approving ask lasts %s, which the SDK may refuse at its bound", askApproveFor)
}
}
// After review (security finding 9): a warrant is acted on only for an ask open in the controller's own record,
// once the claim stands, and never when given after the ask expired.
func TestAWarrantIsActedOnlyForAnOpenAskItClaimsBeforeItExpired(t *testing.T) {
r := newAskerRig(t)
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, heldCondition().Key, "Release")
late := w
late.At = r.store[w.Ask].Ask.Expires.Add(time.Minute)
body, _ := json.Marshal(late)
_ = r.a.Decided(context.Background(), body)
if len(r.called) != 0 {
t.Fatalf("acted on a warrant given after the ask expired: %v", r.called)
}
// Claimed already by another delivery: nothing done here.
kept := r.store[w.Ask]
kept.Acted = "acting"
r.store[w.Ask] = kept
body, _ = json.Marshal(w)
_ = r.a.Decided(context.Background(), body)
if len(r.called) != 0 {
t.Fatalf("acted though the claim was another's: %v", r.called)
}
// Cancelled in its own record: refused.
kept.Acted, kept.State = "", askCancelled
r.store[w.Ask] = kept
_ = r.a.Decided(context.Background(), body)
if len(r.called) != 0 {
t.Errorf("acted on a cancelled ask: %v", r.called)
}
}
// novox/hq ADR 0259 §6: a warrant authorises the act its option bound when the controller asked, and no
// other. A record of the act changed after the ask — another delivery, another machine, another argument —
// is refused and nothing is performed.
func TestAWarrantPerformsOnlyTheActItsOptionBound(t *testing.T) {
for name, change := range map[string]func(*conditions.Action){
"another argument": func(a *conditions.Action) {
a.Arguments = map[string]string{"id": "novox/mesh-controller@000000000000"}
},
"another verb": func(a *conditions.Action) { a.Verb = "mesh-delivery.stop" },
"another machine": func(a *conditions.Action) { a.Machine = "anchor" },
} {
t.Run(name, func(t *testing.T) {
r := newAskerRig(t)
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, heldCondition().Key, "Release")
kept := r.store[w.Ask]
acts := append([]conditions.Action(nil), kept.Actions...)
i := kept.Options[w.Option]
change(&acts[i])
kept.Actions = acts
r.store[w.Ask] = kept
answerWith(t, r, w)
if len(r.called)+len(r.acts) != 0 {
t.Errorf("performed an act the option did not bind: %v %v", r.called, r.acts)
}
})
}
// Every option of an ask binds its act.
q, _ := askOf("x", heldCondition(), partsOf(heldCondition())[0], time.Now())
for _, o := range q.Options {
if o.Binds == "" {
t.Errorf("the option %s binds nothing", o.ID)
}
}
}
// Failure is loud (novox/hq ADR 0259, the self-review of 2026-10-09): a condition that needs the operator and
// could not be asked on any channel — no router, or the router refused the ask — is a condition of its own,
// cleared once it can be asked again.
func TestAnAskThatCannotBeDeliveredIsSaid(t *testing.T) {
r := newAskerRig(t)
var raised [][]conditions.Observation
r.a.raise = func(_ context.Context, obs []conditions.Observation) error {
raised = append(raised, obs)
return nil
}
last := func() []conditions.Observation { return raised[len(raised)-1] }
routerHere := false
r.a.routerHere = func(context.Context) (bool, error) { return routerHere, nil }
channels := "channel/telegram=telegram@anchor[choice]own:true"
r.a.channels = func(context.Context) string { return channels }
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
if got := last(); len(got) != 1 || got[0].Kind != "asks-undelivered" ||
!strings.Contains(got[0].Summary, heldCondition().Key) || !strings.Contains(got[0].Summary, "no router") {
t.Fatalf("no router, said as %+v", got)
}
routerHere = true
_ = r.a.reconcile(context.Background())
if got := last(); len(got) != 0 {
t.Fatalf("asked, and still said undelivered: %+v", got)
}
first := r.asksSent(t)[0]
refusal, _ := json.Marshal(asks.Warrant{Ask: first.ID, Asker: "mesh-controller", Outcome: asks.OutcomeRefused,
Words: "no channel can carry any of its answers now", At: r.now})
if err := r.a.Decided(context.Background(), refusal); err != nil {
t.Fatal(err)
}
_ = r.a.reconcile(context.Background())
if got := last(); len(got) != 1 || !strings.Contains(got[0].Summary, "no channel can carry") {
t.Fatalf("the router's refusal, said as %+v", got)
}
if why, ok := conditions.PlainWords(conditions.Words{Headline: last()[0].Headline, Explanation: last()[0].Explanation,
Needs: last()[0].Needs, Resolved: last()[0].Resolved}, ""); !ok {
t.Errorf("not plain: %s", why)
}
r.open = nil
_ = r.a.reconcile(context.Background())
if got := last(); len(got) != 0 {
t.Errorf("nothing needs asking, and still said: %+v", got)
}
}
// The review of 2026-10-09 (M1): an acknowledging answer never shares an ask with an authorising one. A
// condition offering Restart and Silence is asked twice — Restart alone, about the condition, and Silence
// alone, apart — so Silence chosen on a channel that only acknowledges leaves the Restart ask open.
func TestAnAcknowledgementNeverSharesAnAskWithAnApproval(t *testing.T) {
r := newAskerRig(t)
key := "module.shanks.plex.down"
c := conditions.Condition{Key: key, Kind: "module-down", Severity: conditions.Urgent, Headline: "Plex down on shanks",
Explanation: "Needs you: restart it, or silence this.", Needs: "restart it, or silence this.",
Actions: []conditions.Action{
{Label: "Restart", Verb: "node-service-manager.restart", Machine: "shanks", Level: conditions.LevelApprove,
Arguments: map[string]string{"unit": "plex"}},
conditions.SilenceAction(key)}}
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 2 {
t.Fatalf("asked %d time(s): %+v", len(sent), sent)
}
for _, q := range sent {
if err := q.Check(r.now); err != nil {
t.Errorf("%s: %v", q.About, err)
}
levels := map[asks.Level]bool{}
for _, o := range q.Options {
levels[o.Level] = true
}
if len(levels) != 1 {
t.Errorf("the ask about %s mixes levels: %+v", q.About, q.Options)
}
}
byAbout := map[string]asks.Ask{}
for _, q := range sent {
byAbout[q.About] = q
}
if q := byAbout[key]; len(q.Options) != 1 || q.Options[0].Label != "Restart" {
t.Errorf("the condition's own ask: %+v", q)
}
if q := byAbout[key+".acknowledge"]; len(q.Options) != 1 || q.Options[0].Level != asks.Acknowledge {
t.Errorf("the acknowledging ask: %+v", q)
}
// Silence chosen: performed, and the Restart ask stays open, never asked twice.
answerWith(t, r, r.warrantFor(t, key, "Silence for a week"))
if len(r.silenced) != 1 || len(r.called) != 0 {
t.Fatalf("silenced %v called %v", r.silenced, r.called)
}
_ = r.a.reconcile(context.Background())
open := 0
for _, a := range r.store {
if a.State == askOpen && a.Condition == key {
open++
if a.Part != "" || a.Ask.Options[0].Label != "Restart" {
t.Errorf("the open ask is %+v", a)
}
}
}
if open != 1 || len(r.asksSent(t)) != 2 {
t.Errorf("after the silence: %d open, %d asked", open, len(r.asksSent(t)))
}
// And the approval still answers: Restart chosen on a channel that proves who answered is performed.
answerWith(t, r, r.warrantFor(t, key, "Restart"))
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "node-service-manager.restart@shanks") {
t.Errorf("the approval kept through a silence was not performed: %v", r.called)
}
}
// novox/hq issue 353: the controller's grant to ask is composed from the router's assignment and reaches the
// bus only when its machine is pushed. Between the two, the bus refuses every ask (measured 2026-10-09, 17:54 to
// 17:56 local: seven refusals of mesh.seat.operator-channel.accept.ask.mesh-controller). So nothing is asked
// while the bus's user list is behind, it is said once, the conditions that need the operator are raised as
// undelivered with what to do, and the asks go out once the bus holds the grant.
func TestNothingIsAskedWhileTheBusLacksTheControllersGrant(t *testing.T) {
r := newAskerRig(t)
var said []string
r.a.logf = func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }
var raised [][]conditions.Observation
r.a.raise = func(_ context.Context, obs []conditions.Observation) error {
raised = append(raised, obs)
return nil
}
held := false
r.a.grantHeld = func(context.Context) (bool, string, error) {
return held, "the bus's user list on anchor is behind what the mesh composes; `push anchor` carries it", nil
}
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
_ = r.a.reconcile(context.Background())
if len(r.asksSent(t)) != 0 {
t.Error("asked while the bus lacks the grant")
}
n := 0
for _, s := range said {
if strings.Contains(s, "does not hold the controller's grant") {
n++
}
}
if n != 1 {
t.Errorf("said %d times: %q", n, said)
}
if len(raised) == 0 || len(raised[len(raised)-1]) != 1 ||
!strings.Contains(raised[len(raised)-1][0].Summary, "`push anchor` carries it") ||
raised[len(raised)-1][0].Kind != "asks-undelivered" {
t.Fatalf("not said as a condition with what to do: %+v", raised)
}
held = true
_ = r.a.reconcile(context.Background())
if sent := r.asksSent(t); len(sent) != 1 || sent[0].About != heldCondition().Key {
t.Errorf("not asked once the bus holds the grant: %+v", sent)
}
if last := raised[len(raised)-1]; len(last) != 0 {
t.Errorf("the undelivered condition was not cleared: %+v", last)
}
}
// novox/hq issue 369: the router refused while its channels had not yet said they could send; they could twenty
// minutes later, and "Questions for you not delivered" repeated that refusal for eleven hours, escalating on it.
// The ask is made again; while the router's word on it is awaited the condition stands unchanged (neither
// cleared nor raised again); once the router took it, the condition clears; a new refusal is said in its own
// words.
func TestARefusalIsAskedAgainAndTheUndeliveredConditionClearsOnceTaken(t *testing.T) {
r := newAskerRig(t)
var raised [][]conditions.Observation
r.a.raise = func(_ context.Context, obs []conditions.Observation) error {
raised = append(raised, obs)
return nil
}
last := func() []conditions.Observation { return raised[len(raised)-1] }
r.a.channels = func(context.Context) string { return "channel/telegram=telegram@anchor[choice]own:true" }
r.open = []conditions.Condition{unitsCondition()}
refuse := func(words string) {
t.Helper()
sent := r.asksSent(t)
refusal, _ := json.Marshal(asks.Warrant{Ask: sent[len(sent)-1].ID, Asker: "mesh-controller",
Outcome: asks.OutcomeRefused, Words: words, At: r.now})
if err := r.a.Decided(context.Background(), refusal); err != nil {
t.Fatal(err)
}
}
_ = r.a.reconcile(context.Background())
refuse("no channel can carry any of its answers now: telegram: telegram has not said whether it can send")
_ = r.a.reconcile(context.Background())
if got := last(); len(got) != 1 || !strings.Contains(got[0].Summary, "telegram has not said") {
t.Fatalf("the refusal, said as %+v", got)
}
// The wait ends: asked again; until the router's word on it is in, the condition stands as it was.
r.now = r.now.Add(askEvery)
_ = r.a.reconcile(context.Background())
if n := len(r.asksSent(t)); n != 2 {
t.Fatalf("not asked again: %d asks", n)
}
if got := last(); len(got) != 1 {
t.Fatalf("cleared while the router's word on the new ask was awaited: the condition would flap")
}
r.now = r.now.Add(askVerdictWait / 2)
_ = r.a.reconcile(context.Background())
if got := last(); len(got) != 1 {
t.Fatalf("cleared inside the verdict wait: the condition would flap")
}
// Refused again, now for a reason of today: said in those words.
refuse("no channel can carry any of its answers now: telegram: no account is linked")
_ = r.a.reconcile(context.Background())
if got := last(); len(got) != 1 || !strings.Contains(got[0].Summary, "no account is linked") ||
strings.Contains(got[0].Summary, "has not said") {
t.Fatalf("an old refusal's words repeated: %+v", got)
}
// Asked again after the doubled wait, and taken: no refusal comes, and the condition clears.
r.now = r.now.Add(2 * askEvery)
_ = r.a.reconcile(context.Background())
if n := len(r.asksSent(t)); n != 3 {
t.Fatalf("not asked again after the doubled wait: %d asks", n)
}
r.now = r.now.Add(askVerdictWait)
_ = r.a.reconcile(context.Background())
if got := last(); len(got) != 0 {
t.Fatalf("taken, and still said undelivered: %+v", got)
}
if n := len(r.asksSent(t)); n != 3 {
t.Fatalf("an ask taken was asked again: %d asks", n)
}
// The operator answers it; the condition stays open (its fix takes a while): the old refusal is not said again.
taken := r.asksSent(t)[2]
r.store.Change(context.Background(), taken.ID, func(x *asked) bool {
x.State, x.Ended = string(asks.OutcomeChosen), r.now
return true
})
for i := 0; i < 5; i++ {
r.now = r.now.Add(askEvery)
_ = r.a.reconcile(context.Background())
if got := last(); len(got) != 0 {
t.Fatalf("an answered ask brought its old refusal back: %+v", got)
}
}
// Its explanation opens with one verdict, not two.
r.open = append(r.open, heldCondition())
_ = r.a.reconcile(context.Background())
refuse("no channel can carry any of its answers now")
_ = r.a.reconcile(context.Background())
if e := conditions.Verdict(last()[0].Needs, last()[0].Explanation); strings.Count(e, "Needs you") != 1 {
t.Errorf("the explanation says its verdict twice: %q", e)
}
}
+339
View File
@@ -0,0 +1,339 @@
package main
// The asker on the bus: its asks in the controller's bucket `asked`, its asks and cancels published on the
// seat under the controller's name, the verbs a warrant chooses called with the controller's grant, and the
// router's record of its asks read under its name (novox/hq ADR 0259).
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// askerFrom is the serving controller's asker; nil in any other process.
var askerFrom *asker
// askWithin is how long a verb a warrant chose is given to answer.
const askWithin = time.Minute
type busAsked struct{ conn *nats.Conn }
func (b busAsked) kv(ctx context.Context) (jetstream.KeyValue, error) {
js, err := jetstream.New(b.conn)
if err != nil {
return nil, err
}
return js.KeyValue(ctx, broker.AskedBucket)
}
func (b busAsked) Get(ctx context.Context, id string) (*asked, error) {
kv, err := b.kv(ctx)
if err != nil {
return nil, err
}
e, err := kv.Get(ctx, id)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
var r asked
return &r, json.Unmarshal(e.Value(), &r)
}
// Create keeps a new ask under its id, and only where none is kept: never over another.
func (b busAsked) Create(ctx context.Context, r asked) error {
kv, err := b.kv(ctx)
if err != nil {
return err
}
body, err := json.Marshal(r)
if err != nil {
return err
}
_, err = kv.Create(ctx, r.ID, body)
return err
}
// Change applies change to the ask kept under id by compare-and-set on its key's revision (the review of
// 2026-10-09, L2): read, changed, and written only over the revision read; when another write came between,
// read again and asked again, at most askChangeTries times. change says whether to write at all.
func (b busAsked) Change(ctx context.Context, id string, change func(*asked) bool) (bool, error) {
kv, err := b.kv(ctx)
if err != nil {
return false, err
}
for try := 0; try < askChangeTries; try++ {
e, err := kv.Get(ctx, id)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return false, nil
}
if err != nil {
return false, err
}
var r asked
if err := json.Unmarshal(e.Value(), &r); err != nil {
return false, err
}
if !change(&r) {
return false, nil
}
body, err := json.Marshal(r)
if err != nil {
return false, err
}
if _, err := kv.Update(ctx, id, body, e.Revision()); err != nil {
var api *jetstream.APIError
if errors.Is(err, jetstream.ErrKeyExists) || (errors.As(err, &api) && api.ErrorCode == jetstream.JSErrCodeStreamWrongLastSequence) {
continue
}
return false, err
}
return true, nil
}
return false, fmt.Errorf("the ask %s changed under every one of %d tries", id, askChangeTries)
}
func (b busAsked) All(ctx context.Context) ([]asked, error) {
kv, err := b.kv(ctx)
if err != nil {
return nil, err
}
lister, err := kv.ListKeys(ctx)
if err != nil {
return nil, err
}
defer func() { _ = lister.Stop() }()
var out []asked
for k := range lister.Keys() {
e, err := kv.Get(ctx, k)
if err != nil {
continue
}
var r asked
if json.Unmarshal(e.Value(), &r) == nil {
out = append(out, r)
}
}
return out, nil
}
// callAction performs an action's verb as the controller, through the grant that names it.
func callAction(conn *nats.Conn) func(ctx context.Context, a conditions.Action, args map[string]string) error {
return func(ctx context.Context, a conditions.Action, args map[string]string) error {
seat, verb, ok := strings.Cut(a.Verb, ".")
if !ok {
return fmt.Errorf("%q names no seat and verb", a.Verb)
}
body := map[string]any{}
for k, v := range args {
body[k] = v
}
if seat == catalogue.DeliverySeat {
_, err := askDeliveryOwner(ctx, conn, verb, body)
return err
}
granted := false
for _, v := range broker.VerbsTheControllerActsOnAWarrant {
granted = granted || (v.Seat == seat && v.Verb == verb)
}
if !granted {
return fmt.Errorf("%s: %w", a.Verb, errNotGranted)
}
var answer link.Answer
var err error
if a.Machine != "" {
answer, err = link.AskSeatTool(ctx, conn, seat, verb, a.Machine, body, askWithin)
} else {
answer, err = link.AskMeshSeatTool(ctx, conn, seat, verb, body, askWithin)
}
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("%s refused: %s", a.Verb, answer.Error)
}
return nil
}
}
// routerRecordOf reads the router's record of one of the controller's asks, under its name, and answers
// how it ended when it did: the bucket is the one the asks seat's declarer names as its records.
func routerRecordOf(conn *nats.Conn, inv *inventory.Inventory) func(ctx context.Context, id string) (*asks.Warrant, error) {
return func(ctx context.Context, id string) (*asks.Warrant, error) {
bucket, err := asksRecords(ctx, inv)
if err != nil || bucket == "" {
return nil, err
}
reply, err := conn.RequestWithContext(ctx, "$JS.API.DIRECT.GET.KV_"+bucket+".$KV."+bucket+"."+askerName+"."+id, nil)
if err != nil {
return nil, err
}
if reply.Header.Get("Status") != "" {
return nil, nil // none, or not readable: the event says it
}
var rec struct {
State string `json:"state"`
Warrant *asks.Warrant `json:"warrant"`
}
if json.Unmarshal(reply.Data, &rec) != nil || rec.State == "open" || rec.Warrant == nil {
return nil, nil
}
return rec.Warrant, nil
}
}
// asksRecords is the bucket the asks seat's declarer keeps its record of asks in.
func asksRecords(ctx context.Context, inv *inventory.Inventory) (string, error) {
declared, err := inv.Catalogue(ctx)
if err != nil {
return "", err
}
for _, m := range declared {
for _, s := range m.DefinesSeats {
if s.Name == broker.AsksSeat && len(s.Records) > 0 {
return broker.BucketName(m.Module, s.Records[0]), nil
}
}
}
return "", nil
}
// routerHereIn says whether a module declaring the asks seat, with its ask named by its caller, is assigned:
// without it nothing takes an ask, and asking would only fill a queue nobody reads.
func routerHereIn(inv *inventory.Inventory) func(ctx context.Context) (bool, error) {
return func(ctx context.Context) (bool, error) {
entries, err := inv.Catalogued(ctx)
if err != nil {
return false, err
}
for _, e := range entries {
for _, s := range e.Manifest.DefinesSeats {
if s.Name == broker.AsksSeat && s.NamedByCaller("ask") && len(e.On) > 0 {
return true, nil
}
}
}
return false, nil
}
}
// grantHeldIn says whether the bus holds the controller's grant to ask (novox/hq issue 353): the user list the
// machine holding the bus was last sent is the one the mesh composes now (brokerBehind, the same judgement a
// push makes to send that machine first). While it is behind, the controller's ask is refused by the bus,
// whatever the record says of the router, so nothing is asked and the operator is told to push that machine.
// Judged at most every grantLookEvery: composing the list resolves the bus's machine whole.
func grantHeldIn(open *stores) func(ctx context.Context) (bool, string, error) {
var mu sync.Mutex
var at time.Time
var held bool
var why string
return func(ctx context.Context) (bool, string, error) {
mu.Lock()
defer mu.Unlock()
if !at.IsZero() && time.Since(at) < grantLookEvery {
return held, why, nil
}
machine, behind, err := brokerBehind(ctx, open, nil)
if err != nil {
return false, "", err
}
at, held, why = time.Now(), !behind, ""
if behind {
why = fmt.Sprintf("the bus's user list on %s is behind what the mesh composes, so the bus has not been "+
"given the controller's grant to ask; `push %s` carries it", machine, machine)
}
return held, why, nil
}
}
// grantLookEvery is how often the bus's user list is judged against the one its machine was last sent.
const grantLookEvery = 30 * time.Second
// channelsIn is what the channels are now, as a fingerprint: each module claiming a kind of the channel
// bench, where, promising what, and whether of its own account. An ask the router refused is asked again
// once this changes.
func channelsIn(inv *inventory.Inventory) func(ctx context.Context) string {
return func(ctx context.Context) string {
entries, err := inv.Catalogued(ctx)
if err != nil {
return ""
}
var parts []string
for _, e := range entries {
for _, c := range e.Manifest.Claims {
if c.Kind == "" || !catalogue.KindedBenches[c.Name] {
continue
}
on := append([]string(nil), e.On...)
sort.Strings(on)
caps := append([]string(nil), c.Capabilities...)
sort.Strings(caps)
parts = append(parts, fmt.Sprintf("%s/%s=%s@%s[%s]own:%t", c.Name, c.Kind, e.Manifest.Module,
strings.Join(on, ","), strings.Join(caps, ","), e.Manifest.RunsAs != ""))
}
}
sort.Strings(parts)
return strings.Join(parts, ";")
}
}
// startAsking makes the serving controller's asker and hands it the router's words.
func startAsking(ctx context.Context, open *stores, server *link.Server, conn *nats.Conn, keeper *conditions.Keeper) {
js, err := jetstream.New(conn)
if err != nil {
fmt.Printf("the operator cannot be asked: %v\n", err)
return
}
a := &asker{
open: keeper.Open,
silence: func(ctx context.Context, key string, d time.Duration, by, why string) error {
_, err := keeper.Silence(ctx, key, d, by, why)
return err
},
store: busAsked{conn: conn},
publish: func(ctx context.Context, subject string, body []byte, id string) error {
_, err := js.Publish(ctx, subject, body, jetstream.WithMsgID(id))
return err
},
call: callAction(conn),
// A proposed settings layer is set by this controller itself on the warrant (novox/hq ADR 0277).
setLayer: setLayerIn(open),
record: func(ctx context.Context, act link.HandAct) error {
_, err := link.RecordHandAct(ctx, conn, act)
return err
},
routerRecord: routerRecordOf(conn, open.inventory),
routerHere: routerHereIn(open.inventory),
grantHeld: grantHeldIn(open),
channels: channelsIn(open.inventory),
raise: func(ctx context.Context, obs []conditions.Observation) error {
return keeper.Reconcile(ctx, sourceAsker, obs)
},
now: time.Now,
logf: func(format string, args ...any) { fmt.Printf(format+"\n", args...) },
}
if err := server.Decides(a); err != nil {
fmt.Printf("the operator's answers cannot be heard, so nothing is asked: %v\n", err)
return
}
askerFrom = a
go a.keep(ctx)
}
File diff suppressed because it is too large Load Diff
+902
View File
@@ -0,0 +1,902 @@
package main
import (
"bytes"
"context"
"encoding/json"
"io"
"os"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The suite's tests of a merge, but for the merge window's own (this file's), plan the merge as it is heard: a
// window of nothing closes at once, so a merge with no walk open is cut into its walk at once, as every merge
// was planned before novox/hq ADR 0276. The window's tests set it through the controller's settings.
func init() { mergeWindowDefault = 0 }
// windowed is a mesh whose controller's settings give a merge window of 90 seconds and at most 10 minutes, as its
// settings file says them, and
// modules built from the catalogue (app, notes) and from three repositories of their own.
func windowed(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
settings := t.TempDir() + "/merge-window.json"
if err := os.WriteFile(settings, []byte(`{"merge-window": "90s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, m := range []struct{ module, repository, path string }{
{"app", "novox/mesh-catalog", "modules/app"},
{"notes", "novox/mesh-catalog", "modules/notes"},
{"one", "novox/one", ""},
{"two", "novox/two", ""},
{"three", "novox/three", ""},
} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1"},
inventory.Source{Repository: m.repository, Seat: "git", Path: m.path, Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
}
return open
}
// t0 is the moment a test's clock starts: merges are dated by it, and heard after it.
var t0 = time.Now().UTC().Truncate(time.Second)
// catalogueMerge is a merge of the catalogue changing one module's directory, made at a moment.
func catalogueMerge(commit, module string, made time.Time) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit,
MergedAt: made.Format(time.RFC3339Nano), Paths: []string{"modules/" + module + "/module.json"},
ModuleDirs: []string{"modules/" + module}, ModuleDirsSaid: true}
}
// repoMerge is a merge of a repository of one module's own.
func repoMerge(repo, commit string, made time.Time) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Commit: commit,
MergedAt: made.Format(time.RFC3339Nano), Paths: []string{"main.go"}}
}
// hear is a merge heard at a moment.
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
if err := (following{open: open}).hearMerge(t.Context(), m, now); err != nil {
t.Fatal(err)
}
}
// cutAt is the cutter looking at a moment.
func cutAt(t *testing.T, open *stores, now time.Time) {
t.Helper()
if err := cutBatchesHeld(t.Context(), open, now); err != nil {
t.Fatal(err)
}
}
// walks is every plan record that is a walk, newest first, and the batches not yet cut.
func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
t.Helper()
recent, err := open.inventory.RecentPlans(t.Context(), 50)
if err != nil {
t.Fatal(err)
}
for _, p := range recent {
if p.Batch() {
batches = append(batches, p)
} else {
walks = append(walks, p)
}
}
return walks, batches
}
// The replay of issue 362: two catalogue merges 18 seconds apart are one batch, cut once into one walk at the
// later commit, which names both merges — the earlier carried by the later.
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, claude, t0.Add(time.Second))
hear(t, open, dunst, t0.Add(18*time.Second))
if ws, bs := walks(t, open); len(ws) != 0 || len(bs) != 1 {
t.Fatalf("within the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
cutAt(t, open, t0.Add(18*time.Second+89*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window closed: %+v", ws)
}
cutAt(t, open, t0.Add(18*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if w.Commit != dunst.Commit || len(w.Commits) != 1 || w.Commits[0].Commit != dunst.Commit {
t.Fatalf("the walk is at %s %+v, not the later commit", w.Commit, w.Commits)
}
for _, m := range []string{"app", "notes"} {
if _, in := w.Modules[m]; !in {
t.Fatalf("the walk does not build %s, which one of its merges moved: %v", m, w.Modules)
}
}
want := []inventory.PlanMerge{{Repository: "novox/mesh-catalog", Commit: claude.Commit, Carried: dunst.Commit},
{Repository: "novox/mesh-catalog", Commit: dunst.Commit}}
if w.Delivery == nil || !slices.EqualFunc(w.Delivery.Merges, want, sameMerge) {
t.Fatalf("the walk answers %+v, want %+v", w.Delivery, want)
}
if len(*asked) != 2 || (*asked)[0][2] != "main" || (*asked)[1][2] != "main" {
t.Fatalf("the walk did not ask its modules at the branch, which holds the commit it carries: %v", *asked)
}
// Heard again, as the bus may hand it over twice: never a second merge, nor a second walk.
hear(t, open, claude, t0.Add(5*time.Minute))
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 {
t.Fatalf("a merge heard twice made %d walk(s) and %d batch(es)", len(ws), len(bs))
}
}
// Merges of three repositories in one window are one walk, one commit for each.
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if len(w.Commits) != 3 || len(w.Delivery.Merges) != 3 {
t.Fatalf("the walk carries %+v and answers %+v", w.Commits, w.Delivery.Merges)
}
for _, r := range []string{"one", "two", "three"} {
if w.CommitOf("novox/"+r) != "c-"+r {
t.Fatalf("the walk carries %s at %q", r, w.CommitOf("novox/"+r))
}
if _, in := w.Modules[r]; !in {
t.Fatalf("the walk does not build %s: %v", r, w.Modules)
}
}
// Each module's ask is recorded at its own repository's commit.
for _, r := range []string{"one", "two", "three"} {
q, found, err := open.inventory.BuildRequestByID(t.Context(), w.Modules[r].Build)
if err != nil || !found || q.Commit != "c-"+r {
t.Fatalf("%s's build was not recorded at its own commit: %+v %v %v", r, q, found, err)
}
}
}
// Each merge restarts the window: a merge at second 80 keeps the batch open until second 170; the window
// closes at most ten minutes after its first merge however busy.
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
hear(t, open, repoMerge("two", "c2", t0.Add(80*time.Second)), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
}
cutAt(t, open, t0.Add(170*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || len(ws[0].Commits) != 2 {
t.Fatalf("the window did not close 90 seconds after its last merge: %+v", ws)
}
busy := windowed(t)
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, repoMerge(repo, "c"+string(rune('a'+i)), last), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
}
break
}
}
ws, _ := walks(t, busy)
if len(ws) != 1 {
t.Fatalf("ten minutes of merges a minute apart were never cut: %d walk(s)", len(ws))
}
}
// `plans` lists the batch being assembled first, in the operator's words: how long is left, when at the latest,
// what is grouped, and that the plan is not yet calculated. It says so on the bus as plan-moved, too.
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
inventory.PlanSaved = func(p inventory.Plan) { said = append(said, p) }
t.Cleanup(func() { inventory.PlanSaved = was })
now := time.Now().UTC()
claude := catalogueMerge("553b7191claude", "app", now.Add(-20*time.Second))
claude.Number, claude.Title = 174, "claude-code: an agent proposes a section"
dunst := catalogueMerge("48bda475dunst", "notes", now.Add(-2*time.Second))
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, repoMerge("one", "a6bc0931one", now), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
for _, want := range []string{"assembling: ", " s left (at the latest ", "grouped: novox/mesh-catalog@48bda475 " +
"(answers 553b7191), novox/one@a6bc0931; plan not yet calculated"} {
if !strings.Contains(first, want) {
t.Fatalf("plans' first line %q does not say %q", first, want)
}
}
// Under it, one line per repository: the pull request, what it answers, what it moves.
if len(lines) < 3 || strings.TrimSpace(lines[1]) != "mesh-catalog #175 dunst: the font the operator chose (answers "+
"#174 claude-code: an agent proposes a section) · app, notes" ||
strings.TrimSpace(lines[2]) != "one a6bc0931 · one" {
t.Fatalf("the grouped list reads %q", lines[1:4])
}
if len(said) == 0 || said[len(said)-1].State != inventory.PlanAssembling || said[len(said)-1].Delivery.Batch == nil ||
len(said[len(said)-1].Delivery.Merges) != 3 {
t.Fatalf("the batch was not said as assembling with its merges: %+v", said)
}
}
// captured is what a command printed.
func captured(t *testing.T, run func() error) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
was := os.Stdout
os.Stdout = w
runErr := run()
os.Stdout = was
_ = w.Close()
var b bytes.Buffer
_, _ = io.Copy(&b, r)
if runErr != nil {
t.Fatal(runErr)
}
return b.String()
}
// One walk at a time: a merge heard while a walk runs joins the next batch, which is cut when the walk ends;
// the walk that started is never superseded.
func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first batch was not cut: %+v", ws)
}
first := ws[0]
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(5*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID {
t.Fatalf("the merge during the walk: %d walk(s), batches %+v", len(ws), bs)
}
if got, _ := open.inventory.PlanByID(t.Context(), first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the next merge", got.State)
}
if !strings.HasPrefix(batchWords(bs[0], t0.Add(5*time.Minute)), "queued behind "+first.ID) {
t.Fatalf("a queued batch reads %q", batchWords(bs[0], t0.Add(5*time.Minute)))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(6*time.Minute))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut when the walk ended: %+v %+v", ws, bs)
}
}
// A walk waiting for its delivery's word is folded into the next batch's walk, which answers its merges; it
// names the walk that took it over.
func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
waiting := ws[0]
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(2*time.Minute+90*time.Second))
ws, bs := walks(t, open)
if len(bs) != 0 || len(ws) != 2 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
folded, err := open.inventory.PlanByID(ctx, waiting.ID)
if err != nil {
t.Fatal(err)
}
newer := ws[0]
if folded.State != inventory.PlanSuperseded || folded.Delivery.TakenOverBy != newer.ID {
t.Fatalf("the waiting walk is %s, taken over by %q", folded.State, folded.Delivery.TakenOverBy)
}
if len(newer.Delivery.Merges) != 2 || newer.CommitOf("novox/one") != "c1" || newer.CommitOf("novox/two") != "c2" {
t.Fatalf("the newer walk answers %+v", newer.Delivery.Merges)
}
if _, in := newer.Modules["one"]; !in {
t.Fatalf("what the folded walk was to build is not built: %v", newer.Modules)
}
}
// A merge heard after a later merge of its branch was walked is answered by that walk when it builds all it
// moves — named at once on a walk done — and joins the next batch when it does not.
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, later, t0.Add(18*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
// Moves only notes, which the done walk built from the branch after it: answered there, at once.
hear(t, open, catalogueMerge("553b7191early", "notes", t0), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if got.State != inventory.PlanDone || len(got.Delivery.Merges) != 2 ||
!sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: "553b7191early",
Carried: later.Commit}) {
t.Fatalf("the late merge is not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// Moves app, which that walk never built: the next batch walks it.
hear(t, open, catalogueMerge("1111aaaaapp", "app", t0.Add(time.Second)), t0.Add(16*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "1111aaaaapp" {
t.Fatalf("a late merge moving what the walk never built did not join the next batch: %+v", bs)
}
}
// A failed walk marks nothing delivered: its earlier merges are walked alone, on their own commit, the newest
// first; the first delivered answers the older ones, and the search stops.
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
oldest := catalogueMerge("aaaa0001", "app", t0.Add(-3*time.Minute))
middle := catalogueMerge("bbbb0002", "app", t0.Add(-3*time.Minute+10*time.Second))
newest := catalogueMerge("cccc0003", "notes", t0.Add(-3*time.Minute+20*time.Second))
for i, m := range []link.SourceMoved{oldest, middle, newest} {
hear(t, open, m, t0.Add(time.Duration(i)*10*time.Second+time.Second))
}
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
failed := ws[0]
failed.State, failed.Note = inventory.PlanFailed, "notes failed to build in tier 0"
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, failed.ID)
if got.State != inventory.PlanFailed || len(got.Delivery.Merges) != 3 {
t.Fatalf("the failed walk's record changed: %s %+v", got.State, got.Delivery.Merges)
}
ws, _ = walks(t, open)
alone := ws[0]
if alone.ID == failed.ID || alone.Commit != middle.Commit || len(alone.Delivery.Merges) != 1 ||
!sameMerge(alone.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: middle.Commit}) {
t.Fatalf("the newest earlier merge was not walked alone on its own commit: %+v", alone)
}
if _, in := alone.Modules["app"]; !in || (*asked)[len(*asked)-1] != [3]string{"novox/mesh-catalog", "modules/app",
middle.Commit} {
t.Fatalf("the merge walked alone does not build app at its own commit: %v, asked %v", alone.Modules, *asked)
}
// Retried, the failed walk would name merges the search answers now.
if _, err := retryPlan(ctx, open, failed.ID); err == nil || !strings.Contains(err.Error(), "walked alone") {
t.Fatalf("a failed walk whose merges are searched was retried: %v", err)
}
alone.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &alone); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(4*time.Minute))
ws, bs := walks(t, open)
if ws[0].ID != alone.ID || len(bs) != 0 {
t.Fatalf("the search went on after a merge was delivered: %+v", ws[0])
}
done, _ := open.inventory.PlanByID(ctx, alone.ID)
if len(done.Delivery.Merges) != 2 || !sameMerge(done.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog",
Commit: oldest.Commit, Carried: middle.Commit}) {
t.Fatalf("the oldest merge is not answered by the walk that delivered the one after it: %+v", done.Delivery.Merges)
}
// A stopped walk starts no search: a person ended it.
hear(t, open, catalogueMerge("dddd0004", "app", t0.Add(5*time.Minute)), t0.Add(5*time.Minute))
hear(t, open, catalogueMerge("eeee0005", "notes", t0.Add(5*time.Minute+time.Second)), t0.Add(5*time.Minute+time.Second))
cutAt(t, open, t0.Add(8*time.Minute))
ws, _ = walks(t, open)
if _, err := stopWalk(ctx, open.inventory, ws[0].ID, "mesh-delivery for jochen", "not now"); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(9*time.Minute))
if again, _ := walks(t, open); again[0].ID != ws[0].ID {
t.Fatalf("a stopped walk's earlier merge was walked alone: %+v", again[0])
}
}
// A delivery group's order holds inside the walk (ADR 0249 unchanged): merges of two repositories from one head
// branch name, the node-engine's before the controller's, are tiered so; without the group, one tier.
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: "1"},
inventory.Source{Repository: "novox/" + m, Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
host := repoMerge("mesh-host", "h1", t0)
ctl := repoMerge("mesh-controller", "k1", t0.Add(time.Second))
host.Head, ctl.Head = "feat/together", "feat/together"
if !grouped {
ctl.Head = "feat/alone"
}
hear(t, open, ctl, t0.Add(2*time.Second))
hear(t, open, host, t0.Add(3*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walks", len(ws))
}
tiers := ws[0].Tiers
if grouped {
if len(tiers) != 2 || !slices.Equal(tiers[0], []string{"mesh-host"}) ||
!slices.Equal(tiers[1], []string{"mesh-controller"}) {
t.Fatalf("the group's order is not the walk's tiers: %v", tiers)
}
} else if len(tiers) != 1 {
t.Fatalf("two merges of no group were ordered: %v", tiers)
}
}
}
// A restarted controller resumes the window where it stood: the batch, its merges and their times are in the
// store, a merge handed over again is not doubled, and the batch is cut when its window closes.
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
hear(t, open, repoMerge("two", "c2", t0.Add(30*time.Second)), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, repoMerge("one", "c1", t0), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
}
cutAt(t, again, t0.Add(120*time.Second))
ws, bs := walks(t, again)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller did not cut the batch at its window: %+v %+v", ws, bs)
}
}
// The window's settings reach the controller in its settings file, read at every look; one that says no
// duration, or no file, is the default.
func TestTheWindowIsTheControllersSetting(t *testing.T) {
file := t.TempDir() + "/merge-window.json"
t.Setenv(mergeWindowFileVar, file)
for _, c := range []struct {
content string
window, atMost time.Duration
}{
{`{"merge-window": "60s", "merge-window-at-most": "5m"}`, time.Minute, 5 * time.Minute},
{`{"merge-window": 30, "merge-window-at-most": ""}`, 30 * time.Second, mergeWindowAtMostDefault},
{`{"merge-window": "soon"}`, mergeWindowDefault, mergeWindowAtMostDefault},
} {
if err := os.WriteFile(file, []byte(c.content), 0o600); err != nil {
t.Fatal(err)
}
if w, m := mergeWindowOf(); w != c.window || m != c.atMost {
t.Errorf("%s reads as %s and %s, want %s and %s", c.content, w, m, c.window, c.atMost)
}
}
t.Setenv(mergeWindowFileVar, file+".gone")
if w, m := mergeWindowOf(); w != mergeWindowDefault || m != mergeWindowAtMostDefault {
t.Errorf("no file reads as %s and %s", w, m)
}
}
// S16 names every merge a waiting walk answers, never one commit a supersession may have ended (issue 362).
func TestAWaitingWalkNamesTheMergesItAnswers(t *testing.T) {
f := calm(t0)
f.waits = []waitFacts{{id: "plan-2", repository: "novox/mesh-catalog", commit: "48bda475dunst", awaits: "mesh-delivery",
since: t0.Add(-31 * time.Minute), merges: []inventory.PlanMerge{
{Repository: "novox/mesh-catalog", Commit: "553b7191claude", Carried: "48bda475dunst"},
{Repository: "novox/mesh-catalog", Commit: "48bda475dunst"}}}}
got := watchWaits(f)
if len(got) != 1 || !strings.Contains(got[0].Summary, "novox/mesh-catalog@553b7191") ||
!strings.Contains(got[0].Summary, "novox/mesh-catalog@48bda475") {
t.Fatalf("the waiting walk does not name both merges: %+v", got)
}
}
// A merge heard after a later merge of its branch was cut is named by that walk however its modules read since
// the cut, in a repository of one module (review of this change); a walk that never built what it moves names
// nothing, and the merge joins the next batch.
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
hear(t, open, c2, t0)
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0.Add(-60*time.Second)), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the late merge was not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// A walk of two that built only two: a late merge of one is not its to name.
hear(t, open, repoMerge("two", "d2", t0.Add(16*time.Minute)), t0.Add(16*time.Minute))
cutAt(t, open, t0.Add(18*time.Minute))
ws, _ = walks(t, open)
w = ws[0]
delete(w.Modules, "two")
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("two", "d1", t0.Add(15*time.Minute)), t0.Add(20*time.Minute))
if got, _ := open.inventory.PlanByID(ctx, w.ID); len(got.Delivery.Merges) != 1 {
t.Fatalf("a walk that never built what the late merge moves named it: %+v", got.Delivery.Merges)
}
if _, bs := walks(t, open); len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "d1" {
t.Fatalf("the late merge did not join the next batch: %+v", bs)
}
}
// One walk at a time holds against the delivery's word too: a waiting walk let go beside a started one takes the
// word and starts once that one ended, asking nothing before.
func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(2*time.Minute))
started, _ := walks(t, open)
waiting := inventory.Plan{ID: "plan-waiting", Repository: "novox/two", Branch: "main", Commit: "d1", Created: t0,
State: inventory.PlanBuilding, Tiers: [][]string{{"two"}}, Modules: map[string]*inventory.PlanModule{"two": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if _, err := letGo(ctx, open.inventory, waiting.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatalf("the word was refused beside %s: %v", started[0].ID, err)
}
before := len(*asked)
advanceHeld(ctx, open)
got, _ := open.inventory.PlanByID(ctx, waiting.ID)
if len(*asked) != before || !strings.Contains(got.Note, "starts once "+started[0].ID) {
t.Fatalf("a walk let go beside a started one asked (%d → %d) or does not say it waits: %q", before, len(*asked), got.Note)
}
// Read as a wait, an hour on: its note on the line, never LATE, and no tier of it late for S3.
later := time.Now().Add(time.Hour)
if line := planLine(got, later); !strings.Contains(line, "starts once "+started[0].ID) || strings.Contains(line, "LATE") {
t.Fatalf("a deferred walk reads %q", line)
}
facts, _, err := gatherPlans(ctx, open.inventory, later, nil)
if err != nil {
t.Fatal(err)
}
for _, f := range facts {
if f.id == got.ID {
t.Fatalf("a deferred walk is watched as a tier running late: %+v", f)
}
}
done := started[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) != before+1 {
t.Fatalf("the walk did not start once the started one ended: asked %v", *asked)
}
}
// A file a merge of the batch removed and a later one brought back is not removed; one removed last is.
func TestARemovedFileIsTheLastMergesWord(t *testing.T) {
ev := func(commit string, paths, removed []string) inventory.BatchedMerge {
m := link.SourceMoved{Owner: "novox", Repo: "one", Base: "main", Commit: commit, Paths: paths, Removed: removed}
b, _ := json.Marshal(m)
return inventory.BatchedMerge{Repository: "novox/one", Branch: "main", Commit: commit, Event: b,
Merged: t0.Add(time.Duration(len(commit)) * time.Second)}
}
got := combinedMerges([]inventory.BatchedMerge{
ev("a", []string{"x/module.json", "y/module.json"}, []string{"x/module.json", "y/module.json"}),
ev("bb", []string{"x/module.json"}, nil),
ev("ccc", []string{"z.go"}, nil)})
if len(got) != 1 || got[0].Commit != "ccc" || !slices.Equal(got[0].Removed, []string{"y/module.json"}) ||
len(got[0].Paths) != 3 {
t.Fatalf("combined as %+v", got)
}
}
// The catch-up hands over what the bus lost after its branch's later merges were cut, and the record keeps it:
// an earlier merge is named by the walk that carried it; a merge made before a cut and heard after it, which
// no later merge carries, joins the next batch — neither reads as history and is dropped (review of this change).
func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
base := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
hear(t, open, repoMerge("one", "c2", base.Add(10*time.Second)), base.Add(11*time.Second))
cutAt(t, open, base.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
lost := []link.AnnouncedMerge{
{SourceMoved: repoMerge("one", "c1", base), At: base.Add(time.Second)},
{SourceMoved: repoMerge("one", "c3lost", base.Add(20*time.Second)), At: base.Add(21 * time.Second)},
}
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return nil, nil, err
}
read, err := readForPlanning(ctx, open.inventory)
return entries, read, err
}
if err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announcedList(lost), open.inventory}, catalogued,
(following{open}).SourceMoved, t.Logf); err != nil {
t.Fatal(err)
}
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the earlier merge the catch-up handed over is not named by the walk that carried it: %+v",
got.Delivery.Merges)
}
if _, kept, err := open.inventory.MergeOf(ctx, "novox/one", "c3lost"); err != nil || !kept {
t.Fatalf("the merge made before the cut and heard after it was dropped: %v %v", kept, err)
}
}
// A merge on the controller's own path never shares a batch with one that waits for mesh-delivery's word
// (decided during the build, 2026-10-10): the two are batches of their own, cut one after the other; a walk on
// the own path folds no waiting catalogue walk but batches its merges again behind it; the catalogue's walk
// still waits for the word, so no catalogue delivery skips its turn.
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("aaaa0001", "app", t0), t0)
hear(t, open, repoMerge("mesh-controller", "k1", t0.Add(10*time.Second)), t0.Add(10*time.Second))
hear(t, open, catalogueMerge("bbbb0002", "notes", t0.Add(20*time.Second)), t0.Add(20*time.Second))
_, bs := walks(t, open)
var ownBatch, catalogueBatch inventory.Plan
for _, b := range bs {
if b.OwnPath() {
ownBatch = b
} else {
catalogueBatch = b
}
}
if len(bs) != 2 || ownBatch.ID == "" || catalogueBatch.ID == "" || len(catalogueBatch.Delivery.Merges) != 2 ||
len(ownBatch.Delivery.Merges) != 1 {
t.Fatalf("a controller merge and two catalogue merges in one window: %+v", bs)
}
if !strings.Contains(batchWords(ownBatch, t0.Add(30*time.Second)), "own path") {
t.Fatalf("the own-path batch does not say so: %q", batchWords(ownBatch, t0.Add(30*time.Second)))
}
// Both windows closed, the controller's batch is cut first and starts; the catalogue batch queues behind it,
// is cut when the controller's walk ended, and waits for its word with both merges.
cutAt(t, open, t0.Add(2*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the controller's batch was not cut first into a started walk: %+v", ws)
}
ownWalk := ws[0]
for _, m := range []string{"app", "notes"} {
if _, in := ownWalk.Modules[m]; in {
t.Fatalf("the controller's walk builds %s, a catalogue module: it skipped its turn", m)
}
}
if len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != ownWalk.ID || bs[0].OwnPath() {
t.Fatalf("the catalogue batch does not queue behind the controller's walk: %+v", bs)
}
ownWalk.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &ownWalk); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
ws, bs = walks(t, open)
if len(bs) != 0 || ws[0].ID != catalogueBatch.ID || !ws[0].Waiting() || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the catalogue batch was not cut into a waiting walk once the controller's ended: %+v %+v", ws, bs)
}
for _, m := range []string{"app", "notes"} {
if _, in := ws[0].Modules[m]; !in {
t.Fatalf("the catalogue walk does not build %s: %v", m, ws[0].Modules)
}
}
for _, a := range *asked {
if a[1] == "modules/app" || a[1] == "modules/notes" {
t.Fatalf("a catalogue module was asked without the word: %v", *asked)
}
}
// A catalogue walk waiting for its word when an own-path batch is cut keeps waiting beside the own-path
// walk, is not folded into it, and its word is taken meanwhile: it starts once the own-path walk ended.
catalogueWalk := ws[0]
hear(t, open, repoMerge("mesh-controller", "k2", t0.Add(4*time.Minute)), t0.Add(4*time.Minute))
cutAt(t, open, t0.Add(6*time.Minute))
ws, _ = walks(t, open)
kept, _ := open.inventory.PlanByID(ctx, catalogueWalk.ID)
if !kept.Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k2" || ws[0].Waiting() {
t.Fatalf("the waiting catalogue walk was not kept waiting beside the controller's walk: %s %+v", kept.State, ws)
}
if _, in := ws[0].Modules["app"]; in {
t.Fatalf("the controller's walk folded the waiting catalogue walk in: %v", ws[0].Modules)
}
if _, err := letGo(ctx, open.inventory, catalogueWalk.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatal(err)
}
before := len(*asked)
advanceHeld(ctx, open)
if len(*asked) != before {
t.Fatalf("the catalogue walk started beside the controller's: asked %v", (*asked)[before:])
}
own2 := ws[0]
own2.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &own2); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) < before+1 || (*asked)[before][1] != "modules/app" {
t.Fatalf("the catalogue walk did not start once the controller's ended: %v", (*asked)[before:])
}
}
// `plans` names a walk by what it moves (asked by the operator, 2026-10-10): the repository's pull request and
// title, the modules it moves and the machines running them, then the state words; a record naming no pull
// request is named by its commit, and a title is cut at fifty runes.
func TestAPlanLineNamesWhatItMoves(t *testing.T) {
now := time.Date(2026, 10, 10, 12, 0, 0, 0, time.UTC)
p := inventory.Plan{ID: "plan-1", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c1c1c1c1c1",
State: inventory.PlanBuilding, Tiers: [][]string{{"messenger", "telegram"}}, TierEntered: now.Add(-2 * time.Minute),
Modules: map[string]*inventory.PlanModule{"messenger": {State: "asked"}, "telegram": {State: "asked"}},
Commits: []inventory.PlanCommit{{Repository: "novox/mesh-catalog", Branch: "main", Commit: "c1c1c1c1c1"}},
Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{{Repository: "novox/mesh-catalog",
Commit: "c1c1c1c1c1", Number: 175, Title: "a tap shows its outcome", Moves: []string{"telegram", "messenger"}}}}}
running := func(module string) []string {
if module == "messenger" || module == "telegram" {
return []string{"novox"}
}
return nil
}
if got, want := planLineOn(p, now, pauseView{}, tierAtLeast, running),
"mesh-catalog #175 a tap shows its outcome · messenger, telegram → novox · tier 1 of 1, building for 2m0s"; got != want {
t.Fatalf("the line reads %q, want %q", got, want)
}
if got := planLine(p, now); !strings.HasPrefix(got, "mesh-catalog #175 a tap shows its outcome · messenger, telegram · tier") {
t.Fatalf("without the machines the line reads %q", got)
}
old := p
old.Commits, old.Delivery = nil, nil
if got := planLine(old, now); !strings.HasPrefix(got, "mesh-catalog c1c1c1c1 · tier 1 of 1") {
t.Fatalf("a record naming no pull request reads %q", got)
}
long := p
long.Delivery.Merges[0].Title = strings.Repeat("abcdefghij", 6)
if got := planHeadline(long, nil); !strings.Contains(got, "#175 "+strings.Repeat("abcdefghij", 4)+"abcdefghi…") {
t.Fatalf("a long title is not cut at fifty runes: %q", got)
}
}
// sameMerge compares what a walk names of a merge: its repository, commit and the commit carrying it.
func sameMerge(a, b inventory.PlanMerge) bool {
return a.Repository == b.Repository && a.Commit == b.Commit && a.Carried == b.Carried
}
// A catalogue merge heard after a later merge of its branch was walked without the word (a merge that touched
// the bus too, on the controller's own path) is not answered by that walk while the delivery seat has a holder:
// its delivery would skip its turn. It joins the next catalogue batch.
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
module string
claims []catalogue.Claim
}{
{"nats", nil},
{"mesh-delivery", []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1", Claims: m.claims},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/" + m.module, Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
mixed := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "m1xed", MergedAt: t0.Format(time.RFC3339Nano),
Paths: []string{"modules/nats/module.json", "modules/app/module.json"}, ModuleDirs: []string{"modules/nats", "modules/app"},
ModuleDirsSaid: true}
hear(t, open, mixed, t0.Add(time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() {
t.Fatalf("the mixed merge's walk waited, or was not cut: %+v", ws)
}
done := ws[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("ear1ier", "app", t0.Add(-30*time.Second)), t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, done.ID)
_, bs := walks(t, open)
if len(got.Delivery.Merges) != 1 || len(bs) != 1 || bs[0].OwnPath() || bs[0].Delivery.Merges[0].Commit != "ear1ier" {
t.Fatalf("the late catalogue merge was answered by the own-path walk (%+v) rather than the next catalogue batch (%+v)",
got.Delivery.Merges, bs)
}
}
+103
View File
@@ -0,0 +1,103 @@
package main
import (
"context"
"fmt"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// **Every wait of a batch has a bound and a condition** (novox/hq ADR 0276 decision 9): a batch still
// assembling a minute past its maximum while no walk is open is a cut the controller failed to make (S18),
// and a batch waiting behind an open walk longer than that walk's bound waits on a walk gone wrong (S19).
// The kinds S18 and S19 raise.
const (
kindBatchNotCut = "batch-not-cut"
kindBatchBehindWalk = "batch-behind-walk"
)
// batchFacts is one batch not yet cut, as the watchdogs read it.
type batchFacts struct {
id, state, grouped string
// atMost is when its window closes at the latest; closed when it closed.
atMost, closed time.Time
// behind is the open walk it waits behind, and walkBound that walk's bound: a tier's bound for each of its
// tiers.
behind string
walkBound time.Duration
}
// gatherBatches is every batch not yet cut, with the bound of the walk it waits behind.
func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time) ([]batchFacts, error) {
batches, err := inv.Batches(ctx)
if err != nil || len(batches) == 0 {
return nil, err
}
bounds, err := measuredTierBounds(ctx, inv, now)
if err != nil {
return nil, err
}
var out []batchFacts
for _, b := range batches {
f := batchFacts{id: b.ID, state: b.State, grouped: groupedWords(b)}
if w := b.Delivery; w != nil && w.Batch != nil {
f.atMost, f.closed, f.behind = w.Batch.AtMost, w.Batch.ClosesAt, w.Batch.Behind
if f.atMost.Before(f.closed) {
f.closed = f.atMost
}
}
if f.behind != "" {
if walk, err := inv.PlanByID(ctx, f.behind); err == nil {
f.walkBound = bounds.of(walk.Repository) * time.Duration(max(1, len(walk.Tiers)))
}
}
out = append(out, f)
}
return out, nil
}
// watchBatchesNotCut is S18: a batch still assembling a minute past its maximum, while no walk is open.
func watchBatchesNotCut(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, b := range f.batches {
if b.state != inventory.PlanAssembling || b.atMost.IsZero() || f.now.Sub(b.atMost) <= batchLateAfter {
continue
}
late := f.now.Sub(b.atMost)
out = append(out, conditions.Observation{Scope: conditions.ScopePlan, ID: b.id, Kind: kindBatchNotCut,
Severity: conditions.Warning,
Summary: fmt.Sprintf("the batch %s is still assembling %s past its latest close (%s), with no walk open: "+
"the controller did not cut it; grouped: %s", b.id, ago(late), b.atMost.UTC().Format(time.RFC3339), b.grouped),
Said: fmt.Sprintf("assembling since %s past its maximum", ago(late)),
Headline: "Merged changes are not being delivered",
Explanation: fmt.Sprintf("Merges collected for one delivery should have been planned %s ago and were not. "+
"Nothing is lost; the mesh keeps them until it plans them.", humanDuration(late)),
Resolved: "Merged changes are being delivered again"})
}
return out
}
// watchBatchesBehind is S19: a batch waiting behind an open walk longer than that walk's bound, naming it.
func watchBatchesBehind(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, b := range f.batches {
if b.state != inventory.PlanQueued || b.behind == "" || b.walkBound <= 0 || f.now.Sub(b.closed) <= b.walkBound {
continue
}
in := f.now.Sub(b.closed)
out = append(out, conditions.Observation{Scope: conditions.ScopePlan, ID: b.id, Kind: kindBatchBehindWalk,
Severity: conditions.Warning,
Summary: fmt.Sprintf("the batch %s has waited %s behind the walk %s, longer than that walk's bound (%s); "+
"grouped: %s; `plans %s` says where that walk stands", b.id, ago(in), b.behind, ago(b.walkBound),
b.grouped, b.behind),
Said: fmt.Sprintf("queued behind %s for %s", b.behind, ago(in)),
Headline: "Merged changes wait behind a slow delivery",
Explanation: fmt.Sprintf("Merges collected for the next delivery have waited %s for the delivery before "+
"them, which is taking longer than it should. Nothing is lost.", humanDuration(in)),
Resolved: "Merged changes no longer wait behind a slow delivery"})
}
return out
}
+38 -6
View File
@@ -166,6 +166,10 @@ func buildFrom(result link.BuildResult) inventory.Build {
for _, r := range result.Read {
kept.Read = append(kept.Read, inventory.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
// What it was made from, as files (novox/hq ADR 0267): the planner maps the next merge onto it.
for _, s := range result.Sources {
kept.Sources = append(kept.Sources, inventory.BuildSource{Repository: s.Repository, Ref: s.Ref, Paths: s.Paths})
}
var announced []inventory.Artifact
for _, made := range result.Made {
announced = append(announced, inventory.Artifact{
@@ -451,6 +455,10 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
if err != nil {
return "", err
}
// Through a verb, only a repository the catalogue builds from (novox/hq ADR 0266).
if err := verbMayAsk(ctx, source, repository); err != nil {
return "", err
}
ident, err := openIdentity(ctx)
if err != nil {
@@ -500,8 +508,10 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
// never before, so an ask that failed never reads as a build in flight. A dry run registers nothing, and
// is not kept.
keep := !dryRun && asker != ""
// Asked at the terminal is what lets its outcome register a module from a repository the catalogue does
// not build it from (novox/hq ADR 0266); never through a verb.
asked := inventory.BuildRequest{ID: request.ID, Repository: source.Repository, Seat: source.Seat, Path: path,
Ref: ref, For: asker}
Ref: ref, For: asker, AtTerminal: startedAtTheTerminal()}
if wait == 0 {
// Asked and not waited for (novox/hq issue 176): the outcome is the role's event, and the
@@ -631,6 +641,21 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return manifest, kept, fmt.Errorf("%s built %s (%s), and the mesh does not register it: %w",
result.On, result.Repository, short(result.Commit), err)
}
// **Only from the repository the catalogue builds the module from** (novox/hq ADR 0266): else the trunk
// below is the trunk of whatever repository was built, which may be one an agent made — and a module named
// `sudo` from it would be what the next push sends. Another repository is the operator's, at the terminal.
trunk := result.Trunk
if was, err := inv.SourceOf(ctx, manifest.Module); err == nil && followedBranch(was.Ref) != "" {
trunk = followedBranch(was.Ref)
}
if trunk == "" {
trunk = "main"
}
repoID, err := mayRegisterFrom(ctx, inv, manifest.Module, recorded, result.ID, result.Path, trunk)
if err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
manifest.Module, short(result.Commit), err)
}
// **Only a commit on the trunk is published** (novox/hq ADR 0238): a commit off its repository's
// default branch — a pull request's head, a feature branch built by hand, a `rebuild` or `replay
// --register` of one — is for checking, and is never a module's version; nothing could then send it.
@@ -678,6 +703,10 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
}
return manifest, kept, err
}
// Which repository it is registered from, by the forge's own id (novox/hq ADR 0266).
if err := inv.SetSourceIdentity(ctx, manifest.Module, repoID); err != nil {
return manifest, kept, err
}
// The keep set just moved, and new bytes just landed (novox/hq ADR 0189). Asked here rather
// than on a timer of its own: this is the only moment either is true. Never fatal — the build
// worked and the module is registered.
@@ -719,6 +748,9 @@ func buildAndShow(ctx context.Context, source buildSource, path, ref string, wai
if err != nil {
return err
}
if err := verbMayAsk(ctx, source, repository); err != nil {
return err
}
ident, err := openIdentity(ctx)
if err != nil {
return err
@@ -879,6 +911,10 @@ func heldBy(ctx context.Context) map[string]string {
// the mesh runs on today this needs the controller's own connection, so it is handed one; on the bus
// being built it dials, because a build request is a one-shot and holds nothing else.
func askOverOn(seat string) (link.Builders, error) {
// The serving controller asks on its own connection (novox/hq issue 327).
if serving := servingBus.Load(); serving != nil {
return link.BuildsOn(serving, seat), nil
}
address, err := broker.BusAddress()
if err != nil {
return nil, err
@@ -937,11 +973,7 @@ func buildSeatAmong(entries []inventory.Entry) string {
// with a consumer of its own that is gone when this returns, so nothing accumulates in the server
// for the reading, and filtered by subject, so one build's lines are all that travel.
func buildLog(ctx context.Context, id string) error {
address, err := broker.BusAddress()
if err != nil {
return err
}
js, err := broker.Dial(address)
js, err := aBus()
if err != nil {
return fmt.Errorf("cannot reach the bus to read a build's log: %w", err)
}
+458
View File
@@ -0,0 +1,458 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"regexp"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Where a build may register a module from (novox/hq ADR 0266).
//
// A build's outcome registers its module, and a registered module is what the next push sends. Before this,
// an outcome registered whatever its manifest named, from whichever repository it was built from: an agent
// that made a repository of its own, committed `modules/sudo/module.json` granting itself a rule without a
// password, and asked the `build` verb for it had its repository registered as the module `sudo` — whose next
// push made the agent root on every node. A fork of the node-engine did the same everywhere. The trunk rule
// (ADR 0238) did not stop it: the trunk it checked was the trunk of the repository built, which was the
// agent's own.
//
// So **an outcome registers a module only from the repository the catalogue already builds that module
// from**; and a module new to the catalogue only from a repository the catalogue already builds another
// module from — whose trunk takes a reviewed, approved merge, which is how a merge adds a module (novox/hq
// issue 300). Anything else — a module moved to another repository, a module from a repository the
// catalogue has never built — is the operator's, at the controller's terminal: allowed only when the build
// request was kept as asked there. Judged at the take-in, which every outcome reaches whoever hears it and
// whichever verb asked it (`build`, `rebuild`, `replay --register`, `assign` with build), and before the ask
// for a call through a verb, so an agent cannot have a build node run a repository the catalogue does not
// build from at all.
// errNotItsSource is an outcome refused for where it was built from.
var errNotItsSource = errors.New("not built from the repository the catalogue builds it from")
// servedVar marks every process the serving controller starts — each verb's command, each child — and the serving
// process itself, so none of them can read as the operator at the terminal (novox/hq ADR 0266). Set by serve
// before it answers anything, inherited by every child through os.Environ.
const servedVar = "MESH_SERVED_BY_THE_CONTROLLER"
// startedAtTheTerminal says this process was started at the controller's terminal: not the serving controller,
// not anything it started, not a verb's command, not a seat call's. The serving controller marks its own
// environment (servedVar), so a build asked in it, or by any process it starts, never reads as the terminal's;
// runVerb also names the verb and the caller.
//
// **And a line mesh-cli asked as the controller's terminal** (novox/hq ADR 0272 §4): the serving controller runs it
// without the served mark and without a verb, names its caller, and marks it with cliTerminalVar — a mark only the
// mesh-cli path sets and every other command line the controller runs is stripped of (commandEnvironment).
func startedAtTheTerminal() bool {
if os.Getenv(servedVar) != "" || os.Getenv(verbVar) != "" {
return false
}
return os.Getenv(link.CallerVar) == "" || os.Getenv(cliTerminalVar) != ""
}
// cliTerminalVar marks a command line the serving controller runs as the controller's terminal for mesh-cli.
const cliTerminalVar = "MESH_CLI_TERMINAL"
// markServed marks this process, and so everything it starts, as the serving controller's.
func markServed() {
if err := os.Setenv(servedVar, "1"); err != nil {
panic("the serving controller could not mark its environment: " + err.Error())
}
}
// sourceForms compares sources however each is spelled: a path on a seat's holder (ADR 0111) or a URL — a
// build asked of a seat's path is registered with the URL composed from it when its outcome does not echo
// the seat. A seat's path is composed into its URL where the seat's holder is known, so both spellings of
// one repository are one; where it is not, a seat's path matches only the same seat's same path.
type sourceForms struct {
bases map[string]string // the seat's clone base, `scheme://host:port`, by seat; "" where it is not known
base func(seat string) string
}
// newSourceForms reads the seats' bases from the mesh once, when first needed.
func newSourceForms(ctx context.Context, inv *inventory.Inventory) *sourceForms {
f := &sourceForms{bases: map[string]string{}}
var world *catalogue.World
f.base = func(seat string) string {
if b, known := f.bases[seat]; known {
return b
}
if world == nil {
w := catalogue.World{}
if shelf, err := inv.Catalogue(ctx); err == nil {
if read, err := theRestOfTheMesh(ctx, inv, shelf, ""); err == nil {
w = read
}
}
world = &w
}
b, err := seatBase(*world, seat)
if err != nil {
b = ""
}
f.bases[seat] = b
return b
}
return f
}
// canonical is one spelling of a repository: lower case, no `.git`, no trailing slash, and a seat's path as
// the URL its holder serves it at where that is known.
func (f *sourceForms) canonical(repository, seat string) string {
trim := func(s string) string {
s = strings.TrimSpace(strings.ToLower(s))
s = strings.TrimRight(s, "/")
return strings.TrimRight(strings.TrimSuffix(s, ".git"), "/")
}
if seat == "" {
return trim(repository)
}
if b := f.base(seat); b != "" {
return trim(b + "/" + strings.Trim(repository, "/"))
}
return "seat:" + seat + ":" + trim(strings.Trim(repository, "/"))
}
// same says two sources are one repository.
func (f *sourceForms) same(aRepository, aSeat, bRepository, bSeat string) bool {
if aRepository == "" || bRepository == "" {
return false
}
return f.canonical(aRepository, aSeat) == f.canonical(bRepository, bSeat)
}
// buildsFrom says the catalogue builds some module from this repository.
func (f *sourceForms) buildsFrom(entries []inventory.Entry, repository, seat string) bool {
for _, e := range entries {
if e.Provided || e.Source.Repository == "" {
continue
}
if f.same(e.Source.Repository, e.Source.Seat, repository, seat) {
return true
}
}
return false
}
// forgeFacts is what the mesh's forge says of a repository a module is registered from (novox/hq ADR 0266).
type forgeFacts struct {
// ID is the forge's own id of the repository: what tells it from one deleted and made again by its name.
ID int64
// Guarded is whether the branch is protected as a module's trunk must be: no direct push, at least one
// required status, and no administrator merging past one. Why says what is missing when it is not.
Guarded bool
Why string
}
// askTheForge asks the forge, through its module's tools on the bus, for a repository's id and its branch's
// protection. A variable so a test needs no forge.
var askTheForge = func(ctx context.Context, owner, repo, branch string) (forgeFacts, error) {
js, err := aBus()
if err != nil {
return forgeFacts{}, err
}
defer js.Close()
bus := link.OverNATS{Conn: js.Conn()}
ask := func(tool string, args map[string]any, into any) error {
raw, _ := json.Marshal(args)
answer, err := link.Ask(ctx, bus, "gitea", tool, raw, 30*time.Second)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("gitea.%s: %s", tool, answer.Error)
}
return json.Unmarshal(answer.Result, into)
}
var found struct {
Result struct {
ID int64 `json:"id"`
FullName string `json:"full_name"`
} `json:"result"`
}
if err := ask("gitea_api", map[string]any{"path": "/repos/" + owner + "/" + repo}, &found); err != nil {
return forgeFacts{}, err
}
if found.Result.ID == 0 {
return forgeFacts{}, fmt.Errorf("the forge named no id for %s/%s", owner, repo)
}
var rules struct {
Rules []protectionRule `json:"rules"`
}
if err := ask("gitea_branch_protection_get", map[string]any{"owner": owner, "repo": repo}, &rules); err != nil {
return forgeFacts{}, err
}
facts := judgedRule(rules.Rules, branch)
facts.ID = found.Result.ID
return facts, nil
}
// protectionRule is a branch protection rule as the forge's tool summarises it.
type protectionRule struct {
Rule string `json:"rule"`
Push bool `json:"push"`
RequiredStatuses []string `json:"required_statuses"`
AdminMayOverride bool `json:"admin_may_override"`
}
// judgedRule judges the one rule the forge applies to a branch, as the forge picks it: the rule named for the
// branch, else the first glob rule, in the forge's order, that covers it (gitea's first matching rule). Never a
// later rule that happens to be stronger: the forge does not read it.
func judgedRule(rules []protectionRule, branch string) forgeFacts {
var applied *protectionRule
for i := range rules {
if rules[i].Rule == branch {
applied = &rules[i]
break
}
}
if applied == nil {
for i := range rules {
if ruleCovers(rules[i].Rule, branch) {
applied = &rules[i]
break
}
}
}
switch r := applied; {
case r == nil:
return forgeFacts{Why: fmt.Sprintf("no protection rule covers %s", branch)}
case r.Push:
return forgeFacts{Why: fmt.Sprintf("the rule %s lets a person push to %s directly", r.Rule, branch)}
case len(r.RequiredStatuses) == 0:
return forgeFacts{Why: fmt.Sprintf("the rule %s requires no status before a merge into %s", r.Rule, branch)}
case r.AdminMayOverride:
return forgeFacts{Why: fmt.Sprintf("the rule %s lets an administrator merge into %s past a status", r.Rule, branch)}
}
return forgeFacts{Guarded: true}
}
// ruleCovers says a protection rule's name — a branch, or a glob of them — covers a branch, as the forge reads it.
func ruleCovers(rule, branch string) bool {
if rule == branch {
return true
}
if !strings.ContainsAny(rule, "*?[") {
return false
}
var re strings.Builder
re.WriteString("^")
for i := 0; i < len(rule); i++ {
switch c := rule[i]; {
case c == '*' && i+1 < len(rule) && rule[i+1] == '*':
re.WriteString(".*")
i++
case c == '*':
re.WriteString("[^/]*")
case c == '?':
re.WriteString("[^/]")
default:
re.WriteString(regexp.QuoteMeta(string(c)))
}
}
re.WriteString("$")
ok, _ := regexp.MatchString(re.String(), branch)
return ok
}
// onTheForge is a source's owner and name on the mesh's own forge (the git seat's holder), and whether it is
// there at all.
func (f *sourceForms) onTheForge(repository, seat string) (owner, name string, ok bool) {
var rest string
switch {
case seat == gitSeat:
rest = strings.Trim(repository, "/")
case seat == "":
base := f.base(gitSeat)
if base == "" {
return "", "", false
}
url, prefix := f.canonical(repository, ""), f.canonical(base, "")+"/"
if !strings.HasPrefix(url, prefix) {
return "", "", false
}
// The case the forge spells it with: the URL as given, past the base.
rest = strings.TrimSuffix(strings.Trim(repository[len(prefix):], "/"), ".git")
default:
return "", "", false
}
parts := strings.Split(rest, "/")
if len(parts) != 2 || parts[0] == "" || parts[1] == "" {
return "", "", false
}
return parts[0], parts[1], true
}
// mayRegisterFrom says whether a build's outcome may register module from the source it was built from, and the
// forge's id of that repository to record (novox/hq ADR 0266). Through any verb, only:
//
// - from the module's registered repository — the same repository by the forge's own id, not only its name; or,
// for a module new to the catalogue, from a repository the catalogue builds another module from;
// - and from a repository on the mesh's forge whose trunk is protected as a trunk must be: no direct push, at
// least one required status, no administrator merging past one.
//
// Anything else only when the build request was kept as asked at the controller's terminal, for this very
// repository and path. path is the module's directory as built; branch the trunk it is registered from.
func mayRegisterFrom(ctx context.Context, inv *inventory.Inventory, module string, built inventory.Source,
buildID, path, branch string) (int64, error) {
forms := newSourceForms(ctx, inv)
was, err := inv.SourceOf(ctx, module)
isNew := errors.Is(err, inventory.ErrNoSuchModule)
if err != nil && !isNew {
return 0, err
}
terminal, err := askedHereFor(ctx, inv, forms, buildID, built, path)
if err != nil {
return 0, err
}
refuse := func(why string) (int64, error) {
return 0, fmt.Errorf("%w: %s. A module is registered from such a source only by a build asked at the "+
"controller's terminal, never through a verb: a registered module is what the next push sends, and whoever "+
"may call a verb includes agents (novox/hq ADR 0266)", errNotItsSource, why)
}
var why string
var alongside []inventory.Entry // the modules a new one's repository already builds
switch {
case !isNew && forms.same(was.Repository, was.Seat, built.Repository, built.Seat):
case !isNew:
registered := was.Repository
if registered == "" {
registered = "no repository (it was handed over by hand)"
}
why = fmt.Sprintf("%s is built from %s, and this build is of %s", module, sourceWords(registered, was.Seat),
sourceWords(built.Repository, built.Seat))
default:
entries, err := inv.Catalogued(ctx)
if err != nil {
return 0, err
}
for _, e := range entries {
if !e.Provided && e.Source.Repository != "" &&
forms.same(e.Source.Repository, e.Source.Seat, built.Repository, built.Seat) {
alongside = append(alongside, e)
}
}
if len(alongside) == 0 {
why = fmt.Sprintf("%s is new to the catalogue, and %s is no repository the catalogue builds a module from",
module, sourceWords(built.Repository, built.Seat))
}
}
if why != "" && !terminal {
return refuse(why)
}
owner, name, onForge := forms.onTheForge(built.Repository, built.Seat)
if !onForge {
if terminal {
fmt.Printf("%s: %s is not on the mesh's forge — registered, as asked at the controller's terminal\n",
buildID, sourceWords(built.Repository, built.Seat))
return 0, nil
}
return refuse(fmt.Sprintf("%s is not on the mesh's forge, so whether its trunk is protected cannot be read",
sourceWords(built.Repository, built.Seat)))
}
facts, err := askTheForge(ctx, owner, name, branch)
if err != nil {
if terminal {
fmt.Printf("%s: the forge could not be asked about %s/%s (%v) — registered, as asked at the controller's "+
"terminal\n", buildID, owner, name, err)
return 0, nil
}
return refuse(fmt.Sprintf("the forge could not say whether %s/%s's %s is protected: %v", owner, name, branch, err))
}
if terminal {
if why != "" || !facts.Guarded {
fmt.Printf("%s: %s — registered, as asked at the controller's terminal\n", buildID,
strings.Trim(why+"; "+facts.Why, "; "))
}
return facts.ID, nil
}
if !facts.Guarded {
return refuse(fmt.Sprintf("%s/%s's %s is not protected as a module's trunk must be: %s", owner, name, branch,
facts.Why))
}
// The same repository by the forge's id, not only its name: one deleted and made again is another.
recorded := map[string]int64{}
if !isNew {
id, err := inv.SourceIdentity(ctx, module)
if err != nil {
return 0, err
}
recorded[module] = id
}
for _, e := range alongside {
id, err := inv.SourceIdentity(ctx, e.Manifest.Module)
if err != nil {
return 0, err
}
recorded[e.Manifest.Module] = id
}
for m, id := range recorded {
if id != 0 && id != facts.ID {
return refuse(fmt.Sprintf("%s/%s is not the repository %s was registered from: the forge knows it as "+
"repository %d, and %s was registered from repository %d — one of that name deleted and made again",
owner, name, m, facts.ID, m, id))
}
}
return facts.ID, nil
}
// askedHereFor says the build was asked at the controller's terminal, for this repository and this path: a kept
// request marked so, whose source is the outcome's (novox/hq ADR 0266).
func askedHereFor(ctx context.Context, inv *inventory.Inventory, forms *sourceForms, buildID string,
built inventory.Source, path string) (bool, error) {
r, found, err := inv.BuildRequestByID(ctx, buildID)
if err != nil || !found || !r.AtTerminal {
return false, err
}
if !forms.same(r.Repository, r.Seat, built.Repository, built.Seat) ||
strings.Trim(r.Path, "/") != strings.Trim(path, "/") {
fmt.Printf("%s was asked at the terminal of %s at %q, and its outcome is of %s at %q: not the terminal's\n",
buildID, sourceWords(r.Repository, r.Seat), r.Path, sourceWords(built.Repository, built.Seat), path)
return false, nil
}
return true, nil
}
// sourceWords is a source as a person reads it.
func sourceWords(repository, seat string) string {
if seat == "" {
return repository
}
return buildSource{Repository: repository, Seat: seat}.String()
}
// verbMayAsk refuses, for a call through a verb, a build of a repository the catalogue builds no module from
// (novox/hq ADR 0266): the build node would run what an agent wrote, and its outcome could never be
// registered anyway. url is the repository as it is cloned. At the terminal anything may be asked.
func verbMayAsk(ctx context.Context, source buildSource, url string) error {
if startedAtTheTerminal() {
return nil
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return err
}
forms := newSourceForms(ctx, open.inventory)
if forms.buildsFrom(entries, source.Repository, source.Seat) || forms.buildsFrom(entries, url, "") {
return nil
}
return terminalRefusal("%s is no repository the catalogue builds a module from, and a build of any other is "+
"asked at the controller's terminal only, never through a verb: a build node runs what the repository "+
"says, and its outcome would register a module the next push sends — whoever may call a verb includes "+
"agents (novox/hq ADR 0266). Nothing was asked", source)
}
+384
View File
@@ -0,0 +1,384 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"hash/fnv"
"os"
"os/exec"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The routes to root a review of ADR 0266 found (novox/hq ADR 0266 §7): an agent makes a repository of its
// own — or forks one the mesh builds from — commits a module.json naming a module the mesh runs everywhere
// (`sudo`, granting itself a rule without a password; `mesh-host`, the node-engine), and asks the `build` verb
// for it. Its outcome was registered under that name from the agent's repository, and the next push sent it.
// onTrunk is an outcome of a commit on its repository's trunk, as the build seat says it.
func onTrunk(id, repository, seat, path string, manifest map[string]any) link.BuildResult {
raw, _ := json.Marshal(manifest)
r := link.BuildResult{ID: id, Repository: "http://forge.internal:20000/" + repository + ".git", Path: path,
Ref: "main", On: "anchor", Commit: "c0ffee0123456789", Manifest: raw,
Trunk: "main", OnTrunk: true, Branches: []string{"main"}}
if seat != "" {
r.Source = &link.SourceOnSeat{Seat: seat, Repository: repository}
}
return r
}
// keptAsked keeps a build request as the asker would: through a verb, or at the terminal.
func keptAsked(t *testing.T, inv *inventory.Inventory, id, repository, path string, atTerminal bool) {
t.Helper()
if err := inv.RecordBuildRequest(t.Context(), inventory.BuildRequest{ID: id, Repository: repository, Seat: "git",
Path: path, For: "build", AtTerminal: atTerminal}); err != nil {
t.Fatal(err)
}
}
// theCatalogue is a mesh whose sudo is built from the catalogue repository and whose node-engine from its own.
func theCatalogue(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
for _, b := range []link.BuildResult{
onTrunk("build-sudo", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "1"}),
onTrunk("build-host", "novox/mesh-host", "git", "", map[string]any{"module": "mesh-host", "version": "1"}),
} {
keptAsked(t, open.inventory, b.ID, b.Source.Repository, b.Path, true)
if _, _, err := takeIn(ctx, open.inventory, b); err != nil {
t.Fatal(err)
}
}
return open
}
func TestABuildFromAnAgentsRepositoryIsNotRegisteredAsAModuleTheMeshHolds(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
for _, c := range []struct {
name, repository, path, module string
}{
{"its own repository naming sudo", "agent/sudo", "modules/sudo", "sudo"},
{"a fork of the catalogue", "agent/mesh-catalog", "modules/sudo", "sudo"},
{"a fork of the node-engine", "agent/mesh-host", "", "mesh-host"},
} {
t.Run(c.name, func(t *testing.T) {
id := "build-" + strings.ReplaceAll(c.repository, "/", "-")
keptAsked(t, open.inventory, id, c.repository, c.path, false) // through the build verb
evil := onTrunk(id, c.repository, "git", c.path, map[string]any{"module": c.module, "version": "evil"})
_, _, err := takeIn(ctx, open.inventory, evil)
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a build of %s was taken in as %s: %v", c.repository, c.module, err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf[c.module].Version; got != "1" {
t.Fatalf("%s is now %q, from %s", c.module, got, c.repository)
}
if _, found, _ := open.inventory.BuildByID(ctx, id); !found {
t.Errorf("the refused build %s is not recorded", id)
}
})
}
}
func TestANewModuleFromARepositoryTheCatalogueDoesNotBuildFromIsNotRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-new", "agent/tools", "", false)
_, _, err := takeIn(ctx, open.inventory, onTrunk("build-new", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from an agent's repository was taken in: %v", err)
}
// And one asked of nobody here — an outcome on the bus no request was kept for — the same.
_, _, err = takeIn(ctx, open.inventory, onTrunk("build-unasked", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome nobody asked for was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["agent-tools"].Module != "" {
t.Fatal("the refused module is in the catalogue")
}
}
// The operator at the terminal may still move a module, or add one from a new repository.
func TestAtTheTerminalAnotherRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-moved", "novox/sudo", "", true)
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-moved", "novox/sudo", "git", "",
map[string]any{"module": "sudo", "version": "2"})); err != nil {
t.Fatalf("a move the operator asked for at the terminal was refused: %v", err)
}
if src, _ := open.inventory.SourceOf(ctx, "sudo"); src.Repository != "novox/sudo" {
t.Fatalf("sudo is built from %q", src.Repository)
}
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-external",
Repository: "http://forge.internal:20000/someone/app.git", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-external", "someone/app", "", "",
map[string]any{"module": "app", "version": "1"})); err != nil {
t.Fatalf("a new module the operator asked for at the terminal was refused: %v", err)
}
}
// The delivery's flow is untouched: a merge's rebuild of a module from its own repository, and a merge adding
// a module to a repository the catalogue builds from (novox/hq issue 300), are registered with no terminal.
func TestADeliveryFromTheRegisteredRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
rebuilt := onTrunk("build-plan", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: rebuilt.ID,
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/sudo", For: "plan"}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, rebuilt); err != nil {
t.Fatalf("a plan's build of the module's own repository was refused: %v", err)
}
added := onTrunk("build-merge", "novox/mesh-catalog", "git", "modules/zram", map[string]any{"module": "zram", "version": "1"})
if _, _, err := takeIn(ctx, open.inventory, added); err != nil {
t.Fatalf("a module a merge added to the catalogue repository was refused: %v", err)
}
shelf, _ := open.inventory.Catalogue(ctx)
if shelf["sudo"].Version != "2" || shelf["zram"].Module == "" {
t.Fatalf("not registered: sudo %q, zram %q", shelf["sudo"].Version, shelf["zram"].Module)
}
}
// Through a verb, a build of a repository the catalogue builds nothing from is not even asked: the build node
// would run what the agent wrote.
func TestAVerbAsksNoBuildOfARepositoryTheCatalogueDoesNotBuildFrom(t *testing.T) {
theCatalogue(t)
ctx := t.Context()
t.Setenv(verbVar, "build")
t.Setenv(link.CallerVar, "node-tools.anchor, through the mesh-controller seat")
err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "http://forge.internal:20000/agent/sudo.git")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("a verb's build of an agent's repository was asked: %v", err)
}
if err := verbMayAsk(ctx, buildSource{Repository: "novox/mesh-catalog", Seat: "git"},
"http://forge.internal:20000/novox/mesh-catalog.git"); err != nil {
t.Fatalf("a verb's build of the catalogue repository was refused: %v", err)
}
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
if err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, ""); err != nil {
t.Fatalf("the terminal was refused: %v", err)
}
}
// asTheOperator keeps a build as asked at the controller's terminal, as the operator's first build of a module
// from a repository the catalogue does not yet build from is (novox/hq ADR 0266), and hands it back.
func asTheOperator(t *testing.T, inv *inventory.Inventory, b link.BuildResult) link.BuildResult {
t.Helper()
repository, seat := b.Repository, ""
if b.Source != nil {
repository, seat = b.Source.Repository, b.Source.Seat
}
if err := inv.RecordBuildRequest(t.Context(), inventory.BuildRequest{ID: b.ID, Repository: repository, Seat: seat,
Path: b.Path, For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
return b
}
// A rollback puts back only a build of the module's own repository: an agent's build of the module's name,
// recorded and refused, at the very commit the machine ran before (a fork carries it), is never registered by
// the back door of a failed gate.
func TestARollbackNeverPutsBackABuildFromAnotherRepository(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
fork := onTrunk("build-1791500000000000000", "agent/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "evil"})
fork.Commit = "c0ffee0123456789" // the commit sudo was registered at
keptAsked(t, inv, fork.ID, "agent/mesh-catalog", "modules/sudo", false)
if _, _, err := takeIn(ctx, inv, fork); !errors.Is(err, errNotItsSource) {
t.Fatalf("the fork's build was taken in: %v", err)
}
// Asked after the registered build, whenever the test runs: an id naming a fixed moment read as older
// than the registered build once the clock passed it (2026-10-10 02:40 UTC), and the test failed on main.
failed := onTrunk(fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano()), "novox/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "2"})
failed.Commit = "badbadbad0123456"
if _, _, err := takeIn(ctx, inv, failed); err != nil {
t.Fatal(err)
}
record, _, err := inv.BuildByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
previous, found, err := inv.PreviousBuild(ctx, "sudo", "c0ffee0123456789", record)
if err != nil {
t.Fatal(err)
}
if found && previous.ID == fork.ID {
t.Fatalf("a rollback would put back the fork's build %s", previous.ID)
}
if !found || previous.ID != "build-sudo" {
t.Fatalf("a rollback puts back %q (found %v), want the registered build-sudo", previous.ID, found)
}
}
// theForge is what the forge says in a test, by owner/name: a repository not named here is protected as a
// trunk must be, with an id of its own.
var theForge sync.Map
func init() {
askTheForge = func(_ context.Context, owner, repo, _ string) (forgeFacts, error) {
if said, ok := theForge.Load(owner + "/" + repo); ok {
switch f := said.(type) {
case error:
return forgeFacts{}, f
case forgeFacts:
return f, nil
}
}
h := fnv.New32a()
_, _ = h.Write([]byte(owner + "/" + repo))
return forgeFacts{ID: int64(h.Sum32()), Guarded: true}, nil
}
}
// A module's trunk the forge does not protect — direct pushes, no required status — or a forge that cannot say,
// registers nothing through a verb: the trunk rule means nothing on a branch anyone pushes to.
func TestATrunkTheForgeDoesNotProtectRegistersNothing(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/unguarded", forgeFacts{ID: 7, Why: "the rule main lets a person push to main directly"})
t.Cleanup(func() { theForge.Delete("novox/unguarded") })
first := onTrunk("build-unguarded-1", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/unguarded", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatalf("at the terminal: %v", err)
}
again := onTrunk("build-unguarded-2", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "2"})
if _, _, err := takeIn(ctx, open.inventory, again); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "push to main directly") {
t.Fatalf("a rebuild from an unprotected trunk was taken in: %v", err)
}
theForge.Store("novox/unguarded", errors.New("nothing serves gitea.gitea_api"))
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-unguarded-3", "novox/unguarded", "git", "",
map[string]any{"module": "unguarded", "version": "3"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a forge that could not say was read as a protected trunk: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["unguarded"].Version != "1" {
t.Fatalf("unguarded is %q", shelf["unguarded"].Version)
}
}
// A repository deleted and made again under the module's repository's name is another repository: the forge's
// id, recorded at registration, tells them apart.
func TestARepositoryMadeAgainUnderItsNameIsNotTheModulesSource(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/remade", forgeFacts{ID: 100, Guarded: true})
t.Cleanup(func() { theForge.Delete("novox/remade") })
first := onTrunk("build-remade-1", "novox/remade", "git", "", map[string]any{"module": "remade", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/remade", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "remade"); id != 100 {
t.Fatalf("the forge's id was not recorded: %d", id)
}
theForge.Store("novox/remade", forgeFacts{ID: 101, Guarded: true}) // deleted, and made again by an agent
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-2", "novox/remade", "git", "",
map[string]any{"module": "remade", "version": "evil"})); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "made again") {
t.Fatalf("a repository made again under the name was taken in: %v", err)
}
// And a new module from it, beside the one registered from the first, the same.
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-3", "novox/remade", "git", "modules/other",
map[string]any{"module": "other", "version": "1"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from a repository made again was taken in: %v", err)
}
}
// The terminal's mark is the operator's for the repository and path they asked: an outcome of another, under that
// build's id, is not theirs.
func TestATerminalRequestCoversOnlyWhatItAsked(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-asked", Repository: "novox/app",
Seat: "git", Path: "modules/app", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
other := onTrunk("build-asked", "agent/sudo", "git", "modules/app", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, other); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another repository under a terminal request's id was taken in: %v", err)
}
elsewhere := onTrunk("build-asked", "novox/app", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, elsewhere); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another path under a terminal request's id was taken in: %v", err)
}
}
// The serving controller, and everything it starts, is never the terminal: a build asked in it reads as asked
// through the mesh even when no verb and no caller is named.
func TestTheServingControllerIsNeverTheTerminal(t *testing.T) {
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
t.Setenv(servedVar, "")
if !startedAtTheTerminal() {
t.Fatal("a process started by hand is not the terminal")
}
markServed()
if startedAtTheTerminal() {
t.Fatal("the serving controller reads as the terminal")
}
child := exec.Command(os.Args[0], "-test.run=^$")
child.Env = os.Environ()
if !slices.Contains(child.Env, servedVar+"=1") {
t.Fatal("what the serving controller starts does not carry its mark")
}
}
// The forge applies one rule to a branch: the rule named for it, else the first glob that covers it. A stronger
// rule later in the list is not what guards the branch, and is not read as if it were (the confirmation review).
func TestTheRuleJudgedIsTheOneTheForgeApplies(t *testing.T) {
guarded := protectionRule{Rule: "*", RequiredStatuses: []string{"mesh/merge-gate"}}
open := protectionRule{Rule: "main", Push: true, RequiredStatuses: []string{"mesh/merge-gate"}}
if f := judgedRule([]protectionRule{guarded, open}, "main"); f.Guarded {
t.Error("an exact rule letting pushes was passed over for a glob that guards")
}
if f := judgedRule([]protectionRule{{Rule: "ma*", RequiredStatuses: nil}, {Rule: "*", RequiredStatuses: []string{"x"}}},
"main"); f.Guarded || !strings.Contains(f.Why, "ma*") {
t.Errorf("the first glob covering the branch was not the one judged: %+v", f)
}
if f := judgedRule([]protectionRule{{Rule: "release/*"}, {Rule: "main", RequiredStatuses: []string{"x"}}}, "main"); !f.Guarded {
t.Errorf("the rule named for the branch was not judged: %+v", f)
}
if f := judgedRule(nil, "main"); f.Guarded {
t.Error("no rule is no protection")
}
}
// An id the forge could not give keeps the id already recorded.
func TestAnUnknownIdentityKeepsTheRecordedOne(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 100); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 0); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "sudo"); id != 100 {
t.Fatalf("the recorded id became %d", id)
}
}
+4 -4
View File
@@ -19,11 +19,11 @@ func TestABuildHeardIsRecordedAndRegistered(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "shop", "version": "3"})
m, _, err := takeIn(ctx, open.inventory, link.BuildResult{
m, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, link.BuildResult{
ID: "b-1", Repository: "http://forge.internal:20000/novox/shop.git", Path: "modules/shop",
Ref: "main", On: "anchor", Commit: "abcdef0123", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/shop"},
})
}))
if err != nil {
t.Fatal(err)
}
@@ -78,7 +78,7 @@ func TestABuildAtACommitKeepsTheBranchTheModuleFollows(t *testing.T) {
Path: "modules/unifi", Ref: ref, On: "anchor", Commit: commit, Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, result("b-1", "main", "1111111aaaa")); err != nil {
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result("b-1", "main", "1111111aaaa"))); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, result("b-2", "9c97a8a", "9c97a8a1d2c3")); err != nil {
@@ -117,7 +117,7 @@ func TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer(t *testing.T) {
Path: "modules/postgres", Ref: "main", On: "anchor", Commit: "efff5415", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, result(newer, "4bcd5f73")); err != nil {
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result(newer, "4bcd5f73"))); err != nil {
t.Fatal(err)
}
_, _, err := takeIn(ctx, open.inventory, result(older, "0ab07fa9"))
+9 -2
View File
@@ -151,6 +151,13 @@ func busCommand(ctx context.Context, args []string) error {
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
sub, args = args[0], args[1:]
}
// The view's credential, a terminal line like a person's (bus_view.go).
switch sub {
case "view-credential":
return busViewCredential(ctx, args)
case "view-revoke":
return busViewRevoke(ctx, args)
}
set := flag.NewFlagSet("bus", flag.ContinueOnError)
snapshot := set.String("snapshot-taken", "", "where the streams' snapshot a person took is, while the mesh takes none itself")
reversible := set.Bool("reversible", false, "the new version can be undone by putting the old one back")
@@ -160,14 +167,14 @@ func busCommand(ctx context.Context, args []string) error {
if rest, err := parseAround(set, args); err != nil {
return err
} else if len(rest) > 0 {
return errors.New("bus [upgrade --why … --reversible|--irreversible [--snapshot-taken <where>]]")
return errors.New(busUsage)
}
switch sub {
case "":
return busStatus(ctx)
case "upgrade":
default:
return fmt.Errorf("bus says what a bus upgrade would do, or `bus upgrade` — not %q", sub)
return fmt.Errorf("bus says what a bus upgrade would do, or `bus upgrade`, `bus view-credential`, `bus view-revoke` — not %q", sub)
}
// Everything refused before anything is done.
if err := why.require("bus upgrade"); err != nil {
+117
View File
@@ -0,0 +1,117 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
"strconv"
"strings"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// The view's credential: the one read-only user a page in a browser connects to the bus as, over the
// bus module's WebSocket listener (novox/hq research 036, gap G1; broker.KindView).
//
// **A terminal line, like a person's credential** (operator.go): printed once, never stored — the mesh
// keeps a hash — and revoked by forgetting the row, which the next composition of the user list makes
// real. There is one view; issuing it again rotates its password.
const busUsage = "bus [upgrade --why … --reversible|--irreversible [--snapshot-taken <where>] | view-credential | view-revoke]"
// busWebSocketPort is the port the bus module's WebSocket listener is published on, mirrored from the
// nats module's manifest (its `bus-websocket` opening), because the credential names where to connect
// and the controller does not read the module's configuration. Reached across the overlay only: the
// opening is from the mesh, and the mesh's filter admits nothing else.
const busWebSocketPort = 4223
func busViewCredential(ctx context.Context, args []string) error {
if len(args) != 0 {
return errors.New("bus view-credential takes nothing: there is one view, and this prints its credential once")
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
// Refused here rather than at the next composition, where it would stop the whole file.
if _, err := broker.PermissionsFor(broker.Principal{Kind: broker.KindView, PasswordHash: "x"}); err != nil {
return err
}
password, err := inv.MintBusPassword(ctx, inventory.BusUser{Username: broker.ViewUser, Kind: inventory.BusView})
if err != nil {
return err
}
where, err := broker.FromEnvironment()
if err != nil && !errors.Is(err, broker.ErrNotConfigured) {
return err
}
host := where.Address
if h, _, err := net.SplitHostPort(where.Address); err == nil {
host = h
}
websocket := ""
if host != "" {
websocket = "ws://" + net.JoinHostPort(host, strconv.Itoa(busWebSocketPort))
}
held, err := json.Marshal(struct {
WebSocket string `json:"websocket,omitempty"`
URL string `json:"url,omitempty"`
Fingerprint string `json:"fingerprint,omitempty"`
User string `json:"user"`
Password string `json:"password"`
InboxPrefix string `json:"inbox_prefix"`
Hears []string `json:"hears"`
Reads string `json:"reads"`
HowToRead string `json:"how_to_read"`
}{
WebSocket: websocket, URL: "nats://" + where.Address, Fingerprint: where.Fingerprint,
User: broker.ViewUser, Password: password,
// The client must make its inboxes under the view's own prefix: its subscribe grant is
// `_INBOX.view.>` and no wider (design 25 §4), and a client's default inbox is not under it.
InboxPrefix: "_INBOX." + broker.ViewUser,
Hears: broker.ViewHears, Reads: broker.ViewBucket,
HowToRead: "direct reads only, no watch (a consumer is refused): list with a request to $JS.API.DIRECT.GET.KV_" +
broker.ViewBucket + ` carrying {"multi_last":["$KV.` + broker.ViewBucket + `.>"]}, answered until a 204 status; ` +
"read one key with $JS.API.DIRECT.GET.KV_" + broker.ViewBucket + ".$KV." + broker.ViewBucket + ".<number> " +
"(nats.js: kvm.open(bucket, {allow_direct: true}), never create); re-read the key an event's number names",
})
if err != nil {
return err
}
fmt.Printf("issued the view, which hears %s and reads the bucket %s, and nothing else\n",
strings.Join(broker.ViewHears, ", "), broker.ViewBucket)
fmt.Println(" this is the only time the credential is printed; the mesh keeps a hash")
fmt.Println(" it works once the bus has been told, which is the next push to the machine holding mesh-broker —")
fmt.Println(" and while a new build of the bus module waits for that machine, the next `bus upgrade` a person starts,")
fmt.Println(" which is also what brings the WebSocket listener it connects through")
fmt.Println()
fmt.Println(string(held))
return nil
}
func busViewRevoke(ctx context.Context, args []string) error {
if len(args) != 0 {
return errors.New("bus view-revoke takes nothing: there is one view")
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
if err := open.inventory.ForgetBusUser(ctx, broker.ViewUser); err != nil {
return err
}
// **Revoked at the next composition, not now** — as a person is (operator revoke): the bus's users
// are a file, and the credential stops working when the file no longer names it.
fmt.Println("the view is forgotten, and its credential stops working at the next composition — " +
"push the machine holding mesh-broker to make it so")
return nil
}
+195
View File
@@ -0,0 +1,195 @@
package main
import (
"fmt"
"sort"
"strings"
"sync"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// A send held for the bus's planned step (novox/hq issue 336).
//
// **A wait only a person can end is said to that person at once.** No send replaces the bus but its planned
// step (sendToEach, ADR 0236), and the step is a person's `bus upgrade`. So while a new bus build waits, every
// walk whose tier sends to the bus's machine is refused, and tries again each tick. The refusal was kept only
// as the walk's note; nothing was raised, and the operator found the hold by asking why a walk did not move.
// Row S17 raises it on the first tick after the first refusal, in the bus's scope, for the operator: what
// waits, behind which bus build, since when, and the verb. A walk held only by it is not late, so S3 leaves it
// out, as it leaves out a walk under a paused build seat. It clears once the bus's machine has been sent the
// build the mesh holds — the step was taken, and no send is refused for the bus any more — whatever the walks'
// notes still say. Whether the new bus came up healthy is the step's own condition (probe DB).
// kindBusStepWaiting is S17's kind: bus.<module>.step-waiting.
const kindBusStepWaiting = "bus-step-waiting"
// busFacts is a new bus build waiting for its planned step, and the sends held for it.
type busFacts struct {
module, to string
// from is the build each machine of the bus runs; machines those whose bus the step would replace.
from map[string]string
machines []string
waits []busWaitFacts
}
// busWaitFacts is one walk whose send was refused because it would replace the bus.
type busWaitFacts struct {
plan, repository, commit string
// modules are what its tier sends: what waits.
modules []string
// since is the first refusal: as this controller saw it, or, read back, the save that kept the refusal.
since time.Time
}
// busRefusedFirst is when this controller first saw each walk refused for the bus's step. The walk's note
// keeps the refusal across a restart; this keeps its moment more exactly than the walk's last save.
var busRefusedFirst = &firstSeen{at: map[string]time.Time{}}
type firstSeen struct {
mu sync.Mutex
at map[string]time.Time
}
// mark keeps the first moment an id was seen.
func (s *firstSeen) mark(id string, at time.Time) {
s.mu.Lock()
defer s.mu.Unlock()
if _, seen := s.at[id]; !seen {
s.at[id] = at
}
}
// of is when an id was first seen; zero when it was not.
func (s *firstSeen) of(id string) time.Time {
s.mu.Lock()
defer s.mu.Unlock()
return s.at[id]
}
// keepOnly forgets every id not given: a walk no longer held is not held since then.
func (s *firstSeen) keepOnly(ids map[string]bool) {
s.mu.Lock()
defer s.mu.Unlock()
for id := range s.at {
if !ids[id] {
delete(s.at, id)
}
}
}
// refusedForTheBus is whether an error is the refusal of a send for the bus's planned step.
func refusedForTheBus(err error) bool {
return err != nil && strings.Contains(err.Error(), errBusWaits.Error())
}
// busWaitsOf is the sends held for the bus's step, from the open walks: each walk not waiting for its
// delivery's word whose note keeps a refusal for this very bus build, while the build would still replace
// the bus on a machine. first is when this controller first saw a walk refused, zero when it did not.
func busWaitsOf(plans []inventory.Plan, b busPending, first func(string) time.Time) busFacts {
f := busFacts{module: b.module, to: b.to, from: b.from}
for _, n := range b.machines {
if b.moves(n) {
f.machines = append(f.machines, n)
}
}
if len(f.machines) == 0 {
return f
}
for _, p := range plans {
if !p.Open() || p.Waiting() || !strings.Contains(p.Note, errBusWaits.Error()) || !strings.Contains(p.Note, short(b.to)) {
continue
}
since := p.Updated
if seen := first(p.ID); !seen.IsZero() && (since.IsZero() || seen.Before(since)) {
since = seen
}
var modules []string
if p.Tier >= 0 && p.Tier < len(p.Tiers) {
modules = append(modules, p.Tiers[p.Tier]...)
} else {
modules = planModules(p)
}
sort.Strings(modules)
f.waits = append(f.waits, busWaitFacts{plan: p.ID, repository: p.Repository, commit: p.Commit, modules: modules,
since: since})
}
sort.Slice(f.waits, func(i, j int) bool { return f.waits[i].since.Before(f.waits[j].since) })
return f
}
// heldByTheBus is the walks of a bus's facts, by id: what S3 leaves out.
func (b busFacts) heldByTheBus() map[string]bool {
out := map[string]bool{}
for _, w := range b.waits {
out[w.plan] = true
}
return out
}
// busUpgradeVerb is the call that ends the wait, as the summary names it: through the mesh MCP server, with
// why, saying whether the new version can be undone ("reversible": "true") or not ("irreversible": "true").
const busUpgradeVerb = "`mesh_call mesh-controller.bus {\"upgrade\": \"true\", \"why\": \"…\", \"reversible\" or \"irreversible\": \"true\"}`"
// watchBusWaits is S17: a send held for the bus's planned step, said at once to the operator.
func watchBusWaits(f *signalFacts) []conditions.Observation {
b := f.bus
if len(b.waits) == 0 || len(b.machines) == 0 {
return nil
}
since := b.waits[0].since
var walks, what []string
for _, w := range b.waits {
walks = append(walks, fmt.Sprintf("%s (%s %s, tier: %s)", w.plan, w.repository, short(w.commit),
strings.Join(w.modules, ", ")))
what = append(what, deliveryWhat(w.modules, w.repository))
}
var from []string
for _, n := range b.machines {
from = append(from, short(orNotKnown(b.from[n])))
}
machines := namesWords(b.machines, 3)
in := f.now.Sub(since)
return []conditions.Observation{{Scope: conditions.ScopeBus, ID: b.module, Token: "step-waiting",
Kind: kindBusStepWaiting, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Machine: b.machines[0], Also: b.machines[1:],
Summary: fmt.Sprintf("sends to %s wait for the bus's planned step: a new bus build (%s %s → %s) would replace "+
"the bus there, which only a person's %s does; waiting since %s: %s", strings.Join(b.machines, ", "),
b.module, strings.Join(sortedUnique(from), ", "), short(b.to), busUpgradeVerb,
since.UTC().Format(time.RFC3339), strings.Join(walks, "; ")),
Said: fmt.Sprintf("%d walk(s) refused since %s", len(b.waits), since.UTC().Format(time.RFC3339)),
Headline: clipWords("Sends to "+machines+" wait for a bus upgrade", conditions.HeadlineMax),
Explanation: clipWords(fmt.Sprintf("A new version of the mesh's message system is built, and only a person "+
"installs it. Until then nothing else is sent to %s: %s waits, for %s so far.", machines,
namesWords(sortedUnique(what), 3), humanDuration(in)), conditions.ExplanationMax),
Needs: "start the bus upgrade " + FromMeshMCPServer,
Resolved: "Resolved: the bus upgrade started, and sends go on"}}
}
// sortedUnique is a list sorted, each once.
func sortedUnique(xs []string) []string {
seen := map[string]bool{}
var out []string
for _, x := range xs {
if !seen[x] {
seen[x] = true
out = append(out, x)
}
}
sort.Strings(out)
return out
}
// clipWords keeps a plain sentence within a bound, at a word.
func clipWords(s string, n int) string {
if len(s) <= n {
return s
}
cut := strings.LastIndex(s[:n-1], " ")
if cut <= 0 {
cut = n - 1
}
return s[:cut] + "…"
}
+159
View File
@@ -0,0 +1,159 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 336: a send held because it would replace the bus outside its planned step waits for a
// person, so a person is told at once — what waits, behind which bus build, since when, and the verb that
// ends it — and the wait is not read as a walk running late.
// aBusOnAnchor is a new bus build waiting for its step on anchor: nats 88135ad0 there, 32307bd1 held.
func aBusOnAnchor() busPending {
return busPending{module: "nats", machines: []string{"anchor"}, from: map[string]string{"anchor": "88135ad0aaaa"},
to: "32307bd1bbbb", same: map[string]bool{}}
}
// refusedNote is the note a walk keeps when its send was refused for the bus, as advanceHeld writes it.
func refusedNote(b busPending, tier int) string {
held := "sending anchor would replace the bus (nats " + short(b.from["anchor"]) + " → " + short(b.to) +
"), which is a planned step: `bus upgrade --why …` snapshots its streams first and checks them after (novox/hq ADR 0236)"
return "tier " + string(rune('0'+tier)) + ": " + errBusWaits.Error() + ": " + held + " — tried again"
}
func TestASendHeldForTheBusStepIsFoundFromTheWalksItHolds(t *testing.T) {
now := time.Date(2026, 10, 8, 18, 30, 0, 0, time.UTC)
b := aBusOnAnchor()
walk := func(id, repo, note string, updated time.Time) inventory.Plan {
return inventory.Plan{ID: id, Repository: repo, Commit: "c0ffee001122", State: inventory.PlanRolling, Tier: 0,
Tiers: [][]string{{"mesh-host"}}, Modules: map[string]*inventory.PlanModule{"mesh-host": {}}, Note: note,
Updated: updated}
}
for _, c := range []struct {
name string
bus busPending
plans []inventory.Plan
first map[string]time.Time
want []string
since time.Time
}{
{"a walk refused for the bus", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-27*time.Minute))}, nil,
[]string{"plan-1"}, now.Add(-27 * time.Minute)},
{"refused earlier than its last save, as this controller saw it", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-5*time.Minute))},
map[string]time.Time{"plan-1": now.Add(-28 * time.Minute)}, []string{"plan-1"}, now.Add(-28 * time.Minute)},
{"a walk refused for another reason", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", "tier 0: the build seat is paused — tried again", now)}, nil,
nil, time.Time{}},
{"refused for an older bus build than the one held now", func() busPending { o := b; o.to = "99999999cccc"; return o }(),
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"the bus already runs the build held: its step started", func() busPending {
o := aBusOnAnchor()
o.from = map[string]string{"anchor": o.to}
return o
}(), []inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"a walk waiting for its delivery's word asks no send", b, func() []inventory.Plan {
p := walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)
p.Delivery = &inventory.PlanDelivery{Awaits: "mesh-delivery"}
return []inventory.Plan{p}
}(), nil, nil, time.Time{}},
} {
t.Run(c.name, func(t *testing.T) {
got := busWaitsOf(c.plans, c.bus, func(id string) time.Time { return c.first[id] })
var ids []string
for _, w := range got.waits {
ids = append(ids, w.plan)
}
if strings.Join(ids, ",") != strings.Join(c.want, ",") {
t.Fatalf("held for the bus: %v, want %v", ids, c.want)
}
if len(c.want) > 0 {
if !got.waits[0].since.Equal(c.since) {
t.Errorf("waiting since %s, want %s", got.waits[0].since, c.since)
}
if strings.Join(got.machines, ",") != "anchor" || got.to != b.to || got.module != "nats" {
t.Errorf("behind %+v", got)
}
}
})
}
}
// **Said at once, not as lateness, and cleared when the step starts**: the walk held only for the bus raises
// the bus's condition on the first tick, before any tier bound, naming what waits, the bus build from and to,
// since when and `bus upgrade`, for the operator; S3 says nothing of it even past its bound; and the
// condition clears once the bus's machine runs the new build.
func TestASendHeldForTheBusStepIsSaidAtOnceAndNotAsLateness(t *testing.T) {
now := time.Date(2026, 10, 8, 18, 30, 0, 0, time.UTC)
b := aBusOnAnchor()
held := func(entered time.Duration) *signalFacts {
f := calm(now)
f.plans = []planFacts{{id: "plan-1", repository: "novox/mesh-host", commit: "c0ffee00", tier: 0, tiers: 1,
entered: now.Add(-entered), bound: 30 * time.Minute, waiting: refusedNote(b, 0), bus: true}}
f.bus = busFacts{module: "nats", to: b.to, from: b.from, machines: []string{"anchor"},
waits: []busWaitFacts{{plan: "plan-1", repository: "novox/mesh-host", commit: "c0ffee001122",
modules: []string{"mesh-host"}, since: now.Add(-time.Minute)}}}
return f
}
f := held(time.Minute)
got := watchBusWaits(f)
if len(got) != 1 {
t.Fatalf("a send held for the bus a minute ago raised %+v", got)
}
o := got[0]
if o.Key() != "bus.nats.step-waiting" || o.Kind != kindBusStepWaiting || o.Resolver != conditions.ResolverOperator {
t.Fatalf("raised %s (%s), resolver %q", o.Key(), o.Kind, o.Resolver)
}
for _, want := range []string{"anchor", "mesh-host", "88135ad0", "32307bd1", "mesh_call mesh-controller.bus",
`"upgrade": "true"`, `"reversible"`, `"irreversible"`, "2026-10-08T18:29:00Z", "plan-1"} {
if !strings.Contains(o.Summary, want) {
t.Errorf("its summary does not say %q: %s", want, o.Summary)
}
}
if !strings.Contains(o.Explanation, "mesh-host") || !strings.Contains(o.Headline, "bus upgrade") || o.Needs == "" {
t.Errorf("its words do not say what waits and what the operator does: %+v", o)
}
if why, ok := conditions.PlainWords(conditions.Words{Headline: o.Headline, Explanation: o.Explanation,
Resolved: o.Resolved, Needs: o.Needs}, "anchor"); !ok {
t.Errorf("its words are not plain: %s", why)
}
// Past S3's bound: still the bus's wait, never a stalled walk.
late := held(45 * time.Minute)
if s3 := watchPlans(late); len(s3) != 0 {
t.Fatalf("a walk held only by the bus step was said stalled: %+v", s3)
}
// A walk held for something else past its bound is still stalled.
other := held(45 * time.Minute)
other.plans[0].bus = false
if s3 := watchPlans(other); len(s3) != 1 {
t.Fatalf("a walk held for something else past its bound raised %+v", s3)
}
// Through the keeper: open at the first tick, cleared when the step started.
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store, Teller: &conditions.Told{},
Now: func() time.Time { return now }})
defer k.Close(context.Background())
w := &watchdogs{keeper: k, started: now.Add(-time.Hour)}
w.see(t.Context(), held(time.Minute))
open, err := k.Open(t.Context())
if err != nil || len(open) != 1 || open[0].Key != "bus.nats.step-waiting" {
t.Fatalf("after the first tick, open: %+v (%v)", open, err)
}
started := held(time.Minute)
started.bus = busFacts{module: "nats", to: b.to}
started.plans[0].bus = false
w.see(t.Context(), started)
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("the step started, and open: %+v", open)
}
}
+48
View File
@@ -53,9 +53,26 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
if err != nil {
return nil, err
}
// And the work queues of seats that name their caller or their kind, with each holder's worker
// (novox/hq ADR 0259 §3): an ask queues until the router takes it, a channel's work until that kind
// takes it.
trafficStreams, trafficWorkers, err := seatTrafficObjects(ctx, inv)
if err != nil {
return nil, err
}
// Every one tried, and every failure named: one module's consumer the bus refuses is no reason
// the modules after it in the list hear nothing (novox/hq issue 208, where this runs on each send).
var failed []error
for _, s := range trafficStreams {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the work queue %s: %w", s.Name, err))
}
}
for _, c := range trafficWorkers {
if err := r.EnsureConsumer(c); err != nil {
failed = append(failed, fmt.Errorf("the worker %s on %s: %w", c.Name, c.Stream, err))
}
}
for _, c := range consumers {
if err := r.EnsureConsumer(c.Consumer); err != nil {
failed = append(failed, fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err))
@@ -130,6 +147,37 @@ func moduleConsumers(ctx context.Context, inv *inventory.Inventory) ([]broker.Mo
return broker.ConsumersOf(users), nil
}
// seatTrafficObjects is the work queues and workers of seats that name their caller or their kind, from
// the records the user list is composed from.
func seatTrafficObjects(ctx context.Context, inv *inventory.Inventory) ([]broker.Stream, []broker.Consumer, error) {
records, err := inv.BusRecords(ctx)
if err != nil {
return nil, nil, err
}
users, err := broker.Users(records)
if err != nil {
return nil, nil, err
}
streams, workers := broker.SeatTrafficObjects(users)
// And the queue of every such seat the catalogue declares, held or not: work queues from registration,
// so what is submitted before a holder is assigned waits for it (the correctness review of 2026-10-08).
declared, err := inv.DeclaredTrafficSeats(ctx)
if err != nil {
return nil, nil, err
}
have := map[string]bool{}
for _, s := range streams {
have[s.Name] = true
}
for _, s := range broker.TrafficQueues(declared) {
if !have[s.Name] {
streams = append(streams, s)
have[s.Name] = true
}
}
return streams, workers, nil
}
// moduleConsumerCount is how many modules hear what they consume, for the raise's one line.
func moduleConsumerCount(ctx context.Context, inv *inventory.Inventory) (int, error) {
consumers, err := moduleConsumers(ctx, inv)
-2
View File
@@ -37,8 +37,6 @@ func TestAPushAnswersBeforeItSends(t *testing.T) {
}{
{"push", map[string]any{"node": "anchor", "why": "w"}, true},
{"push", map[string]any{"why": "w"}, true},
{"command", map[string]any{"command": "push anchor --why w"}, true},
{"command", map[string]any{"command": "push --behind --why=w"}, true},
{"command", map[string]any{"command": "builds"}, false},
{"status", map[string]any{}, false},
{"assign", map[string]any{"node": "anchor", "module": "m"}, false},
+15 -2
View File
@@ -5,6 +5,7 @@ import (
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -50,7 +51,13 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
}
}
switch {
case name == "nats":
case (name == "nats" || catalogue.ProvidesBus(e.Manifest)) && len(e.On) > 0:
// Said before the merge (novox/hq issue 336): a new bus build holds every send to the bus's machine
// until a person takes the step, so merging it is a promise to take it.
p.Steps = append(p.Steps, fmt.Sprintf("%s: %s is never sent by an ordinary send, and until %s runs, "+
"nothing else is sent to %s either — unless the new build turns out the same as the one running there "+
"(issue 280)", busUpgradeNeeded, name, busUpgradeVerb, strings.Join(e.On, ", ")))
case name == "nats" || catalogue.ProvidesBus(e.Manifest):
p.Steps = append(p.Steps, "a planned bus step: the bus is upgraded by `bus upgrade`, never by an ordinary send")
case waits && len(e.On) > 0:
p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s)", name,
@@ -81,6 +88,9 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
return p
}
// busUpgradeNeeded is how a change plan says that merging it holds the bus's machine for a person's step.
const busUpgradeNeeded = "merging this needs a person's bus upgrade"
// summaryOf is a change plan in one line: what it builds, where it goes, and whether the bus moves.
func summaryOf(p link.ChangePlan) string {
if len(p.Moved) == 0 && len(p.New) == 0 {
@@ -110,7 +120,10 @@ func summaryOf(p link.ChangePlan) string {
}
bus := "no bus step"
for _, s := range p.Steps {
if strings.HasPrefix(s, "a planned bus step") {
switch {
case strings.HasPrefix(s, busUpgradeNeeded):
bus = busUpgradeNeeded
case strings.HasPrefix(s, "a planned bus step") && bus == "no bus step":
bus = "a bus step"
}
}
+6 -2
View File
@@ -45,7 +45,7 @@ func TestAChangePlanSaysWhatEachMachineReceives(t *testing.T) {
}
p = plan("modules/nats/Dockerfile", "modules/photos/x.js")
if !strings.Contains(p.Summary, "a bus step") || !strings.Contains(p.Summary, "2 wait(s) for a person") ||
if !strings.Contains(p.Summary, "needs a person's bus upgrade") || !strings.Contains(p.Summary, "2 wait(s) for a person") ||
!strings.Contains(p.Summary, "(+1 dependent(s))") {
t.Errorf("the bus and a held module read %q", p.Summary)
}
@@ -58,7 +58,11 @@ func TestAChangePlanSaysWhatEachMachineReceives(t *testing.T) {
t.Errorf("the deploy plan reads %v", got)
}
text := strings.Join(p.Steps, "\n")
for _, want := range []string{"a planned bus step", "photos waits for a person", "nats provides mesh-bus"} {
// Said before the merge (novox/hq issue 336): merging it holds every send to the bus's machine until a
// person runs the bus's step, and the verb that does.
for _, want := range []string{"merging this needs a person's bus upgrade", "nothing else is sent to anchor",
"mesh_call mesh-controller.bus", "the same as the one running there", "photos waits for a person",
"nats provides mesh-bus"} {
if !strings.Contains(text, want) {
t.Errorf("the steps do not say %q:\n%s", want, text)
}
+40 -1
View File
@@ -32,6 +32,9 @@ import (
// provision it wants that a manifest given here offers. An overflow is refused in the pull request
// that introduces it — a new requirement, a lowered bound, a longer slug — instead of on the
// provider's machine when a real machine's name first meets the module's.
// SomeManifestsVar, set by the merge gate, says the manifests given are only some of their repository's.
const SomeManifestsVar = "MESH_MODULE_CHECK_SOME"
func moduleCheck(paths []string, out io.Writer) error {
return moduleCheckFor(paths, catalogue.DefaultLongestMachine, out)
}
@@ -65,6 +68,13 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
}
// A definition names no installation (novox/hq ADR 0112, ADR 0155): judged here, in the
// catalogue-wide test, and at registration, which refuses in the same words.
if wrong := catalogue.TrustProblems(m); len(wrong) > 0 {
for _, p := range wrong {
fmt.Fprintf(out, "%s: %s\n", path, p)
}
failed += len(wrong)
faulted[m.Module] = true
}
if named := catalogue.InstallationProblems(m); len(named) > 0 {
for _, p := range named {
fmt.Fprintf(out, "%s: %s\n", path, p)
@@ -78,12 +88,24 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
// Between the manifests: a seat declared twice, a use of a seat nothing declares, a claim on
// a seat that does not exist. Run only over what parsed, because a problem inside one manifest
// has already been said and would be said again here in a worse form.
//
// **Over some of a repository's manifests, a seat none of them declares is a note** (novox/hq issue 364), as
// this command's own word says above: the merge gate passes only the manifests a change touches, and says so
// with SomeManifestsVar, so a module that uses or claims a seat another module declares (the operator
// channel's, a channel bench) was refused there for a manifest it was not given. Given every manifest — the
// catalogue's own check, and registration — it stays a refusal.
some := os.Getenv(SomeManifestsVar) != ""
problems := catalogue.CatalogueProblems(shelf)
sort.Strings(problems)
for _, p := range problems {
if some && catalogue.IsUndeclaredSeat(p) {
fmt.Fprintf(out, "note: %s among the manifests given; registration judges it against the whole catalogue, "+
"and passing the declaring module's manifest too judges it here\n", p)
continue
}
fmt.Fprintln(out, p)
failed++
}
failed += len(problems)
// Between the manifests too: an identity against the bounds of the provisions it wants, which
// only the provider's manifest states.
@@ -130,6 +152,13 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
}
}
// **A file that asks for a setting and does not say whether it is trusted counts as trusted** (novox/hq issue
// 339): listed and counted, never refused, so an author can opt out a file nothing trusts.
unsaid := 0
for _, name := range names {
unsaid += len(catalogue.UnsaidTrust(shelf[name]))
}
for _, name := range names {
m := shelf[name]
if faulted[name] {
@@ -171,6 +200,11 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
if len(checks) > 0 {
fmt.Fprintf(out, ", ready: %s", strings.Join(checks, "; "))
}
if missing := catalogue.UnsaidTrust(m); len(missing) > 0 {
fmt.Fprintf(out, "; WARNING: %s ask(s) for a setting and do(es) not say whether it is trusted, so it counts as "+
"trusted: set at the terminal alone; say %q false where nothing trusts it (novox/hq issue 339)",
strings.Join(missing, ", "), catalogue.TrustedField)
}
if missing := catalogue.Undeclared(m); len(missing) > 0 {
fmt.Fprintf(out, "; WARNING: %s stay(s) up and say(s) not how it is ready — judged by liveness alone, "+
"refused from %s (ADR 0240 rule 8)", strings.Join(missing, ", "), catalogue.HealthRequiredFrom.Format("2006-01-02"))
@@ -179,6 +213,7 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
}
// The count the catalogue keeps (ADR 0240 rule 8), in a line its merge check reads.
fmt.Fprintf(out, "%s %d\n", UndeclaredHealthLine, undeclared)
fmt.Fprintf(out, "%s %d\n", UnsaidTrustLine, unsaid)
if failed > 0 {
return fmt.Errorf("%d problem(s) in %d manifest(s)", failed, len(paths))
}
@@ -193,6 +228,10 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
// `health` in, over the manifests given: the catalogue's merge check compares it with the number it keeps.
const UndeclaredHealthLine = "long-running resources without health:"
// UnsaidTrustLine starts the line `module check` says the count of files that ask for a setting and do not say
// whether it is trusted, and so count as trusted (novox/hq issue 339).
const UnsaidTrustLine = "files asking for a setting without saying whether it is trusted:"
// checkNow is the clock `module check` judges the date by; a test sets it.
var checkNow = time.Now
+3 -1
View File
@@ -82,7 +82,9 @@ func checkHereCommand(ctx context.Context, args []string) error {
if _, err := git("fetch", "--quiet", "origin", *base); err != nil {
return fmt.Errorf("cannot fetch %s to say what the change touches: %w", *base, err)
}
changedText, err := git("diff", "--name-only", "origin/"+*base+"...HEAD")
// Without rename detection, so a file moved out of a build source is said under its old name as well:
// its going is a change to the build that held it (novox/hq ADR 0267).
changedText, err := git("diff", "--name-only", "--no-renames", "origin/"+*base+"...HEAD")
if err != nil {
return err
}
+20
View File
@@ -103,3 +103,23 @@ func TestTheControllersManifestServesEveryVerbOfItsSeat(t *testing.T) {
t.Fatalf("the controller's own module.json fails module check: %v\n%s", err, out.String())
}
}
// A module that uses a seat another module declares (novox/hq issue 364): refused over the whole catalogue when the
// declarer is missing, a note when the gate says it gives only the manifests a change touches.
func TestASeatAnotherModuleDeclaresIsANoteOverSomeManifests(t *testing.T) {
dir := t.TempDir()
user := filepath.Join(dir, "user.json")
os.WriteFile(user, []byte(`{"module":"asker","version":"1","uses":["operator-channel"]}`), 0o600)
var out bytes.Buffer
if err := moduleCheck([]string{user}, &out); err == nil {
t.Fatalf("a use of a seat nothing given declares passed the whole-catalogue check:\n%s", out.String())
}
t.Setenv(SomeManifestsVar, "1")
out.Reset()
if err := moduleCheck([]string{user}, &out); err != nil {
t.Fatalf("over some manifests the use was refused:\n%s", out.String())
}
if !strings.Contains(out.String(), "note: asker uses the seat \"operator-channel\"") {
t.Fatalf("the note was not said:\n%s", out.String())
}
}
+1 -1
View File
@@ -143,7 +143,7 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
if err != nil {
return err
}
read, err := inv.ReadRepositories(ctx)
read, err := readForPlanning(ctx, inv)
if err != nil {
return err
}
+30 -26
View File
@@ -5,6 +5,7 @@ import (
"errors"
"flag"
"fmt"
"io"
"os"
"slices"
"strconv"
@@ -46,7 +47,7 @@ func keeperOn(ctx context.Context, conn *nats.Conn) (*conditions.Keeper, error)
Say: func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) },
// What status leads with changed: composed again soon (a nudge outside the serving controller
// does nothing).
Changed: statusFrom.nudge,
Changed: func() { statusFrom.nudge(); askerFrom.nudge() },
// Written under the lease, carrying its epoch (novox/hq to-be 45 §6).
Epoch: func() (uint64, error) { return theLease.epoch(context.WithoutCancel(ctx)) }}), nil
}
@@ -107,19 +108,20 @@ func conditionsCommand(ctx context.Context, args []string) error {
}
switch sub {
case "list":
return listConditions(ctx, args)
return listConditions(ctx, args, os.Stdout)
case "show":
return showCondition(ctx, args)
return showCondition(ctx, args, os.Stdout)
case "silence":
return silenceCondition(ctx, args)
case "history":
return conditionHistory(ctx, args)
return conditionHistory(ctx, args, os.Stdout)
}
return errors.New(conditionsUsage)
}
func listConditions(ctx context.Context, args []string) error {
func listConditions(ctx context.Context, args []string, w io.Writer) error {
set := flag.NewFlagSet("conditions", flag.ContinueOnError)
usageTo(set, w)
scope := set.String("scope", "", "only this scope: "+strings.Join(conditions.Scopes, ", "))
severity := set.String("severity", "", "only urgent, or only warning")
machine := set.String("machine", "", "only those about this machine")
@@ -144,20 +146,20 @@ func listConditions(ctx context.Context, args []string) error {
}
}
if *asJSON {
return printJSON(map[string]any{"conditions": inBrief(out), "open": len(open), "counted": counted(out),
return printJSONTo(w, map[string]any{"conditions": inBrief(out), "open": len(open), "counted": counted(out),
"note": "urgent first, then oldest first; a condition clears when observation says so, never by hand; " +
"each with its newest evidence — `conditions key=<key>` gives one whole"})
}
if len(out) == 0 {
if len(open) == 0 {
fmt.Println("no open conditions")
fmt.Fprintln(w, "no open conditions")
} else {
fmt.Printf("none of the %d open condition(s) is about that\n", len(open))
fmt.Fprintf(w, "none of the %d open condition(s) is about that\n", len(open))
}
return nil
}
for _, line := range conditionLines(out, time.Now()) {
fmt.Println(line)
fmt.Fprintln(w, line)
}
return nil
}
@@ -233,8 +235,9 @@ func conditionLines(list []conditions.Condition, now time.Time) []string {
return out
}
func showCondition(ctx context.Context, args []string) error {
func showCondition(ctx context.Context, args []string, w io.Writer) error {
set := flag.NewFlagSet("conditions show", flag.ContinueOnError)
usageTo(set, w)
asJSON := set.Bool("json", false, "as data")
rest, err := parseAround(set, args)
if err != nil {
@@ -266,34 +269,34 @@ func showCondition(ctx context.Context, args []string) error {
"history --key %s` what became of it", key, key)
}
if *asJSON {
return printJSON(c)
return printJSONTo(w, c)
}
now := time.Now()
fmt.Printf("%s %s\n %s\n\n", strings.ToUpper(string(c.Severity)), c.Key, c.Summary)
fmt.Printf(" kind %s\n about %s %s", c.Kind, c.Subject.Scope, c.Subject.ID)
fmt.Fprintf(w, "%s %s\n %s\n\n", strings.ToUpper(string(c.Severity)), c.Key, c.Summary)
fmt.Fprintf(w, " kind %s\n about %s %s", c.Kind, c.Subject.Scope, c.Subject.ID)
if c.Subject.Machine != "" {
fmt.Printf(", on %s", c.Subject.Machine)
fmt.Fprintf(w, ", on %s", c.Subject.Machine)
}
fmt.Printf("\n raised by %s\n since %s (%s ago), observed %d time(s), last %s ago\n",
fmt.Fprintf(w, "\n raised by %s\n since %s (%s ago), observed %d time(s), last %s ago\n",
c.Source, c.Raised.Local().Format("2006-01-02 15:04:05"), roughly(now.Sub(c.Raised)), c.Observations,
now.Sub(c.LastObserved).Round(time.Second))
if c.Count > 1 {
fmt.Printf(" raised %d times, each within ten minutes of clearing\n", c.Count)
fmt.Fprintf(w, " raised %d times, each within ten minutes of clearing\n", c.Count)
}
fmt.Printf(" resolved by %s\n", resolverWords(c.Resolver))
fmt.Fprintf(w, " resolved by %s\n", resolverWords(c.Resolver))
if c.Silenced != nil {
fmt.Printf(" silenced until %s by %s: %s\n", c.Silenced.Until.Local().Format("2006-01-02 15:04"),
fmt.Fprintf(w, " silenced until %s by %s: %s\n", c.Silenced.Until.Local().Format("2006-01-02 15:04"),
c.Silenced.By, c.Silenced.Why)
}
if len(c.Tried) > 0 {
fmt.Println("\n tried:")
fmt.Fprintln(w, "\n tried:")
for _, t := range c.Tried {
fmt.Printf(" %s %s — %s: %s\n", t.At.Local().Format("2006-01-02 15:04"), orHealer(t.By), t.What, t.Outcome)
fmt.Fprintf(w, " %s %s — %s: %s\n", t.At.Local().Format("2006-01-02 15:04"), orHealer(t.By), t.What, t.Outcome)
}
}
fmt.Println("\n evidence, newest first:")
fmt.Fprintln(w, "\n evidence, newest first:")
for _, e := range c.Evidence {
fmt.Printf(" %s %s\n", e.At.Local().Format("2006-01-02 15:04:05"), e.Said)
fmt.Fprintf(w, " %s %s\n", e.At.Local().Format("2006-01-02 15:04:05"), e.Said)
}
return nil
}
@@ -378,8 +381,9 @@ func parseFor(s string) (time.Duration, error) {
return d, nil
}
func conditionHistory(ctx context.Context, args []string) error {
func conditionHistory(ctx context.Context, args []string, w io.Writer) error {
set := flag.NewFlagSet("conditions history", flag.ContinueOnError)
usageTo(set, w)
days := set.Int("days", 7, "how many days back, at most 90")
key := set.String("key", "", "only this condition")
asJSON := set.Bool("json", false, "as data")
@@ -420,10 +424,10 @@ func conditionHistory(ctx context.Context, args []string) error {
if out == nil {
out = []conditions.Event{}
}
return printJSON(map[string]any{"history": out, "days": *days})
return printJSONTo(w, map[string]any{"history": out, "days": *days})
}
if len(out) == 0 {
fmt.Printf("nothing was raised, changed or cleared in the last %d day(s)\n", *days)
fmt.Fprintf(w, "nothing was raised, changed or cleared in the last %d day(s)\n", *days)
return nil
}
for _, e := range out {
@@ -440,7 +444,7 @@ func conditionHistory(ctx context.Context, args []string) error {
line += " — " + e.Why
}
}
fmt.Println(line)
fmt.Fprintln(w, line)
}
return nil
}
+2 -2
View File
@@ -86,7 +86,7 @@ func TestASilenceThroughTheVerbHoldsAndTheConditionStaysOpen(t *testing.T) {
func TestUnreadableConditionsAreNotAWellMesh(t *testing.T) {
open := aMesh(t)
_, store := withConditionsInMemory(t)
store.Fail = errors.New("the bus is away")
store.SetFail(errors.New("the bus is away"))
asked, err := theThreeQuestions(t.Context(), open)
if err != nil {
t.Fatal(err)
@@ -98,7 +98,7 @@ func TestUnreadableConditionsAreNotAWellMesh(t *testing.T) {
if !strings.HasPrefix(said, "the open conditions could NOT be read") || strings.Contains(said, "no open conditions") {
t.Fatalf("%s", said)
}
store.Fail = nil
store.SetFail(nil)
asked, _ = theThreeQuestions(t.Context(), open)
said = printed(t, func() error { return printStatus(asked) })
if asked.well() && !strings.Contains(said, "no open conditions;") {
+162
View File
@@ -0,0 +1,162 @@
package main
import (
"context"
"errors"
"fmt"
"strconv"
"strings"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// What a consumer gave up on, answered by the serving controller (novox/hq issue 330).
//
// **In this process, on its own connection**: DEAD_LETTERS is read and changed on the bus, and the
// serving controller is already on it. A verb run as a fresh process would open a connection of its own
// for each call (novox/hq issue 327).
// busHandles are the serving controller's connection and JetStream handle.
type busHandles struct {
conn *nats.Conn
js nats.JetStreamContext
}
// defaultDeadLetters is how many the list says when not asked for more.
const defaultDeadLetters = 50
// causeDeadLetter is the cause a delivery or drop gives when the caller gives none.
const causeDeadLetter = "dead-letter"
// deadLettersAnswer is what `dead-letters` answers: the list, one whole, or what came of delivering one
// again or dropping it.
func deadLettersAnswer(ctx context.Context, a *verbArguments) (any, error) {
serving := servingBus.Load()
if serving == nil {
return nil, errors.New("this controller is not serving, so it does not read DEAD_LETTERS: ask again, " +
"and the serving controller answers")
}
on := &busHandles{conn: serving.Conn(), js: serving.Context()}
// The shape first, and every argument it reads; one given beside it is refused before anything is
// done, as every verb refuses what it would pass over (novox/hq issue 244).
var deliver, drop, why, cause, idText, consumer, limit string
switch {
case a.given["deliver"] != "" || a.given["drop"] != "":
deliver, drop, why, cause = a.str("deliver"), a.str("drop"), a.str("why"), a.str("cause")
case a.given["id"] != "":
idText = a.str("id")
default:
consumer, limit = a.str("consumer"), a.str("limit")
}
if unused := a.unused(); len(unused) > 0 {
return nil, fmt.Errorf("dead-letters did not use %s together with %s, and an argument a verb would pass "+
"over is refused: nothing was done", quoteAll(unused), quoteAll(a.usedGiven()))
}
switch {
case deliver != "" && drop != "":
return nil, errors.New("dead-letters delivers one again or drops one, not both. Nothing was done")
case deliver != "" || drop != "":
act, text := "deliver", deliver
if drop != "" {
act, text = "drop", drop
}
if strings.TrimSpace(why) == "" {
return nil, fmt.Errorf("dead-letters %s is a hand act, and says why: why is required and recorded in "+
"the hand-act log (novox/hq to-be 45 §7). Nothing was done", act)
}
id, err := deadLetterID(text)
if err != nil {
return nil, err
}
return actOnDeadLetter(ctx, on, act, id, why, cause)
case idText != "":
id, err := deadLetterID(idText)
if err != nil {
return nil, err
}
return link.DeadLetterNamed(on.js, id)
}
most := defaultDeadLetters
if limit != "" {
n, err := strconv.Atoi(limit)
if err != nil || n <= 0 {
return nil, fmt.Errorf("limit is a number of dead letters, not %q", limit)
}
most = n
}
held, total, err := link.DeadLetters(on.js, consumer, most)
if err != nil {
return nil, err
}
answer := map[string]any{"dead_letters": held, "held": total,
"note": "newest first; with id, one whole; deliver or drop one with why"}
if total == 0 {
answer["note"] = "no consumer gave up on a message that is still kept"
}
return answer, nil
}
// deadLetterID is a dead letter's id as a caller wrote it.
func deadLetterID(text string) (uint64, error) {
id, err := strconv.ParseUint(strings.TrimSpace(text), 10, 64)
if err != nil || id == 0 {
return 0, fmt.Errorf("a dead letter's id is its number in %s, as dead-letters lists it, not %q",
broker.DeadLettersStream, text)
}
return id, nil
}
// actOnDeadLetter delivers one again or drops it, recorded in the hand-act log before it is done. A log
// that cannot be written is said, and the act still happens: the log is never the reason a person's act
// is refused (handacts.go).
func actOnDeadLetter(ctx context.Context, on *busHandles, act string, id uint64, why, cause string) (any, error) {
d, err := link.DeadLetterNamed(on.js, id)
if err != nil {
return nil, err
}
if act == "deliver" {
// Refused before it is recorded: an act that cannot be done is not an act.
if _, err := link.AgainTo(on.js, d); err != nil {
return nil, err
}
}
if cause == "" {
cause = causeDeadLetter
}
by := link.CallerIn(ctx)
if by == "" {
by = "a seat call whose caller the bus did not name"
}
answer := map[string]any{"dead_letter": d.ID, "consumer": d.Who, "subject": d.Subject}
recorded, logErr := link.RecordHandAct(ctx, on.conn, link.HandAct{Verb: "dead-letters " + act,
Args: []string{strconv.FormatUint(id, 10), d.Stream + "." + d.Consumer}, Why: why, Cause: cause,
Condition: conditions.Key(conditions.ScopeBus, d.Stream+"."+d.Consumer, link.AdvisoryMaxDeliveries),
By: by + ", through the " + catalogue.ControllerSeatName + " seat"})
if logErr != nil {
answer["unrecorded"] = "the hand-act log could not be written, and the act was done all the same: " + logErr.Error()
} else {
answer["recorded"] = recorded.ID
}
switch act {
case "deliver":
_, to, err := link.DeliverAgain(on.js, id)
if err != nil {
return nil, err
}
answer["delivered_on"] = to
answer["done"] = fmt.Sprintf("dead letter %d was delivered again to %s, and nobody else; it is no longer kept",
id, consumerWho(d.Stream, d.Consumer))
case "drop":
if _, err := link.DropDeadLetter(on.js, id); err != nil {
return nil, err
}
answer["done"] = fmt.Sprintf("dead letter %d, which %s gave up on, was dropped for good", id,
consumerWho(d.Stream, d.Consumer))
}
return answer, nil
}
+132
View File
@@ -0,0 +1,132 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// The serving controller's bus, with the mesh's streams, for the verb to read and act on.
func servingDeadLetters(t *testing.T) *broker.JetStream {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
if err := js.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
before := servingBus.Load()
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(before) })
return js
}
func askDeadLetters(t *testing.T, args map[string]any) (any, error) {
t.Helper()
a, err := readArguments("dead-letters", args)
if err != nil {
return nil, err
}
return deadLettersAnswer(context.Background(), a)
}
func TestDeadLettersListsDropsAndRecordsWhy(t *testing.T) {
js := servingDeadLetters(t)
if _, err := js.Context().Publish("mesh.mod.gitea.event.pull.merged", []byte(`{"n":1}`)); err != nil {
t.Fatal(err)
}
d, err := link.KeepDeadLetter(js.Context(), []byte(`{"stream":"EVENTS","consumer":"media_sonarr","stream_seq":1,"deliveries":5}`))
if err != nil {
t.Fatal(err)
}
answer, err := askDeadLetters(t, map[string]any{})
if err != nil {
t.Fatal(err)
}
listed := answer.(map[string]any)
if listed["held"] != 1 || len(listed["dead_letters"].([]link.DeadLetter)) != 1 {
t.Fatalf("listed %v", listed)
}
// Refused before anything is done: an act without why, an argument the shape passes over, both acts.
for _, args := range []map[string]any{
{"drop": "1"},
{"drop": "1", "why": "x", "limit": "3"},
{"drop": "1", "deliver": "1", "why": "x"},
{"why": "x"},
{"id": "nought"},
} {
if _, err := askDeadLetters(t, args); err == nil {
t.Errorf("%v was done", args)
}
}
if _, total, _ := link.DeadLetters(js.Context(), "", 0); total != 1 {
t.Fatalf("a refused call changed what is kept: %d left", total)
}
done, err := askDeadLetters(t, map[string]any{"drop": "1", "why": "the media server took the download in by hand"})
if err != nil {
t.Fatal(err)
}
if said := done.(map[string]any); said["recorded"] == nil || !strings.Contains(said["done"].(string), "dropped") {
t.Fatalf("answered %v", said)
}
acts, err := link.HandActs(context.Background(), js.Conn(), time.Now().Add(-time.Minute))
if err != nil || len(acts) != 1 || acts[0].Verb != "dead-letters drop" || acts[0].Cause != causeDeadLetter ||
!strings.Contains(acts[0].Condition, "media_sonarr") {
t.Fatalf("recorded %+v (%v)", acts, err)
}
if !personsDecision(acts[0]) {
t.Error("dropping a dead letter is counted as a repair, so S15 would want a healer for it")
}
if _, err := askDeadLetters(t, map[string]any{"id": "1"}); err == nil {
t.Errorf("dead letter %d is still answered after it was dropped", d.ID)
}
}
// Open while DEAD_LETTERS holds a message for the consumer, in words the operator reads in one pass:
// what is held, and where to act.
func TestAConsumersDeadLettersAreSaidUntilActedOn(t *testing.T) {
f := &signalFacts{now: time.Now(), deadLetters: map[string]int{"EVENTS.media_sonarr": 4, "EVENTS.controller": 1}}
found := watchDeadLetters(f)
if len(found) != 2 {
t.Fatalf("said %d conditions", len(found))
}
for _, o := range found {
if o.Kind != "max-deliveries" || o.Severity != conditions.Warning || o.Needs == "" {
t.Errorf("%+v", o)
}
if why, ok := conditions.PlainWords(conditions.Words{Headline: o.Headline, Needs: o.Needs,
Explanation: o.Explanation, Resolved: o.Resolved}, "media"); !ok {
t.Errorf("%q is not plain: %s", o.Headline, why)
}
if !strings.Contains(o.Needs, "mesh MCP server") {
t.Errorf("does not say where to act: %q", o.Needs)
}
}
sonarr := found[1]
if sonarr.ID != "EVENTS.media_sonarr" || sonarr.Machine != "media" ||
sonarr.Headline != "Sonarr on media could not handle 4 messages" ||
!strings.Contains(sonarr.Summary, "DEAD_LETTERS") {
t.Errorf("%+v", sonarr)
}
if found[0].Headline != "The controller could not handle a message" {
t.Errorf("%q", found[0].Headline)
}
// None held, none said: it clears when they are delivered again or dropped.
if left := watchDeadLetters(&signalFacts{now: time.Now()}); len(left) != 0 {
t.Fatalf("%v", left)
}
}
+9 -1
View File
@@ -103,6 +103,8 @@ func letGo(ctx context.Context, inv *inventory.Inventory, id, by, why string) (i
return p, fmt.Errorf("%s was let go by %s at %s already", p.ID, p.Delivery.By,
p.Delivery.Go.Local().Format("15:04:05"))
}
// **One walk at a time** (novox/hq ADR 0276): the word is taken, and the walk starts once no other walk is
// started (advanceOnce), so the delivery's owner says it once and is not refused.
now := time.Now().UTC()
p.Delivery.Go, p.Delivery.By, p.Delivery.Why = &now, by, why
p.Note = "let go by " + by + "; its first tier is asked next"
@@ -574,6 +576,12 @@ func deliveryCommand(ctx context.Context, args []string) error {
if walks, err = inv.OpenPlans(ctx); err != nil {
return err
}
// And the batch being assembled (novox/hq ADR 0276): never a walk to link or let go, shown.
batches, err := inv.Batches(ctx)
if err != nil {
return err
}
walks = append(walks, batches...)
recent, err := inv.RecentPlans(ctx, *limit)
if err != nil {
return err
@@ -604,7 +612,7 @@ func theGraph(ctx context.Context, inv *inventory.Inventory) ([]inventory.Entry,
if err != nil {
return nil, nil, nil, err
}
read, err := inv.ReadRepositories(ctx)
read, err := readForPlanning(ctx, inv)
if err != nil {
return nil, nil, nil, err
}
+8 -6
View File
@@ -33,12 +33,14 @@ var deliveryOwnerWithin = 10 * time.Second
// stalledLine is one delivery past its bound, as mesh-delivery's `stalled` says it.
type stalledLine struct {
ID string `json:"id"`
State string `json:"state"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
ID string `json:"id"`
// Number is the pull request's, for a line of a head the forge never announced (novox/hq issue 347).
Number int `json:"number,omitempty"`
State string `json:"state"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
}
// operatorsOnly is whether the table leaves H2 nothing to do for the line: the state is the operator's.
@@ -136,12 +136,13 @@ func TestTheDeliveryOwnerIsAskedOverTheBus(t *testing.T) {
if err != nil {
t.Fatal(err)
}
for _, verb := range []string{"stalled", "close"} {
// And release and stop, which the operator's warrant chooses (novox/hq ADR 0259).
for _, verb := range []string{"stalled", "close", "release", "stop"} {
if !slices.Contains(granted.Publish, link.SeatToolSubject(catalogue.DeliverySeat, verb)) {
t.Errorf("the controller may not ask %s.%s", catalogue.DeliverySeat, verb)
}
}
if _, err := askDeliveryOwner(t.Context(), nil, "stop", nil); err == nil || !strings.Contains(err.Error(), "grant") {
if _, err := askDeliveryOwner(t.Context(), nil, "retire-history", nil); err == nil || !strings.Contains(err.Error(), "grant") {
t.Fatalf("a verb the grant does not name was asked: %v", err)
}
conn, err := nats.Connect(testbus.URL(t))
+8 -5
View File
@@ -227,7 +227,9 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
t.Fatalf("with no holder on record the walk waited (%v) or asked %v", p.Waiting(), *asked)
}
// The holder on record: the next merge waits, asking nothing, and an advance asks nothing either.
// The holder on record: the next merge waits, asking nothing, and an advance asks nothing either. One walk
// is open at a time (novox/hq ADR 0276): the first ends before the next merge's batch is cut.
finish(p.ID)
if _, err := inv.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
@@ -259,10 +261,11 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
t.Fatalf("the word was not kept: %+v %v", got.Delivery, err)
}
// The delivery's owner's own merge never waits for it — and takes over what the older walk had not
// built (app, folded in: ADR 0218), which goes with it on the controller's own path.
// The delivery's owner's own merge never waits for it. A walk that started is never taken over (novox/hq
// ADR 0276): the merge's batch is cut once it ended.
finish(p.ID)
p = merge("c3cccccccc", "modules/mesh-delivery/main.go")
if p.Waiting() || len(*asked) != 4 {
if p.Waiting() || len(*asked) != 3 {
t.Fatalf("mesh-delivery's own walk waited for mesh-delivery: %v %v", p.Waiting(), *asked)
}
@@ -280,7 +283,7 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
}
advanceHeld(ctx, open)
got, _ = inv.PlanByID(ctx, p.ID)
if got.Waiting() || !strings.HasPrefix(got.Delivery.By, "a person") || len(*asked) < 5 {
if got.Waiting() || !strings.HasPrefix(got.Delivery.By, "a person") || len(*asked) < 4 {
t.Fatalf("a person's word did not start the walk: %+v, asked %v", got.Delivery, *asked)
}
+333
View File
@@ -0,0 +1,333 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/secrets"
)
// A module's own secret given at the operator's desk (novox/hq ADR 0259 §10, ADR 0277).
//
// mesh-controller secret ask <node> <module> <name> [--at <desk>]
//
// **The value never passes through whoever asked for it.** An agent, or the operator at the mesh MCP
// server, calls `secret-ask` (or `give`) with the machine, the module, the secret's name and the desk — never
// a value. The controller makes a sealing keypair for this one call, asks the desk's `node-launcher.secret` to
// prompt the operator without showing what is typed, and is answered with what was typed **sealed to that
// key**: no plaintext on the bus, in a runtime's log or in any call's record. It opens it here, seals it to the
// module's machine exactly as `secret accept` does, and forgets it. What it answers says only that the
// value was taken, or why not.
//
// **Bounded, and the prompt says who asked** (ADR 0277): one open prompt per secret and few an hour, read from
// the store before the prompt opens (inventory.OpenSecretAsk), so an agent cannot keep a prompt in front of the
// operator until they type. The prompt names the module, the secret, the machine and who asked — the caller as
// the bus named it, never a word the caller chose — written by the desk's launcher from those names alone.
//
// **What remains** (ADR 0234's accepted residual risk): on an X11 desk any program of the operator's
// account can read the keys as they are typed. And a program that calls the desk's prompt itself, with a
// key of its own, is answered with what the operator typed into a prompt they did not ask for — as it could
// draw a window of its own. The prompt says who asks and for what, so the operator types only into a
// prompt they started.
// deskPromptWithin is how long the prompt waits for the operator: below the runtime's thirty seconds for
// one call, as the launcher's menu is.
const deskPromptWithin = 25
// deskGive is the desk path, its four reaches given so a test needs no store and no bus.
type deskGive struct {
// declares refuses a module or a secret the mesh would refuse, before anybody is asked to type.
declares func(module, name string) error
// known refuses a machine the mesh does not know, before anybody is asked to type; nil knows every one
// (a test that does not look).
known func(machine string) error
// trusted says a module runs as an account of its own: its secret is never taken at a desk (below). Nil is
// never (a test that does not look).
trusted func(module string) (bool, error)
// ask asks one machine's node-launcher.secret and answers its result, or the holder's refusal.
ask func(machine string, args map[string]any) (json.RawMessage, error)
// accept seals the value as `secret accept` does, and says whether it lives until the module's start.
accept func(value string) (untilStart bool, err error)
// record writes the act in the hand-act log.
record func(link.HandAct) error
// announce raises the condition that says a module's own secret was given (secretGivenObservation), on
// every channel; nil announces nothing (a test that does not look).
announce func(node, module, name, how string) error
// askedBy is who asked, as the bus named the caller: said in the prompt and recorded.
askedBy string
// open records the ask and holds the bounds (one open per secret, few an hour), answering the record's id;
// nil keeps no record (a test that does not look). end closes it with how it ended.
open func(node, module, name, desk string) (int64, error)
end func(id int64, outcome string) error
}
// askedByName is the caller as the prompt names it: the first clause of what the bus said, in the characters
// a name has, at most 80 of them. The desk's launcher refuses anything else, so no words of the caller's own
// reach the prompt.
func askedByName(caller string) string {
first, _, _ := strings.Cut(caller, ",")
var b strings.Builder
for _, r := range strings.TrimSpace(first) {
switch {
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '.', r == '_', r == '/', r == '@',
r == '-', r == ' ':
b.WriteRune(r)
default:
b.WriteRune('-')
}
if b.Len() >= 80 {
break
}
}
name := strings.TrimSpace(b.String())
if name == "" || strings.HasPrefix(name, "-") {
return "an unnamed caller"
}
return name
}
// errNothingGiven is a prompt dismissed, or not answered in time: nothing changes.
var errNothingGiven = errors.New("nothing was given")
// give asks the desk for the value and seals it; it answers the words said to the caller.
func (d deskGive) give(node, module, name, desk string) (string, error) {
for what, v := range map[string]string{"the machine": node, "the module": module, "the secret": name, "the desk": desk} {
if strings.TrimSpace(v) == "" {
return "", fmt.Errorf("%s is not named", what)
}
}
if d.known != nil {
for what, machine := range map[string]string{"the machine the secret is for": node, "the desk": desk} {
if err := d.known(machine); err != nil {
return "", fmt.Errorf("nobody was asked to type anything: %s, %s, is not a machine this mesh knows: %w",
what, machine, err)
}
}
}
if err := d.declares(module, name); err != nil {
return "", fmt.Errorf("nobody was asked to type anything: %w", err)
}
// **A trusted party's secret is not taken at a desk** (the confirmation review of 2026-10-09, N1-give). The
// prompt's answer comes back over the desk machine's bus, on a subject that machine's runtime answers — and
// on a desk machine agents run as the operator, who holds that runtime's credential. An agent could answer
// first, with a bot token of its own sealed to this call's key, and the channel the operator's answers are
// proven on would be the agent's. So the value of a module running as its own account is typed at the
// controller's terminal, where no bus carries it.
if d.trusted != nil {
trusted, err := d.trusted(module)
if err != nil {
return "", fmt.Errorf("nobody was asked to type anything: whether %s is a trusted party could not be read: %w", module, err)
}
if trusted {
return "", fmt.Errorf("nobody was asked to type anything: %s runs as an account of its own and proves the "+
"operator's answers, so its %s is given at the controller's terminal alone — there, run `mesh-controller "+
"secret accept %s %s %s` and type it at its prompt. A desk's prompt is answered over the desk machine's "+
"bus, where an agent may answer first (novox/hq ADR 0259 §10)", module, name, node, module, name)
}
}
// The bounds, read and kept before anybody is asked to type (novox/hq ADR 0277): one open prompt per secret,
// few an hour. How the ask ends is recorded whatever happens below.
outcome := "failed"
if d.open != nil {
id, err := d.open(node, module, name, desk)
if err != nil {
return "", fmt.Errorf("nobody was asked to type anything: %w", err)
}
defer func() {
if d.end != nil {
_ = d.end(id, outcome)
}
}()
}
public, private, err := secrets.Keypair()
if err != nil {
return "", fmt.Errorf("no key could be made to take the value: %w", err)
}
// By name, never by words: the holder writes the prompt from these, and says the controller asks, which
// the bus alone makes true (broker.ControllerOnly). Who asked is the bus's word on the caller, cut to a
// name's characters — never an argument of the call.
raw, err := d.ask(desk, map[string]any{
"module": module,
"secret": name,
"node": node,
"asked_by": askedByName(d.askedBy),
"seal_to": public,
"timeout_seconds": deskPromptWithin,
})
if err != nil {
outcome = "refused"
return "", fmt.Errorf("the desk on %s could not be asked: %w", desk, err)
}
var answer struct {
Sealed string `json:"sealed"`
Cancelled bool `json:"cancelled"`
TimedOut bool `json:"timed_out"`
}
if err := json.Unmarshal(raw, &answer); err != nil {
return "", fmt.Errorf("the desk on %s answered something that is not the prompt's answer", desk)
}
switch {
case answer.TimedOut:
outcome = "timed-out"
return "", fmt.Errorf("%w: the prompt on %s was not answered within %d seconds", errNothingGiven, desk, deskPromptWithin)
case answer.Cancelled:
outcome = "dismissed"
return "", fmt.Errorf("%w: the prompt on %s was dismissed", errNothingGiven, desk)
case answer.Sealed == "":
return "", fmt.Errorf("the desk on %s answered no sealed value", desk)
}
opened, err := secrets.Open(private, answer.Sealed)
if err != nil {
// Never the value, never what failed to open: only that it was not sealed to this call.
return "", fmt.Errorf("the desk on %s answered a value not sealed to this call; nothing was taken", desk)
}
value := asSupplied(string(opened))
for i := range opened {
opened[i] = 0
}
if strings.TrimSpace(value) == "" {
outcome = "empty"
return "", fmt.Errorf("%w: the prompt on %s was answered empty", errNothingGiven, desk)
}
untilStart, err := d.accept(value)
value = ""
if err != nil {
return "", err
}
outcome = "given"
act := link.HandAct{Verb: "secret accept", Args: []string{node, module, name, "--at-desk", desk},
Why: fmt.Sprintf("the operator gave %s for %s on %s at the desk on %s, asked by %s", name, module, node, desk,
askedByName(d.askedBy)),
Cause: "given-at-the-desk"}
recorded := ""
if err := d.record(act); err != nil {
recorded = fmt.Sprintf("\n this act could NOT be recorded in the hand-act log, and is done anyway: %v", err)
}
if d.announce != nil {
if err := d.announce(node, module, name, "at the desk on "+desk); err != nil {
recorded += fmt.Sprintf("\n this change could NOT be announced on the operator's channels: %v", err)
}
}
words := fmt.Sprintf("%s on %s now holds %q, given at the desk on %s and sealed to %s; the mesh cannot read it "+
"back.\n run `push %s` to send it", module, node, name, desk, node, node)
if untilStart {
words += fmt.Sprintf("\n it lives until %s next starts well under the mesh, and is then replaced with a value "+
"the mesh makes (ADR 0228)", module)
}
return words + recorded, nil
}
// askAtDesk is `secret ask <node> <module> <name> [--at <machine>]`, and the terminal's `secret accept … --at-desk
// <machine>`: the desk path, on this controller's stores and bus. The desk is the module's machine unless named.
func askAtDesk(ctx context.Context, node, module, name, desk string) error {
if desk == "" {
desk = node
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
d := deskGive{
askedBy: link.Caller(),
open: func(node, module, name, desk string) (int64, error) {
return open.inventory.OpenSecretAsk(ctx, node, module, name, askedByName(link.Caller()), desk)
},
end: func(id int64, outcome string) error { return open.inventory.EndSecretAsk(ctx, id, outcome) },
declares: func(module, name string) error { return open.inventory.DeclaresOwnSecret(ctx, module, name) },
known: func(machine string) error {
_, err := open.inventory.NodeByName(ctx, machine)
return err
},
trusted: func(module string) (bool, error) { return open.inventory.RunsAsItsOwnAccount(ctx, module) },
ask: func(machine string, args map[string]any) (json.RawMessage, error) {
var result json.RawMessage
err := onTheBus(func(conn *nats.Conn) error {
answer, err := link.AskSeatTool(ctx, conn, "node-launcher", "secret", machine, args,
time.Duration(deskPromptWithin+5)*time.Second)
if err != nil {
return err
}
if answer.Error != "" {
return errors.New(answer.Error)
}
result = answer.Result
return nil
})
return result, err
},
accept: func(value string) (bool, error) {
return open.inventory.AcceptGivenSecret(ctx, node, module, name, value)
},
record: func(act link.HandAct) error {
return onTheBus(func(conn *nats.Conn) error {
_, err := link.RecordHandAct(ctx, conn, act)
return err
})
},
announce: func(node, module, name, how string) error { return announceSecretGiven(ctx, node, module, name, how) },
}
words, err := d.give(node, module, name, desk)
if err != nil {
return err
}
fmt.Println(words)
return nil
}
// kindSecretGiven is the condition every value given for a module's own secret raises (the review of 2026-10-09,
// M4): on every channel, so a bot token changed by somebody else — a channel that now answers for them — is
// heard of. It stays until the operator silences or clears it.
const kindSecretGiven = "secret-given"
// secretGivenObservation is that condition: which secret, of which module on which machine, how and when.
// The summary, for whoever looks closer, names the secret and the time. The words the operator reads are
// held to the plain rule (conditions.PlainWords): no clock time — the channel says when, in the operator's
// time — and the secret's name said as words. Words that broke the rule were replaced by the keeper with
// "needs a look … a problem it calls secret given" (hq issue 359), which told the operator nothing.
func secretGivenObservation(node, module, name, how string, at time.Time) conditions.Observation {
key := node + "." + module + "." + name
where := module + " on " + node
// Within the bounds whatever the names' length: the module and machine, else the module, else the machine.
headline := "New secret given for " + where
for _, h := range []string{"New secret given for " + module, "New secret given on " + node} {
if len(headline) > conditions.HeadlineMax {
headline = h
}
}
resolved := "You saw that " + module + " was given a new secret"
if len(resolved) > conditions.HeadlineMax+20 {
resolved = "You saw that a new secret was given on " + node
}
return conditions.Observation{Scope: conditions.ScopeMachine, ID: key, Token: kindSecretGiven, Kind: kindSecretGiven,
Machine: node, Severity: conditions.Urgent, Source: kindSecretGiven,
Summary: fmt.Sprintf("%s of %s on %s was given %s at %s", name, module, node, how,
at.Local().Format("2006-01-02 15:04")),
Headline: headline,
Explanation: fmt.Sprintf("The secret %s of %s was given %s. If you gave it, nothing else is needed. If you "+
"did not, somebody else now holds what %s acts with.", secretNameWords(name), where, how, module),
Needs: "silence this if you just gave it; if you did not, give it again yourself so that only you hold it.",
Resolved: resolved,
Actions: []conditions.Action{conditions.SilenceAction(conditions.Key(conditions.ScopeMachine, key, kindSecretGiven))}}
}
// secretNameWords is a secret's name as the operator reads it: "telegram-token" is "telegram token".
func secretNameWords(name string) string {
return strings.Join(strings.FieldsFunc(name, func(r rune) bool { return r == '-' || r == '_' || r == '.' }), " ")
}
// announceSecretGiven raises it on this controller's keeper.
func announceSecretGiven(ctx context.Context, node, module, name, how string) error {
return withKeeper(ctx, func(k *conditions.Keeper) error {
_, err := k.Observe(ctx, secretGivenObservation(node, module, name, how, time.Now()))
return err
})
}
+512
View File
@@ -0,0 +1,512 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/secrets"
)
const typed = "123456789:AAEhBP0av28P4XFQnIuR-o-7Xnz1kkUzW3g"
// aDesk is the desk path with a prompt the test answers as the operator would, and what it was asked kept.
func aDesk(t *testing.T, answer func(args map[string]any) (json.RawMessage, error)) (deskGive, *[]string, *[]link.HandAct, *[]map[string]any) {
t.Helper()
var accepted []string
var acts []link.HandAct
var asked []map[string]any
return deskGive{
declares: func(module, name string) error {
if module != "telegram" || name != "telegram-token" {
return errors.New(module + " does not declare " + name + " as an own secret")
}
return nil
},
ask: func(machine string, args map[string]any) (json.RawMessage, error) {
asked = append(asked, args)
return answer(args)
},
accept: func(value string) (bool, error) { accepted = append(accepted, value); return false, nil },
record: func(a link.HandAct) error { acts = append(acts, a); return nil },
}, &accepted, &acts, &asked
}
func sealedTo(t *testing.T, value string) func(args map[string]any) (json.RawMessage, error) {
return func(args map[string]any) (json.RawMessage, error) {
sealed, err := secrets.Seal(args["seal_to"].(string), []byte(value+"\n"))
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(map[string]any{"sealed": sealed})
return raw, nil
}
}
// novox/hq ADR 0259 §10: the value typed at the desk is sealed as `secret accept` seals it, and is in no
// answer, no prompt argument and no act recorded.
func TestASecretGivenAtTheDeskIsSealedAndSaidNowhere(t *testing.T) {
d, accepted, acts, asked := aDesk(t, sealedTo(t, typed))
words, err := d.give("anchor", "telegram", "telegram-token", "laptop")
if err != nil {
t.Fatal(err)
}
if len(*accepted) != 1 || (*accepted)[0] != typed {
t.Fatalf("the value sealed is not what was typed, its line ending taken off")
}
if len(*acts) != 1 || (*acts)[0].Verb != "secret accept" || (*acts)[0].Cause != "given-at-the-desk" ||
!strings.Contains((*acts)[0].Why, "at the desk on laptop") {
t.Errorf("the act: %+v", *acts)
}
raw, _ := json.Marshal(struct {
Words string
Acts []link.HandAct
Asked []map[string]any
}{words, *acts, *asked})
if strings.Contains(string(raw), typed) || strings.Contains(string(raw), "AAEhBP0") {
t.Fatal("the value appears in what was said, asked or recorded")
}
if !strings.Contains(words, "push anchor") || !strings.Contains(words, "given at the desk on laptop") {
t.Errorf("%q", words)
}
if p := (*asked)[0]; p["seal_to"] == "" || p["timeout_seconds"] != deskPromptWithin {
t.Errorf("the prompt was asked %v", p)
}
}
func TestNothingIsAskedForASecretTheMeshWouldRefuse(t *testing.T) {
for _, c := range [][2]string{{"telegram", "chat-id"}, {"nobody", "telegram-token"}} {
d, accepted, _, asked := aDesk(t, sealedTo(t, typed))
if _, err := d.give("anchor", c[0], c[1], "laptop"); err == nil || !strings.Contains(err.Error(), "nobody was asked") {
t.Errorf("%v: %v", c, err)
}
if len(*asked) != 0 || len(*accepted) != 0 {
t.Errorf("%v: the operator was asked anyway", c)
}
}
d, _, _, _ := aDesk(t, sealedTo(t, typed))
if _, err := d.give("anchor", "telegram", "telegram-token", ""); err == nil {
t.Error("no desk was refused nowhere")
}
}
func TestADismissedEmptyLateOrForeignAnswerTakesNothing(t *testing.T) {
for want, answer := range map[string]func(map[string]any) (json.RawMessage, error){
"not answered within 25 seconds": func(map[string]any) (json.RawMessage, error) {
return json.RawMessage(`{"cancelled":true,"timed_out":true}`), nil
},
"was dismissed": func(map[string]any) (json.RawMessage, error) { return json.RawMessage(`{"cancelled":true}`), nil },
"answered empty": sealedTo(t, " "),
"not sealed to this call": func(map[string]any) (json.RawMessage, error) {
other, _, _ := secrets.Keypair()
sealed, _ := secrets.Seal(other, []byte(typed))
raw, _ := json.Marshal(map[string]any{"sealed": sealed})
return raw, nil
},
"could not be asked": func(map[string]any) (json.RawMessage, error) { return nil, errors.New("no session answers") },
} {
d, accepted, acts, _ := aDesk(t, answer)
_, err := d.give("anchor", "telegram", "telegram-token", "laptop")
if err == nil || !strings.Contains(err.Error(), want) || strings.Contains(err.Error(), typed) {
t.Errorf("want %q, got %v", want, err)
}
if len(*accepted) != 0 || len(*acts) != 0 {
t.Errorf("%s: something was taken or recorded", want)
}
}
}
func TestTheGiveVerbRunsTheDeskPathAndTheControllerMayAskTheDesk(t *testing.T) {
argv, err := argvFor("give", map[string]any{"node": "anchor", "module": "telegram", "secret": "telegram-token", "at": "laptop"})
if err != nil || strings.Join(argv, " ") != "secret ask anchor telegram telegram-token --at laptop" {
t.Fatalf("%v %v", argv, err)
}
if _, err := argvFor("give", map[string]any{"node": "anchor", "module": "telegram", "secret": "telegram-token"}); err == nil {
t.Error("give without a desk was taken")
}
// secret-ask is the same line, with the desk the module's machine unless named (novox/hq ADR 0277).
argv, err = argvFor("secret-ask", map[string]any{"node": "anchor", "module": "telegram", "secret": "telegram-token"})
if err != nil || strings.Join(argv, " ") != "secret ask anchor telegram telegram-token" {
t.Fatalf("%v %v", argv, err)
}
argv, err = argvFor("secret-ask", map[string]any{"node": "anchor", "module": "telegram", "secret": "telegram-token", "at": "laptop"})
if err != nil || strings.Join(argv, " ") != "secret ask anchor telegram telegram-token --at laptop" {
t.Fatalf("%v %v", argv, err)
}
perms, err := broker.PermissionsFor(broker.Principal{Kind: broker.KindController})
if err != nil {
t.Fatal(err)
}
found := false
for _, p := range perms.Publish {
found = found || p == "mesh.seat.node-launcher.tool.secret.*"
}
if !found {
t.Error("the controller may not ask the desk's prompt")
}
}
// The review of 2026-10-09 (M4): the desk's prompt says who asks in words the caller does not choose — the
// controller, which the bus alone lets ask it — and what for, from names the controller checked; the prompt
// carries no free text of the caller's.
func TestThePromptIsAskedByNameNeverByWordsTheCallerChose(t *testing.T) {
d, _, _, asked := aDesk(t, sealedTo(t, typed))
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
p := (*asked)[0]
if p["module"] != "telegram" || p["secret"] != "telegram-token" || p["node"] != "anchor" {
t.Errorf("the prompt was not asked by name: %v", p)
}
for _, free := range []string{"prompt", "message"} {
if _, there := p[free]; there {
t.Errorf("the prompt carries the caller's %s: %v", free, p)
}
}
}
// Every value given for a module's own secret is announced as a condition, on every channel (the review of
// 2026-10-09, M4): a bot token changed by somebody else is a channel that now answers for them.
func TestAValueGivenAtTheDeskIsAnnounced(t *testing.T) {
d, _, _, _ := aDesk(t, sealedTo(t, typed))
var said []string
d.announce = func(node, module, name, how string) error {
said = append(said, node+" "+module+" "+name+" "+how)
return nil
}
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
if len(said) != 1 || !strings.Contains(said[0], "anchor telegram telegram-token") || !strings.Contains(said[0], "laptop") {
t.Fatalf("announced %v", said)
}
o := secretGivenObservation("anchor", "telegram", "telegram-token", "at the desk on laptop", time.Date(2026, 10, 9, 12, 3, 0, 0, time.UTC))
if o.Severity != conditions.Urgent || !strings.Contains(o.Explanation, "telegram token") ||
len(o.Actions) == 0 || o.Key() == "" {
t.Errorf("the announcement %+v", o)
}
if strings.Contains(o.Summary+o.Explanation+o.Said, typed) {
t.Error("the announcement carries the value")
}
}
// **The secret-given condition says itself in its own plain words** (hq issue 359): the words it carries pass
// the plain rule, from the controller's terminal and from a desk, so the keeper keeps them — they once held a
// clock time, and the operator read "Novox needs a look … a problem it calls secret given" instead. Its
// severity and its answer stay: it is heard on every channel, and silenced by the operator.
func TestTheSecretGivenConditionSaysItselfInPlainWords(t *testing.T) {
at := time.Date(2026, 10, 9, 23, 32, 0, 0, time.Local)
for _, how := range []string{"at the controller's terminal", "at the desk on laptop"} {
o := secretGivenObservation("anchor", "telegram", "telegram-token", how, at)
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs,
Actions: o.Actions}
if why, ok := conditions.PlainWords(w, o.Machine); !ok {
t.Fatalf("given %s, the words are not plain: %s", how, why)
}
k, _ := withConditionsInMemory(t)
c, err := k.Observe(t.Context(), o)
if err != nil {
t.Fatal(err)
}
if c.Headline != "New secret given for telegram on anchor" || c.Severity != conditions.Urgent ||
!strings.HasPrefix(c.Explanation, conditions.NeedsYou+" silence this") ||
!strings.Contains(c.Explanation, "telegram token of telegram on anchor was given "+how) ||
len(c.Actions) != 1 || c.Actions[0].Label != "Silence for a week" {
t.Errorf("given %s, the keeper said %q / %q (%s, %v)", how, c.Headline, c.Explanation, c.Severity, c.Actions)
}
if strings.Contains(c.Headline+c.Explanation+c.Resolved+c.Needs, typed) {
t.Error("the words carry the value")
}
}
// A long module name keeps the headline and the resolved line within their bounds.
for _, module := range []string{"a-module-with-a-rather-long-name-indeed",
"a-module-with-a-name-so-long-that-no-headline-could-ever-hold-it"} {
o := secretGivenObservation("anchor", module, "api-key", "at the controller's terminal", at)
if why, ok := conditions.PlainWords(conditions.Words{Headline: o.Headline, Explanation: o.Explanation,
Resolved: o.Resolved, Needs: o.Needs, Actions: o.Actions}, o.Machine); !ok {
t.Errorf("the module %s: %s", module, why)
}
}
}
// The bus lets the controller alone ask the desk's prompt (the review of 2026-10-09, M4): the runtime, which
// carries every agent's calls, and a person granted every tool are denied it, however wide their grant.
func TestOnlyTheControllerMayAskTheDesksPrompt(t *testing.T) {
for _, p := range []broker.Principal{
{Kind: broker.KindNodeTools, Node: "laptop"},
{Kind: broker.KindPerson, Module: "operator", Invokes: []string{"*"}},
{Kind: broker.KindModule, Node: "laptop", Module: "lab", Invokes: []string{"seat:node-launcher.secret"}},
} {
perms, err := broker.PermissionsFor(p)
if err != nil {
t.Fatal(err)
}
for _, subject := range []string{"mesh.seat.node-launcher.tool.secret.laptop", "mesh.seat.node-launcher.tool.secret",
"mesh.mod.rofi.tool.node-launcher.secret", "mesh.mod.rofi.tool.node-launcher.secret.laptop"} {
if broker.MayPublish(perms, subject) {
t.Errorf("%s may publish %s", p.Username(), subject)
}
}
}
perms, _ := broker.PermissionsFor(broker.Principal{Kind: broker.KindController})
if !broker.MayPublish(perms, "mesh.seat.node-launcher.tool.secret.laptop") {
t.Error("the controller may not ask the desk's prompt")
}
}
// A value for a secret comes from the terminal or the desk, never through a verb (the review of 2026-10-09,
// M4): `secret accept` with a value, run for a verb, is refused before anything is read.
func TestASecretValueIsNeverAcceptedThroughAVerb(t *testing.T) {
t.Setenv(verbVar, "mesh-controller.command")
for _, args := range [][]string{
{"accept", "anchor", "telegram", "telegram-token", "--from", "/dev/null"},
{"accept", "anchor", "app", "db", "--from", "/dev/null", "--provider", "store"},
} {
err := secretCommand(context.Background(), args)
if err == nil || !strings.Contains(err.Error(), "never through a verb") {
t.Errorf("%v: %v", args, err)
}
}
}
// The `give` and `secret-ask` verbs' own line passes the terminal-only rule of ADR 0266, and no `secret accept`
// does: a value, a file, a provider or an extra word is still the terminal's alone (novox/hq ADR 0277).
func TestOnlyTheGiveLinePassesTheTerminalRuleForSecrets(t *testing.T) {
for _, argv := range [][]string{
{"secret", "ask", "anchor", "telegram", "telegram-token", "--at", "laptop"},
{"secret", "ask", "anchor", "telegram", "telegram-token"},
} {
if err := terminalOnly(argv); err != nil {
t.Errorf("%v refused: %v", argv, err)
}
}
for _, argv := range [][]string{
{"secret", "accept", "anchor", "telegram", "telegram-token", "--at-desk", "laptop"},
{"secret", "ask", "anchor", "telegram", "telegram-token", "--from", "/tmp/x"},
{"secret", "ask", "anchor", "telegram", "telegram-token", "--at", "laptop", "--local"},
{"secret", "ask", "anchor", "telegram", "--at", "laptop"},
{"secret", "accept", "anchor", "telegram", "telegram-token"},
{"secret", "accept", "anchor", "telegram", "telegram-token", "--from", "/tmp/x"},
{"secret", "accept", "anchor", "telegram", "telegram-token", "--at-desk", "laptop", "--local"},
{"secret", "accept", "anchor", "telegram", "--provider", "--at-desk", "laptop"},
{"secret", "export", "anchor", "telegram", "telegram-token", "--at-desk", "laptop"},
} {
if err := terminalOnly(argv); err == nil {
t.Errorf("%v passed the terminal rule", argv)
}
}
}
// The confirmation review of 2026-10-09, N1-give: a desk's prompt is answered over the desk machine's bus, and
// on a desk machine agents run as the operator, who holds its runtime's credential — so a trusted party's
// secret (a module running as an account of its own: the Telegram bot's token) is never taken at a desk.
// Refused before anybody is asked to type, whoever called, naming the terminal's line.
func TestATrustedPartysSecretIsNeverTakenAtADesk(t *testing.T) {
d, accepted, acts, asked := aDesk(t, sealedTo(t, typed))
d.trusted = func(module string) (bool, error) { return module == "telegram", nil }
_, err := d.give("anchor", "telegram", "telegram-token", "laptop")
if err == nil || !strings.Contains(err.Error(), "controller's terminal alone") ||
!strings.Contains(err.Error(), "secret accept anchor telegram telegram-token") {
t.Fatalf("a trusted party's secret was taken at the desk, or refused without the line: %v", err)
}
if len(*asked)+len(*accepted)+len(*acts) != 0 {
t.Errorf("asked %v, accepted %d, recorded %v", *asked, len(*accepted), *acts)
}
d.trusted = func(string) (bool, error) { return false, errors.New("the store did not answer") }
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err == nil || len(*asked) != 0 {
t.Errorf("a module not known to be untrusted was asked at the desk: %v", err)
}
}
// And who may answer the desk's prompt at all: only the runtime of the machine it is asked on, carrying the
// launcher that holds the seat there — never the controller, another machine's runtime, or a module's own
// account (the confirmation review of 2026-10-09, N1-give).
func TestOnlyTheDeskMachinesLauncherMayAnswerItsPrompt(t *testing.T) {
launcher := broker.Declared{Module: "rofi", Holds: []broker.Seat{{Name: "node-launcher", Scope: "node",
Serves: []string{"run", "secret"}}}}
subject := "mesh.seat.node-launcher.tool.secret.laptop"
for _, c := range []struct {
p broker.Principal
answers bool
}{
{broker.Principal{Kind: broker.KindNodeTools, Node: "laptop", Module: broker.RuntimeModule, Carries: []broker.Declared{launcher}}, true},
{broker.Principal{Kind: broker.KindNodeTools, Node: "anchor", Module: broker.RuntimeModule, Carries: []broker.Declared{launcher}}, false},
{broker.Principal{Kind: broker.KindController}, false},
{broker.Principal{Kind: broker.KindModule, Node: "laptop", Module: "lab"}, false},
{broker.Principal{Kind: broker.KindNode, Node: "laptop"}, false},
} {
perms, err := broker.PermissionsFor(c.p)
if err != nil {
t.Fatal(err)
}
if got := broker.MaySubscribe(perms, subject); got != c.answers {
t.Errorf("%s may answer %s: %v, want %v", c.p.Username(), subject, got, c.answers)
}
}
}
// N1-give at the controller's terminal (the confirmation review of 2026-10-09): a trusted party's secret is
// announced before it is kept, and not kept when the announcement fails; another module's is kept first and
// a failed announcement is said, not undone.
func TestATrustedPartysSecretGivenAtTheTerminalIsAnnouncedBeforeItIsKept(t *testing.T) {
var order []string
announce := func(fail bool) func() error {
return func() error {
order = append(order, "announce")
if fail {
return errors.New("no channel")
}
return nil
}
}
keep := func() (bool, error) { order = append(order, "keep"); return false, nil }
order = nil
if _, unannounced, err := keepGiven(true, announce(false), keep); err != nil || unannounced != nil ||
strings.Join(order, ",") != "announce,keep" {
t.Errorf("trusted: %v %v, order %v; want announced, then kept", unannounced, err, order)
}
order = nil
if _, _, err := keepGiven(true, announce(true), keep); err == nil || strings.Join(order, ",") != "announce" {
t.Errorf("trusted, announcement failed: %v, order %v; want refused and nothing kept", err, order)
}
order = nil
if _, unannounced, err := keepGiven(false, announce(true), keep); err != nil || unannounced == nil ||
strings.Join(order, ",") != "keep,announce" {
t.Errorf("not trusted: %v %v, order %v; want kept, then the failed announcement said", unannounced, err, order)
}
order = nil
failing := func() (bool, error) { order = append(order, "keep"); return false, errors.New("store away") }
if _, _, err := keepGiven(false, announce(false), failing); err == nil || strings.Join(order, ",") != "keep" {
t.Errorf("not trusted, keep failed: %v, order %v; want refused and nothing announced", err, order)
}
}
// A machine the mesh does not know, as the secret's or as the desk, is refused before anybody is asked to type.
func TestAGiveNamingAMachineTheMeshDoesNotKnowAsksNobody(t *testing.T) {
for _, unknown := range []string{"elsewhere", "nodesk"} {
d, accepted, acts, asked := aDesk(t, func(map[string]any) (json.RawMessage, error) {
t.Fatal("the desk was asked")
return nil, nil
})
d.known = func(machine string) error {
if machine == unknown {
return errors.New("no node " + machine)
}
return nil
}
node, desk := "anchor", "laptop"
if unknown == "elsewhere" {
node = unknown
} else {
desk = unknown
}
_, err := d.give(node, "telegram", "telegram-token", desk)
if err == nil || !strings.Contains(err.Error(), "nobody was asked") || !strings.Contains(err.Error(), unknown) {
t.Errorf("%s: %v", unknown, err)
}
if len(*accepted)+len(*acts)+len(*asked) != 0 {
t.Errorf("%s: something happened: %v %v %v", unknown, *accepted, *acts, *asked)
}
}
}
// novox/hq ADR 0277: the prompt names who asked — the controller's word on the bus's caller, cut to a name's
// characters — and never a word the caller chose: there is no argument for it.
func TestThePromptNamesWhoAskedFromTheBussWordAlone(t *testing.T) {
d, _, acts, asked := aDesk(t, sealedTo(t, typed))
d.askedBy = "g14/claude-code, through the mesh-controller seat"
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
if got := (*asked)[0]["asked_by"]; got != "g14/claude-code" {
t.Errorf("asked_by %q", got)
}
if len(*acts) != 1 || !strings.Contains((*acts)[0].Why, "asked by g14/claude-code") {
t.Errorf("the record: %+v", *acts)
}
for in, want := range map[string]string{
"jochen at a shell on novox": "jochen at a shell on novox",
"laptop/agent": "laptop/agent",
"Your bank asks\nType your PIN, now": "Your bank asks-Type your PIN",
"": "an unnamed caller",
"<b>x</b>": "an unnamed caller",
strings.Repeat("a", 100): strings.Repeat("a", 80),
"--prompt, something else": "an unnamed caller",
} {
if got := askedByName(in); got != want {
t.Errorf("askedByName(%q) = %q, want %q", in, got, want)
}
}
// The verb's schema has no argument that reaches the prompt's words.
for _, verb := range []string{"give", "secret-ask"} {
for _, free := range []string{"asked_by", "prompt", "message", "value", "from"} {
if _, err := argvFor(verb, map[string]any{"node": "anchor", "module": "telegram", "secret": "telegram-token",
"at": "laptop", free: "x"}); err == nil {
t.Errorf("%s takes %s", verb, free)
}
}
}
}
// An ask for a secret is bounded before anybody is asked to type (ADR 0277): the record refuses it, nothing is
// asked; and how every ask ends is recorded.
func TestASecretAskIsBoundedAndItsEndRecorded(t *testing.T) {
d, accepted, _, asked := aDesk(t, sealedTo(t, typed))
var ended []string
d.open = func(node, module, name, desk string) (int64, error) { return 7, nil }
d.end = func(id int64, outcome string) error {
ended = append(ended, fmt.Sprintf("%d %s", id, outcome))
return nil
}
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
d.open = func(node, module, name, desk string) (int64, error) {
return 0, errors.New("an ask for telegram-token of telegram on anchor is still open")
}
_, err := d.give("anchor", "telegram", "telegram-token", "laptop")
if err == nil || !strings.Contains(err.Error(), "nobody was asked to type anything") || !strings.Contains(err.Error(), "still open") {
t.Errorf("a second ask: %v", err)
}
if len(*asked) != 1 || len(*accepted) != 1 {
t.Errorf("asked %d, accepted %d", len(*asked), len(*accepted))
}
d.open = func(node, module, name, desk string) (int64, error) { return 8, nil }
d.ask = func(string, map[string]any) (json.RawMessage, error) {
return json.RawMessage(`{"cancelled":true}`), nil
}
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); !errors.Is(err, errNothingGiven) {
t.Errorf("a dismissed prompt: %v", err)
}
if strings.Join(ended, "; ") != "7 given; 8 dismissed" {
t.Errorf("ended: %v", ended)
}
}
// A secret ask cannot be turned into a secret read: the line carries no value, the answer carries none, and every
// other `secret` line is the terminal's.
func TestASecretAskIsNeverASecretRead(t *testing.T) {
t.Setenv(verbVar, "mesh-controller.secret-ask")
for _, args := range [][]string{
{"ask", "anchor", "telegram", "telegram-token", "the-value"},
{"ask", "anchor", "telegram"},
{"ask", "anchor", "telegram", "telegram-token", "--from", "/dev/null"},
} {
if err := secretCommand(context.Background(), args); err == nil {
t.Errorf("secret %v was taken", args)
}
}
for _, args := range [][]string{{"recover", "anchor", "telegram", "telegram-token"}, {"export"}} {
if err := terminalOnly(append([]string{"secret"}, args...)); err == nil {
t.Errorf("secret %v passed the terminal rule", args)
}
}
}
+15
View File
@@ -116,6 +116,21 @@ var probeRegistry = []probe{
{ID: probeDeliveriesID, Asserts: "no delivery is held past its state's bound unsaid: mesh-delivery's " +
"`stalled`, each with the transition its table lets healer H2 take", From: "ADR 0239",
Kind: kindDeliveryStalled, Phase: 3, run: probeDeliveries},
// A client of the bus reconnecting in a loop (novox/hq issue 327), from the server's record of closed
// connections, which the bus's own module reads.
{ID: probeReconnectsID, Asserts: "no user of the bus had its connection dropped more than twelve times in the " +
"last hour: the bus module's nats_closed_connections", From: "issue 327", Kind: kindBusReconnects,
Phase: 1, run: probeReconnects},
// The account agents run as (novox/hq ADR 0266): where a machine names one, its node-engine has judged it
// unable to become root without a person — what ADR 0259 §8 rests an authorised answer on.
{ID: agentAccountProbe, Asserts: "every machine that names an agent account has it judged, on its node-engine's " +
"newest statement, unable to become root without a person", From: "ADR 0266, ADR 0259 §8",
Kind: kindAgentCanBecomeRoot, Phase: 1, run: probeAgentAccounts},
// Root where the trusted parties run (novox/hq ADR 0259 §8): while an agent can become root there without a
// person, an answer proven there proves nothing.
{ID: "D-root", Asserts: "no agent can become root without a person on a machine where the router or a channel " +
"proving its sender runs: not by its own account, and not through a tool that runs its command as an account " +
"that can", From: "ADR 0259 §8", Kind: kindRootNotFree, Phase: 2, run: probeAgentRoot},
{ID: "DW", Asserts: "the watchdogs of the signals table ran within three of their intervals",
From: "ADR 0227 rule 6: the watchers are watched", Kind: "watchdogs-silent", Phase: 1, run: probeWatchdogs},
// The core's health definitions (novox/hq to-be 45 §8, ADR 0236): what a core component's new build is
+14 -2
View File
@@ -205,7 +205,7 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
if err != nil {
return snapshot.Facts{}, err
}
read, err := inv.ReadRepositories(ctx)
read, err := readForPlanning(ctx, inv)
if err != nil {
return snapshot.Facts{}, err
}
@@ -269,7 +269,8 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
engines := map[string]bool{}
for _, n := range nodes {
m := snapshot.Machine{Name: scrub.Machine(n.Name), Length: len(n.Name), Adopted: n.Adopted,
AccountHome: scrub.Text(n.AccountHome), NodeEngine: n.HostVersion, PublicDomain: domains[n.Name]}
AccountHome: scrub.Text(n.AccountHome), NodeEngine: n.HostVersion, PublicDomain: domains[n.Name],
AgentAccount: scrub.Account(n.AgentAccount), AgentAccountHome: scrub.Text(n.AgentAccountHome)}
switch n.Account {
case "", "root":
m.Account = n.Account
@@ -410,7 +411,18 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
Commit: e.Source.BuiltFrom, Provided: e.Provided, RollOut: current[e.Manifest.Module].RollOut,
Manifest: raw}
for _, r := range read[e.Manifest.Module] {
// The module's own build source is said apart: a gate that predates it would read an own
// entry among Reads as a context of its own repository.
if r.Own {
if len(r.Paths) > 0 {
mod.Sources = append(mod.Sources, snapshot.BuildSource{Own: true, Paths: r.Paths})
}
continue
}
mod.Reads = append(mod.Reads, snapshot.RepositoryName(r.Repository))
if len(r.Paths) > 0 {
mod.Sources = append(mod.Sources, snapshot.BuildSource{Repository: snapshot.RepositoryName(r.Repository), Paths: r.Paths})
}
}
f.Modules = append(f.Modules, mod)
if e.Provided || e.Source.Repository == "" {
+156
View File
@@ -0,0 +1,156 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A directory the node-engine uses as found (novox/hq issue 339) waits for a person to hand it over at the
// machine. Found before this send, it is no fault of the build: the gate passes with the wait carried, so an
// urgent fix of that module still goes through. Found by this send, the send brought it, and the gate holds.
func foundDirectory(module string, since time.Time) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: module, Resource: module + ".data", Kind: link.KindDirectory,
Target: "/srv/" + module, State: link.StateUnhealthy, Since: since,
Reason: link.ReasonUsedAsFound + " owned by 1000:1000, mode 700, as found; root, mode 755 was declared and " +
"not given it — `nox node hand-over laptop <directory>` on the control-node, with its path, hands it to the mesh"}
}
func TestADirectoryFoundBeforeTheSendIsAWaitForAPerson(t *testing.T) {
now := time.Now()
sent := now.Add(-time.Minute)
f := gateFacts{now: now, health: map[string]inventory.NodeHealth{"laptop": {Node: "laptop", HeardAt: now,
Resources: []inventory.ResourceHealth{foundDirectory("notes", sent.Add(-24*time.Hour))}}}}
h, why := moduleHealthWord("notes", "laptop", sent, f)
if h != healthPerson || !strings.Contains(why, "notes.data") || !strings.Contains(why, "hand-over") {
t.Fatalf("a directory found before the send reads %v %q; want a wait for a person", h, why)
}
// Found by this very send: the send brought it, and it is not passed.
f.health["laptop"] = inventory.NodeHealth{Node: "laptop", HeardAt: now,
Resources: []inventory.ResourceHealth{foundDirectory("notes", sent.Add(time.Second))}}
if h, why := moduleHealthWord("notes", "laptop", sent, f); h != healthNotYet {
t.Fatalf("a directory this send found reads %v %q; want not yet", h, why)
}
// A container down beside the old wait is a fault, as before.
f.health["laptop"] = inventory.NodeHealth{Node: "laptop", HeardAt: now, Resources: []inventory.ResourceHealth{
foundDirectory("notes", sent.Add(-time.Hour)),
{Module: "notes", Resource: "notes.web", Kind: "container", Target: "notes", State: link.StateUnhealthy, Reason: "down"}}}
if h, why := moduleHealthWord("notes", "laptop", sent, f); h != healthNotYet {
t.Fatalf("a container down beside the wait reads %v %q; want not yet", h, why)
}
}
// The whole walk: a module whose directory was used as found long before still gets its fix to every machine,
// its pass kept and the wait said; a directory this very send found holds it and puts it back.
func TestAFixGoesThroughPastADirectoryFoundBefore(t *testing.T) {
for _, c := range []struct {
name string
found time.Duration // when the directory was found, against now
passes bool
}{
{"found a day before the send", -24 * time.Hour, true},
{"found by this send", time.Hour, false},
} {
t.Run(c.name, func(t *testing.T) {
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
releaseHeard = func(context.Context, *stores) (map[string]bool, error) {
return map[string]bool{"anchor": true, "laptop": true}, nil
}
backlogFacts := gatherGateFacts
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
f, err := backlogFacts(ctx, open, component)
// The used-as-found condition, raised after the send (its second statement): its own kind, never a
// fault the gate reads as the build's.
f.judged, f.openErr = true, nil
f.open = append(f.open, conditions.Condition{Key: usedAsFoundKey("app", "anchor"), Kind: kindUsedAsFound,
Subject: conditions.Subject{Scope: conditions.ScopeModule, ID: "app.anchor", Machine: "anchor"},
Raised: time.Now()})
f.health = map[string]inventory.NodeHealth{}
for _, n := range []string{"anchor", "laptop"} {
f.health[n] = inventory.NodeHealth{Node: n, HeardAt: time.Now(), Resources: []inventory.ResourceHealth{
{Module: "app", Resource: "app.web", Kind: "container", Target: "app", State: link.StateHealthy},
foundDirectory("app", time.Now().Add(c.found)),
{Module: "late", Resource: "late.web", Kind: "container", Target: "late", State: link.StateHealthy}}}
}
return f, err
}
wasSettle, wasEvery, wasBound := gateSettle, gateEvery, gateBound
t.Cleanup(func() { gateSettle, gateEvery, gateBound = wasSettle, wasEvery, wasBound })
gateSettle, gateEvery, gateBound = 0, 0, 300*time.Millisecond
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
advancePlans(ctx, b.open)
if p := b.release(t); p.State != inventory.PlanRolling {
break
}
time.Sleep(20 * time.Millisecond)
}
p := b.release(t)
v, found, err := inv.GateOf(ctx, "build-app-c2")
if c.passes {
if p.State != inventory.PlanDone || err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("the walk is %s (%s); app's verdict %+v: want the fix through", p.State, p.Note, v)
}
if !strings.Contains(v.Why+p.Note, "hand it over") {
t.Errorf("the wait is not carried: verdict %q, walk %q", v.Why, p.Note)
}
return
}
if p.State == inventory.PlanDone {
t.Fatalf("a directory this send found let the walk through: %s", p.Note)
}
})
}
}
// The condition says the wait in its own kind: the operator's, never urgent, and cleared once handed over.
func TestADirectoryUsedAsFoundIsItsOwnCondition(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
rs := map[string][]inventory.ResourceHealth{"notes": {foundDirectory("notes", time.Now().Add(-time.Hour))}}
for i := 0; i < 2; i++ {
if err := judgeModuleHealth(ctx, nil, k, "laptop", rs, map[string]int{"notes": i + 1}, time.Now()); err != nil {
t.Fatal(err)
}
}
open, _ := k.Open(ctx)
var got *conditions.Condition
for i, c := range open {
if c.Key == usedAsFoundKey("notes", "laptop") {
got = &open[i]
}
if c.Kind == kindModuleUnhealthy {
t.Fatalf("raised as a fault: %+v", c)
}
}
if got == nil || got.Resolver != conditions.ResolverOperator || got.Severity == conditions.Urgent ||
!strings.Contains(got.Summary, "notes.data") {
t.Fatalf("the condition: %+v", got)
}
// Long open is still not urgent: only a person can hand it over, and nothing is broken by the wait.
if err := judgeModuleHealth(ctx, nil, k, "laptop", rs, map[string]int{"notes": 3}, time.Now().Add(48*time.Hour)); err != nil {
t.Fatal(err)
}
open, _ = k.Open(ctx)
for _, c := range open {
if c.Key == usedAsFoundKey("notes", "laptop") && c.Severity == conditions.Urgent {
t.Fatal("a directory used as found became urgent")
}
}
if err := judgeModuleHealth(ctx, nil, k, "laptop", map[string][]inventory.ResourceHealth{}, nil, time.Now()); err != nil {
t.Fatal(err)
}
open, _ = k.Open(ctx)
for _, c := range open {
if c.Key == usedAsFoundKey("notes", "laptop") {
t.Fatal("not cleared once handed over")
}
}
}
+162 -6
View File
@@ -87,6 +87,9 @@ const (
healthWaiting
healthNotYet
healthBroken
// healthSuperseded is a judging that cannot go on: the machine was sent another build of the module
// after the gate's send (novox/hq issue 352). No verdict on the build judged, and nothing put back.
healthSuperseded
)
// served is what one machine's node tools answered the bus's discovery with.
@@ -122,6 +125,39 @@ type gateFacts struct {
// groupsAdded is, per module, whether the move judged puts an account in a group its previous build did
// not (issue 318 review): the only move whose wait for a new login is excused.
groupsAdded map[string]bool
// sent is, per machine, the declaration the gate's own send carried there (novox/hq issue 352): a
// report is held against it, never against the send made last. sentBuilds is what each machine was
// last sent of every module, and judged the commit of each module this gate judges: a machine last
// sent another build of the module is not running the build judged.
sent map[string]inventory.SentDeclaration
sentBuilds map[string]map[string]string
commits map[string]string
}
// reportedOn says a machine's last report is on what the gate sent it (novox/hq issue 352): on that
// declaration, or one it was sent after it — or, for a gate kept before sends were kept on it, on the
// declaration last sent. On 2026-10-09 a release's gate read the control node's report against a newer
// send another plan had just made there, and failed three builds the machine had reported healthy as
// "has not reported on what it was sent".
func (f gateFacts) reportedOn(machine string, r inventory.Reported) bool {
if sent, kept := f.sent[machine]; kept {
return sent.ReportsOn(r)
}
return r.Current
}
// supersededOn says the machine was last sent another build of the module than the one this gate judges
// (novox/hq issue 352): the judging cannot go on, whatever the machine reports. On 2026-10-09 a controller
// put back by one gate judged another gate's newer controller build passed on the same machine, reading
// the put-back build's health as the newer one's.
func (f gateFacts) supersededOn(module, machine string) (string, bool) {
judged, known := f.commits[module]
sent, has := f.sentBuilds[machine][module]
if !known || !has || judged == "" || sent == "" || sameCommit(sent, judged) {
return "", false
}
return fmt.Sprintf("%s was sent %s %s after this gate's %s: the build judged no longer runs there, and "+
"this judging is superseded by that send's", machine, module, short(sent), short(judged)), true
}
// gatherGateFacts reads what a judging needs, from the store, the bus and this controller's memory. A
@@ -199,9 +235,12 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
return healthBroken, fmt.Sprintf("the witness on %s judged the %s %s and %s: %s", machine, r.Component,
short(r.From), r.Outcome, r.Why)
}
if why, superseded := f.supersededOn(module, machine); superseded {
return healthSuperseded, why
}
r, said := f.reports[machine]
switch {
case !said || r.At == nil || !r.Current:
case !said || r.At == nil || !f.reportedOn(machine, r):
return healthNotYet, fmt.Sprintf("%s has not reported on what it was sent", machine)
case r.Outcome == inventory.OutcomeFailed || r.Outcome == inventory.OutcomeRefused:
return healthBroken, fmt.Sprintf("%s %s what it was sent", machine, r.Outcome)
@@ -209,16 +248,19 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
return healthNotYet, fmt.Sprintf("%s reported %q", machine, r.Outcome)
}
// **No new condition about it**: about the machine itself, or naming the module on that machine,
// raised since the judging began. The gate's own are not evidence about the build.
// raised since the judging began. The gate's own are not evidence about the build. A fault that was
// there at the send and reopened since is not new; one that had cleared before the send and came back
// after it is (OpenAt, novox/hq issue 348).
if f.judged {
if f.openErr != nil {
return healthNotYet, "what is wrong cannot be read, so whether the build made anything wrong is not known: " +
firstLine(f.openErr.Error())
}
for _, c := range f.open {
// A wait for a person's new login is the module's reading, not a fault raised since the send: the
// gate reads it from the statement below (ADR 0254).
if c.Source == gateProbe || c.Raised.Before(since) || c.Kind == kindReloginNeeded {
// A wait for a person's new login, or for a directory used as found to be handed over, is the module's
// reading, not a fault raised since the send: the gate reads it from the statement below (ADR 0254,
// novox/hq issue 339).
if c.Source == gateProbe || c.OpenAt(since) || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound {
continue
}
onIt := c.Subject.Machine == machine || slices.Contains(c.Subject.Also, machine) ||
@@ -329,7 +371,8 @@ func aboutTheMachine(machine string, moved []string, since time.Time, f gateFact
for _, c := range f.open {
aboutIt := c.Subject.Scope == conditions.ScopeMachine && (c.Subject.ID == machine || c.Subject.Machine == machine ||
slices.Contains(c.Subject.Also, machine))
if !aboutIt || c.Source == gateProbe || c.Raised.Before(since) {
// A directory used as found waits for a person, whatever the send did (novox/hq issue 339).
if !aboutIt || c.Source == gateProbe || c.OpenAt(since) || c.Kind == kindUsedAsFound {
kept = append(kept, c)
continue
}
@@ -434,6 +477,21 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
return "", err
}
facts.groupsAdded = movesAddingGroups(ctx, open.inventory, g, pairs, shelf)
facts.sent = g.Sent
facts.commits, facts.sentBuilds = judgedCommits(g, pairs), map[string]map[string]string{}
// A module this gate put back at once (putBackBroken) was sent its earlier build by the gate itself:
// not another send, and not a judging superseded.
for _, m := range g.Returned {
delete(facts.commits, m)
}
for _, j := range pairs {
if _, read := facts.sentBuilds[j.node]; read {
continue
}
if builds, known, err := open.inventory.SentBuilds(ctx, j.node); err == nil && known {
facts.sentBuilds[j.node] = builds
}
}
// **What is wrong with a machine itself is the machine's** (novox/hq issue 281): read once for each
// machine judged, apart from what is wrong with a module there, and never pinned on the module the
// gate happens to be kept on.
@@ -470,6 +528,13 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
if _, seen := reading[j.module]; !seen {
modules = append(modules, j.module)
}
if h == healthSuperseded {
// Decided at once (novox/hq issue 352): nothing of this gate's can be judged on a machine that
// was sent another build of it, and nothing is put back — the later send is what runs there.
g.Failing, g.Last = nil, ""
decide(g, inventory.GateSuperseded, said, now)
return g.Verdict, nil
}
if h == healthBroken && !slices.Contains(g.Broken, j.module) {
g.Broken = append(g.Broken, j.module)
if g.BrokenWhy == "" {
@@ -573,6 +638,40 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
return g.Verdict, nil
}
// judgedCommits is the commit of each module a gate judges: the gate's own To for its module, and each
// carried move's. Pure.
func judgedCommits(g *inventory.PlanGate, pairs []judged) map[string]string {
out := map[string]string{}
for _, c := range g.Carried {
if c.To != "" {
out[c.Module] = c.To
}
}
if g.To != "" {
for _, j := range pairs {
if _, has := out[j.module]; !has && !slices.ContainsFunc(g.Carried, func(c inventory.CarriedMove) bool { return c.Module == j.module }) {
out[j.module] = g.To
}
}
}
return out
}
// sentNow is what each machine was just sent, read after a send for the gate to keep (novox/hq issue
// 352): a machine whose send is not on record is left out, and its report is read as before.
func sentNow(ctx context.Context, inv *inventory.Inventory, machines []string) map[string]inventory.SentDeclaration {
out := map[string]inventory.SentDeclaration{}
for _, n := range machines {
if s, found, err := inv.SentTo(ctx, n); err == nil && found {
out[n] = s
}
}
if len(out) == 0 {
return nil
}
return out
}
// whyFor is a passing gate's why as one module's verdict says it: the send's, and that module's own wait
// for a person, never another's (issue 318 review).
func whyFor(g *inventory.PlanGate, module string) string {
@@ -798,6 +897,16 @@ func gateFailed(ctx context.Context, open *stores, p *inventory.Plan, module str
"back", module, short(state.Previous), inventory.KeptBuilds))
return
}
if g.Component == lease.ComponentController {
// **The controller is never put back to a build older than the store's schema** (novox/hq issue
// 352): the build before it carries fewer migrations than the failed one applied, starts behind
// its own records, and judges the next gate with what it can read. The current build is kept and
// the condition says so; a person decides.
if why, ok := controllerSchemaAllows(ctx, inv, previous); !ok {
notBack(why)
return
}
}
if err := inv.RestoreModule(ctx, previous); err != nil {
notBack(err.Error())
return
@@ -831,6 +940,53 @@ func gateFailed(ctx context.Context, open *stores, p *inventory.Plan, module str
sayRollback(ctx, open, module, g, "")
}
// controllerSchemaAllows says the store's schema lets this build of the controller be put back: the
// build recorded, when it served, a reach at or past the highest migration the store has applied. One
// that never recorded a reach is not proved safe, and is refused as such (novox/hq issue 352). Why
// says what is kept and why when it is not.
func controllerSchemaAllows(ctx context.Context, inv *inventory.Inventory, previous inventory.Build) (string, bool) {
applied, err := inv.SchemaApplied(ctx)
if err != nil {
return "what the store's schema reaches cannot be read, so whether the build before it can read it is not " +
"known; the current build is kept: " + err.Error(), false
}
build := versionOfBuild(previous)
if build == "" {
return fmt.Sprintf("the build before it (%s) names no bundle to know it by, so whether it can read the store's "+
"schema (migration %04d) is not known; the current build is kept, and a person decides", short(previous.Commit), applied), false
}
reach, known, err := inv.SchemaReachOf(ctx, build)
if err != nil {
return "what the build before it knows of the store's schema cannot be read; the current build is kept: " + err.Error(), false
}
if !known {
return fmt.Sprintf("the build before it (%s, %s) never recorded how far it reads the store's schema — a "+
"controller records that when it serves — so it is not proved to read migration %04d, which the store "+
"has applied; a controller older than its store starts behind its own records and judges with what it "+
"can read, so the current build is kept, and a person decides", short(previous.Commit), build, applied), false
}
if reach < applied {
return fmt.Sprintf("the build before it (%s, %s) reads the store's schema up to migration %04d, and the store "+
"is at %04d: a controller older than its store starts behind its own records and judges with what it "+
"can read, so the current build is kept, and a person decides", short(previous.Commit), build, reach, applied), false
}
return "", true
}
// versionOfBuild is the version a build's bundle is delivered as — its archive's digest, short, as the
// catalogue names it (`${version}`) — read from the build's artifacts; empty when none is a bundle.
func versionOfBuild(b inventory.Build) string {
for _, a := range b.Made {
if a.Kind != catalogue.ArtifactBundle && a.Kind != catalogue.ArtifactArchive {
continue
}
if _, hex, found := strings.Cut(a.Reference, "sha256:"); found && len(hex) >= 12 {
return hex[:12]
}
}
return ""
}
// rollbacks is what a failed send puts back, sent together (novox/hq issue 281): a gate that judged one
// send judges what it moved as one, and what it found wanting goes back in one send per machine — not
// in a send for each module, which is the churn that failed the gate in the first place.
+4 -2
View File
@@ -113,9 +113,10 @@ func aGateMesh(t *testing.T) *gateMesh {
}
for node, h := range g.health {
if h == healthNotYet {
// Not reported on the send: neither the send made last, nor the gate's own (issue 352).
f.rolledBack[node] = nil
r := f.reports[node]
r.Current = false
r.Current, r.Declared, r.ReportedSequence = false, "", 0
f.reports[node] = r
}
}
@@ -221,7 +222,8 @@ func TestABuildThatFailsItsGateIsRolledBackOnItsFirstMachineAndGoesNoFurther(t *
t.Fatalf("sent again after the rollback: %v", g.sent)
}
_, _, err = takeIn(ctx, inv, link.BuildResult{ID: "build-2", Repository: "novox/mesh-catalog", Path: "modules/app",
Commit: "c2", Manifest: mustJSON(t, catalogue.Manifest{Module: "app", Version: "c2"})})
Commit: "c2", Manifest: mustJSON(t, catalogue.Manifest{Module: "app", Version: "c2"}),
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}})
if err == nil || !strings.Contains(err.Error(), "failed its gate") {
t.Fatalf("the failed build was registered again: %v", err)
}
+36 -16
View File
@@ -6,6 +6,7 @@ import (
"errors"
"flag"
"fmt"
"io"
"os"
"slices"
"sort"
@@ -14,7 +15,6 @@ import (
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
@@ -60,11 +60,18 @@ var handActVerbs = []handActVerb{
// a person's word (ADR 0242), which the push itself reads from what it carried (recorded_push.go).
{Verb: "push", Decision: "a recorded build moves only by a person's push: that push is the word its " +
"upgrade policy asks for (ADR 0242)", DecidedWhen: pushedRecorded},
{Verb: "plans stop"},
// Stopping or starting a walk the operator chose on a warrant (novox/hq ADR 0259) is their decision.
{Verb: "plans stop", Decision: "the operator's answer to an ask is their decision, not a repair (ADR 0259)",
DecidedFor: []string{conditions.CauseOperatorAnswer}},
{Verb: "plans close"},
// A walk started by a person instead of its delivery's owner (novox/hq ADR 0239): the owner down, or
// not trusted with it — either is a repair the owner should have made.
{Verb: "plans go"},
// not trusted with it — either is a repair the owner should have made. Unless the operator chose it on
// a warrant (ADR 0259).
{Verb: "plans go", Decision: "the operator's answer to an ask is their decision, not a repair (ADR 0259)",
DecidedFor: []string{conditions.CauseOperatorAnswer}},
// An act the operator chose on a warrant (novox/hq ADR 0259): asked by the controller, answered on a
// channel that proved who answered, performed by the controller as itself.
{Verb: handActWarrant, Decision: "the operator chose it, answering what the controller asked (ADR 0259)"},
{Verb: "broker consumer-reset"},
// Silencing the same condition twice says the condition, or what it watches, wants mending — unless
// it is the operator's answer on a notification: a decision to live with it (novox/hq ADR 0258).
@@ -82,6 +89,11 @@ var handActVerbs = []handActVerb{
{Verb: "retire approve", Decision: "nothing is retired past its bound without a person (ADR 0230)"},
{Verb: "retire reject", Decision: "keeping a consumer active is a person's word (ADR 0230)"},
{Verb: "cleanup delete", Decision: "nothing retired is deleted without a person (ADR 0230)"},
// What becomes of a message a consumer gave up on (novox/hq issue 330): kept until a person says.
{Verb: "dead-letters deliver", Decision: "a message a consumer gave up on is delivered again only on a " +
"person's word (issue 330)"},
{Verb: "dead-letters drop", Decision: "a message a consumer gave up on is let go only on a person's word " +
"(issue 330)"},
// The sweep run on a person's word rather than after a build: the same decision the records make, at
// a moment the person chose (ADR 0251) — never a repair.
{Verb: "collect", Decision: "letting the store go of what the records keep for no reason, now rather " +
@@ -98,6 +110,9 @@ var handActVerbs = []handActVerb{
// to that judgement. Several values rotate for one leak, and a leak that recurs is a defect of the
// module that prints them, an issue against it, not a healer that rotates. A rotation for any other
// cause — a credential that stopped working — counts: a schedule or a healer could take it over.
// A value given at the desk (novox/hq ADR 0259 §10): an outside party's key, such as a bot token, which
// only a person can give. Their word, never a repair.
{Verb: "secret accept", Decision: "a value an outside party issued is given by a person, at their desk"},
{Verb: "secret rotate", Decision: "a value a person judged disclosed is replaced on their word",
DecidedFor: []string{causeLeakedInLogs}},
}
@@ -149,14 +164,10 @@ func onTheBus(f func(*nats.Conn) error) error {
if handActConn != nil {
return f(handActConn)
}
address, err := broker.BusAddress()
js, err := aBus()
if err != nil {
return err
}
js, err := broker.Dial(address)
if err != nil {
return fmt.Errorf("cannot reach the bus: %w", err)
}
defer js.Close()
return f(js.Conn())
}
@@ -240,6 +251,9 @@ func handActCommand(ctx context.Context, args []string) error {
if len(args) > 0 && args[0] == "drill" {
return handActDrill(ctx, args[1:])
}
if len(args) > 0 && args[0] == "warrant" {
return handActWarrantCommand(ctx, args[1:])
}
if len(args) > 0 && args[0] != "list" && !strings.HasPrefix(args[0], "-") {
return errors.New("hand-act record <what> --why <text> --cause <word> | hand-act drill <what> --why <text> " +
"| hand-acts [--days N] [--json]")
@@ -247,7 +261,13 @@ func handActCommand(ctx context.Context, args []string) error {
if len(args) > 0 && args[0] == "list" {
args = args[1:]
}
return listHandActs(ctx, args, os.Stdout)
}
// listHandActs is `hand-acts`: what was done by hand lately, and the causes done more than once.
func listHandActs(ctx context.Context, args []string, w io.Writer) error {
set := flag.NewFlagSet("hand-acts", flag.ContinueOnError)
usageTo(set, w)
days := set.Int("days", 14, "how many days back")
asJSON := set.Bool("json", false, "as data")
if _, err := parseAround(set, args); err != nil {
@@ -265,27 +285,27 @@ func handActCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
fmt.Println(string(body))
fmt.Fprintln(w, string(body))
return nil
}
if len(acts) == 0 {
fmt.Printf("nothing was done by hand in the last %d day(s)\n", *days)
fmt.Fprintf(w, "nothing was done by hand in the last %d day(s)\n", *days)
return nil
}
for i := len(acts) - 1; i >= 0; i-- {
a := acts[i]
fmt.Printf("%s %s %s %s\n by %s — %s (cause: %s", a.At.Local().Format("2006-01-02 15:04"), a.ID,
fmt.Fprintf(w, "%s %s %s %s\n by %s — %s (cause: %s", a.At.Local().Format("2006-01-02 15:04"), a.ID,
a.Verb, strings.Join(a.Args, " "), a.By, a.Why, a.Cause)
if a.Condition != "" {
fmt.Printf(", condition %s", a.Condition)
fmt.Fprintf(w, ", condition %s", a.Condition)
}
fmt.Println(")")
fmt.Fprintln(w, ")")
if pushedRecorded(a) {
carried := strings.Join(a.Carried, "; ")
if carried == "" {
carried = recordedBefore[a.ID]
}
fmt.Printf(" a push of recorded builds, no repair: %s\n", carried)
fmt.Fprintf(w, " a push of recorded builds, no repair: %s\n", carried)
}
}
if len(repeated) > 0 {
@@ -294,7 +314,7 @@ func handActCommand(ctx context.Context, args []string) error {
causes = append(causes, fmt.Sprintf("%s ×%d", c, n))
}
sort.Strings(causes)
fmt.Printf("\ndone by hand more than once in a fortnight — a healer is wanted (to-be 45 S15): %s\n",
fmt.Fprintf(w, "\ndone by hand more than once in a fortnight — a healer is wanted (to-be 45 S15): %s\n",
strings.Join(causes, ", "))
}
return nil
-5
View File
@@ -22,10 +22,6 @@ func TestARepairByHandWithoutAReasonIsRefused(t *testing.T) {
{"plans", map[string]any{"close": "plan-1"}},
{"plans", map[string]any{"stop": "plan-1"}},
{"hand-act", map[string]any{"what": "restarted the proxy", "cause": "proxy-stuck"}},
{"command", map[string]any{"command": "push anchor"}},
{"command", map[string]any{"command": "plans close plan-1"}},
{"command", map[string]any{"command": "broker consumer-reset EVENTS controller"}},
{"command", map[string]any{"command": "hand-act record restarted --cause x"}},
} {
argv, err := argvFor(c.verb, c.args)
if c.verb == "plans" && err == nil {
@@ -65,7 +61,6 @@ func TestARepairByHandCarriesItsReason(t *testing.T) {
{"plans", map[string]any{"retry": "plan-1"}, "plans retry plan-1"},
{"hand-act", map[string]any{"what": "restarted", "why": "hung", "cause": "proxy", "condition": "machine.a.silent"},
"hand-act record restarted --why hung --cause proxy --condition machine.a.silent"},
{"command", map[string]any{"command": "push anchor --why stuck"}, "push anchor --why stuck"},
{"command", map[string]any{"command": "plans plan-1"}, "plans plan-1"},
} {
argv, err := argvFor(c.verb, c.args)
+169
View File
@@ -0,0 +1,169 @@
package main
import (
"bytes"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A hand-over's line is judged before anything is asked (novox/hq issue 356): a node's name and the directory's
// absolute path exactly as the engine states it — no `..`, no doubled or trailing separator, nothing relative.
func TestAHandOverLineIsJudgedBeforeItIsAsked(t *testing.T) {
for _, args := range [][]string{
{},
{"laptop"},
{"laptop", "/srv/notes", "extra"},
{"", "/srv/notes"},
{"--node", "/srv/notes"},
{"laptop", "srv/notes"},
{"laptop", "/srv/../etc"},
{"laptop", "/srv//notes"},
{"laptop", "/srv/notes/"},
{"laptop", "/srv/notes/."},
} {
if _, _, err := handOverLine(args); err == nil {
t.Errorf("%q was taken", args)
}
}
node, path, err := handOverLine([]string{"laptop", "/srv/notes"})
if err != nil || node != "laptop" || path != "/srv/notes" {
t.Fatalf("read as %q %q %v", node, path, err)
}
}
// Who hands over is the line's caller in the words every verb's caller is recorded in — the operator through
// mesh-cli — or the controller's terminal when nobody is named.
func TestAHandOverNamesWhoAsked(t *testing.T) {
t.Setenv(link.CallerVar, " jo through mesh-cli on anchor ")
if by := handOverBy(); by != "jo through mesh-cli on anchor" {
t.Fatalf("by %q", by)
}
t.Setenv(link.CallerVar, "")
if by := handOverBy(); by != "the controller's terminal" {
t.Fatalf("by %q", by)
}
}
// **A hand-over is the controller's terminal's alone** (novox/hq issue 356, ADR 0266): through any verb, and
// through mesh-cli outside the terminal, `node hand-over` is refused and nothing runs — at the next apply root
// gives the directory to the account the module declares, and whoever may call a verb includes agents.
func TestAHandOverIsRefusedThroughEveryVerb(t *testing.T) {
line := []string{"node", "hand-over", "laptop", "/srv/notes"}
if err := terminalOnly(line); err == nil {
t.Fatal("node hand-over passed as a verb's line")
}
if _, err := argvFor("command", map[string]any{"command": "node hand-over laptop /srv/notes"}); err == nil {
t.Fatal("the command verb composed node hand-over")
}
if _, err := ordinaryLine(line); err == nil {
t.Fatal("node hand-over composed as an ordinary mesh-cli line")
}
if _, err := argvFor("node", map[string]any{"node": "laptop", "hand-over": "/srv/notes"}); err == nil {
t.Fatal("the node verb composed a hand-over")
}
}
// The condition's words are plain and name no machine's binary; the operator's line, with the node and the path
// (novox/hq ADR 0272), is in the summary the module's health gives (moduleHealthWord) and in the evidence.
func TestTheUsedAsFoundConditionNamesTheOperatorsLine(t *testing.T) {
rs := []inventory.ResourceHealth{foundDirectory("notes", time.Now().Add(-24*time.Hour))}
o := usedAsFoundObservation("notes", "laptop", "notes on laptop uses notes.data as found", rs)
if strings.Contains(o.Needs, "mesh-host") || strings.Contains(o.Explanation, "mesh-host") ||
!strings.Contains(o.Needs, "control-node") {
t.Fatalf("the condition's words: %q %q", o.Needs, o.Explanation)
}
if why, ok := conditions.PlainWords(conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Needs: o.Needs,
Resolved: o.Resolved}, "laptop"); !ok {
t.Fatalf("not plain: %s", why)
}
f := gateFacts{now: time.Now(), health: map[string]inventory.NodeHealth{"laptop": {Node: "laptop", HeardAt: time.Now(),
Resources: rs}}}
h, why := moduleHealthWord("notes", "laptop", time.Now().Add(-time.Hour), f)
if h != healthPerson || !strings.Contains(why, "`nox node hand-over laptop <directory>` on the control-node") ||
strings.Contains(why, "mesh-host") || strings.Contains(why, "/srv/") {
t.Fatalf("the module's health reads %v %q; want the operator's line", h, why)
}
}
// **Nothing is asked of a node the mesh does not know, and the engine's refusal is the command's failure**
// (review of issue 356): a refused hand-over never exits as a success.
func TestAHandOverAsksOnlyAKnownNodeAndFailsOnARefusal(t *testing.T) {
t.Setenv(link.CallerVar, "jo through mesh-cli on anchor")
asked := 0
ask := func(answer link.HandOverAnswer) func(node, path, by string) (link.HandOverAnswer, error) {
return func(node, path, by string) (link.HandOverAnswer, error) {
asked++
if node != "laptop" || path != "/srv/notes" || by != "jo through mesh-cli on anchor" {
t.Fatalf("asked %q %q %q", node, path, by)
}
return answer, nil
}
}
unknown := func(string) error { return errors.New("no node called laptop") }
known := func(string) error { return nil }
var out bytes.Buffer
err := handOverAsked([]string{"laptop", "/srv/notes"}, unknown, ask(link.HandOverAnswer{Said: "x"}), &out)
if err == nil || asked != 0 || !strings.Contains(err.Error(), "nothing was asked") {
t.Fatalf("an unknown node: %v, asked %d", err, asked)
}
err = handOverAsked([]string{"laptop", "/srv/../etc"}, known, ask(link.HandOverAnswer{Said: "x"}), &out)
if err == nil || asked != 0 {
t.Fatalf("a refused line was asked: %v, asked %d", err, asked)
}
err = handOverAsked([]string{"laptop", "/srv/notes"}, known,
ask(link.HandOverAnswer{Refused: "/srv/notes is not used as found; nothing was handed over"}), &out)
if err == nil || !strings.Contains(err.Error(), "laptop refused: /srv/notes is not used as found") || out.Len() != 0 {
t.Fatalf("a refusal: %v, printed %q", err, out.String())
}
err = handOverAsked([]string{"laptop", "/srv/notes"}, known, ask(link.HandOverAnswer{Said: "handed over"}), &out)
if err != nil || !strings.HasPrefix(out.String(), "handed over\n") || !strings.Contains(out.String(), "`nox push laptop`") {
t.Fatalf("a record: %v, printed %q", err, out.String())
}
failing := func(string, string, string) (link.HandOverAnswer, error) {
return link.HandOverAnswer{}, errors.New("no engine")
}
if err := handOverAsked([]string{"laptop", "/srv/notes"}, known, failing, &out); err == nil {
t.Fatal("an ask that failed was a success")
}
}
// The setuid search's line asks only a known node, one name and nothing else, and fails on a refusal
// (novox/hq issue 361).
func TestASetuidSearchAsksOnlyAKnownNodeAndFailsOnARefusal(t *testing.T) {
t.Setenv(link.CallerVar, "jo through mesh-cli on anchor")
asked := 0
ask := func(answer link.HandOverAnswer) func(node, by string) (link.HandOverAnswer, error) {
return func(node, by string) (link.HandOverAnswer, error) {
asked++
if node != "novox" || by != "jo through mesh-cli on anchor" {
t.Fatalf("asked %q %q", node, by)
}
return answer, nil
}
}
known := func(string) error { return nil }
var out bytes.Buffer
for _, args := range [][]string{nil, {"novox", "extra"}, {"-x"}} {
if err := setuidSearchAsked(args, known, ask(link.HandOverAnswer{Said: "x"}), &out); err == nil || asked != 0 {
t.Fatalf("%v was asked: %v", args, err)
}
}
if err := setuidSearchAsked([]string{"novox"}, func(string) error { return errors.New("no node called novox") },
ask(link.HandOverAnswer{Said: "x"}), &out); err == nil || asked != 0 {
t.Fatalf("an unknown node: %v", err)
}
if err := setuidSearchAsked([]string{"novox"}, known, ask(link.HandOverAnswer{Refused: "no; no search was started"}),
&out); err == nil || !strings.Contains(err.Error(), "novox refused") || out.Len() != 0 {
t.Fatalf("a refusal: %v, printed %q", err, out.String())
}
if err := setuidSearchAsked([]string{"novox"}, known, ask(link.HandOverAnswer{Said: "a new search starts"}),
&out); err != nil || !strings.HasPrefix(out.String(), "a new search starts\n") {
t.Fatalf("a start: %v, printed %q", err, out.String())
}
}
+1 -1
View File
@@ -625,7 +625,7 @@ func planStale(ctx context.Context, inv *inventory.Inventory, p inventory.Plan)
}
for _, newer := range recent {
if newer.ID == p.ID || !newer.Created.After(p.Created) || !repositoryMatches(newer.Repository, p.Repository) ||
newer.Branch != p.Branch || newer.State == inventory.PlanSuperseded {
newer.Branch != p.Branch || newer.State == inventory.PlanSuperseded || newer.Batch() {
continue
}
return inventory.PlanSuperseded, fmt.Sprintf("superseded by %s (%s %s), a newer merge of the same repository "+
+25
View File
@@ -45,3 +45,28 @@ func TestModuleCheckCountsTheUndeclaredAndRefusesThemFromTheDate(t *testing.T) {
t.Errorf("the refusal does not name the resource:\n%s", out.String())
}
}
// A file that asks for a setting without saying whether it is trusted counts as trusted (novox/hq issue 339):
// `module check` lists and counts it, and never refuses it — there is nothing unsafe to refuse.
func TestModuleCheckListsUnmarkedFilesAndNeverRefusesThem(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "module.json")
os.WriteFile(path, []byte(`{"module":"power","resources":[
{"id":"logind","type":"file","path":"/etc/systemd/logind.conf.d/power.conf","mode":"0644","trusted":true,
"content":"HandleLidSwitch=${setting:lid}\n"},
{"id":"note","type":"file","path":"/var/lib/power/note","mode":"0644","content":"${setting:greeting}\n"}]}`), 0o600)
defer func() { checkNow = time.Now }()
for _, at := range []time.Time{time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC), time.Date(2036, 1, 1, 0, 0, 0, 0, time.UTC)} {
checkNow = func() time.Time { return at }
var out bytes.Buffer
if err := moduleCheck([]string{path}, &out); err != nil {
t.Fatalf("refused at %v: %v\n%s", at, err, out.String())
}
for _, want := range []string{"note ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted",
UnsaidTrustLine + " 1"} {
if !strings.Contains(out.String(), want) {
t.Errorf("the check does not say %q:\n%s", want, out.String())
}
}
}
}
+26
View File
@@ -0,0 +1,26 @@
package main
import (
"os"
"golang.org/x/sys/unix"
)
// hideTyping turns a terminal's echo off while a secret is typed at it, and gives back what restores it. On
// anything that is not a terminal (a pipe, a file) it does nothing.
func hideTyping(f *os.File) func() {
fd := int(f.Fd())
before, err := unix.IoctlGetTermios(fd, unix.TCGETS)
if err != nil {
return func() {}
}
hidden := *before
hidden.Lflag &^= unix.ECHO
if err := unix.IoctlSetTermios(fd, unix.TCSETS, &hidden); err != nil {
return func() {}
}
return func() {
_ = unix.IoctlSetTermios(fd, unix.TCSETS, before)
_, _ = os.Stderr.WriteString("\n")
}
}
@@ -92,3 +92,26 @@ func TestHoldingNeedsAnAnswer(t *testing.T) {
}
}
}
// The control-node withholds the login shell's `execute` (novox/hq ADR 0268): its holder there serves
// nothing on the seat, and must not be judged silent for it, as the store's rows read it back.
func TestALoginShellWithholdingExecuteIsNotSilent(t *testing.T) {
defer catalogue.UseSeats(catalogue.DefaultSeats())
var rows []catalogue.Seat
for _, s := range catalogue.DefaultSeats() {
stored := s
stored.Serves = nil
for _, v := range s.Serves {
v.Optional = false // the store never keeps the mark
stored.Serves = append(stored.Serves, v)
}
rows = append(rows, stored)
}
catalogue.UseSeats(rows)
recorded := []catalogue.Held{{Claim: catalogue.LoginShellSeat, Scope: catalogue.ScopeNode, Node: "anchor", Module: "zsh"}}
expected := holdersToHear(catalogue.SeatsWithAProtocol(), recorded, nil, map[string]bool{"anchor": true}, nil, time.Now())
if _, asked := expected[catalogue.LoginShellSeat]; asked {
t.Fatalf("the login shell's holder is expected to answer, so withholding execute would be said silent: %v",
expected[catalogue.LoginShellSeat])
}
}
+221
View File
@@ -0,0 +1,221 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq issue 348: on 2026-10-09 the control node's resolver stopped answering on its private address
// at 10:57:57 UTC; the machine's network condition was raised at 10:58:45. The node-engine and the
// controller were sent at 10:59:34. The new node-engine's first statement judged the names once — unknown,
// "one look failed; a second decides" — and that statement cleared the condition, though its last evidence
// still said "connection refused". The next look raised it again at 11:00:23, after the send, and both
// builds failed their gate at 11:10 with "raised since it was sent" and were put back, for a fault that
// began before they were sent.
// namesRefused is the control node's names part as its node-engine said it in the outage: its own
// resolver, at its own address, refusing.
func namesRefused(state string, streak int) link.NetworkPart {
p := link.NetworkPart{Part: link.PartNames, State: state, Since: h0, Streak: streak}
if state == link.StateUnhealthy {
p.Reason = "1 of its 2 resolvers do not answer as the mesh's do"
p.Said = "10.77.0.1 — anchor.internal (IPv4): read udp 10.77.0.1:35244->10.77.0.1:53: read: connection refused"
p.Toward = []string{"10.77.0.1"}
}
return p
}
func networkSaying(parts ...link.NetworkPart) *link.NetworkHealth {
state := link.StateHealthy
for _, p := range parts {
switch {
case p.State == link.StateUnhealthy:
state = link.StateUnhealthy
case p.State == link.StateUnknown && state == link.StateHealthy:
state = link.StateUnknown
}
}
return &link.NetworkHealth{State: state, Since: h0, Parts: parts}
}
// TestReplay348 replays the statements of the outage: the condition raised before the send is not
// cleared by the restarted engine's first, undecided statement, and the gate does not count it against
// the builds sent after it began.
func TestReplay348(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv, k := open.inventory, conditionsFrom
say := func(at time.Time, n *link.NetworkHealth) {
t.Helper()
if err := stateHealth(ctx, inv, k, "anchor", link.Health{Contract: link.ReadinessContract, At: at, Network: n}, at); err != nil {
t.Fatal(err)
}
}
network := func() (conditions.Condition, bool) {
t.Helper()
list, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
for _, c := range list {
if c.Key == "machine.anchor.network" {
return c, true
}
}
return conditions.Condition{}, false
}
// 10:58:45 — the second failing look: raised.
say(h0, networkSaying(namesRefused(link.StateUnhealthy, 2)))
raised, ok := network()
if !ok {
t.Fatal("the resolver refusing on the control node raised nothing")
}
time.Sleep(5 * time.Millisecond)
sent := time.Now().UTC()
time.Sleep(5 * time.Millisecond)
// 10:59:42 — the restarted engine's first statement: one look failed, a second decides.
say(h0.Add(time.Minute), networkSaying(namesRefused(link.StateUnknown, 1)))
if _, ok := network(); !ok {
t.Fatal("a statement that judged nothing yet cleared the condition: the restarted engine's first look " +
"said the fault was gone while it still refused")
}
// 11:00:23 — its second look: unhealthy again, the same raising.
say(h0.Add(2*time.Minute), networkSaying(namesRefused(link.StateUnhealthy, 2)))
again, ok := network()
if !ok || !again.Raised.Equal(raised.Raised) || again.Count != 1 {
t.Fatalf("the same raising was not kept: raised %s (first %s), count %d", again.Raised, raised.Raised, again.Count)
}
// The gate on the control node, for a build sent after the fault began.
open2, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
f := gateFacts{judged: true, open: open2}
if w := aboutTheMachine("anchor", []string{"mesh-host"}, sent, f); w.whole != "" || len(w.on) != 0 {
t.Fatalf("a fault from before the send held the build: %+v", w)
}
// Decided healthy: cleared.
say(h0.Add(3*time.Minute), networkSaying(link.NetworkPart{Part: link.PartNames, State: link.StateHealthy, Since: h0}))
if c, ok := network(); ok {
t.Fatalf("a statement that decides the names healthy left %s open", c.Key)
}
}
// A fault there at the send, cleared and reopened after it, is not raised since the send; one that cleared
// before the send and came back after it is — a send that breaks a recovered machine fails its gate (review
// of mesh-controller PR 179, A2). Read through OpenAt, by both of the gate's readings.
func TestAFaultThatFlappedAfterTheSendIsNotTheSendsAndOneThatRecoveredBeforeItIs(t *testing.T) {
since := h0
network := func(first time.Time, gaps ...conditions.Gap) conditions.Condition {
raised := since.Add(time.Minute)
if len(gaps) > 0 {
raised = gaps[len(gaps)-1].Reopened
}
return conditions.Condition{Key: "machine.anchor.network", Kind: kindMachineNetwork,
Subject: conditions.Subject{Scope: conditions.ScopeMachine, ID: "anchor", Machine: "anchor"},
Summary: "anchor's network is not healthy", Source: sourceNetwork, First: first, Gaps: gaps, Raised: raised}
}
held := func(c conditions.Condition) bool {
return aboutTheMachine("anchor", []string{"mesh-controller"}, since, gateFacts{judged: true,
open: []conditions.Condition{c}}).whole != ""
}
// The day's case: raised before the send, cleared 8 s after it, reopened 49 s after it.
flapped := network(since.Add(-49*time.Second),
conditions.Gap{Cleared: since.Add(8 * time.Second), Reopened: since.Add(49 * time.Second)})
if held(flapped) {
t.Fatal("a fault there at the send, flapping after it, held the machine")
}
// Recovered before the send, broken again after it: the send's.
recovered := network(since.Add(-time.Hour),
conditions.Gap{Cleared: since.Add(-30 * time.Second), Reopened: since.Add(20 * time.Second)})
if !held(recovered) {
t.Fatal("a machine recovered at the send and broken after it passed the gate")
}
// An older gap, before the send, and the fault there at the send: not the send's.
twice := network(since.Add(-time.Hour),
conditions.Gap{Cleared: since.Add(-50 * time.Minute), Reopened: since.Add(-45 * time.Minute)},
conditions.Gap{Cleared: since.Add(10 * time.Second), Reopened: since.Add(30 * time.Second)})
if held(twice) {
t.Fatal("a fault there at the send, with an older gap, held the machine")
}
// Raised after the send, never cleared: the send's.
if !held(network(time.Time{})) {
t.Fatal("a fault raised after the send held nothing")
}
// And a module's own, through judgeHealth.
at := since.Add(2 * time.Minute)
g := gateFacts{judged: true, now: at, reports: map[string]inventory.Reported{"anchor": {Node: "anchor",
Outcome: inventory.OutcomeApplied, At: &at, Current: true}}, engines: map[string]string{},
served: map[string]served{}, rolledBack: map[string][]lease.Rollback{},
open: []conditions.Condition{{Key: "provider.app.anchor.x.failing", Subject: conditions.Subject{
Scope: conditions.ScopeProvider, ID: "app.anchor.x", Machine: "anchor"}, Summary: "failing",
First: since.Add(-time.Hour), Raised: since.Add(time.Minute),
Gaps: []conditions.Gap{{Cleared: since.Add(5 * time.Second), Reopened: since.Add(time.Minute)}}}}}
if _, why := judgeHealth("app", "", catalogue.Manifest{Module: "app"}, "anchor", since, g); strings.HasPrefix(why, "raised since it was sent") {
t.Fatalf("a module's own fault there at the send: %s", why)
}
g.open[0].Gaps[0].Cleared = since.Add(-5 * time.Second)
if _, why := judgeHealth("app", "", catalogue.Manifest{Module: "app"}, "anchor", since, g); !strings.HasPrefix(why, "raised since it was sent") {
t.Fatalf("a module's own fault, recovered at the send and back after it, was not counted: %s", why)
}
}
// Only an undecided part holds a condition that names it; a condition about another part clears, and a
// statement unknown as a whole holds every part (review of PR 179, A4). Pure.
func TestAnUndecidedPartHoldsOnlyWhatNamesIt(t *testing.T) {
f := netFacts(map[string]*inventory.NetworkHealth{
"anchor": aNetwork(link.StateUnknown, inventory.NetworkPart{Part: link.PartNames, State: link.StateUnknown, Streak: 1},
inventory.NetworkPart{Part: link.PartRoute, State: link.StateHealthy}),
"laptop": aNetwork(link.StateHealthy, inventory.NetworkPart{Part: link.PartNames, State: link.StateHealthy}),
"spare": aNetwork(link.StateStarting, inventory.NetworkPart{Part: link.PartTunnel, State: link.StateHealthy}),
"other": aNetwork(link.StateStarting, inventory.NetworkPart{Part: link.PartTunnel, State: link.StateStarting}),
})
u := undecidedParts(f)
if !u["anchor"][link.PartNames] || u["anchor"][link.PartRoute] || u["laptop"] != nil || !u["spare"]["*"] ||
!u["other"][link.PartTunnel] || u["other"]["*"] {
t.Fatalf("undecided: %v", u)
}
about := func(machine, said string, also ...string) conditions.Condition {
return conditions.Condition{Subject: conditions.Subject{Scope: conditions.ScopeMachine, ID: machine,
Machine: machine, Also: also}, Evidence: []conditions.Evidence{{Said: said}}}
}
for _, c := range []struct {
c conditions.Condition
held bool
}{
{about("anchor", "names since 2026-10-09 10:58:45 UTC: 10.77.0.1 — refused"), true},
{about("anchor", "route since 2026-10-09 10:58:45 UTC: no default route"), false},
{about("laptop", "names since 2026-10-09 10:58:45 UTC: refused"), false},
{about("spare", "route since …: no default route"), true},
{about("hub", "anchor: names: refused", "anchor"), true},
{about("hub", "anchor: tunnel: no handshake", "anchor"), false},
} {
if got := heldUndecided(c.c, u); got != c.held {
t.Errorf("%s %q held %v, want %v", c.c.Subject.Machine, c.c.Evidence[0].Said, got, c.held)
}
}
}
// A release walks its modules without a record per module: D10 counts what its tier names as rolling,
// so the node-engine a release walks is not "behind, and no plan is rolling it out" on its first machine.
func TestAReleaseRollsOutWhatItsTierNames(t *testing.T) {
plans := []inventory.Plan{
{ID: "release-1", State: inventory.PlanRolling, Tiers: [][]string{{"mesh-host"}}, Modules: map[string]*inventory.PlanModule{}},
{ID: "plan-2", State: inventory.PlanRolling, Modules: map[string]*inventory.PlanModule{"letta": {}}},
}
got := rollingModules(plans)
if !got["mesh-host"] || !got["letta"] || len(got) != 2 {
t.Fatalf("rolling: %v", got)
}
}
+356
View File
@@ -0,0 +1,356 @@
package main
import (
"context"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
)
// novox/hq issue 352: on 2026-10-09 a release's gate on the control node read the machine's report against a
// newer send another plan had just made there — not against its own send — and failed three builds the
// machine had reported healthy ("has not reported on what it was sent"), put them back on every machine,
// to a controller older than the store's schema, and that controller then judged the newer plan's
// controller passed from the put-back build's health.
// TestReplay352 replays the walk on the backlog fixture: the release sends anchor and anchor reports;
// another send reaches anchor, unreported; the gate still passes. And a send that moves a judged module
// to another build supersedes the judging: no verdict, nothing put back.
func TestReplay352(t *testing.T) {
t.Run("a newer send to the judged machine does not unreport the gate's", testANewerSendDoesNotUnreportTheGatesOwn)
t.Run("a send that moves the module supersedes the judging", testASendThatMovesTheModuleSupersedesTheJudging)
}
func testANewerSendDoesNotUnreportTheGatesOwn(t *testing.T) {
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
advancePlans(ctx, b.open) // anchor is sent, and the fixture reports it applied
// 16:31:37 — another plan sends anchor a newer declaration, which it has not reported on.
carried, _, _ := inv.SentBuilds(ctx, "anchor")
if err := inv.RecordSent(ctx, nodeID(t, b.open, "anchor"), "d-anchor-newer", carried); err != nil {
t.Fatal(err)
}
reports, _ := inv.LastReports(ctx)
for _, r := range reports {
if r.Node == "anchor" && r.Current {
t.Fatal("the fixture's newer send reads as reported")
}
}
gateEvery, gateBound = 0, 0 // past the bound at once: before the fix, "has not reported" fails it here
for i := 0; i < 4; i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
if p.State == inventory.PlanFailed || strings.Contains(p.Note, "has not reported") {
t.Fatalf("the release failed on the newer send: %s %s", p.State, p.Note)
}
if g := p.Release.Gate; g != nil && (g.Sent == nil || g.Sent["anchor"].Digest == "") {
t.Fatalf("the gate does not keep what it sent: %+v", g)
}
if v, found, err := inv.GateOf(ctx, "build-app-c2"); err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("app's pass on anchor was not kept: %+v %v %v", v, found, err)
}
if current, _ := inv.CurrentBuilds(ctx); current["app"].Commit != "c2" {
t.Fatalf("app was put back to %s", current["app"].Commit)
}
}
// A plan's own first send waits while a release judges the same module on that machine with another build.
func TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere(t *testing.T) {
b := aBacklog(t)
ctx := t.Context()
advancePlans(ctx, b.open) // the release judges app c2 on anchor
_, _, err := gatedSend(ctx, b.open, "anchor", []inventory.CarriedMove{{Module: "app", Node: "anchor", From: "c2", To: "c3", Build: "build-app-c3"}})
if !errors.Is(err, errWalkedElsewhere) || !strings.Contains(err.Error(), "release-") {
t.Fatalf("a newer build of a judged module was sent under the release's gate: %v", err)
}
if len(b.sent) != 1 {
t.Fatalf("sent %v", b.sent)
}
}
// A merge plan's judging is superseded the same way: another send moved its module on the first machine.
func TestAPlansJudgingIsSupersededByASendThatMovesItsModule(t *testing.T) {
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
advancePlans(ctx, g.open) // anchor is sent app c2 first
if err := inv.RecordSent(ctx, nodeID(t, g.open, "anchor"), "d-anchor-c3", map[string]string{"app": "c3"}); err != nil {
t.Fatal(err)
}
gateEvery = 0
advancePlans(ctx, g.open)
p := g.plan(t)
if p.State != inventory.PlanSuperseded || !strings.Contains(p.Note, "superseded") || !strings.Contains(p.Note, "c3") {
t.Fatalf("the plan is %s: %s", p.State, p.Note)
}
// Nothing put back: the registered build stands, the build is not marked, and the plan's gate made no
// rollback (a release may walk what the other send left waiting on anchor; that is not a put-back).
if r := p.Modules["app"].Gate.Rollback; r != "" {
t.Fatalf("a superseded judging made a rollback: %q", r)
}
if current, _ := inv.CurrentBuilds(ctx); current["app"].Commit != "c2" {
t.Fatalf("app was put back to %s", current["app"].Commit)
}
if failed, _ := inv.GateFailed(ctx, "build-2"); failed {
t.Fatal("a superseded build was marked failed")
}
}
func testASendThatMovesTheModuleSupersedesTheJudging(t *testing.T) {
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
advancePlans(ctx, b.open)
// Another send moves app on anchor to a build this gate does not judge.
carried, _, _ := inv.SentBuilds(ctx, "anchor")
carried["app"] = "c3"
if err := inv.RecordSent(ctx, nodeID(t, b.open, "anchor"), "d-anchor-c3", carried); err != nil {
t.Fatal(err)
}
gateEvery = 0
advancePlans(ctx, b.open)
p := b.release(t)
if p.State != inventory.PlanSuperseded || !strings.Contains(p.Note, "superseded") || !strings.Contains(p.Note, "c3") {
t.Fatalf("the release is %s: %s", p.State, p.Note)
}
if !reflect.DeepEqual(b.sent, [][]string{{"anchor"}}) {
t.Fatalf("sent %v: a superseded judging puts nothing back", b.sent)
}
if _, found, _ := inv.GateOf(ctx, "build-app-c2"); found {
t.Fatal("a superseded judging kept a verdict")
}
if current, _ := inv.CurrentBuilds(ctx); current["app"].Commit != "c2" {
t.Fatalf("app was put back to %s", current["app"].Commit)
}
}
// A report is on the gate's own send: the declaration itself, or one sequenced after it; a gate kept
// without its send reads the report against the send made last, as before. Pure.
func TestAReportIsHeldAgainstTheGatesOwnSend(t *testing.T) {
sent := inventory.SentDeclaration{Digest: "d-490", Sequence: 490}
for _, c := range []struct {
r inventory.Reported
want bool
}{
{inventory.Reported{Declared: "d-490", Current: false}, true},
{inventory.Reported{Declared: "d-491", ReportedSequence: 491, Current: true}, true},
{inventory.Reported{Declared: "d-489", ReportedSequence: 489, Current: false}, false},
{inventory.Reported{Declared: "other", ReportedSequence: 490}, true}, // the same sequence, said by another digest
{inventory.Reported{Declared: "d-495", ReportedSequence: 495, Current: true}, true}, // the last send: this one or a later one
{inventory.Reported{Declared: "", Current: false}, false},
} {
if got := sent.ReportsOn(c.r); got != c.want {
t.Errorf("%+v on %+v: %v", c.r, sent, got)
}
}
byDigest := inventory.SentDeclaration{Digest: "d-1"}
if !byDigest.ReportsOn(inventory.Reported{Declared: "d-1"}) || byDigest.ReportsOn(inventory.Reported{ReportedSequence: 5}) ||
!byDigest.ReportsOn(inventory.Reported{Current: true}) {
t.Fatal("a send kept without a sequence is matched by its digest and by the last send alone")
}
f := gateFacts{sent: map[string]inventory.SentDeclaration{"anchor": sent}}
if !f.reportedOn("anchor", inventory.Reported{Declared: "d-490"}) || f.reportedOn("anchor", inventory.Reported{Declared: "d-1"}) {
t.Fatal("a gate that kept its send read the report against something other than it")
}
if !f.reportedOn("laptop", inventory.Reported{Current: true}) || f.reportedOn("laptop", inventory.Reported{Current: false}) {
t.Fatal("a gate that did not keep its send does not read the report against the send made last")
}
// Through the merge plan's first-machine wait too.
at := time.Now()
state := inventory.PlanModule{First: []string{"anchor"}, FirstAt: &at,
Gate: &inventory.PlanGate{Machines: []string{"anchor"}, Sent: map[string]inventory.SentDeclaration{"anchor": sent}}}
reports := []inventory.Reported{{Node: "anchor", At: &at, Outcome: inventory.OutcomeApplied, Current: false, Declared: "d-490"}}
if step := nextRollout(state, []string{"anchor", "laptop"}, false, reports, at.Add(time.Minute), time.Hour); step.waiting != "" || step.failed != "" {
t.Fatalf("the first machine's report on the plan's own send read as none: %+v", step)
}
reports[0].Declared = "d-480"
if step := nextRollout(state, []string{"anchor", "laptop"}, false, reports, at.Add(time.Minute), time.Hour); step.waiting == "" {
t.Fatalf("a report on an older send read as the plan's: %+v", step)
}
}
// A gate judges only the build the machine was last sent: last sent another build of the module, the
// judging is superseded, whatever the machine reports. Pure.
func TestAGateJudgesOnlyTheBuildTheMachineWasLastSent(t *testing.T) {
at := time.Now()
f := gateFacts{now: at, reports: map[string]inventory.Reported{"anchor": {Node: "anchor", Outcome: inventory.OutcomeApplied,
At: &at, Current: true}}, engines: map[string]string{}, served: map[string]served{}, rolledBack: map[string][]lease.Rollback{},
commits: map[string]string{"mesh-controller": "e6b00e2e"}, sentBuilds: map[string]map[string]string{"anchor": {"mesh-controller": "ef26d4cb"}}}
taken := at.Add(-30 * time.Second)
f.holder = &lease.Holder{Taken: taken, Health: &lease.Health{Ready: true}}
h, why := judgeHealth("mesh-controller", lease.ComponentController, catalogue.Manifest{}, "anchor", at.Add(-time.Minute), f)
if h != healthSuperseded || !strings.Contains(why, "ef26d4cb") || !strings.Contains(why, "e6b00e2e") {
t.Fatalf("a controller build the machine no longer runs: %v %q", h, why)
}
f.sentBuilds["anchor"]["mesh-controller"] = "e6b00e2e"
if h, why := judgeHealth("mesh-controller", lease.ComponentController, catalogue.Manifest{}, "anchor", at.Add(-time.Minute), f); h == healthSuperseded {
t.Fatalf("the build sent read as another: %q", why)
}
delete(f.sentBuilds, "anchor")
if h, why := judgeHealth("mesh-controller", lease.ComponentController, catalogue.Manifest{}, "anchor", at.Add(-time.Minute), f); h == healthSuperseded {
t.Fatalf("a machine whose send is not known read as superseded: %q", why)
}
g := &inventory.PlanGate{To: "c2", Carried: []inventory.CarriedMove{{Module: "late", Node: "anchor", To: "c5"}}}
if got := judgedCommits(g, []judged{{"app", "anchor"}, {"late", "anchor"}}); got["app"] != "c2" || got["late"] != "c5" {
t.Fatalf("judged commits %v", got)
}
}
// A move of another build of a module to a machine where a release or a plan is judging that module
// waits for that judging; the same build to that machine is already there. Pure.
func TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine(t *testing.T) {
at := time.Now()
release := inventory.Plan{ID: "release-1", State: inventory.PlanRolling, Release: &inventory.PlanRelease{
Gate: &inventory.PlanGate{Machines: []string{"novox"}, Carried: []inventory.CarriedMove{
{Module: "mesh-controller", Node: "novox", From: "ef26d4cb", To: "2913c54c"}}}}}
merge := inventory.Plan{ID: "plan-1", State: inventory.PlanRolling, Modules: map[string]*inventory.PlanModule{
"app": {First: []string{"anchor"}, FirstAt: &at, Commit: "c2", Gate: &inventory.PlanGate{Machines: []string{"anchor"}}}}}
f := moveFacts{plans: []inventory.Plan{release, merge}}
for _, c := range []struct {
module, node, to, want string
}{
{"mesh-controller", "novox", "e6b00e2e", "release-1"}, // the day's case: a newer controller to the judged machine
{"mesh-controller", "novox", "2913c54c", ""}, // the same build: already there
{"mesh-controller", "ace", "2913c54c", "release-1"}, // another machine while the first is judged
{"mesh-host", "novox", "x", ""}, // a module the release does not carry
{"app", "anchor", "c2", ""},
{"app", "anchor", "c3", "plan-1"},
{"app", "laptop", "c2", "plan-1"},
} {
if got := f.walkedBy(c.module, c.node, c.to); got != c.want {
t.Errorf("%s %s to %s: walked by %q, want %q", c.module, c.to, c.node, got, c.want)
}
}
release.Release.Gate.Verdict = inventory.GatePassed
merge.Modules["app"].Gate.Verdict = inventory.GatePassed
if f.walkedBy("mesh-controller", "novox", "e6b00e2e") != "" || f.walkedBy("app", "laptop", "c3") != "" {
t.Fatal("a passed judging still holds a move")
}
release.Release.Gate.Verdict = ""
f.plans[0].State = inventory.PlanSuperseded
if f.walkedBy("mesh-controller", "novox", "e6b00e2e") != "" {
t.Fatal("a closed release still holds a move")
}
}
// The controller is never put back to a build that reaches less of the store's schema than the store
// has, or to one that never said what it reaches: the current build is kept, and the condition says so.
func TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
keeper, _ := withConditionsInMemory(t)
told := &conditions.Told{}
was := doctorFrom
doctorFrom = &doctor{open: open, keeper: keeper, teller: told}
t.Cleanup(func() { doctorFrom = was })
wasSend := sendRollout
var sent [][]string
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
sent = append(sent, names)
return names, nil
}
t.Cleanup(func() { sendRollout = wasSend })
build := func(id, commit, digest string, asked time.Time) inventory.Build {
manifest, _ := json.Marshal(catalogue.Manifest{Module: "mesh-controller", Version: commit})
b := inventory.Build{ID: id, Module: "mesh-controller", Commit: commit, Repository: "novox/mesh-controller", Path: ".",
Manifest: manifest, Asked: asked, At: asked, Made: []inventory.Artifact{{Name: "controller", Kind: catalogue.ArtifactBundle,
Reference: "mesh-artifact://mesh-controller/controller/blobs/sha256:" + digest}}}
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: commit},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Path: ".", BuiltFrom: commit, Head: commit, Asked: asked}); err != nil {
t.Fatal(err)
}
return b
}
previous := build("build-old", "ef26d4cb", strings.Repeat("1", 64), time.Now().Add(-2*time.Hour))
failed := build("build-new", "e6b00e2e", strings.Repeat("2", 64), time.Now().Add(-time.Minute))
if versionOfBuild(previous) != strings.Repeat("1", 12) {
t.Fatalf("the build's version is %q", versionOfBuild(previous))
}
applied, err := inv.SchemaApplied(ctx)
if err != nil || applied < 87 {
t.Fatalf("the store's schema reaches %d (%v)", applied, err)
}
// The build before never recorded what it reads: not proved, refused.
if why, ok := controllerSchemaAllows(ctx, inv, previous); ok || !strings.Contains(why, "never recorded") {
t.Fatalf("an unknown reach: %v %q", ok, why)
}
// It reads less than the store has: refused, naming both.
if err := inv.RecordSchemaReach(ctx, versionOfBuild(previous), applied-1); err != nil {
t.Fatal(err)
}
if why, ok := controllerSchemaAllows(ctx, inv, previous); ok || !strings.Contains(why, "is at") {
t.Fatalf("a reach behind the store: %v %q", ok, why)
}
// Through the gate: the failed build is marked, nothing is put back, nothing is sent, the condition is urgent.
at := time.Now().Add(-5 * time.Minute)
state := &inventory.PlanModule{Build: failed.ID, Commit: failed.Commit, Previous: previous.Commit, First: []string{"anchor"}, FirstAt: &at}
p := inventory.Plan{ID: "plan-352", Repository: "novox/mesh-controller", Branch: "main", Commit: failed.Commit, Created: at,
State: inventory.PlanRolling, Tiers: [][]string{{"mesh-controller"}}, Modules: map[string]*inventory.PlanModule{"mesh-controller": state}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
gateFailed(ctx, open, &p, "mesh-controller", state, []string{"anchor"}, "not healthy within 10m0s of its apply")
if state.Gate.Rollback != inventory.NotRolledBack || !strings.Contains(p.Note, "NOT put back") || !strings.Contains(p.Note, "the current build is kept") {
t.Fatalf("rollback %q: %s", state.Gate.Rollback, p.Note)
}
if len(sent) != 0 {
t.Fatalf("sent %v: nothing is put back", sent)
}
if current, _ := inv.CurrentBuilds(ctx); current["mesh-controller"].Commit != failed.Commit {
t.Fatalf("the module was put back to %s", current["mesh-controller"].Commit)
}
if marked, _ := inv.GateFailed(ctx, failed.ID); !marked {
t.Fatal("the failed build is not marked failed at its gate")
}
open2, _ := keeper.Open(ctx)
var found bool
for _, c := range open2 {
if c.Kind == kindRollbackFailed && c.Severity == conditions.Urgent && strings.Contains(c.Summary, "current build is kept") {
found = true
}
}
if !found {
t.Fatalf("no urgent rollback-failed condition saying the current build is kept: %+v", open2)
}
// Reaching the store: allowed.
if err := inv.RecordSchemaReach(ctx, versionOfBuild(previous), applied); err != nil {
t.Fatal(err)
}
if why, ok := controllerSchemaAllows(ctx, inv, previous); !ok {
t.Fatalf("a build that reads the whole schema was refused: %q", why)
}
// A build with no bundle to know it by: refused.
if why, ok := controllerSchemaAllows(ctx, inv, inventory.Build{Commit: "x"}); ok || !strings.Contains(why, "names no bundle") {
t.Fatalf("a build without a bundle: %v %q", ok, why)
}
}
// The lease's holder names the build the declaration told it it is, and the version stamp only without one.
func TestTheHolderNamesTheBuildTheDeclarationToldIt(t *testing.T) {
t.Setenv(RunningBuildVar, " ad62528c47c7 ")
if h := holderOf("x"); h.Build != "ad62528c47c7" {
t.Fatalf("the holder's build is %q", h.Build)
}
t.Setenv(RunningBuildVar, "")
if h := holderOf("x"); h.Build != version {
t.Fatalf("without a declared version the holder's build is %q", h.Build)
}
if reach, err := schemaReach(); err != nil || reach < 87 {
t.Fatalf("this build's reach is %d (%v)", reach, err)
}
}
+55 -1
View File
@@ -98,8 +98,9 @@ func judgeNetworks(ctx context.Context, inv *inventory.Inventory, k *conditions.
problems = append(problems, err.Error())
}
}
undecided := undecidedParts(f)
for _, c := range open {
if !slices.Contains(networkKinds, c.Kind) || said[c.Key] {
if !slices.Contains(networkKinds, c.Kind) || said[c.Key] || heldUndecided(c, undecided) {
continue
}
why := "no machine says it any more"
@@ -116,6 +117,59 @@ func judgeNetworks(ctx context.Context, inv *inventory.Inventory, k *conditions.
return nil
}
// undecidedParts is, per machine, every part of its newest statement not yet judged: starting, or unknown
// — one look failed and a second decides (novox/hq issue 348). Such a part does not say its fault is gone.
// A statement whose parts are all decided but whose whole is unknown or starting holds every part. Pure.
//
// On 2026-10-09 the control node's resolver refused every question from 10:58 to 11:18 UTC. A build of
// the node-engine sent at 10:59:34 restarted it; its first statement judged the names once (unknown, "one
// look failed; a second decides"), and that statement cleared the control node's network condition while
// its last evidence still said "connection refused". The second look raised it again forty seconds later —
// after the send — and the gate failed the build for a fault from before it.
func undecidedParts(f networkFacts) map[string]map[string]bool {
out := map[string]map[string]bool{}
for m, h := range f.healths {
if h.Network == nil {
continue
}
parts := map[string]bool{}
for _, p := range h.Network.Parts {
if p.State == link.StateUnknown || p.State == link.StateStarting {
parts[p.Part] = true
}
}
if len(parts) == 0 && (h.Network.State == link.StateUnknown || h.Network.State == link.StateStarting) {
parts["*"] = true
}
if len(parts) > 0 {
out[m] = parts
}
}
return out
}
// heldUndecided says an open network condition is kept rather than cleared: a part its newest evidence
// names is undecided in the newest statement of a machine it is about. A condition about other parts
// clears as before. Pure.
func heldUndecided(c conditions.Condition, undecided map[string]map[string]bool) bool {
said := ""
if len(c.Evidence) > 0 {
said = c.Evidence[0].Said
}
for _, m := range append([]string{c.Subject.Machine}, c.Subject.Also...) {
parts := undecided[m]
if parts["*"] {
return true
}
for part := range parts {
if strings.Contains(said, part+" since ") || strings.Contains(said, part+": ") {
return true
}
}
}
return false
}
// pointed is one machine's failing part that points at another machine.
type pointed struct {
from string
+25
View File
@@ -15,6 +15,7 @@ import (
"os/signal"
"syscall"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/identity"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
@@ -53,6 +54,9 @@ func run() error {
return fmt.Errorf("no command given")
}
// Every connection this process dials says what it is (novox/hq issue 327).
broker.ConnectionName = connectionName(args[0], os.Getenv(verbVar))
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
// Whatever this process holds of the controller's lease is given back as it ends (novox/hq to-be
@@ -74,6 +78,8 @@ func run() error {
return rotateCommand(ctx, args[1:])
case "ask":
return askCommand(ctx, args[1:])
case "rehearse":
return rehearseCommand(ctx, args[1:])
case "builds":
return buildsCommand(ctx, args[1:])
// The build queue, controlled by hand (novox/hq ADR 0219).
@@ -212,6 +218,10 @@ func run() error {
func usage() {
fmt.Fprint(os.Stderr, `mesh-controller — the control plane
On a node the operator types these as 'mesh-cli <command>' (in zsh, 'nox <command>'): asked
through the node's engine, and run as the controller's terminal only on the control-node
(novox/hq ADR 0272).
migrate bring each context's schema up to date
prepare the same, asked the way the mesh asks any module (ADR 0135)
node add <name> [--adopted] create a node record; --adopted: the machine is in use
@@ -416,3 +426,18 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
statusFrom.nudge()
return nil
}
// verbVar carries the seat verb a command runs for, from the serving controller to the process it starts.
const verbVar = "MESH_VERB"
// connectionName is what this process's connections say they are in the bus's list (novox/hq issue 327):
// the serving controller, a verb's own process and which verb, or a command run at a shell and which.
func connectionName(command, verb string) string {
switch {
case command == "serve":
return "mesh-controller serving"
case verb != "":
return "mesh-controller verb " + verb
}
return "mesh-controller command " + command
}
+17 -1
View File
@@ -1013,6 +1013,11 @@ func raiseFromFacts(ctx context.Context, open *stores, f snapshot.Facts, shelf m
return notes, err
}
}
if m.AgentAccount != "" {
if err := inv.SetAgentAccount(ctx, m.Name, m.AgentAccount, m.AgentAccountHome); err != nil {
return notes, err
}
}
if m.PublicDomain != "" {
if err := inv.SetPublicDomain(ctx, m.Name, m.PublicDomain); err != nil {
return notes, err
@@ -1199,7 +1204,18 @@ func graphOfFacts(f snapshot.Facts) ([]inventory.Entry, map[string][]inventory.R
entries = append(entries, inventory.Entry{Manifest: manifest, Provided: mod.Provided,
Source: inventory.Source{Repository: mod.Repository, Path: mod.Path, BuiltFrom: mod.Commit}})
for _, r := range mod.Reads {
read[mod.Name] = append(read[mod.Name], inventory.ReadRepository{Repository: r})
entry := inventory.ReadRepository{Repository: r}
for _, s := range mod.Sources {
if !s.Own && s.Repository == r && len(s.Paths) > 0 {
entry.Paths = s.Paths
}
}
read[mod.Name] = append(read[mod.Name], entry)
}
for _, s := range mod.Sources {
if s.Own && len(s.Paths) > 0 {
read[mod.Name] = append(read[mod.Name], inventory.ReadRepository{Own: true, Paths: s.Paths})
}
}
}
var edges []inventory.Edge
+271
View File
@@ -0,0 +1,271 @@
package main
import (
"bytes"
"context"
"errors"
"fmt"
"os/exec"
"slices"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The operator's command, `mesh-cli` (alias `nox`), answered here (novox/hq ADR 0272).
//
// mesh-cli asks the node-engine on its own machine; the engine reads the asking account from the kernel and asks
// this controller on its own node's subject. Here it is judged — the controller's terminal, an ordinary call, or
// nothing — and run as every verb's line is: a fresh process of this binary, with this process's environment.
//
// **The terminal** is a line from a node's operator account, on the control-node (§4). Phase 2 adds a node whose
// agents run under an account of their own, judged unable to become root (ADR 0266, not built yet); until then no
// other node is the terminal, because agents there run as its operator. **An ordinary call** is a line from that
// account elsewhere: it meets every refusal of the generic `command` verb and runs with `MESH_VERB=mesh-cli`, so a
// terminal-only change is refused with its reason. **Any other account is refused**, root included: those two
// accounts already reach the mesh's verbs through the mesh MCP server, and no other account gains anything here.
// cliVerb is what a line from mesh-cli that is not the terminal runs as: the verb its process names, so every
// terminal-only refusal applies and says it came through mesh-cli.
const cliVerb = "mesh-cli"
// cliServers are commands that serve until stopped. Run for mesh-cli they would hold a second server under this
// one until the call's bound killed it.
var cliServers = map[string]bool{"serve": true, "api": true, "board": true}
// cliVerdict is how a line is run, or why it is not.
type cliVerdict struct {
terminal bool
// why says why the line is or is not the terminal, in words the operator reads under the answer.
why string
// refused says why nothing runs.
refused string
}
// judgeCLI decides how a line from mesh-cli runs (ADR 0272 §4). nodes is every node the mesh knows; control is
// every node the controller's module is assigned to, which is exactly one in a mesh that is well.
func judgeCLI(node string, asked link.CLIAsked, nodes []inventory.Node, control []string) cliVerdict {
var record *inventory.Node
for i := range nodes {
if nodes[i].Name == node {
record = &nodes[i]
break
}
}
switch {
case record == nil:
return cliVerdict{refused: fmt.Sprintf("%s is not a node this mesh knows, so nothing ran", node)}
case asked.UID == 0 || asked.Account == "root":
return cliVerdict{refused: "mesh-cli answers the operator's own account, never root: run it as yourself, " +
"not under sudo. Nothing ran"}
case record.Account == "":
return cliVerdict{refused: fmt.Sprintf("the mesh does not know %s's operator account (`node account <node> <login>`), "+
"so no account there is answered. Nothing ran", node)}
case asked.Account != record.Account:
return cliVerdict{refused: fmt.Sprintf("mesh-cli answers %s's operator account (%s) only, and this line came "+
"from %s. Nothing ran", node, record.Account, asked.Account)}
}
if len(control) != 1 {
return cliVerdict{why: fmt.Sprintf("not the controller's terminal: the mesh names %d control-nodes, and the "+
"terminal is the one control-node's operator account", len(control))}
}
if control[0] == node {
// **A person at a terminal, not a service of the operator's** (review of ADR 0272): the tool runner and the
// account's user units run as the operator too, and only a login session is the terminal.
if asked.Session == "" {
return cliVerdict{why: fmt.Sprintf("not the controller's terminal: %s on %s asked from no login session — a "+
"service or a user unit running as the operator, not a person at a terminal", asked.Account, node)}
}
return cliVerdict{terminal: true, why: fmt.Sprintf("the controller's terminal: %s on the control-node %s, "+
"login session %s", asked.Account, node, asked.Session)}
}
return cliVerdict{why: fmt.Sprintf("not the controller's terminal: agents on %s may run as %s, so a "+
"terminal-only change is made from the control-node %s (novox/hq ADR 0272)", node, asked.Account, control[0])}
}
// controlNodes is every node the controller's module is assigned to.
func controlNodes(ctx context.Context, inv *inventory.Inventory, nodes []inventory.Node) ([]string, error) {
var out []string
for _, n := range nodes {
modules, err := inv.Assigned(ctx, n.Name)
if err != nil {
return nil, err
}
if slices.Contains(modules, controllerModule) {
out = append(out, n.Name)
}
}
return out, nil
}
// controllerModule is the module that runs the controller, and so marks the control-node.
const controllerModule = "mesh-controller"
// answerMeshCLI is the serving controller's answer to mesh-cli.
func answerMeshCLI(inv *inventory.Inventory) link.CLIHandler {
return func(ctx context.Context, node string, asked link.CLIAsked) link.CLIAnswer {
if handingOver.Load() {
return link.CLIRefusal("this controller is stopping and runs no new command; ask again in a moment. Nothing ran")
}
nodes, err := inv.Nodes(ctx)
if err != nil {
return link.CLIRefusal("the mesh's nodes cannot be read, so nothing ran: " + err.Error())
}
control, err := controlNodes(ctx, inv, nodes)
if err != nil {
return link.CLIRefusal("which node is the control-node cannot be read, so nothing ran: " + err.Error())
}
return runForMeshCLI(ctx, node, asked, judgeCLI(node, asked, nodes, control))
}
}
// cliJournal says one line in the controller's journal (its standard output); a variable so a test can read it.
var cliJournal = func(line string) { fmt.Println(line) }
// runForMeshCLI runs a judged line and answers what it said. Every line is said in the journal first: its call,
// who asked on which node, its command word only, and how it runs (review of ADR 0272).
func runForMeshCLI(ctx context.Context, node string, asked link.CLIAsked, v cliVerdict) link.CLIAnswer {
how := "as an ordinary call"
switch {
case v.refused != "":
how = "refused: " + v.refused
case v.terminal:
how = "as the controller's terminal"
}
call := link.CallIDIn(ctx)
if call == "" {
call = "(no call)"
}
cliJournal(fmt.Sprintf("mesh-cli %s: %s on %s asked %q, %s", call, asked.Account, node, asked.Line[0], how))
if v.refused != "" {
return link.CLIRefusal(v.refused)
}
// Standard input is the terminal's alone: a secret given at the terminal reaches `secret accept`, and no ordinary
// call is handed what the asker's standard input held (novox/hq ADR 0259 §10, ADR 0272).
if len(asked.Stdin) > 0 && !v.terminal {
return link.CLIAnswer{Exit: 1, Why: v.why, Refused: "standard input is given to a line that runs as the " +
"controller's terminal alone, and this one does not. Nothing ran"}
}
if cliServers[asked.Line[0]] {
return link.CLIAnswer{Exit: 1, Why: v.why, Refused: fmt.Sprintf("%s serves until stopped, and is not a "+
"command line mesh-cli runs. Nothing ran", asked.Line[0])}
}
verb, line := "", asked.Line
if !v.terminal {
composed, err := ordinaryLine(asked.Line)
if err != nil {
return link.CLIAnswer{Exit: 1, Why: v.why, Refused: err.Error()}
}
verb, line = cliVerb, composed
}
cmd := selfCommand(ctx, line)
cmd.Env = commandEnvironment(fmt.Sprintf("%s through mesh-cli on %s", asked.Account, node), verb, v.terminal)
// No standard input unless mesh-cli carried one for a terminal line: a command that reads one gets nothing, and
// fails saying so (ADR 0272 §5).
cmd.Stdin = nil
if len(asked.Stdin) > 0 {
cmd.Stdin = bytes.NewReader(asked.Stdin)
}
var stdout, stderr bytes.Buffer
cmd.Stdout, cmd.Stderr = &stdout, &stderr
err := cmd.Run()
answer := link.CLIAnswer{Stdout: stdout.Bytes(), Stderr: stderr.Bytes(), Terminal: v.terminal, Why: v.why}
var exit *exec.ExitError
switch {
case err == nil:
case errors.As(err, &exit):
answer.Exit = exit.ExitCode()
if answer.Exit < 0 {
// Killed: by the call's bound, or by this controller stopping.
answer.Exit = 1
answer.Stderr = append(answer.Stderr, []byte(fmt.Sprintf("\nmesh-cli: the command was stopped (%v)\n",
exit))...)
}
default:
answer.Exit = 1
answer.Refused = fmt.Sprintf("could not run %s from this controller's own build: %v", strings.Join(asked.Line, " "), err)
}
return answer
}
// settingsForms is what an ordinary `settings` line may say: the settings verb's own forms.
const settingsForms = "settings set <module> <values> [--replace] [--node <node>], settings clear <module> " +
"[--node <node>], settings show <module> [--history] [--node <node>], or settings preferences [<module>] " +
"[--node <node>]"
// ordinaryLine is the command line an ordinary call runs (ADR 0272 §4): a `settings` line composed exactly as the
// settings verb composes its own, so its refusals — the terminal-only keys among them — are that verb's (review of
// ADR 0272); any other line as the generic `command` verb takes it, with its refusals.
func ordinaryLine(argv []string) ([]string, error) {
if len(argv) == 0 || argv[0] != "settings" {
// What the generic verb runs, and nothing it would refuse: its reading forms only, and never a command that
// is the terminal's alone (novox/hq ADR 0266) — the two checks argvFor makes of the `command` verb's line,
// made of the words as given rather than re-split from one string.
if err := refusedAsTheGenericCommand(argv); err != nil {
return nil, err
}
if err := terminalOnly(argv); err != nil {
return nil, err
}
return argv, nil
}
args := map[string]any{}
var words []string
rest := argv[1:]
for i := 0; i < len(rest); i++ {
w := rest[i]
switch {
case w == "--replace" || w == "-replace":
args["replace"] = "true"
case w == "--history" || w == "-history":
args["history"] = "true"
case w == "--node" || w == "-node":
if i+1 >= len(rest) {
return nil, fmt.Errorf("--node names no node; settings takes %s. Nothing ran", settingsForms)
}
i++
args["node"] = rest[i]
case strings.HasPrefix(w, "--node=") || strings.HasPrefix(w, "-node="):
_, args["node"], _ = strings.Cut(w, "=")
case strings.HasPrefix(w, "-"):
return nil, fmt.Errorf("settings takes no %s through mesh-cli outside the terminal; it takes %s. "+
"Nothing ran", w, settingsForms)
default:
words = append(words, w)
}
}
wrong := fmt.Errorf("through mesh-cli outside the terminal, settings takes the settings verb's forms: %s. "+
"Nothing ran", settingsForms)
if len(words) == 0 {
return nil, wrong
}
switch words[0] {
case "set":
if len(words) != 3 {
return nil, wrong
}
args["module"], args["values"] = words[1], words[2]
case "clear":
if len(words) != 2 {
return nil, wrong
}
args["module"], args["clear"] = words[1], "true"
case "show":
if len(words) != 2 {
return nil, wrong
}
args["module"] = words[1]
case "preferences":
if len(words) > 2 {
return nil, wrong
}
args["list"] = "preferences"
if len(words) == 2 {
args["module"] = words[1]
}
default:
return nil, wrong
}
return argvFor("settings", args)
}
+99
View File
@@ -0,0 +1,99 @@
package main
import (
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"fmt"
"strings"
"sync"
"testing"
)
// secretAcceptWants makes the test binary, run as a command line, read a secret as `secret accept` reads it and
// say whether it is the value whose SHA-256 the variable names (TestMain).
const secretAcceptWants = "MESH_TEST_SECRET_ACCEPT_WANTS"
// readAsSecretAccept is that process: `secret accept <node> <module> <name> [--from -]`, the value read by
// valueFor, compared by digest, and only the verdict printed.
func readAsSecretAccept(want string, argv []string) int {
if len(argv) < 5 || argv[0] != "secret" || argv[1] != "accept" {
fmt.Printf("not a secret accept line: %q\n", argv)
return 2
}
from := ""
if len(argv) == 7 && argv[5] == "--from" {
from = argv[6]
}
value, err := valueFor(argv[2], argv[3], argv[4], from)
if err != nil {
fmt.Printf("secret accept read nothing: %v\n", err)
return 1
}
sum := sha256.Sum256([]byte(asSupplied(value)))
if hex.EncodeToString(sum[:]) != want {
fmt.Printf("secret accept read something else (%d bytes)\n", len(value))
return 1
}
fmt.Println("secret accept read the value it was given")
return 0
}
// novox/hq ADR 0259 §10, ADR 0272: what mesh-cli's standard input held reaches `secret accept` on a line that runs
// as the controller's terminal, and appears nowhere else — not in the answer, not in the journal, not in the calls
// record; an ordinary line carrying it is refused and nothing runs.
func TestStandardInputReachesSecretAcceptAtTheTerminalAndNowhereElse(t *testing.T) {
token := "123456789:AAEhBOweik6ad9r_QxGivenAtTheTerminal"
sum := sha256.Sum256([]byte(token))
t.Setenv(secretAcceptWants, hex.EncodeToString(sum[:]))
var journal []string
var mu sync.Mutex
was := cliJournal
cliJournal = func(line string) { mu.Lock(); journal = append(journal, line); mu.Unlock() }
t.Cleanup(func() { cliJournal = was })
ctx := context.Background()
for _, line := range [][]string{
{"secret", "accept", "anchor", "telegram", "telegram-token", "--from", "-"},
{"secret", "accept", "anchor", "telegram", "telegram-token"}, // the prompt's path, a line on standard input
} {
asked := cliAsked("operator", 1000, line...)
asked.Stdin = []byte(token + "\n")
a := runForMeshCLI(ctx, "control", asked, cliVerdict{terminal: true, why: "the terminal"})
if a.Exit != 0 || !strings.Contains(string(a.Stdout), "read the value it was given") {
t.Fatalf("%q: secret accept did not read what mesh-cli carried: %+v (%s)", line, a, a.Stdout)
}
if body, _ := json.Marshal(a); strings.Contains(string(body), "AAEh") || strings.Contains(string(body), base64.StdEncoding.EncodeToString([]byte(token))) {
t.Fatalf("the answer carries the secret: %s", body)
}
}
// Without standard input, the line reads nothing, as before.
a := runForMeshCLI(ctx, "control", cliAsked("operator", 1000, "secret", "accept", "anchor", "telegram",
"telegram-token", "--from", "-"), cliVerdict{terminal: true, why: "the terminal"})
if a.Exit == 0 {
t.Fatalf("a line with no standard input read a value: %+v", a)
}
// An ordinary call is never handed it: refused, and nothing ran.
asked := cliAsked("operator", 1000, "status")
asked.Stdin = []byte(token)
a = runForMeshCLI(ctx, "laptop", asked, cliVerdict{why: "not the terminal"})
if a.Exit == 0 || !strings.Contains(a.Refused, "terminal alone") || len(a.Stdout) != 0 {
t.Fatalf("an ordinary line was given standard input: %+v", a)
}
mu.Lock()
defer mu.Unlock()
for _, l := range journal {
if strings.Contains(l, "AAEh") {
t.Fatalf("the journal says the secret: %s", l)
}
}
if len(journal) == 0 {
t.Fatal("the lines were not said in the journal at all")
}
}
+305
View File
@@ -0,0 +1,305 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// echoEnvironment makes the test binary, run as a command line, say the verb and caller it was given (TestMain).
const echoEnvironment = "MESH_TEST_ECHO_ENVIRONMENT"
var cliNodes = []inventory.Node{
{Name: "control", Account: "operator"},
{Name: "laptop", Account: "operator"},
{Name: "unnamed"},
}
func cliAsked(account string, uid uint32, line ...string) link.CLIAsked {
return link.CLIAsked{Line: line, Account: account, UID: uid, Session: "session-1.scope"}
}
// Who is the controller's terminal, and who is an ordinary call or refused (novox/hq ADR 0272 §4).
func TestMeshCLIIsTheTerminalOnlyForTheControlNodesOperator(t *testing.T) {
control := []string{"control"}
cases := []struct {
name, node string
asked link.CLIAsked
control []string
terminal bool
refused string
why string
}{
{"the control-node's operator", "control", cliAsked("operator", 1000, "status"), control, true, "", "the controller's terminal"},
{"another node's operator", "laptop", cliAsked("operator", 1000, "status"), control, false, "", "agents on laptop may run as operator"},
{"another account", "control", cliAsked("agent", 1001, "status"), control, false, "operator account (operator) only", ""},
{"root", "control", cliAsked("root", 0, "status"), control, false, "never root", ""},
{"a node with no operator account", "unnamed", cliAsked("operator", 1000, "status"), control, false, "does not know unnamed's operator account", ""},
{"a node the mesh does not know", "elsewhere", cliAsked("operator", 1000, "status"), control, false, "not a node this mesh knows", ""},
{"two control-nodes", "control", cliAsked("operator", 1000, "status"), []string{"control", "laptop"}, false, "", "2 control-nodes"},
}
for _, c := range cases {
v := judgeCLI(c.node, c.asked, cliNodes, c.control)
if v.terminal != c.terminal {
t.Errorf("%s: terminal %v, want %v (%+v)", c.name, v.terminal, c.terminal, v)
}
if c.refused == "" && v.refused != "" || c.refused != "" && !strings.Contains(v.refused, c.refused) {
t.Errorf("%s: refused %q, want %q", c.name, v.refused, c.refused)
}
if c.why != "" && !strings.Contains(v.why, c.why) {
t.Errorf("%s: why %q, want %q", c.name, v.why, c.why)
}
}
}
// The terminal runs its line without MESH_VERB, even where this process carries one; an ordinary call names
// mesh-cli; both record who asked through mesh-cli.
func TestTheTerminalRunsWithoutAVerbAndAnOrdinaryCallNamesMeshCLI(t *testing.T) {
t.Setenv(echoEnvironment, "1")
t.Setenv("MESH_VERB", "leaked-from-the-serving-process")
// The serving controller marks itself (ADR 0266); its terminal line for mesh-cli is still the terminal's.
t.Setenv(servedVar, "1")
ctx := context.Background()
a := runForMeshCLI(ctx, "control", cliAsked("operator", 1000, "status"), cliVerdict{terminal: true, why: "the terminal"})
if a.Exit != 0 || a.Refused != "" || !a.Terminal {
t.Fatalf("the terminal's line did not run: %+v", a)
}
if got := string(a.Stdout); !strings.Contains(got, `verb=""`) || !strings.Contains(got, "operator through mesh-cli on control") ||
!strings.Contains(got, "terminal=true") {
t.Fatalf("the terminal's line ran with %s", got)
}
a = runForMeshCLI(ctx, "laptop", cliAsked("operator", 1000, "status"), cliVerdict{why: "not the terminal"})
if a.Exit != 0 || a.Terminal || a.Why != "not the terminal" {
t.Fatalf("an ordinary line did not run as one: %+v", a)
}
if got := string(a.Stdout); !strings.Contains(got, `verb="mesh-cli"`) || !strings.Contains(got, "terminal=false") {
t.Fatalf("an ordinary line ran with %s", got)
}
}
// An ordinary call meets every refusal of the generic command verb, and nothing runs; a server is never run.
func TestAnOrdinaryCallMeetsTheCommandVerbsRefusals(t *testing.T) {
t.Setenv(echoEnvironment, "1")
ctx := context.Background()
ordinary := cliVerdict{why: "not the terminal"}
a := runForMeshCLI(ctx, "laptop", cliAsked("operator", 1000, "cleanup", "delete", "x"), ordinary)
if a.Refused == "" || len(a.Stdout) != 0 || a.Exit != 1 || a.Why != "not the terminal" {
t.Fatalf("a repair without --why ran as an ordinary call: %+v", a)
}
a = runForMeshCLI(ctx, "laptop", cliAsked("operator", 1000, "settings", "set", "claude-code", "{}"), ordinary)
if a.Refused != "" || !strings.Contains(string(a.Stdout), `verb="mesh-cli"`) {
t.Fatalf("an ordinary settings set did not run through the settings verb's path with MESH_VERB set: %+v", a)
}
for _, server := range []string{"serve", "api", "board"} {
a := runForMeshCLI(ctx, "control", cliAsked("operator", 1000, server), cliVerdict{terminal: true})
if a.Refused == "" || len(a.Stdout) != 0 {
t.Fatalf("%s was run for mesh-cli: %+v", server, a)
}
}
a = runForMeshCLI(ctx, "control", cliAsked("agent", 1001, "status"), cliVerdict{refused: "agent is not answered"})
if a.Refused != "agent is not answered" || len(a.Stdout) != 0 {
t.Fatalf("a refused line ran: %+v", a)
}
}
// **Which node is the terminal does not follow a verb** (review of ADR 0272): the controller's module is assigned
// and unassigned at the terminal alone, so no caller of `assign` can move the terminal to a node of its choosing.
// Asked through the acts themselves, as the verbs ask them; refused before any store is touched (none is given).
func TestTheControllersModuleIsMovedAtTheTerminalAlone(t *testing.T) {
t.Setenv("MESH_VERB", "assign")
ctx := context.Background()
if _, err := assignWith(ctx, nil, "laptop", assignOptions{}, "zsh", "mesh-controller"); err == nil ||
!strings.Contains(err.Error(), "terminal") {
t.Fatalf("assigning the controller through a verb was not refused: %v", err)
}
if _, err := assign(ctx, nil, "laptop", "mesh-controller"); err == nil || !strings.Contains(err.Error(), "terminal") {
t.Fatalf("assigning the controller through a verb was not refused: %v", err)
}
t.Setenv("MESH_VERB", "unassign")
if _, err := unassign(ctx, nil, "control", "mesh-controller"); err == nil || !strings.Contains(err.Error(), "terminal") {
t.Fatalf("unassigning the controller through a verb was not refused: %v", err)
}
// Another module through a verb, and the controller's at the terminal, are not refused for it.
if err := refusedMovingTheController([]string{"zsh"}); err != nil {
t.Fatalf("another module was refused: %v", err)
}
t.Setenv("MESH_VERB", "")
if err := refusedMovingTheController([]string{"mesh-controller"}); err != nil {
t.Fatalf("the terminal was refused: %v", err)
}
}
// An ordinary `settings set|clear` goes down the settings verb's own path: composed as that verb composes it, and
// run with MESH_VERB set, so its refusals — the terminal-only keys among them — are the settings command's own,
// not a blanket refusal of the generic command (review of ADR 0272).
func TestAnOrdinarySettingsLineTakesTheSettingsVerbsPath(t *testing.T) {
cases := []struct {
line []string
want string
err string
}{
{[]string{"settings", "set", "zsh", `{"execute":"withhold"}`, "--node", "laptop"}, `settings set zsh {"execute":"withhold"} --node laptop`, ""},
{[]string{"settings", "set", "zsh", "{}", "--replace"}, "settings set zsh {} --replace", ""},
{[]string{"settings", "clear", "zsh", "--node", "laptop"}, "settings clear zsh --node laptop", ""},
{[]string{"settings", "show", "zsh", "--history"}, "settings show zsh --history", ""},
{[]string{"settings", "preferences"}, "settings preferences", ""},
{[]string{"settings", "set", "zsh"}, "", "settings"},
{[]string{"settings", "set", "zsh", "{}", "--sideways"}, "", "--sideways"},
}
for _, c := range cases {
argv, err := ordinaryLine(c.line)
if c.err != "" {
if err == nil || !strings.Contains(err.Error(), c.err) {
t.Errorf("%q: refused with %v, want %q", c.line, err, c.err)
}
continue
}
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%q: composed %q (%v), want %q", c.line, argv, err, c.want)
}
}
// Anything else still meets the generic command verb's refusals.
if _, err := ordinaryLine([]string{"retire", "delete", "x"}); err == nil {
t.Error("a repair composed as an ordinary line")
}
}
// Every line is said in the controller's journal, with its call, who asked where and how it ran — its command word
// only, never the rest of the line (review of ADR 0272).
func TestEveryMeshCLILineIsSaidInTheJournal(t *testing.T) {
t.Setenv(echoEnvironment, "1")
var said []string
was := cliJournal
cliJournal = func(line string) { said = append(said, line) }
t.Cleanup(func() { cliJournal = was })
ctx := link.WithCallID(context.Background(), "call-1")
runForMeshCLI(ctx, "control", cliAsked("operator", 1000, "settings", "set", "x", `{"password":"s3cret"}`),
cliVerdict{terminal: true, why: "the terminal"})
runForMeshCLI(ctx, "control", cliAsked("agent", 1001, "status"), cliVerdict{refused: "agent is not answered"})
all := strings.Join(said, "\n")
if len(said) != 2 || !strings.Contains(all, "call-1") || !strings.Contains(all, "operator on control") ||
!strings.Contains(all, "as the controller's terminal") || !strings.Contains(all, "refused") {
t.Fatalf("the journal said %q", said)
}
if strings.Contains(all, "s3cret") {
t.Fatalf("the journal carries the line's values: %q", said)
}
}
// **An ordinary line runs only what the generic command verb would** (review of ADR 0272, ADR 0266): its reading
// forms, and never a command that is the terminal's alone. Each of these, from another node's operator, is refused
// and runs nothing.
func TestAnOrdinaryLineRunsNothingTheCommandVerbWouldRefuse(t *testing.T) {
t.Setenv(echoEnvironment, "1")
for _, line := range [][]string{
{"node", "account", "control", "x"},
{"node", "agent-account", "control", "x", "--clear"},
{"token", "issue", "laptop"},
{"secret", "export", "x"},
{"operator", "key", "set", "x"},
{"assign", "laptop", "zsh"},
} {
if _, err := ordinaryLine(line); err == nil {
t.Errorf("%q composed as an ordinary line", line)
}
a := runForMeshCLI(context.Background(), "laptop", cliAsked("operator", 1000, line...), cliVerdict{why: "not the terminal"})
if a.Refused == "" || len(a.Stdout) != 0 {
t.Errorf("%q ran as an ordinary line: %+v", line, a)
}
}
if argv, err := ordinaryLine([]string{"status"}); err != nil || argv[0] != "status" {
t.Fatalf("a read was refused: %v", err)
}
}
// **`calls` never shows a mesh-cli line's answer** (review of ADR 0272): the verb's own answer is read for a line
// served over a bus, and neither the answer's text nor the base64 JSON writes its bytes in is there.
func TestTheCallsVerbNeverShowsAMeshCLILinesAnswer(t *testing.T) {
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer conn.Close()
stop, err := link.OverNATS{Conn: conn}.ServeCLI(func(context.Context, string, link.CLIAsked) link.CLIAnswer {
return link.CLIAnswer{Stdout: []byte("s3cret-join")}
}, nil)
if err != nil {
t.Fatal(err)
}
defer stop()
body, _ := json.Marshal(link.CLIAsked{Line: []string{"token", "issue", "x"}, Account: "operator"})
msg, err := conn.Request(link.CLISubject("control"), body, 5*time.Second)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(msg.Data), base64.StdEncoding.EncodeToString([]byte("s3cret-join"))) {
t.Fatalf("the asker was not given its answer: %s", msg.Data)
}
recent, _ := link.Calls.Recent()
var id string
for _, c := range recent {
if c.Seat == link.CLISeat {
id = c.ID
break
}
}
shown, err := callsAnswer(link.Calls, id)
if err != nil {
t.Fatal(err)
}
said, _ := json.Marshal(shown)
if strings.Contains(string(said), "s3cret-join") ||
strings.Contains(string(said), base64.StdEncoding.EncodeToString([]byte("s3cret-join"))) {
t.Fatalf("calls showed a mesh-cli line's answer: %s", said)
}
}
// **Only the terminal's line carries the terminal's mark** (review of ADR 0272): a mark the serving controller's
// environment holds — leaked, or set by anything — is stripped from every other command line it runs, a verb's and
// an ordinary mesh-cli line alike, so neither reads as the terminal.
func TestTheTerminalsMarkIsStrippedFromEveryOtherLine(t *testing.T) {
t.Setenv(echoEnvironment, "1")
t.Setenv(cliTerminalVar, "1")
t.Setenv(servedVar, "1")
for _, env := range [][]string{commandEnvironment("someone", "status", false)} {
for _, kv := range env {
if strings.HasPrefix(kv, cliTerminalVar+"=") {
t.Fatalf("a verb's line carries the terminal's mark: %s", kv)
}
}
}
a := runForMeshCLI(context.Background(), "laptop", cliAsked("operator", 1000, "status"), cliVerdict{why: "not the terminal"})
if got := string(a.Stdout); !strings.Contains(got, "terminal=false") || !strings.Contains(got, `verb="mesh-cli"`) {
t.Fatalf("an ordinary line with the mark in the serving environment ran as %s", got)
}
a = runForMeshCLI(context.Background(), "control", cliAsked("operator", 1000, "status"), cliVerdict{terminal: true})
if got := string(a.Stdout); !strings.Contains(got, "terminal=true") {
t.Fatalf("the terminal's line ran as %s", got)
}
}
// **Only a login session is the terminal** (review of ADR 0272): the operator's account on the control-node, asking
// from a service — the tool runner, a user unit — and not a login session, is an ordinary call.
func TestOnlyALoginSessionIsTheTerminal(t *testing.T) {
control := []string{"control"}
v := judgeCLI("control", link.CLIAsked{Line: []string{"status"}, Account: "operator", UID: 1000}, cliNodes, control)
if v.terminal || v.refused != "" || !strings.Contains(v.why, "login session") {
t.Fatalf("a line from no login session reads %+v", v)
}
v = judgeCLI("control", link.CLIAsked{Line: []string{"status"}, Account: "operator", UID: 1000, Session: "session-3.scope"},
cliNodes, control)
if !v.terminal {
t.Fatalf("a line from a login session on the control-node reads %+v", v)
}
}
+58 -4
View File
@@ -40,6 +40,50 @@ type merges interface {
AnnouncedMerges(ctx context.Context, since time.Time) ([]link.AnnouncedMerge, error)
}
// unheardMerges is the announcements of merges the controller has not heard (novox/hq ADR 0276): a merge kept
// in a batch is not acted on until its batch is cut, which can be past mergeGrace behind a long walk, and is
// not missed for that.
type unheardMerges struct {
merges
inv *inventory.Inventory
}
func (u unheardMerges) AnnouncedMerges(ctx context.Context, since time.Time) ([]link.AnnouncedMerge, error) {
all, err := u.merges.AnnouncedMerges(ctx, since)
if err != nil {
return nil, err
}
var out []link.AnnouncedMerge
for _, a := range all {
_, kept, err := u.inv.MergeOf(ctx, a.Owner+"/"+a.Repo, a.Commit)
if err != nil {
return nil, err
}
if !kept {
out = append(out, a)
}
}
return out, nil
}
// owedToTheRecord says a merge read as history is still owed to the record (novox/hq ADR 0276): the merges
// reader knows, when it keeps them (unheardMerges).
func owedToTheRecord(ctx context.Context, announced merges, m link.SourceMoved, entries []inventory.Entry,
read map[string][]inventory.ReadRepository) bool {
u, ok := announced.(unheardMerges)
if !ok {
return false
}
merged, err := time.Parse(time.RFC3339Nano, m.MergedAt)
if err != nil {
return false
}
raw := m
raw.MergedAt = ""
owed, err := owedLate(ctx, u.inv, m, merged, wouldMove(raw, entries, read))
return err == nil && owed
}
// catchingUpOnMerges reads back the forge's announcements on a timer, until the context ends.
func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) {
f := following{open}
@@ -48,7 +92,7 @@ func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) {
if err != nil {
return nil, nil, err
}
read, err := open.inventory.ReadRepositories(ctx)
read, err := readForPlanning(ctx, open.inventory)
return entries, read, err
}
failing := ""
@@ -61,7 +105,7 @@ func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) {
case <-tick.C:
}
watchedMerges.begin()
err := catchUpOnMerges(ctx, time.Now(), announced, catalogued, f.SourceMoved, func(format string, args ...any) {
err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announced, open.inventory}, catalogued, f.SourceMoved, func(format string, args ...any) {
fmt.Printf(format+"\n", args...)
})
// What the pass found is what S5 says (novox/hq to-be 45 §3); a pass that could not read
@@ -102,10 +146,15 @@ func catchUpOnMerges(ctx context.Context, now time.Time, announced merges,
return err
}
for _, a := range all {
// A merge the forge said no time of is dated by its announcement, so a packaging module's look can
// make it history once acted on, and the catch-up does not act on it again every pass (ADR 0267).
if a.SourceMoved.MergedAt == "" && !a.At.IsZero() {
a.SourceMoved.MergedAt = a.At.UTC().Format(time.RFC3339)
}
if now.Sub(a.At) < mergeGrace {
continue
}
if len(wouldMove(a.SourceMoved, entries, read)) == 0 {
if len(wouldMove(a.SourceMoved, entries, read)) == 0 && !owedToTheRecord(ctx, announced, a.SourceMoved, entries, read) {
continue
}
if entries, read, err = catalogued(ctx); err != nil {
@@ -113,7 +162,12 @@ func catchUpOnMerges(ctx context.Context, now time.Time, announced merges,
}
moves := wouldMove(a.SourceMoved, entries, read)
if len(moves) == 0 {
continue
if !owedToTheRecord(ctx, announced, a.SourceMoved, entries, read) {
continue
}
raw := a.SourceMoved
raw.MergedAt = ""
moves = wouldMove(raw, entries, read)
}
var names []string
for _, e := range moves {
+94 -4
View File
@@ -73,7 +73,7 @@ func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Ke
for _, r := range h.Resources {
kept := inventory.ResourceHealth{Module: r.Module, Resource: r.Resource, Kind: r.Kind, Target: r.Target,
State: r.State, Reason: r.Reason, Since: r.Since, Streak: r.Streak, Restarts: r.Restarts,
Check: r.Check, Needs: r.Needs, Account: r.Account}
Check: r.Check, Needs: r.Needs, Account: r.Account, Root: r.Root}
resources = append(resources, kept)
if r.State == link.StateUnhealthy && r.Module != "" {
unhealthy[r.Module] = append(unhealthy[r.Module], kept)
@@ -135,7 +135,8 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
}
standing := map[string]conditions.Condition{}
for _, c := range open {
if (c.Kind == kindModuleUnhealthy || c.Kind == kindReloginNeeded) && c.Subject.Machine == node {
if (c.Kind == kindModuleUnhealthy || c.Kind == kindReloginNeeded || c.Kind == kindUsedAsFound) &&
c.Subject.Machine == node {
standing[c.Key] = c
}
}
@@ -158,6 +159,21 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
heldOn := map[string]string{}
providers := map[catalogue.Chosen]bool{}
for _, m := range modules {
// **A directory used as found is said as that** (novox/hq issue 339): the operator's to hand over at the
// machine, never urgent — nothing is broken by the wait that a person was not told of — and its own kind,
// so the gate never reads it as a fault of the build that happened to be sent beside it.
if said, waits := foundWait(m, node, unhealthy[m]); waits {
o := usedAsFoundObservation(m, node, said, unhealthy[m])
seen[o.Key()] = true
became[m] = kindUsedAsFound
if _, isOpen := standing[o.Key()]; streaks[m] < moduleUnhealthyAfter && !isOpen {
continue
}
if _, err := k.Observe(ctx, o); err != nil {
problems = append(problems, err.Error())
}
continue
}
// **A wait for a person's new login is said as that** (novox/hq ADR 0254): one plain sentence to the
// operator, never urgent, cleared on the first statement that no longer says it.
if said, waits := personWait(m, node, unhealthy[m]); waits {
@@ -209,6 +225,9 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
if c.Kind == kindReloginNeeded {
why = fmt.Sprintf("%s says %s no longer waits for a new login", node, module)
}
if c.Kind == kindUsedAsFound {
why = fmt.Sprintf("%s says no directory of %s is used as found any more", node, module)
}
if on, held := heldOn[key]; held {
why = fmt.Sprintf("what %s finds on %s waits on %s, which is unhealthy: held under its condition", module, node, on)
}
@@ -383,6 +402,7 @@ func moduleUnhealthyObservation(module, node string, rs []inventory.ResourceHeal
Explanation: fmt.Sprintf("%s on %s is not healthy: %s. It clears as soon as it runs again.", module, node,
namesWords(plain, 3)),
Needs: needs,
Actions: moduleActions(node, rs),
Resolved: fmt.Sprintf("%s works again on %s", module, node)}
}
@@ -499,6 +519,7 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
if waits && !f.groupsAdded[module] {
waits = false
}
var found []string
for _, r := range h.Resources {
if r.Module != module {
continue
@@ -506,6 +527,14 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
if waits && r.State == link.StateUnhealthy {
continue
}
// **A directory used as found before this send waits for a person** (novox/hq issue 339): the node-engine
// left its owner and mode, and only someone at the machine can hand it over. It is no fault of this
// build, so it does not hold the module's walk — an urgent fix still goes through — and the verdict
// carries the wait. Found by this very send, the send brought it, and it is judged as unhealthy.
if usedAsFound(r) && r.Since.Before(since) {
found = append(found, r.Resource)
continue
}
switch r.State {
case link.StateHealthy:
case link.StateStarting:
@@ -521,12 +550,25 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
reasonAfter(r.Reason))
}
}
if waits {
return healthPerson, wait
if waits || len(found) > 0 {
var said []string
if waits {
said = append(said, wait)
}
if len(found) > 0 {
said = append(said, fmt.Sprintf("on %s, %s uses %s as found and waits for the operator to hand it over "+
"(`nox node hand-over %s <directory>` on the control-node)", machine, module, strings.Join(found, ", "), machine))
}
return healthPerson, strings.Join(said, "; ")
}
return healthGood, ""
}
// usedAsFound is a directory the node-engine states it uses as found (novox/hq issue 339).
func usedAsFound(r inventory.ResourceHealth) bool {
return r.Kind == link.KindDirectory && r.State == link.StateUnhealthy && strings.HasPrefix(r.Reason, link.ReasonUsedAsFound)
}
func reasonAfter(s string) string {
if s == "" {
return ""
@@ -595,3 +637,51 @@ func addsAccountGroups(from catalogue.Manifest, hadFrom bool, to catalogue.Manif
}
return false
}
// kindUsedAsFound is a module's condition while the node-engine uses one of its directories as found (novox/hq
// issue 339): its own kind, the operator's, never urgent, and never read by the gate as a fault of a build.
const kindUsedAsFound = "directory-used-as-found"
// usedAsFoundKey is a module's used-as-found condition on a machine.
func usedAsFoundKey(module, node string) string {
return conditions.Key(conditions.ScopeModule, module+"."+node, kindUsedAsFound)
}
// foundWait is whether everything unhealthy of a module on a machine is a directory used as found, and that in
// one sentence. Anything else unhealthy beside it is judged as a fault, with the directory among its resources.
func foundWait(module, node string, rs []inventory.ResourceHealth) (string, bool) {
var ids, why []string
for _, r := range rs {
if r.Module != module || r.State != link.StateUnhealthy {
continue
}
if !usedAsFound(r) {
return "", false
}
ids = append(ids, r.Resource)
why = append(why, strings.TrimSpace(strings.TrimPrefix(r.Reason, link.ReasonUsedAsFound)))
}
if len(ids) == 0 {
return "", false
}
return fmt.Sprintf("%s on %s uses %s as found: %s", module, node, strings.Join(ids, ", "),
strings.Join(why, "; ")), true
}
// usedAsFoundObservation is a module whose directory the node-engine uses as found, in words: the operator's, a
// warning however long it stays, its summary naming the directories and their owners; the paths are evidence.
func usedAsFoundObservation(module, node, said string, rs []inventory.ResourceHealth) conditions.Observation {
o := moduleUnhealthyObservation(module, node, rs)
o.Token, o.Kind, o.Resolver, o.Severity, o.Summary = kindUsedAsFound, kindUsedAsFound, conditions.ResolverOperator,
conditions.Warning, said
o.Headline = fmt.Sprintf("%s waits for a directory on %s", module, node)
o.Explanation = fmt.Sprintf("A directory of %s was already on %s, with another owner or mode than %s declares. "+
"The mesh left it as it was rather than hand it to an account, so %s may not be able to use it.",
module, node, module, module)
// Plain words (ADR 0253): the line itself — `nox node hand-over <node> <path>` on the control-node (ADR 0272,
// issue 356) — is in the summary and the evidence, which name the directory; a path is never in these.
o.Needs = "hand the directory over from the control-node, as the operator; the details name it and the line to type."
o.Resolved = fmt.Sprintf("%s's directory on %s is the mesh's", module, node)
o.Actions = nil
return o
}
+139 -8
View File
@@ -10,10 +10,12 @@ import (
"os"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/overlay"
)
@@ -397,8 +399,9 @@ func assignCommand(ctx context.Context, verb string, args []string) error {
func settingsCommand(ctx context.Context, args []string) error {
if len(args) == 0 {
return errors.New("settings show <module> [--node <node>] [--history], settings set <module> <file> " +
"[--node <node>] [--replace], settings clear <module> [--node <node>], or settings preferences " +
"[<module>] [--node <node>]")
"[--node <node>] [--replace], settings clear <module> [--node <node>], settings preferences " +
"[<module>] [--node <node>], settings propose <module> <file | --clear> [--node <node>] [--replace], " +
"or settings proposals [<id>]")
}
open, err := openStores(ctx)
if err != nil {
@@ -412,12 +415,38 @@ func settingsCommand(ctx context.Context, args []string) error {
// **What a set removes is refused unless meant** (novox/hq ADR 0217). A layer is replaced whole,
// and on 2026-10-05 setting one placement dropped a machine's whole layer for a module without a
// word (novox/hq issue 304). Adding and changing keys needs nothing; removing one needs this.
replace := set.Bool("replace", false, "for set: remove the keys the new layer does not name")
replace := set.Bool("replace", false, "for set and propose: remove the keys the new layer does not name")
history := set.Bool("history", false, "for show: the layers this one replaced, the latest first")
clear := set.Bool("clear", false, "for propose: propose that the layer be removed")
positionals, err := parseAround(set, args[1:])
if err != nil {
return err
}
switch args[0] {
case "propose":
// A trusted setting, proposed by anyone and set only on the operator's warrant (novox/hq ADR 0277):
// the stores are opened there, so the proposal and the verb's refusals below never meet.
if len(positionals) < 1 || len(positionals) > 2 {
return errors.New("settings propose <module> <settings.json | {…}> [--node <node>] [--replace], or settings propose <module> --clear [--node <node>]")
}
values := ""
if len(positionals) == 2 {
values = positionals[1]
}
return proposeCommand(ctx, positionals[0], *node, values, *clear, *replace)
case "proposals":
if len(positionals) > 1 || *node != "" || *clear || *replace || *history {
return errors.New("settings proposals [<id>]")
}
id := ""
if len(positionals) == 1 {
id = positionals[0]
}
return proposalsCommand(ctx, id)
}
if *clear {
return errors.New("--clear is for settings propose; a layer is cleared with settings clear")
}
where := "the whole mesh"
if *node != "" {
@@ -445,6 +474,9 @@ func settingsCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
if err := refuseTerminalSettingsThroughAVerb(ctx, inv, before, values, positionals[0], where); err != nil {
return err
}
added, changed, removed := settingsChange(before, values)
if len(removed) > 0 && !*replace {
return fmt.Errorf("%s on %s: this layer would no longer set %s. A layer is replaced whole; "+
@@ -452,7 +484,7 @@ func settingsCommand(ctx context.Context, args []string) error {
"removal is meant (novox/hq ADR 0217). Nothing was changed",
positionals[0], where, strings.Join(removed, ", "), positionals[0], nodeFlag(*node))
}
if err := inv.SetSettings(ctx, *node, positionals[0], values); err != nil {
if err := inv.SetSettingsBy(ctx, *node, positionals[0], values, setByWords()); err != nil {
return err
}
fmt.Printf("%s on %s:\n", positionals[0], where)
@@ -493,8 +525,12 @@ func settingsCommand(ctx context.Context, args []string) error {
}
for _, p := range past {
shown, _ := json.MarshalIndent(p.Values, " ", " ")
fmt.Printf("%s on %s, until %s (%s):\n %s\n", positionals[0], where,
p.ReplacedAt.Local().Format("2006-01-02 15:04:05"), p.ReplacedBy, shown)
by := ""
if p.SetBy != "" {
by = "; " + p.SetBy
}
fmt.Printf("%s on %s, until %s (%s%s):\n %s\n", positionals[0], where,
p.ReplacedAt.Local().Format("2006-01-02 15:04:05"), p.ReplacedBy, by, shown)
}
return nil
}
@@ -510,6 +546,14 @@ func settingsCommand(ctx context.Context, args []string) error {
return err
}
fmt.Println(string(shown))
// And who set it (novox/hq ADR 0277): a layer the operator approved on their phone says so.
if setBy, setAt, has, err := inv.LayerOrigin(ctx, *node, positionals[0]); err != nil {
return err
} else if has && setBy != "" {
fmt.Printf(" %s\n", setBy)
} else if has {
fmt.Printf(" set at %s; who set it was not kept\n", setAt.Local().Format("2006-01-02 15:04"))
}
}
// Every value the module gives a default or a layer sets, and where it came from (novox/hq
// ADR 0262): the default, the mesh's layer, or this node's. Said after the layer, which stays
@@ -596,17 +640,33 @@ func settingsCommand(ctx context.Context, args []string) error {
if len(positionals) != 1 {
return errors.New("settings clear <module> [--node <node>]")
}
if err := inv.ClearSettings(ctx, *node, positionals[0]); err != nil {
before, _, err := inv.Layer(ctx, *node, positionals[0])
if err != nil {
return err
}
if err := refuseTerminalSettingsThroughAVerb(ctx, inv, before, nil, positionals[0], where); err != nil {
return err
}
if err := inv.ClearSettingsBy(ctx, *node, positionals[0], setByWords()); err != nil {
return err
}
fmt.Printf("%s on %s is back to what the module says\n", positionals[0], where)
return nil
default:
return fmt.Errorf("settings has no %q; it has show, set, clear and preferences", args[0])
return fmt.Errorf("settings has no %q; it has show, set, clear, preferences, propose and proposals", args[0])
}
}
// setByWords is who sets a layer from this process, as the layer keeps it (novox/hq ADR 0277): the caller at the
// controller's terminal, or through which verb.
func setByWords() string {
if verb, through := throughAVerb(); through {
return "set by " + link.Caller() + " through " + verb + " at " + time.Now().Local().Format("2006-01-02 15:04")
}
return "set at the controller's terminal by " + link.Caller() + " at " + time.Now().Local().Format("2006-01-02 15:04")
}
// describeEffective says each setting's value on a machine or the whole mesh, where it came from, and
// the module's default when a layer overrides it.
func describeEffective(module, where string, values []catalogue.SettingSource) string {
@@ -988,6 +1048,9 @@ func whereItComesFrom(repository, ref, commit, path string, self bool) (inventor
// definition that got past the check — written elsewhere, or checked by nobody — is refused here
// in the same words. A name meant on purpose is declared with its reason and passes.
func namesNoInstallation(m catalogue.Manifest) error {
if problems := catalogue.TrustProblems(m); len(problems) > 0 {
return fmt.Errorf("%s", strings.Join(problems, "; "))
}
named := catalogue.InstallationProblems(m)
if len(named) == 0 {
return nil
@@ -1051,3 +1114,71 @@ func declaresTools(m catalogue.Manifest) bool {
}
return false
}
// The keys no verb may change are catalogue.TerminalKeys (novox/hq issue 339). `places` says where the
// node-engine creates and, as root, owns a module's directories, with an owner the setting names; `accesses` says
// which of the machine's paths are mounted into a module's container; a provider's trust anchors say what every
// consumer trusts. Set through a verb, any of them lets any caller of the mesh's verbs — an agent among them —
// have root hand it a directory, mount one of the machine's into a container it reaches, or have the mesh trust
// an authority of its own. They are the operator's, typed at the controller's terminal.
// throughAVerb says whether this process runs a seat verb's command line: the serving controller names the
// verb in the environment of every command it runs for one (runVerb), and a person at the terminal runs none.
// Every verb route reaches a command through runVerb — the named verbs and the generic `command` alike — so
// this is the one place that knows, whatever line the verb composed.
func throughAVerb() (string, bool) {
verb := os.Getenv(verbVar)
return verb, verb != ""
}
// sameLayer says whether two layers hold the same values, an absent layer and an empty one alike.
func sameLayer(a, b map[string]any) bool {
if len(a) == 0 && len(b) == 0 {
return true
}
ra, _ := json.Marshal(a)
rb, _ := json.Marshal(b)
return string(ra) == string(rb)
}
// refuseTerminalSettingsThroughAVerb refuses a layer change through a verb that would add, change or remove
// places or accesses; a change that leaves both as they were is not refused.
func refuseTerminalSettingsThroughAVerb(ctx context.Context, inv *inventory.Inventory, before, after map[string]any,
module, where string) error {
verb, through := throughAVerb()
if !through {
return nil
}
// Judged against the module's definition as the catalogue holds it: what it serves and which of its files
// are trusted. A catalogue that cannot be read refuses rather than judging against nothing.
shelf, err := inv.Catalogue(ctx)
if err != nil {
return fmt.Errorf("which settings of %s are the terminal's cannot be read, so nothing was changed: %w", module, err)
}
// A trusted mergeable file takes any key, so its module's whole layer is the terminal's (novox/hq issue 340).
if files := catalogue.TrustedMergeable(shelf[module]); len(files) > 0 && !sameLayer(before, after) {
return fmt.Errorf("the settings of %s on %s are set at the controller's terminal (`mesh-cli` on the control-node) or on "+
"the operator's warrant, never through a verb alone (this line came through %q): %s merges whatever key a "+
"layer sets into a file root or a consumer trusts, so any key could point the module at a listener of the "+
"caller's, and whoever may call a verb includes agents (novox/hq issue 340; a file nothing trusts says "+
"\"trusted\": false). Propose it instead: the settings verb with propose puts the exact values to the "+
"operator on a channel that proves who answers, and the layer is set on their Approve (novox/hq ADR 0277). "+
"Nothing was changed",
module, where, verb, strings.Join(files, ", "))
}
for _, key := range catalogue.TerminalKeys(shelf[module]) {
was, _ := json.Marshal(before[key])
now, _ := json.Marshal(after[key])
if string(was) == string(now) {
continue
}
return fmt.Errorf("%s of %s on %s is set at the controller's terminal (`mesh-cli` on the control-node) or on the "+
"operator's warrant, never through a verb alone (this line came through %q): it says where root creates and "+
"owns a module's directories, which of the machine's paths are mounted into its container, what the mesh's "+
"consumers trust, or what a file root or a person's session obeys takes, and whoever may call a verb "+
"includes agents (novox/hq issue 339; issue 340 for a mergeable file's own keys). Propose it instead: the "+
"settings verb with propose puts the exact values to the operator on a channel that proves who answers, and "+
"the layer is set on their Approve (novox/hq ADR 0277). Nothing was changed", key, module, where, verb)
}
return nil
}
+2 -2
View File
@@ -426,10 +426,10 @@ func overlayShow(ctx context.Context, open *stores) error {
tunnel.Interface, tunnel.Range, tunnel.Port)
case n.Hub && hubName == n.Name && tunnel.Interface != "":
fmt.Printf(" hub — found a tunnel on %s and did NOT take it over: its key is not the tunnel's; "+
"`mesh-host overlay take --tunnel %s` on the machine takes it", tunnel.Interface, tunnel.Interface)
"`nox-mesh-host overlay take --tunnel %s` on the machine takes it", tunnel.Interface, tunnel.Interface)
case n.Hub:
fmt.Print(" hub — found no tunnel; if the machine runs the predecessor's, " +
"`mesh-host overlay take --tunnel <iface>` there adopts it (novox/hq ADR 0105)")
"`nox-mesh-host overlay take --tunnel <iface>` there adopts it (novox/hq ADR 0105)")
case !n.Reachable():
fmt.Print(" not dialable")
}
+224 -4
View File
@@ -7,7 +7,10 @@ import (
"errors"
"flag"
"fmt"
"io"
"net"
"os"
"path/filepath"
"strings"
"time"
@@ -15,6 +18,7 @@ import (
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/token"
)
@@ -26,7 +30,7 @@ import (
func nodeCommand(ctx context.Context, args []string) error {
if len(args) == 0 {
return errors.New("node add <name>, node list, node show <name>, or " + publicDomainUsage)
return errors.New("node add <name>, node list, node show <name>, " + publicDomainUsage + ", or " + handOverUsage)
}
open, err := openStores(ctx)
if err != nil {
@@ -98,17 +102,175 @@ func nodeCommand(ctx context.Context, args []string) error {
"containers reaching outward. The machine reports which of its links face outside; see " +
"`node show <name>`")
case "agent-account":
// The account agents run as on this machine, when it is not the operator's (novox/hq ADR 0266). Here,
// at the controller's terminal, and nowhere else: no verb and no setting names it, so no agent can
// name itself another account.
return nodeAgentAccount(ctx, inv, args[1:])
case "account":
// The operator's login on this machine (novox/hq to-be 29): what a home-scoped file is
// owned by and which account `ssh <node>` uses. Reports with no argument; sets with one;
// an optional second argument is the home when it is not /home/<account>.
return nodeAccount(ctx, inv, args[1:])
case "hand-over":
// A directory the node-engine uses as found, handed to the mesh (novox/hq issue 356, issue 339). Here, at
// the controller's terminal, and nowhere else: at the next apply root gives the directory to the account
// the module declares, and whoever may call a verb includes agents.
return nodeHandOver(ctx, open, args[1:])
case "setuid-search":
// A fresh search for setuid programs on a node (novox/hq issue 361), after the operator changed by hand
// what the last one found. Here, at the controller's terminal, and nowhere else: a search never makes a
// machine free wrongly, but asked again and again it would keep the machine unjudged and its disks busy,
// and whoever may call a verb includes agents.
return nodeSetuidSearch(ctx, open, args[1:])
default:
return fmt.Errorf("node has no %q; it has add, list, show, public-domain and account", args[0])
return fmt.Errorf("node has no %q; it has add, list, show, public-domain, account, agent-account, hand-over "+
"and setuid-search", args[0])
}
}
const handOverUsage = "node hand-over <node> <directory> — hand a directory the node-engine on <node> uses as found " +
"to the mesh: its next apply gives it the declared owner and mode. The directory's absolute path, as the module's " +
"condition names it"
// handOverLine reads a hand-over's line: the node and the directory's absolute path, exactly as the engine states
// it. Judged before anything is asked, and judged again by the engine, which is the one that acts.
func handOverLine(args []string) (node, path string, err error) {
if len(args) != 2 {
return "", "", errors.New(handOverUsage)
}
node, path = args[0], args[1]
if node == "" || strings.HasPrefix(node, "-") {
return "", "", fmt.Errorf("%q is not a node's name; %s", node, handOverUsage)
}
if !filepath.IsAbs(path) {
return "", "", fmt.Errorf("%q is not an absolute path; %s", path, handOverUsage)
}
if filepath.Clean(path) != path {
return "", "", fmt.Errorf("%q is not the directory's path as the engine states it (no `..`, no doubled or "+
"trailing separator); %s", path, handOverUsage)
}
return node, path, nil
}
// handOverBy is who hands the directory over, in the words a verb's caller is recorded in: the operator through
// mesh-cli on the control-node, or whoever runs this controller's binary at its terminal.
func handOverBy() string {
if by := strings.TrimSpace(os.Getenv(link.CallerVar)); by != "" {
return by
}
return "the controller's terminal"
}
// nodeHandOver asks the node's engine to take a directory it uses as found as the mesh's, and says what came of
// it. The engine records the hand-over or refuses; nothing is recorded here, because the directory is the
// machine's and the engine is the one that reads it. The ask is signed with the mesh's key (issue 356's review).
func nodeHandOver(ctx context.Context, open *stores, args []string) error {
known := func(node string) error {
_, err := open.inventory.NodeByName(ctx, node)
return err
}
ask := func(node, path, by string) (link.HandOverAnswer, error) {
ident, err := open.Identity(ctx)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("the mesh's signing key cannot be read, so nothing was asked of %s: %w",
node, err)
}
address, err := broker.BusAddress()
if err != nil {
return link.HandOverAnswer{}, err
}
js, err := broker.Dial(address)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("cannot reach the bus, so nothing was asked of %s: %w", node, err)
}
defer js.Close()
return link.AskHandOver(ctx, js.Conn(), ident, node, path, by, link.HandOverWithin)
}
return handOverAsked(args, known, ask, os.Stdout)
}
// handOverAsked is the hand-over's line with its two acts given: whether the mesh knows the node, and the ask.
// Nothing is asked of a line or a node that is refused, and the engine's refusal is this command's failure —
// never a success with the refusal printed.
func handOverAsked(args []string, known func(node string) error,
ask func(node, path, by string) (link.HandOverAnswer, error), out io.Writer) error {
node, path, err := handOverLine(args)
if err != nil {
return err
}
if err := known(node); err != nil {
return fmt.Errorf("nothing was asked: %w", err)
}
answer, err := ask(node, path, handOverBy())
if err != nil {
return err
}
if answer.Refused != "" {
return fmt.Errorf("%s refused: %s", node, answer.Refused)
}
fmt.Fprintln(out, answer.Said)
fmt.Fprintf(out, " the module's condition clears once %s applies; `nox push %s` applies it now\n", node, node)
return nil
}
const setuidSearchUsage = "node setuid-search <node> — throw away the node-engine's last search for setuid " +
"programs on <node> and start a full one: after a setuid-root program it found was removed by hand. Until it " +
"completes, root-free says the node is not judged yet"
// nodeSetuidSearch asks the node's engine for a fresh search, signed with the mesh's key as a hand-over is.
func nodeSetuidSearch(ctx context.Context, open *stores, args []string) error {
known := func(node string) error {
_, err := open.inventory.NodeByName(ctx, node)
return err
}
ask := func(node, by string) (link.HandOverAnswer, error) {
ident, err := open.Identity(ctx)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("the mesh's signing key cannot be read, so nothing was asked of %s: %w",
node, err)
}
address, err := broker.BusAddress()
if err != nil {
return link.HandOverAnswer{}, err
}
js, err := broker.Dial(address)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("cannot reach the bus, so nothing was asked of %s: %w", node, err)
}
defer js.Close()
return link.AskSetuidSearch(ctx, js.Conn(), ident, node, by, link.HandOverWithin)
}
return setuidSearchAsked(args, known, ask, os.Stdout)
}
// setuidSearchAsked is the line with its two acts given: whether the mesh knows the node, and the ask. The
// engine's refusal is this command's failure.
func setuidSearchAsked(args []string, known func(node string) error,
ask func(node, by string) (link.HandOverAnswer, error), out io.Writer) error {
if len(args) != 1 || args[0] == "" || strings.HasPrefix(args[0], "-") {
return errors.New(setuidSearchUsage)
}
node := args[0]
if err := known(node); err != nil {
return fmt.Errorf("nothing was asked: %w", err)
}
answer, err := ask(node, handOverBy())
if err != nil {
return err
}
if answer.Refused != "" {
return fmt.Errorf("%s refused: %s", node, answer.Refused)
}
fmt.Fprintln(out, answer.Said)
fmt.Fprintf(out, " the controller's root-free verb shows the search's progress until it completes\n")
return nil
}
// addNode creates a node record, adopted when the operator says so (novox/hq ADR 0100).
func addNode(ctx context.Context, inv *inventory.Inventory, args []string) error {
set := flag.NewFlagSet("node add", flag.ContinueOnError)
@@ -176,6 +338,57 @@ func nodeAccount(ctx context.Context, inv *inventory.Inventory, positionals []st
return nil
}
const agentAccountUsage = "node agent-account <name> — what it is now; " +
"<name> <account> [home] to name the account agents run as (home defaults to /home/<account>); " +
"<name> --clear to have them run as the operator account again"
// nodeAgentAccount reports, names or clears the account agents run as on a node (novox/hq ADR 0266). Read-
// shaped with no account, like public-domain; clearing is asked for by name.
func nodeAgentAccount(ctx context.Context, inv *inventory.Inventory, args []string) error {
set := flag.NewFlagSet("node agent-account", flag.ContinueOnError)
clear := set.Bool("clear", false, "agents run as the operator account again")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) == 0 || len(positionals) > 3 {
return errors.New(agentAccountUsage)
}
node := positionals[0]
switch {
case *clear && len(positionals) > 1:
return fmt.Errorf("name an agent account for %s or --clear, not both", node)
case *clear:
if err := inv.SetAgentAccount(ctx, node, "", ""); err != nil {
return err
}
fmt.Printf("agents on %s run as the operator account again\n", node)
fmt.Printf(" run `push %s` to send it; the agent account itself is kept (the mesh never deletes a login)\n", node)
return nil
case len(positionals) >= 2:
home := ""
if len(positionals) == 3 {
home = positionals[2]
}
if err := inv.SetAgentAccount(ctx, node, positionals[1], home); err != nil {
return err
}
fmt.Printf("agents on %s run as %s\n", node, positionals[1])
fmt.Printf(" run `push %s` to send it; the self-check says once its node-engine has judged it "+
"unable to become root\n", node)
return nil
default:
n, err := inv.NodeByName(ctx, node)
if err != nil {
return err
}
for _, line := range agentAccountLines(ctx, inv, n) {
fmt.Println(strings.TrimPrefix(line, " "))
}
return nil
}
}
const publicDomainUsage = "node public-domain <name> — what it is now; " +
"<name> <domain> to set it; <name> --clear to take it away"
@@ -590,6 +803,10 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
if err := showMode(ctx, inv, node); err != nil {
return err
}
// Whom agents run as here, and whether that account can become root without a person (ADR 0266).
for _, line := range agentAccountLines(ctx, inv, node) {
fmt.Println(line)
}
// The domain its routed names are composed under, when it has one (novox/hq ADR 0066). Shown
// only when set: a machine that serves nothing to the outside has no domain, and saying so of
@@ -684,11 +901,14 @@ func orNotReported(s string) string {
}
// printJSON prints a value as indented JSON, the shape every `--json` answers in.
func printJSON(v any) error {
func printJSON(v any) error { return printJSONTo(os.Stdout, v) }
// printJSONTo is printJSON to a writer of the caller's.
func printJSONTo(w io.Writer, v any) error {
body, err := json.MarshalIndent(v, "", " ")
if err != nil {
return err
}
fmt.Println(string(body))
fmt.Fprintln(w, string(body))
return nil
}
+2 -2
View File
@@ -157,7 +157,7 @@ func TestAMergeRebuildsTheModulesItChanged(t *testing.T) {
{"a file directly among the modules", merge([]string{"modules/README.md"}, false), ""},
{"a root file beside a module's", merge([]string{"merge-check.sh", "modules/keycloak/x.ts"}, false), "keycloak"},
} {
if got := named(whatTheMergeTouched(candidates, known, c.m)); got != c.want {
if got := named(whatTheMergeTouched(candidates, known, c.m, nil)); got != c.want {
t.Errorf("%s: rebuilt %q, wanted %q", c.what, got, c.want)
}
}
@@ -285,7 +285,7 @@ func TestAChangeInsideAModuleIsThatModulesHeldOrNot(t *testing.T) {
{"an old announcer saying nothing", merge(showcase, nil, false), ""},
{"a manifest the merge removed, said or not", merge([]string{"modules/gone/module.json", "modules/gone/x.ts"}, nil, true), ""},
} {
if got := named(whatTheMergeTouched(candidates, known, c.m)); got != c.want {
if got := named(whatTheMergeTouched(candidates, known, c.m, nil)); got != c.want {
t.Errorf("%s: rebuilt %q, wanted %q", c.what, got, c.want)
}
}
+113 -7
View File
@@ -110,6 +110,15 @@ var plainWordings = map[string]func(conditions.Observation) words{
"reaches it. It keeps running what it has.", m),
Resolved: m + " can get new instructions again"}
}),
kindAgentCanBecomeRoot: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "Sessions on " + m + " could become root",
Needs: "take the sessions' own account out of every group and rule that grants root; the details say which.",
Explanation: fmt.Sprintf("Assistant sessions on %s run under an account of their own, so that none "+
"can take over the machine without you. The machine cannot show that this holds now, so an answer "+
"from your phone authorises nothing there until it does.", m),
Resolved: "Sessions on " + m + " cannot become root again"}
}),
"own-address-banned": worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: m + " has banned the mesh",
@@ -204,6 +213,14 @@ var plainWordings = map[string]func(conditions.Observation) words{
}
return reloginWords(orModule(module), machineOr(o, "a machine"), false)
}),
kindUsedAsFound: worded(func(o conditions.Observation) words {
module := ""
if o.Scope == conditions.ScopeModule && o.Machine != "" {
module = strings.TrimSuffix(o.ID, "."+o.Machine)
}
w := usedAsFoundObservation(orModule(module), machineOr(o, "a machine"), o.Summary, nil)
return words{Headline: w.Headline, Explanation: w.Explanation, Needs: w.Needs, Resolved: w.Resolved}
}),
kindProviderFailing: worded(func(o conditions.Observation) words {
thing, consumer := conditions.ThingWords(o), idPart(o, 2)
if consumer == "" {
@@ -345,6 +362,17 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "Its walk across the machines has not moved for longer than usual. Nothing is lost.",
Resolved: "Resolved: the delivery moves again"}
}),
kindBatchNotCut: worded(func(o conditions.Observation) words {
return words{Headline: "Merged changes are not being delivered",
Explanation: "Merges collected for one delivery should have been planned and were not. Nothing is lost.",
Resolved: "Resolved: the merged changes are being delivered"}
}),
kindBatchBehindWalk: worded(func(o conditions.Observation) words {
return words{Headline: "Merged changes wait behind a slow delivery",
Explanation: "Merges collected for the next delivery wait for the delivery before them, which is taking " +
"longer than it should. Nothing is lost.",
Resolved: "Resolved: the merged changes no longer wait"}
}),
kindWalkWaiting: worded(func(o conditions.Observation) words {
return words{Headline: "A delivery is waiting to start",
Explanation: "A merged change is built, and mesh-delivery (the module that decides when a delivery goes " +
@@ -515,6 +543,13 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "The mesh's message system is being upgraded; some things pause until it is done.",
Resolved: "The bus upgrade is done"}
}),
kindBusStepWaiting: worded(func(o conditions.Observation) words {
return words{Headline: "Sends wait for a bus upgrade",
Needs: "start the bus upgrade " + FromMeshMCPServer,
Explanation: "A new version of the mesh's message system is built, and only a person installs it. Until " +
"then nothing else is sent to the machine that runs it.",
Resolved: "Resolved: the bus upgrade started, and sends go on"}
}),
kindBusUpgradeFailed: worded(func(o conditions.Observation) words {
return words{Headline: "The bus upgrade failed",
Needs: "decide whether to put the bus back to the version before; the details say how.",
@@ -543,10 +578,18 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "The bus reports a listener too slow to keep up, so messages to it are late.",
Resolved: "The listener keeps up again"}
}),
kindBusReconnects: worded(func(o conditions.Observation) words {
return words{Headline: "A client keeps losing the bus",
Explanation: "One of the mesh's clients lost its connection to the bus again and again in the last hour. " +
"While it reconnects, what it says and what it is asked waits.",
Resolved: "Resolved: the client stays connected"}
}),
"max-deliveries": worded(func(o conditions.Observation) words {
return words{Headline: "A message could not be handled",
Explanation: "The bus gave up on a message after trying to hand it over too many times.",
Resolved: "Resolved: messages are handled again"}
return words{Headline: "A listener gave up on messages",
Needs: "deliver them again or drop them, from the mesh MCP server.",
Explanation: "A listener on the bus could not handle messages after several tries, so what they asked " +
"for was not done. The mesh keeps them until you deliver them again or drop them.",
Resolved: "Resolved: the messages given up on were delivered again or dropped"}
}),
"refused": worded(func(o conditions.Observation) words {
return words{Headline: "The bus refuses some messages",
@@ -648,6 +691,8 @@ func walkWaitingWords(w waitFacts, in time.Duration, severity conditions.Severit
}
// waitingNeeds is what the operator does about a walk waiting past its urgent bound: nothing before it.
// Start and Stop are also asked of the operator (novox/hq ADR 0259); the condition's own words keep saying
// where they are given without a channel, and the ask's text drops that (askText).
func waitingNeeds(severity conditions.Severity) string {
if severity == conditions.Urgent {
return "start it, or stop it, " + FromMeshMCPServer
@@ -655,6 +700,20 @@ func waitingNeeds(severity conditions.Severity) string {
return ""
}
// waitingActions are the answers to a walk waiting past its urgent bound: start it, or stop it — the plan's
// own verbs, approved by the operator (novox/hq ADR 0259). None before the bound.
func waitingActions(plan string, severity conditions.Severity) []conditions.Action {
if severity != conditions.Urgent || plan == "" {
return nil
}
return []conditions.Action{
{Label: "Start", Verb: "mesh-controller.plans", Level: conditions.LevelApprove,
Arguments: map[string]string{"go": plan, "why": "", "cause": conditions.CauseOperatorAnswer}},
{Label: "Stop", Verb: "mesh-controller.plans", Level: conditions.LevelApprove,
Arguments: map[string]string{"stop": plan, "why": "", "cause": conditions.CauseOperatorAnswer}},
}
}
// moduleNeeds is what the operator can do about a module unhealthy on a machine: log in again where its
// account's groups wait for it (ADR 0252), restart a failed service, or nothing where the mesh restarts it.
// No answer is offered for a restart: a desk click performs only an acknowledgement (ADR 0258).
@@ -669,11 +728,35 @@ func moduleNeeds(node string, rs []inventory.ResourceHealth) string {
}
}
if unit != "" {
// Also asked of the operator (moduleActions); the ask's text drops where (askText).
return fmt.Sprintf("restart its service %s on %s %s", unit, node, FromMeshMCPServer)
}
return ""
}
// moduleActions are the answers to a module unhealthy on a machine: restart its failed service there,
// approved by the operator (novox/hq ADR 0259) — none when the mesh restarts it, or a new login is what it
// waits for.
func moduleActions(node string, rs []inventory.ResourceHealth) []conditions.Action {
for _, r := range rs {
if strings.Contains(r.Reason, "relogin needed") {
return nil
}
}
for _, r := range rs {
if r.Kind != link.KindUnit || r.Target == "" {
continue
}
scope := "system"
if r.Account != "" {
scope = "user"
}
return []conditions.Action{{Label: "Restart", Verb: "node-service-manager.restart", Machine: node,
Level: conditions.LevelApprove, Arguments: map[string]string{"unit": r.Target, "scope": scope}}}
}
return nil
}
// FromMeshMCPServer ends what the operator needs when no notification can do it (ADR 0258), naming the mesh MCP
// server (the glossary's word; "console" is retired): the answer is not an
// acknowledgement, so it is given where the operator is known to be the one asking, until answers are
@@ -716,6 +799,25 @@ func stalledWords(l stalledLine, o conditions.Observation) (headline, explanatio
actions []conditions.Action) {
repository, _, _ := strings.Cut(l.ID, "@")
name := repoName(repository)
if l.State == "unannounced" {
// A pull request the forge never announced (novox/hq issue 347): no delivery exists, so there is nothing to
// stop, release or close; a new commit on its branch is announced and checked.
long := "for too long"
if d, err := time.ParseDuration(l.For); err == nil {
long = "for " + humanDuration(d)
}
if o.Resolver == conditions.ResolverOperator {
needs = "push a new commit to its branch; the forge announces it and the mesh checks it."
}
pull := "A pull request of " + name
if l.Number > 0 {
pull = fmt.Sprintf("Pull request %s #%d", name, l.Number)
}
return fmt.Sprintf("%s has had no merge check %s", pull, long),
fmt.Sprintf("%s has been open %s on a branch that requires the merge check, and the mesh was never asked "+
"to check it: the forge never announced it. It cannot merge until it is checked.", pull, long),
fmt.Sprintf("%s has a merge check now, or is closed", pull), needs, nil
}
held := l.State
if held == "" {
held = "held"
@@ -725,15 +827,19 @@ func stalledWords(l stalledLine, o conditions.Observation) (headline, explanatio
long = "for " + humanDuration(d)
}
if o.Resolver == conditions.ResolverOperator {
// Words only: releasing or stopping a delivery is not an acknowledgement, so no desk click
// performs it (ADR 0258).
// Asked of the operator, approved on a channel that proves who answered (novox/hq ADR 0259); the
// router says where each can be answered, so the words do not.
release := conditions.Action{Label: "Release", Verb: "mesh-delivery.release", Level: conditions.LevelApprove,
Arguments: map[string]string{"id": l.ID, "why": ""}}
stop := conditions.Action{Label: "Stop", Verb: "mesh-delivery.stop", Level: conditions.LevelApprove,
Arguments: map[string]string{"id": l.ID, "why": ""}}
switch held {
case "held":
needs = "release it, or stop it, " + FromMeshMCPServer
needs, actions = "release it, or stop it, "+FromMeshMCPServer, []conditions.Action{release, stop}
case "ready", "checked":
needs = "merge its pull request, or close it."
default:
needs = "stop it " + FromMeshMCPServer
needs, actions = "stop it "+FromMeshMCPServer, []conditions.Action{stop}
}
}
return fmt.Sprintf("Delivery of %s %s %s", name, held, long),
+45 -7
View File
@@ -65,13 +65,21 @@ func TestADeliveryWaitingNeedsNothingUntilItsBoundThenOffersStartAndStop(t *test
t.Errorf("the summary lost the way on for whoever looks closer: %q", got[0].Summary)
}
// Past four hours it is urgent, and offers the controller's own answers.
// Past four hours it is urgent, and asks the operator to start or stop it (novox/hq ADR 0259): the plan's
// own verbs, approved, which the controller performs on the warrant. The router says where to answer.
f.waits[0].since = now.Add(-5 * time.Hour)
got = watchWaits(f)
plainExample(t, got[0], "openrazer delivery waiting to start",
"Needs you: start it, or stop it, from the mesh MCP server; this notification cannot do it. The change to openrazer is merged and built, and mesh-delivery (the "+
"module that decides when a delivery goes out) has not let it start for 5 hours, so mesh-delivery may "+
"be stuck.")
"be stuck.", "Start", "Stop")
for i, want := range []string{"go", "stop"} {
a := got[0].Actions[i]
if a.Verb != "mesh-controller.plans" || a.Arguments[want] != "plan-1791454185265004861" ||
a.Level != conditions.LevelApprove || a.Arguments["cause"] != conditions.CauseOperatorAnswer {
t.Errorf("%s: %+v", a.Label, a)
}
}
// Many modules are counted, not listed in the headline.
f.waits[0].modules = []string{"a", "b", "c", "d"}
@@ -82,16 +90,20 @@ func TestADeliveryWaitingNeedsNothingUntilItsBoundThenOffersStartAndStop(t *test
}
// **A module unhealthy**: "openrazer on g14 is not healthy: its unit openrazer-daemon.service failed in the
// account's own service manager (exit-code)". Restarting is not an acknowledgement, so it is said in words
// and offered as no answer (ADR 0258).
// account's own service manager (exit-code)". Restarting is not an acknowledgement: it is asked of the
// operator at the approve level (novox/hq ADR 0259), so a desk click never performs it (ADR 0258).
func TestAModuleUnhealthyAsksForARestartInWords(t *testing.T) {
o := moduleUnhealthyObservation("openrazer", "g14", []inventory.ResourceHealth{{Kind: link.KindUnit,
Resource: "openrazer-daemon", Target: "openrazer-daemon.service",
Resource: "openrazer-daemon", Target: "openrazer-daemon.service", Account: "jochen",
Reason: "failed in the account's own service manager (exit-code)", Since: time.Now()}})
plainExample(t, o, "openrazer not working on g14",
"Needs you: restart its service openrazer-daemon on g14 from the mesh MCP server; this notification cannot do it. "+
"openrazer on g14 is not healthy: its service openrazer-daemon stopped with an error. It clears as soon "+
"as it runs again.")
"as it runs again.", "Restart")
if a := o.Actions[0]; a.Verb != "node-service-manager.restart" || a.Machine != "g14" || a.Level != conditions.LevelApprove ||
a.Arguments["unit"] != "openrazer-daemon.service" || a.Arguments["scope"] != "user" {
t.Errorf("restart: %+v", a)
}
// An account waiting for a new login (ADR 0252) asks for the login, held to the plain rule.
o = moduleUnhealthyObservation("openrazer", "g14", []inventory.ResourceHealth{{Kind: "account",
Resource: "operator-in-group", Target: "jochen", Reason: "relogin needed: the account is in the group"}})
@@ -154,7 +166,12 @@ func TestADeliveryHeldAsksForReleaseOrStopInWords(t *testing.T) {
Bound: "24h0m0s", H2: "none: the state is the operator's", Says: "it waits for the operator"}})
plainExample(t, got[0], "Delivery of hq held for 36 hours",
"Needs you: release it, or stop it, from the mesh MCP server; this notification cannot do it. A delivery of hq has been held for 36 hours, past its limit.",
)
"Release", "Stop")
for i, verb := range []string{"mesh-delivery.release", "mesh-delivery.stop"} {
if a := got[0].Actions[i]; a.Verb != verb || a.Arguments["id"] != "novox/hq@055550802096" || a.Level != conditions.LevelApprove {
t.Errorf("%+v", a)
}
}
}
// **Every kind the controller raises has plain words**, and its words are plain for a subject of every
@@ -309,3 +326,24 @@ func TestANeedNoNotificationAnswersNamesTheMeshMCPServer(t *testing.T) {
}
check("updates held", releaseHeldWords([]string{"openrazer"}, []string{"g14"}))
}
// **A pull request the forge never announced says what to do about it** (novox/hq issue 347): mesh-delivery says one
// as a stalled line in state `unannounced` — no delivery exists, so there is nothing to stop, release or close —
// and the operator's one act is a new commit on its branch, which the forge announces.
func TestAnUnannouncedPullRequestSaysToPushANewCommit(t *testing.T) {
l := stalledLine{ID: "novox/hq@bfe82315f42c", Number: 243, State: "unannounced", For: "11m0s", Bound: "10m0s",
H2: "none: the forge never announced it, so there is no delivery to close — the operator's",
Says: "novox/hq#243 is open on main, which requires the merge check, and its head has had no merge check"}
obs := stalledObservations([]stalledLine{l})
if len(obs) != 1 || obs[0].Resolver != conditions.ResolverOperator {
t.Fatalf("an unannounced pull request is not the operator's: %+v", obs)
}
o := obs[0]
if strings.Contains(o.Needs, "stop") || strings.Contains(o.Needs, "release") || !strings.Contains(o.Needs, "commit") {
t.Fatalf("it says to %q", o.Needs)
}
if !strings.Contains(o.Headline, "no merge check") || !strings.Contains(o.Headline, "#243") ||
!strings.Contains(o.Explanation, "never") {
t.Fatalf("it reads %q / %q", o.Headline, o.Explanation)
}
}
+21 -1
View File
@@ -136,7 +136,8 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
resolved, err := catalogue.Resolve(shelf, assigned,
catalogue.Node{Name: nodeName, Site: site, Capabilities: capabilities,
At: onNetwork[nodeName], PublicDomain: publicDomain,
Account: who.Account, AccountHome: who.AccountHome}, world)
Account: who.Account, AccountHome: who.AccountHome,
AgentAccount: who.AgentAccount, AgentAccountHome: who.AgentAccountHome}, world)
if err != nil {
// The node's own set does not compose. Marked, because this is the only failure here that
// a mesh-wide gatherer may pass over — see notResolvable.
@@ -885,8 +886,13 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
judgesRoot, err := engineJudgesRoot(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
return catalogue.Rendering{
ReadsHealth: readsHealth,
JudgesRoot: judgesRoot,
BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Foreseen: foreseen, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
@@ -909,6 +915,16 @@ func engineReadsHealth(ctx context.Context, inv *inventory.Inventory, node strin
return had && stated.Contract >= link.ReadinessContract, nil
}
// engineJudgesRoot says whether a machine's node-engine judges a user's declared `root` (novox/hq ADR 0266),
// by its own newest statement, for the same reason as engineReadsHealth: an older engine parses strictly.
func engineJudgesRoot(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.RootContract, nil
}
// zonesInTheMesh is every zone a module in the mesh declares, where the mesh placed it (novox/hq ADR
// 0199): the zone settled from that node's settings, the node's private address, the port the
// answering listen is published on there.
@@ -1223,6 +1239,10 @@ func planCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
// A module left out is not in the body, so the diff alone would say "nothing would change" for
// a module just assigned whose settings cannot compose — success-shaped silence. Said first, with
// why, as push and the plain plan say it (novox/hq ADR 0163, rule 6).
reportLeftOut(args[0], declared)
return writePlanDiff(ctx, open.inventory, args[0], body)
}
if *asJSON {
+28 -7
View File
@@ -9,7 +9,6 @@ import (
"strings"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -98,11 +97,9 @@ func buildSeatPause(ctx context.Context, inv *inventory.Inventory, plans []inven
if len(holders) == 0 {
return pauseView{}
}
address, err := broker.BusAddress()
if err != nil {
return pauseView{}
}
js, err := broker.Dial(address)
// On the serving controller's own connection when this is it: a watchdog tick while a walk waits for a
// build dialled one every 30 seconds (novox/hq issue 327).
js, err := aBus()
if err != nil {
fmt.Fprintf(os.Stderr, "could not reach the bus to read whether the build seat is paused: %v\n", err)
return pauseView{}
@@ -167,12 +164,13 @@ func retryRefusal(p inventory.Plan, plans []inventory.Plan) error {
case p.Open():
return fmt.Errorf("%s is still %s; nothing in it failed to retry — `rebuild <module>` asks one module again", p.ID, p.State)
}
if len(failedIn(p)) > 0 {
if q, found := newerOpenPlan(p, plans); found {
return fmt.Errorf("%s supersedes it: a newer merge of %s (%s at %s) is open, and retrying %s would build "+
"what that one replaced", q.ID, q.Repository, q.ID, short(q.Commit), p.ID)
}
return nil
return oneWalkAtATime(p, plans)
}
stopped := stoppedRollouts(p)
if len(stopped) == 0 {
@@ -196,6 +194,17 @@ func retryRefusal(p inventory.Plan, plans []inventory.Plan) error {
"the older build back", m, q.ID, q.State, q.Repository, short(q.Commit), p.ID)
}
}
return oneWalkAtATime(p, plans)
}
// oneWalkAtATime refuses a retry while another walk is open, started or waiting for its word (novox/hq ADR
// 0276): a walk retried beside it would be two walks at once.
func oneWalkAtATime(p inventory.Plan, plans []inventory.Plan) error {
for _, q := range plans {
if q.ID != p.ID && q.Open() && q.Release == nil {
return fmt.Errorf("%s is open (%s): one walk at a time — retry %s once it ended", q.ID, q.Named(), p.ID)
}
}
return nil
}
@@ -267,6 +276,18 @@ func retryPlan(ctx context.Context, open *stores, id string) (string, error) {
return "", err
}
plans = append(plans, recent...)
// **A failed walk whose earlier merges are walked alone is not retried** (novox/hq ADR 0276): the search for
// the merge that brought the failure answers them now, and a retried walk would name them twice.
if p.Delivery != nil && len(p.Delivery.Merges) > 0 {
kept, err := inv.MergesOf(ctx, p.ID)
if err != nil {
return "", err
}
if len(kept) < len(p.Delivery.Merges) {
return "", fmt.Errorf("%s's earlier merges are walked alone, to find which one brought its failure: "+
"those walks answer them; a newer merge, or `rebuild <module>`, builds again", p.ID)
}
}
if err := retryRefusal(p, plans); err != nil {
return "", err
}
+151
View File
@@ -0,0 +1,151 @@
package main
import (
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// **The planner over edges the store derives**, not edges written by hand: modules registered, builds
// recorded with what they stood on and which repositories they read, the relation answered by
// inventory.Dependencies (dependenciesOf over the records), and the merge planned by reachOfMerge — the
// path a real merge takes, short of the bus.
//
// **A shared repository moves only what a change's files are in the build source of** (novox/hq ADR 0267,
// issues 338 and 363): each build records the build source it said, and a merge is mapped onto those of the
// newest builds. A build that said none (P below, as every build before ADR 0267) is read as before: P moves
// on any merge to the repository it packages, though through no edge. The rows tagged 338 held the opposite
// until ADR 0267 was built.
func TestASharedRepositoryIsPlannedFromTheRecordedBuildSources(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
asked := time.Now().Add(-time.Hour)
sourcesOf := map[string][]inventory.BuildSource{}
register := func(m catalogue.Manifest, repository, path string, against []string, read []inventory.ReadRepository) {
t.Helper()
if err := inv.RegisterModule(ctx, m, inventory.Source{Repository: repository, Seat: "git", Path: path,
Ref: "main", BuiltFrom: "old", Asked: asked}); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-" + m.Module, Repository: repository, Ref: "main",
Module: m.Module, Commit: "old", On: "builder", Path: path, Against: against, Read: read, Asked: asked,
Sources: sourcesOf[m.Module]}); err != nil {
t.Fatal(err)
}
}
controllerRead := []inventory.ReadRepository{{Repository: "novox/mesh-controller", Ref: "main"}}
agent := catalogue.Manifest{Module: "build-agent", Version: "1",
Claims: []catalogue.Claim{{Name: "node-build-agent", Scope: catalogue.ScopeNode}}}
// The shape of issue 338, each build saying its build source (ADR 0267).
gomod := []string{"go.mod", "go.sum"}
sourcesOf["mesh-controller"] = []inventory.BuildSource{{Paths: append([]string{"module.json", "cmd/mesh-controller/",
"internal/conditions/", "internal/broker/"}, gomod...)}}
sourcesOf["build-agent"] = []inventory.BuildSource{
{Paths: []string{"modules/build-agent/Dockerfile", "modules/build-agent/module.json"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: append([]string{"cmd/mesh-builder/", "internal/broker/"}, gomod...)}}
sourcesOf["route-proxy"] = []inventory.BuildSource{
{Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: append([]string{"examples/route-proxy/", "internal/broker/"}, gomod...)}}
register(catalogue.Manifest{Module: "mesh-controller", Version: "1"}, "novox/mesh-controller", "", nil, nil)
register(agent, "novox/mesh-catalog", "modules/build-agent", nil, controllerRead)
register(catalogue.Manifest{Module: "route-proxy", Version: "1"}, "novox/mesh-catalog", "modules/route-proxy", nil, controllerRead)
register(catalogue.Manifest{Module: "gitea", Version: "1"}, "novox/mesh-catalog", "modules/gitea", nil, nil)
// A, B and C in one repository; D built against A's artifact, E declaring B, P packaging the repository.
for _, n := range []string{"a", "b", "c"} {
register(catalogue.Manifest{Module: n, Version: "1"}, "novox/one", "modules/"+n, nil, nil)
}
register(catalogue.Manifest{Module: "d", Version: "1"}, "novox/two", "d",
[]string{catalogue.ArtifactStoreScheme + "a/runtime@sha256:" + strings.Repeat("0", 64)}, nil)
register(catalogue.Manifest{Module: "e", Version: "1", Build: &catalogue.Build{
On: []catalogue.BuildsOn{{Arg: "BASE", Module: "b", Artifact: "runtime"}}}}, "novox/two", "e", nil, nil)
register(catalogue.Manifest{Module: "p", Version: "1"}, "novox/two", "p", nil,
[]inventory.ReadRepository{{Repository: "novox/one", Ref: "main"}})
entries, err := inv.Catalogued(ctx)
if err != nil {
t.Fatal(err)
}
read, err := inv.ReadRepositories(ctx)
if err != nil {
t.Fatal(err)
}
edges, err := inv.Dependencies(ctx)
if err != nil {
t.Fatal(err)
}
// The edges the hand-written rows of TestASharedRepositoryMovesWhatPackagesItAsItDoesToday use are
// the ones derived here.
var shared []inventory.Edge
in338 := map[string]bool{"mesh-controller": true, "build-agent": true, "route-proxy": true, "gitea": true}
for _, e := range edges {
if in338[e.From] && in338[e.To] {
shared = append(shared, e)
}
}
if !reflect.DeepEqual(shared, sharedRepositoryEdges) {
t.Errorf("derived %v\nthe hand-written rows use %v", shared, sharedRepositoryEdges)
}
// Each kind derived from its record: built against (stands-on), build.on (declared); a read draws none.
for _, e := range edges {
if e.Kind == inventory.EdgePackages {
t.Errorf("a packages edge was drawn (ADR 0267 rule 4): %v", e)
}
}
for _, want := range []inventory.Edge{
dep("d", inventory.EdgeStandsOn, "a"),
dep("e", inventory.EdgeDeclared, "b"),
dep("d", inventory.EdgeBuiltBy, "build-agent"),
} {
found := false
for _, e := range edges {
found = found || e == want
}
if !found {
t.Errorf("no %s %s %s derived: %v", want.From, want.Kind, want.To, edges)
}
}
for _, c := range []struct {
what, repo string
paths []string
want string
issue338 bool
}{
{"A and B changed, C untouched: D after A, E after B; P, which said no build source, reads all", "one",
[]string{"modules/a/x.go", "modules/b/x.go"}, "a,b,p | d,e", false},
{"C alone: C, and P, read whole as before; P after nothing", "one",
[]string{"modules/c/x.go"}, "c,p", false},
{"a README of the repository P packages: P, read whole as before", "one",
[]string{"README.md"}, "p", false},
{"the dependent's repository: D alone", "two", []string{"d/main.go"}, "d", false},
{"a README of the controller's repository: no module", "mesh-controller",
[]string{"README.md"}, "", true},
{"the controller's own command: the controller alone", "mesh-controller",
[]string{"cmd/mesh-controller/main.go"}, "mesh-controller", true},
{"a package only the controller builds from: the controller alone", "mesh-controller",
[]string{"internal/conditions/condition.go"}, "mesh-controller", true},
{"the route proxy's program: the route proxy alone", "mesh-controller",
[]string{"examples/route-proxy/main.go"}, "route-proxy", true},
{"a package all three build from: all three", "mesh-controller",
[]string{"internal/broker/broker.go"}, "mesh-controller | build-agent | route-proxy", false},
{"the route proxy's directory in the catalogue: it alone", "mesh-catalog",
[]string{"modules/route-proxy/module.json"}, "route-proxy", false},
{"the build agent's directory: it alone, nothing it builds", "mesh-catalog",
[]string{"modules/build-agent/module.json"}, "build-agent", false},
} {
_, got := planMerge(t, c.repo, c.paths, entries, read, edges)
if got != c.want {
tag := ""
if c.issue338 {
tag = " (issue 338, flipped by ADR 0267)"
}
t.Errorf("%s: planned %q, wanted %q%s", c.what, got, c.want, tag)
}
}
}
+680
View File
@@ -0,0 +1,680 @@
package main
import (
"encoding/json"
"fmt"
"math/rand/v2"
"sort"
"strings"
"testing"
"time"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The delivery planner's rules, as recorded (novox/hq ADR 0162 §1, ADR 0238 §3), held by one table and
// one property over reachOfMerge — the planner's one answer to "what does this merge move, and in which
// order". A row that fails here on main is a planner that breaks a recorded rule: the row stays, the
// expectation is not bent to the code.
//
// The kinds of edge, as the code reads them (release_plan.go):
//
// kind widens the plan orders the tiers
// stands-on yes yes, after its base is built
// declared yes yes, after its base is built
// packages no no — retired by novox/hq ADR 0267; one recorded before is read and ignored
// built-by no yes, after the build machine — except for what the build machine stands
// on, and for the controller whose worker it binds
// worker-of no yes, the build seat's holder after the controller (hq issue 206)
const (
repoOne = "http://forge.internal:20000/novox/one.git"
repoTwo = "http://forge.internal:20000/novox/two.git"
)
// dep is one edge of the catalogue's relation: from depends on to, in the way kind says.
func dep(from, kind, to string) inventory.Edge {
return inventory.Edge{From: from, To: to, Kind: kind}
}
// tiered is a plan's tiers as one line: a tier's modules by comma, tiers by " | ". Empty for no plan.
func tiered(tiers [][]string) string {
var out []string
for _, t := range tiers {
out = append(out, strings.Join(t, ","))
}
return strings.Join(out, " | ")
}
// planMerge is what reachOfMerge plans for a merge of these files into a repository's main: its tiers as
// one line, and the files no build reads. It also holds the plan to its own shape: every module it builds
// is in exactly one tier.
func planMerge(t *testing.T, repo string, paths []string, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, edges []inventory.Edge) (mergeReach, string) {
t.Helper()
m := link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Commit: "head", Paths: paths}
r := reachOfMerge(m, entries, read, edges)
seen := map[string]int{}
for _, tier := range r.Plan.Tiers {
for _, name := range tier {
seen[name]++
}
}
for name := range r.Plan.Modules {
if seen[name] != 1 {
t.Errorf("%s/%v: %s is in %d tiers of %v", repo, paths, name, seen[name], r.Plan.Tiers)
}
}
if len(seen) != len(r.Plan.Modules) {
t.Errorf("%s/%v: the tiers %v hold modules the plan does not (%d)", repo, paths, r.Plan.Tiers, len(r.Plan.Modules))
}
return r, tiered(r.Plan.Tiers)
}
// **A merge moves the modules whose directory it changed, and everything built on them; nothing else.**
// Each row is a catalogue (modules in directories of one or two repositories), its dependencies with
// their kinds, the files one commit changed, and the exact plan: the moved set and its tier order.
func TestAPlanIsWhatTheChangeTouchedAndWhatIsBuiltOnIt(t *testing.T) {
in := func(repo, dir string, names ...string) []inventory.Entry {
var out []inventory.Entry
for _, n := range names {
d := dir + "/" + n
if dir == "" {
d = n
}
out = append(out, fromRepo(n, repo, d))
}
return out
}
// Modules a to f, x and z in novox/one under modules/; g in novox/two at g/.
entries := append(in(repoOne, "modules", "a", "b", "c", "d", "e", "f", "x", "z"), in(repoTwo, "", "g")...)
const (
standsOn = inventory.EdgeStandsOn
declared = inventory.EdgeDeclared
packages = inventory.EdgePackages
builtBy = inventory.EdgeBuiltBy
workerOf = inventory.EdgeWorkerOf
)
// The two real cycles the kinds resolve themselves (ADR 0162 §1, hq issue 206).
runtime := append(in(repoOne, "modules", "runtime", "builder"), fromRepo("controller", repoTwo, ""))
for _, c := range []struct {
what string
entries []inventory.Entry
edges []inventory.Edge
repo string
paths []string
want string // the tiers, " | " between them
unread string
cycle bool
}{
// The operator's case: two modules changed, a third beside them in the same repository untouched,
// each changed one with a dependent.
{what: "A and B changed, C untouched beside them, D on A and E on B",
edges: []inventory.Edge{dep("d", standsOn, "a"), dep("e", declared, "b")},
repo: "one", paths: []string{"modules/a/main.go", "modules/b/module.json"}, want: "a,b | d,e"},
{what: "only A's directory: A and what stands on it",
edges: []inventory.Edge{dep("d", standsOn, "a"), dep("e", declared, "b")},
repo: "one", paths: []string{"modules/a/main.go"}, want: "a | d"},
{what: "only C's directory: C alone, nothing is built on it",
edges: []inventory.Edge{dep("d", standsOn, "a"), dep("e", declared, "b")},
repo: "one", paths: []string{"modules/c/Dockerfile"}, want: "c"},
{what: "only the dependent changed: its base does not move",
edges: []inventory.Edge{dep("d", standsOn, "a")},
repo: "one", paths: []string{"modules/d/main.go"}, want: "d"},
{what: "transitive: F on D on A, A changed",
edges: []inventory.Edge{dep("f", standsOn, "d"), dep("d", standsOn, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a | d | f"},
{what: "transitive across kinds stops at a packages edge: F declared on D, D packages A (ADR 0267)",
edges: []inventory.Edge{dep("f", declared, "d"), dep("d", packages, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a"},
// Each kind alone: X depends on A, A changed (widening), then both changed (ordering).
{what: "stands-on (built against A's artifact) widens", edges: []inventory.Edge{dep("x", standsOn, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a | x"},
{what: "stands-on orders", edges: []inventory.Edge{dep("x", standsOn, "a")},
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a | x"},
{what: "declared (build.on) widens", edges: []inventory.Edge{dep("x", declared, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a | x"},
{what: "declared orders", edges: []inventory.Edge{dep("x", declared, "a")},
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a | x"},
{what: "packages, recorded before ADR 0267, widens nothing", edges: []inventory.Edge{dep("x", packages, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a"},
{what: "packages does not order", edges: []inventory.Edge{dep("x", packages, "a")},
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a,x"},
{what: "built-by never widens", edges: []inventory.Edge{dep("x", builtBy, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a"},
{what: "built-by orders", edges: []inventory.Edge{dep("x", builtBy, "a")},
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a | x"},
{what: "worker-of never widens", edges: []inventory.Edge{dep("x", workerOf, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a"},
{what: "worker-of orders", edges: []inventory.Edge{dep("x", workerOf, "a")},
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a | x"},
{what: "a change to the base alone does not move what it builds", edges: []inventory.Edge{dep("x", builtBy, "a"),
dep("d", builtBy, "a"), dep("e", standsOn, "a")},
repo: "one", paths: []string{"modules/a/module.json"}, want: "a | e"},
// The cycles the kinds resolve: the build machine stands on the runtime image the runtime image is
// built by; the build seat's holder follows the controller that is built by it.
{what: "the build machine's base comes first, built by the build machine that runs", entries: runtime,
edges: []inventory.Edge{dep("runtime", builtBy, "builder"), dep("builder", standsOn, "runtime")},
repo: "one", paths: []string{"modules/runtime/Dockerfile", "modules/builder/main.go"}, want: "runtime | builder"},
{what: "the runtime image alone takes the build machine on it along", entries: runtime,
edges: []inventory.Edge{dep("runtime", builtBy, "builder"), dep("builder", standsOn, "runtime")},
repo: "one", paths: []string{"modules/runtime/Dockerfile"}, want: "runtime | builder"},
{what: "the build machine alone moves alone", entries: runtime,
edges: []inventory.Edge{dep("runtime", builtBy, "builder"), dep("builder", standsOn, "runtime")},
repo: "one", paths: []string{"modules/builder/main.go"}, want: "builder"},
{what: "the build seat's holder follows the controller it binds the worker of", entries: runtime,
edges: []inventory.Edge{dep("controller", builtBy, "builder"), dep("builder", workerOf, "controller")},
repo: "two", paths: []string{"cmd/main.go"}, want: "controller"},
// Two repositories.
{what: "a dependent in another repository follows its base",
edges: []inventory.Edge{dep("g", standsOn, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a | g"},
{what: "a directory of the same name in another repository is not this one's",
edges: []inventory.Edge{dep("g", standsOn, "a")},
repo: "two", paths: []string{"modules/a/x"}, want: "", unread: "modules/a/x"},
{what: "the dependent's own repository moves the dependent alone",
edges: []inventory.Edge{dep("g", standsOn, "a")},
repo: "two", paths: []string{"g/main.go"}, want: "g"},
// A diamond.
{what: "a diamond, one side changed", edges: []inventory.Edge{dep("d", standsOn, "a"), dep("d", standsOn, "b")},
repo: "one", paths: []string{"modules/a/x"}, want: "a | d"},
{what: "a diamond, both sides changed", edges: []inventory.Edge{dep("d", standsOn, "a"), dep("d", standsOn, "b")},
repo: "one", paths: []string{"modules/a/x", "modules/b/x"}, want: "a,b | d"},
{what: "a diamond on one base", edges: []inventory.Edge{dep("d", standsOn, "a"), dep("d", declared, "b"),
dep("a", standsOn, "z"), dep("b", standsOn, "z")},
repo: "one", paths: []string{"modules/z/x"}, want: "z | a,b | d"},
{what: "a diamond of mixed kinds orders on the ordering side only",
edges: []inventory.Edge{dep("d", standsOn, "a"), dep("d", packages, "b")},
repo: "one", paths: []string{"modules/b/x"}, want: "b"},
// A cycle the catalogue should never produce: what remains is one last tier, and said.
{what: "a cycle is one last tier, not lost", edges: []inventory.Edge{dep("a", standsOn, "b"), dep("b", standsOn, "a"),
dep("f", standsOn, "c")},
repo: "one", paths: []string{"modules/a/x", "modules/c/x"}, want: "c | f | a,b", cycle: true},
// Files no build reads.
{what: "a README at the root of a repository whose modules all live below it",
edges: []inventory.Edge{dep("d", standsOn, "a")},
repo: "one", paths: []string{"README.md"}, want: "", unread: "README.md"},
{what: "a directory no module lives in", edges: []inventory.Edge{dep("d", standsOn, "a")},
repo: "one", paths: []string{"modules/lib/x.go", "modules/README.md"}, want: "",
unread: "modules/lib/x.go,modules/README.md"},
{what: "a module's directory beside a root file", edges: []inventory.Edge{dep("d", standsOn, "a")},
repo: "one", paths: []string{"merge-check.sh", "modules/a/x"}, want: "a | d", unread: "merge-check.sh"},
{what: "a directory whose name begins with a module's", edges: []inventory.Edge{dep("d", standsOn, "a")},
repo: "one", paths: []string{"modules/ab/x"}, want: "", unread: "modules/ab/x"},
} {
e := entries
if c.entries != nil {
e = c.entries
}
r, got := planMerge(t, c.repo, c.paths, e, nil, c.edges)
if got != c.want {
t.Errorf("%s: planned %q, wanted %q", c.what, got, c.want)
}
if u := strings.Join(r.Unread, ","); u != c.unread {
t.Errorf("%s: unread %q, wanted %q", c.what, u, c.unread)
}
// A packages edge in the last tier is no cycle; hasCycle said one on main at 8170fc5.
if hasCycle(r.Plan.Tiers, c.edges) != c.cycle {
t.Errorf("%s: a cycle said %v, wanted %v (%v)", c.what, !c.cycle, c.cycle, r.Plan.Tiers)
}
}
}
// **A shared repository moves only what a change's files are in the build source of** (novox/hq ADR 0267,
// issue 338, issue 363). The controller is built from its repository's root as a Go bundle; the route proxy
// and the build seat's holder build images whose context is that repository and which name the package they
// compile. Each newest trunk build said its build source — the import closure of its program — and a merge
// is mapped onto those. The rows tagged 338 held the opposite until ADR 0267 was built: every merge to the
// controller's repository planned all three, in three tiers.
//
// The build sources are this repository's own programs as GoBuildSource reads them (held to that by
// TestThisRepositorysProgramsHaveBuildSourcesOfTheirOwn in internal/builder), cut to what the rows need.
func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
const catalogueRepo = "http://forge.internal:20000/novox/mesh-catalog.git"
const controllerRepo = "http://forge.internal:20000/novox/mesh-controller.git"
entries := []inventory.Entry{
fromRepo("mesh-controller", controllerRepo, ""),
fromRepo("build-agent", catalogueRepo, "modules/build-agent"),
fromRepo("route-proxy", catalogueRepo, "modules/route-proxy"),
fromRepo("gitea", catalogueRepo, "modules/gitea"),
}
gomod := []string{"go.mod", "go.sum", "vendor/modules.txt"}
with := func(paths ...string) []string { return append(append([]string{}, gomod...), paths...) }
read := map[string][]inventory.ReadRepository{
"mesh-controller": {{Own: true, Paths: with("module.json", "cmd/mesh-controller/", "internal/conditions/",
"internal/broker/", "internal/builder/", "internal/inventory/", "internal/inventory/migrations/**",
"vendor/github.com/nats-io/nats.go/")}},
"build-agent": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: with("cmd/mesh-builder/", "internal/broker/",
"internal/builder/", "internal/inventory/", "internal/inventory/migrations/**", "vendor/github.com/nats-io/nats.go/")},
{Own: true, Paths: []string{"modules/build-agent/Dockerfile", "modules/build-agent/module.json"}},
},
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: with("examples/route-proxy/", "internal/broker/",
"vendor/github.com/nats-io/nats.go/")},
{Own: true, Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
},
}
// With no build source said — before each module's first trunk build under ADR 0267, or while an
// earlier merge's build of it is pending — a module is read as before.
unsaid := map[string][]inventory.ReadRepository{
"build-agent": {{Repository: "novox/mesh-controller", Ref: "main"}},
"route-proxy": {{Repository: "novox/mesh-controller", Ref: "main"}},
}
edges := sharedRepositoryEdges
for _, c := range []struct {
what, repo string
paths []string
read map[string][]inventory.ReadRepository
want string
unread string
}{
// The operator's acceptance: a merge of the controller's own code plans the controller alone.
{"the controller's own command: the controller alone (338)", "mesh-controller",
[]string{"cmd/mesh-controller/main.go"}, read, "mesh-controller", ""},
{"a package only the controller builds from: the controller alone (338)", "mesh-controller",
[]string{"internal/conditions/condition.go", "internal/conditions/bus.go"}, read, "mesh-controller", ""},
{"a test beside the controller's command: nothing is built from it", "mesh-controller",
[]string{"cmd/mesh-controller/main_test.go"}, read, "", "cmd/mesh-controller/main_test.go"},
{"a README of the controller's repository: no module (338)", "mesh-controller",
[]string{"README.md"}, read, "", "README.md"},
{"the route proxy's program alone: the route proxy alone (338)", "mesh-controller",
[]string{"examples/route-proxy/main.go"}, read, "route-proxy", ""},
{"the build seat's program alone: its holder alone", "mesh-controller",
[]string{"cmd/mesh-builder/main.go"}, read, "build-agent", ""},
{"a package the build seat's program and the controller build from: both", "mesh-controller",
[]string{"internal/builder/builder.go"}, read, "mesh-controller | build-agent", ""},
{"a migration the controller and the build seat's program embed: both", "mesh-controller",
[]string{"internal/inventory/migrations/0088-a-build-says-its-build-source.sql"}, read,
"mesh-controller | build-agent", ""},
// A package all three build from: all three; the build seat's holder after the controller whose worker
// it binds (worker-of, issue 206), the proxy after the holder that builds it (built-by).
{"a package all three build from: all three", "mesh-controller",
[]string{"internal/broker/broker.go"}, read, "mesh-controller | build-agent | route-proxy", ""},
{"a vendored package all three build from: all three", "mesh-controller",
[]string{"vendor/github.com/nats-io/nats.go/nats.go"}, read, "mesh-controller | build-agent | route-proxy", ""},
{"go.sum: all three", "mesh-controller",
[]string{"go.sum"}, read, "mesh-controller | build-agent | route-proxy", ""},
{"a file added to the proxy's package: the proxy", "mesh-controller",
[]string{"examples/route-proxy/new.go"}, read, "route-proxy", ""},
// Before any build source is said: as before.
{"no build source said: a README moves all three, as before", "mesh-controller",
[]string{"README.md"}, unsaid, "mesh-controller | build-agent | route-proxy", ""},
// In the catalogue, where they live, each by its own build source.
{"the route proxy's recipe: it alone", "mesh-catalog",
[]string{"modules/route-proxy/Dockerfile"}, read, "route-proxy", ""},
{"the route proxy's manifest: it alone", "mesh-catalog",
[]string{"modules/route-proxy/module.json"}, read, "route-proxy", ""},
{"the route proxy's README: nothing", "mesh-catalog",
[]string{"modules/route-proxy/README.md"}, read, "", "modules/route-proxy/README.md"},
{"the build agent's manifest: it alone, nothing it builds", "mesh-catalog",
[]string{"modules/build-agent/module.json"}, read, "build-agent", ""},
{"another module of the catalogue: neither", "mesh-catalog",
[]string{"modules/gitea/index.ts"}, read, "gitea", ""},
} {
r, got := planMerge(t, c.repo, c.paths, entries, c.read, edges)
if got != c.want {
t.Errorf("%s: planned %q, wanted %q", c.what, got, c.want)
}
if u := strings.Join(r.Unread, ","); u != c.unread {
t.Errorf("%s: unread %q, wanted %q", c.what, u, c.unread)
}
}
// Files not all said: everything the repository builds, as before.
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main", Commit: "head",
Paths: []string{"README.md"}, PathsTruncated: true}
if got := tiered(reachOfMerge(m, entries, read, edges).Plan.Tiers); got != "mesh-controller | build-agent | route-proxy" {
t.Errorf("a merge whose files were not all said: planned %q, wanted all three", got)
}
}
// **A module whose recorded build source a plan has overtaken is read whole** (novox/hq ADR 0267): a merge
// that added an import to the route proxy is planned; before a build of it works — still building, failed,
// or its plan closed before reaching it — a merge changing only the newly imported package must still move
// the proxy, since the build source its last build said does not hold that package.
func TestAModuleAPlanOvertookIsReadWhole(t *testing.T) {
built := time.Date(2026, 10, 10, 1, 0, 0, 0, time.UTC)
read := map[string][]inventory.ReadRepository{
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"examples/route-proxy/", "go.mod"}, Built: built},
{Own: true, Paths: []string{"modules/route-proxy/module.json"}, Built: built},
},
"mesh-controller": {{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/"}, Built: built}},
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main", Paths: []string{"internal/newly/imported.go"}}
if readsFrom(read["route-proxy"], m) {
t.Fatal("the said build source holds the new package: the fixture is wrong")
}
proxy := func(state string) map[string]*inventory.PlanModule {
return map[string]*inventory.PlanModule{"route-proxy": {State: state}, "gitea": {State: "built"}}
}
for _, c := range []struct {
what string
plans []inventory.Plan
whole bool
}{
{"no plan", nil, false},
{"a plan still building it", []inventory.Plan{{State: inventory.PlanBuilding, Created: built.Add(-time.Hour),
Modules: proxy("building")}}, true},
{"a plan made after its build that failed before building it", []inventory.Plan{{State: "failed",
Created: built.Add(time.Minute), Modules: proxy("waiting")}}, true},
{"a plan made after its build that built it", []inventory.Plan{{State: inventory.PlanDone,
Created: built.Add(time.Minute), Modules: proxy("built")}}, false},
{"a plan closed before its build", []inventory.Plan{{State: "failed", Created: built.Add(-time.Hour),
Modules: proxy("waiting")}}, false},
} {
view := planningView(read, c.plans)
if readsFrom(view["route-proxy"], m) != c.whole {
t.Errorf("%s: read whole %v, wanted %v", c.what, !c.whole, c.whole)
}
if (ownSource(view["route-proxy"]) == nil) != c.whole {
t.Errorf("%s: its own build source kept %v", c.what, ownSource(view["route-proxy"]) != nil)
}
if ownSource(view["mesh-controller"]) == nil {
t.Errorf("%s: a module no plan holds lost its build source", c.what)
}
}
}
// **A missed merge that moves only a module packaging the repository is acted on** (novox/hq ADR 0267,
// issue 266): the catch-up asks wouldMove, which counts it; once a plan or build of it is made after the
// merge, the merge is history for it, and the catch-up leaves it.
func TestAMissedMergeMovingOnlyAPackagingModuleIsActedOnOnce(t *testing.T) {
merged := time.Date(2026, 10, 10, 1, 0, 0, 0, time.UTC)
entries := []inventory.Entry{
fromRepo("mesh-controller", "http://forge.internal:20000/novox/mesh-controller.git", ""),
fromRepo("route-proxy", "http://forge.internal:20000/novox/mesh-catalog.git", "modules/route-proxy"),
}
read := map[string][]inventory.ReadRepository{
"mesh-controller": {{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/"}, Built: merged.Add(-time.Hour)}},
"route-proxy": {{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"examples/route-proxy/"},
Built: merged.Add(-time.Hour), Looked: merged.Add(-time.Hour)}},
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main", Commit: "c1",
MergedAt: merged.Format(time.RFC3339), Paths: []string{"examples/route-proxy/main.go"}}
if got := wouldMove(m, entries, planningView(read, nil)); len(got) != 1 || got[0].Manifest.Module != "route-proxy" {
t.Fatalf("a missed merge of the proxy's program would move %v", got)
}
// Acted on at once: the plan answering this very merge is a look, however close the clocks.
atOnce := []inventory.Plan{{State: inventory.PlanBuilding, Commit: "c1", Created: merged.Add(2 * time.Second),
Modules: map[string]*inventory.PlanModule{"route-proxy": {State: "building"}}}}
if got := wouldMove(m, entries, planningView(read, atOnce)); len(got) != 0 {
t.Fatalf("a merge whose own plan holds the proxy would move %v again", got)
}
acted := []inventory.Plan{{State: inventory.PlanBuilding, Created: merged.Add(2 * time.Minute),
Modules: map[string]*inventory.PlanModule{"route-proxy": {State: "building"}}}}
if got := wouldMove(m, entries, planningView(read, acted)); len(got) != 0 {
t.Fatalf("a merge acted on for the proxy would move %v again", got)
}
// A plan that closed without building it looked at nothing: the merge is still news for it.
closed := []inventory.Plan{{State: "failed", Created: merged.Add(2 * time.Minute),
Modules: map[string]*inventory.PlanModule{"route-proxy": {State: "waiting"}}}}
if got := wouldMove(m, entries, planningView(read, closed)); len(got) != 1 {
t.Fatalf("a plan that never built the proxy hid the merge from it: %v", got)
}
// A look just before the merge, on clocks a little apart, is no look after it.
skewed := []inventory.Plan{{State: inventory.PlanBuilding, Created: merged.Add(30 * time.Second),
Modules: map[string]*inventory.PlanModule{"route-proxy": {State: "building"}}}}
if got := wouldMove(m, entries, planningView(read, skewed)); len(got) != 1 {
t.Fatalf("a look within the clocks' margin made the merge history: %v", got)
}
}
// sharedRepositoryEdges is what dependenciesOf derives for the catalogue of the test above, sorted as it
// sorts them: no packages edge (novox/hq ADR 0267 rule 4).
var sharedRepositoryEdges = []inventory.Edge{
dep("build-agent", inventory.EdgeWorkerOf, "mesh-controller"),
dep("gitea", inventory.EdgeBuiltBy, "build-agent"),
dep("mesh-controller", inventory.EdgeBuiltBy, "build-agent"),
dep("route-proxy", inventory.EdgeBuiltBy, "build-agent"),
}
// **The planner's invariant, over random catalogues.** For any catalogue whose dependencies form no cycle
// and any set of changed files in one repository:
//
// - the plan is exactly the modules of that repository whose directory holds a changed file (every file,
// for a module built from the root), and everything reachable from them along stands-on and declared —
// never along packages (novox/hq ADR 0267), built-by or worker-of;
// - every stands-on, declared, built-by and worker-of edge with both ends in the plan has the module
// depended on in an earlier tier;
// - no cycle is said.
//
// Seeded, so a failure is replayed by its seed and case.
func TestAPlanIsTheTouchedModulesAndWhatIsReachableAlongTheWideningEdges(t *testing.T) {
kinds := []string{inventory.EdgeStandsOn, inventory.EdgeDeclared, inventory.EdgePackages,
inventory.EdgeBuiltBy, inventory.EdgeWorkerOf}
widens := map[string]bool{inventory.EdgeStandsOn: true, inventory.EdgeDeclared: true}
orders := map[string]bool{inventory.EdgeStandsOn: true, inventory.EdgeDeclared: true,
inventory.EdgeBuiltBy: true, inventory.EdgeWorkerOf: true}
// Directory names drawn from one pool, so two repositories hold directories of the same name, and one
// is a prefix of another.
dirs := []string{"a", "ab", "b", "c", "lib/x", "lib/y", "modules/a", "modules/a/sub"}
files := []string{"README.md", "merge-check.sh", "lib/z.go", "docs/x.md", "modules/README.md"}
falseCycles, firstFalseCycle := 0, ""
for _, seed := range []uint64{1, 2, 3, 0x338, 0x162} {
rng := rand.New(rand.NewPCG(seed, seed^0x9e3779b97f4a7c15))
for n := 0; n < 100; n++ {
repos := 1 + rng.IntN(3)
repoName := func(i int) string { return fmt.Sprintf("r%d", i) }
count := 1 + rng.IntN(12)
var entries []inventory.Entry
repoOf, dirOf := map[string]int{}, map[string]string{}
for i := 0; i < count; i++ {
name := fmt.Sprintf("m%02d", i)
repo := rng.IntN(repos)
dir := dirs[rng.IntN(len(dirs))]
if rng.IntN(12) == 0 {
dir = "" // built from the repository's root
}
repoOf[name], dirOf[name] = repo, dir
entries = append(entries, fromRepo(name, "http://forge.internal:20000/novox/"+repoName(repo)+".git", dir))
}
// A graph with no cycle: a module depends only on modules made before it.
var edges []inventory.Edge
for i := 1; i < count; i++ {
for j := 0; j < i; j++ {
if rng.IntN(4) == 0 {
edges = append(edges, dep(fmt.Sprintf("m%02d", i), kinds[rng.IntN(len(kinds))], fmt.Sprintf("m%02d", j)))
}
}
}
merged := rng.IntN(repos)
var paths []string
for k := 1 + rng.IntN(4); k > 0; k-- {
if rng.IntN(3) == 0 {
paths = append(paths, files[rng.IntN(len(files))])
} else {
paths = append(paths, dirs[rng.IntN(len(dirs))]+"/f.go")
}
}
// What the rules say.
want := map[string]bool{}
for _, e := range entries {
name := e.Manifest.Module
if repoOf[name] != merged {
continue
}
for _, p := range paths {
if d := dirOf[name]; d == "" || p == d || strings.HasPrefix(p, d+"/") {
want[name] = true
}
}
}
for grew := true; grew; {
grew = false
for _, e := range edges {
if widens[e.Kind] && want[e.To] && !want[e.From] {
want[e.From], grew = true, true
}
}
}
r, _ := planMerge(t, repoName(merged), paths, entries, nil, edges)
got := map[string]bool{}
tierOf := map[string]int{}
for i, tier := range r.Plan.Tiers {
for _, name := range tier {
got[name], tierOf[name] = true, i
}
}
replay := func() string {
return fmt.Sprintf("seed %#x case %d: repository %s, files %v\n modules %v\n edges %v\n tiers %v",
seed, n, repoName(merged), paths, describe(entries), edges, r.Plan.Tiers)
}
if !sameSet(got, want) {
t.Fatalf("planned %v, wanted %v\n%s", keys(got), keys(want), replay())
}
for _, e := range edges {
if orders[e.Kind] && got[e.From] && got[e.To] && tierOf[e.To] >= tierOf[e.From] {
t.Fatalf("%s %s %s, and %s is in tier %d, not before %s's %d\n%s", e.From, e.Kind, e.To,
e.To, tierOf[e.To], e.From, tierOf[e.From], replay())
}
}
if hasCycle(r.Plan.Tiers, edges) {
falseCycles++
if firstFalseCycle == "" {
firstFalseCycle = replay()
}
}
}
}
// Said once, after the other invariants have run over every case, so it hides none of them (hasCycle
// counted a packages edge on main at 8170fc5: 19 of these 500 cases).
if falseCycles > 0 {
t.Errorf("a cycle said of a graph with none in %d of 500 cases; "+
"the first:\n%s", falseCycles, firstFalseCycle)
}
}
func sameSet(a, b map[string]bool) bool {
if len(a) != len(b) {
return false
}
for k := range a {
if !b[k] {
return false
}
}
return true
}
func keys(m map[string]bool) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
func describe(entries []inventory.Entry) []string {
var out []string
for _, e := range entries {
out = append(out, fmt.Sprintf("%s@%s:%q", e.Manifest.Module,
strings.TrimSuffix(strings.TrimPrefix(e.Source.Repository, "http://forge.internal:20000/novox/"), ".git"),
e.Source.Path))
}
return out
}
// **The merge gate reads the build sources the snapshot carries** (novox/hq ADR 0267): the gate's plan of a
// change is the merge handler's, so a snapshot taken by a controller that records build sources narrows the
// gate's plan as it narrows the merge's; one without them reads every module as before.
func TestTheGatePlansFromTheBuildSourcesTheSnapshotCarries(t *testing.T) {
manifest := func(name string) json.RawMessage { return json.RawMessage(`{"module":"` + name + `","version":"1"}`) }
facts := snapshot.Facts{Modules: []snapshot.Module{
{Name: "mesh-controller", Repository: "novox/mesh-controller", Manifest: manifest("mesh-controller"),
Sources: []snapshot.BuildSource{{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/", "internal/broker/"}}}},
{Name: "route-proxy", Repository: "novox/mesh-catalog", Path: "modules/route-proxy", Manifest: manifest("route-proxy"),
Reads: []string{"novox/mesh-controller"},
Sources: []snapshot.BuildSource{{Repository: "novox/mesh-controller", Paths: []string{"examples/route-proxy/", "internal/broker/"}},
{Own: true, Paths: []string{"modules/route-proxy/module.json"}}}},
}}
for paths, want := range map[string]string{
"cmd/mesh-controller/main.go": "mesh-controller",
"examples/route-proxy/main.go": "route-proxy",
"internal/broker/broker.go": "mesh-controller,route-proxy",
"README.md": "",
} {
r, err := reachOfChange(facts, "novox/mesh-controller", []string{paths}, "")
if err != nil {
t.Fatal(err)
}
if got := tiered(r.Plan.Tiers); got != want {
t.Errorf("%s: the gate planned %q, wanted %q", paths, got, want)
}
}
// A snapshot without build sources: as before.
for i := range facts.Modules {
facts.Modules[i].Sources = nil
}
r, err := reachOfChange(facts, "novox/mesh-controller", []string{"README.md"}, "")
if err != nil {
t.Fatal(err)
}
if got := tiered(r.Plan.Tiers); got != "mesh-controller,route-proxy" {
t.Errorf("a snapshot without build sources: the gate planned %q for a README", got)
}
}
// **A plan says why each module is in it** (novox/hq ADR 0267, issue 363): the files of its build source the
// merge changed, or why it is read whole — never that it packages a repository as though that moved it.
func TestAPlanSaysWhyEachModuleIsInIt(t *testing.T) {
const controllerRepo = "http://forge.internal:20000/novox/mesh-controller.git"
controller := fromRepo("mesh-controller", controllerRepo, "")
agent := fromRepo("build-agent", "http://forge.internal:20000/novox/mesh-catalog.git", "modules/build-agent")
gitea := fromRepo("gitea", "http://forge.internal:20000/novox/mesh-catalog.git", "modules/gitea")
built := time.Date(2026, 10, 10, 12, 26, 0, 0, time.UTC)
read := map[string][]inventory.ReadRepository{
"mesh-controller": {{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/", "internal/link/"}, Built: built}},
"build-agent": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"cmd/mesh-builder/", "internal/link/"}, Built: built},
{Own: true, Paths: []string{"modules/build-agent/module.json"}, Built: built},
},
}
merge := func(repo string, paths ...string) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Paths: paths}
}
open := []inventory.Plan{{ID: "plan-1", State: inventory.PlanBuilding, Created: built.Add(time.Minute),
Modules: map[string]*inventory.PlanModule{"build-agent": {State: "asked"}}}}
for _, c := range []struct {
what string
e inventory.Entry
read map[string][]inventory.ReadRepository
m link.SourceMoved
says string
}{
{"the controller's own closure", controller, read, merge("mesh-controller", "README.md", "internal/link/handacts.go"),
"its build source changed: 1 changed file(s) in it, e.g. internal/link/handacts.go"},
{"the build seat's closure, through its context", agent, read, merge("mesh-controller", "internal/link/handacts.go"),
"its build source in novox/mesh-controller changed: 1 changed file(s) in it, e.g. internal/link/handacts.go"},
{"an open plan has yet to build it", agent, planningView(read, open), merge("mesh-controller", "cmd/mesh-controller/main.go"),
"read whole: plan plan-1 has not built it yet, so every file of novox/mesh-controller, which its build context is, is its build source"},
{"nothing recorded, built from its root", controller, nil, merge("mesh-controller", "README.md"),
"read whole: no build source recorded, so every file of its repository is its build source"},
{"nothing recorded, in its directory", gitea, nil, merge("mesh-catalog", "modules/gitea/index.ts"),
"read whole: no build source recorded, so its directory is its build source; e.g. modules/gitea/index.ts"},
{"a root manifest is not in a module's directory", gitea, nil, merge("mesh-catalog", "module.json", "modules/gitea/x.ts"),
"read whole: no build source recorded, so its directory is its build source; e.g. modules/gitea/x.ts"},
{"a context on another branch says nothing of this one", agent, map[string][]inventory.ReadRepository{"build-agent": {
{Repository: "novox/mesh-controller", Ref: "release", Paths: []string{"internal/link/"}},
{Repository: "novox/mesh-controller", Ref: "main"}}}, merge("mesh-controller", "internal/link/handacts.go"),
"read whole: no build source recorded, so every file of novox/mesh-controller, which its build context is, is its build source"},
{"files not all said", controller, read, link.SourceMoved{Owner: "novox", Repo: "mesh-controller", PathsTruncated: true,
Paths: []string{"x"}}, "read whole: the merge's changed files were not all said"},
} {
if got := whyMoved(c.e, c.read[c.e.Manifest.Module], c.m); got != c.says {
t.Errorf("%s:\n said %q\n wanted %q", c.what, got, c.says)
}
}
}
+378
View File
@@ -0,0 +1,378 @@
package main
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// Who can become root where the trusted parties run (novox/hq ADR 0259 §8, as reviewed on 2026-10-09).
//
// The router and every channel proving its sender run as accounts of their own, so that no agent reads what
// they hold or speaks as them. **Root on their machine undoes all of it**, and so does an agent running as the
// operator's account there. A machine is **root-free** — an answer proven there may authorise — only when all
// of these are measured, now, and hold:
//
// 1. the machine names an account agents run as (novox/hq ADR 0266), so no agent runs as the operator's
// account, which may become root;
// 2. its node-engine — running as root, which no agent controls — judged that account unable to become root
// without a person, in a statement heard within the last 15 minutes (agentConfined, judgedConfined). The
// engine gives that verdict only from a complete search for setuid programs younger than mesh-host's
// rootsearch.FreshFor; before one, it says "not judged yet", which is no pass (novox/hq issue 361);
// 3. the login shell's `execute` is not served there (novox/hq ADR 0268): its holder's setting withholds it
// **and** the bus was asked and heard no `execute` answered there. `execute` runs commands as the machine's
// runtime account, which the mesh's acting tools give passwordless sudo; that account is taken to become
// root, always, so no measure of it is asked.
//
// **Nothing else is a pass.** A machine that names no agent account, an unknown machine, a store or bus that
// could not be read, a verdict stale or absent — each is not root-free, and says why. The sudo module's own
// measure is no longer part of this judgement: it ran in the machine's runtime, as the very account an agent
// could become, so it could not be believed.
//
// The one judgement (judgeRoot) is read two ways: the self-check raises `root-not-free` on every machine where
// the router or a module of its own account runs and the judgement fails; and the `root-free` verb answers it
// live, to the router, which honours a verified sender only on its pass. A machine holding the operator's
// graphical session, where a messaging client's desktop app may run, is not judged here: the operator accepted
// that gap for now (hq issue 344).
// kindRootNotFree is the condition a trusted party's machine that is not root-free raises. Its own key, apart
// from ADR 0266's agent-can-become-root (Token agent-root, from DA): the two judge different things — DA the
// agent account alone, this the whole of root-free — and one key from two probes flapped between them (the
// confirmation review of 2026-10-09).
const kindRootNotFree = "root-not-free"
// routerSeat is the seat the router holds: where it runs counts as a trusted party's machine.
const routerSeat = "operator-channel"
const (
loginShellSeat = "node-login-shell"
// loginShellVerb is the seat's verb that runs a command, and the name of the setting its holder withholds
// it by (novox/hq ADR 0268).
loginShellVerb = "execute"
// executeServes is the one value of that setting that serves the verb; anything else withholds it.
executeServes = "serve"
)
// loginShellServed judges whether the login shell's execute is served on a machine, failing closed (novox/hq
// ADR 0268): served while the holder's setting there is `serve` (or the holder has no such setting and claims
// the verb), or while the bus heard the verb answered there, or while the bus could not be asked. why says
// which, in words.
func loginShellServed(holder string, claims bool, setting *any, heard, asked bool) (bool, string) {
var why []string
switch {
case setting != nil:
if v, _ := (*setting).(string); v == executeServes {
why = append(why, holder+"'s execute setting there is "+executeServes)
}
case claims:
why = append(why, holder+" claims execute and has no setting that withholds it")
}
if heard {
why = append(why, "the bus hears execute answered there")
} else if !asked {
why = append(why, "the bus could not be asked whether execute is answered there")
}
return len(why) > 0, strings.Join(why, "; ")
}
// rootFacts is what the judgement reads of one machine.
type rootFacts struct {
Machine string
// Unread is every read that failed, in words: any one is a fail.
Unread []string
// AgentNamed is whether the machine names an agent account; Confined whether its node-engine judged it
// unable to become root, freshly; ConfinedWhy the judgement's words either way.
AgentNamed bool
Confined bool
ConfinedWhy string
// Execute is whether the login shell's execute is served there; ExecuteWhy why, in words.
Execute bool
ExecuteWhy string
// SearchPending is the agent account unjudged only because the node-engine's setuid search runs, within
// the quiet the controller gives it (searchQuietFor; ADR 0266's quiet window, issue 361).
SearchPending bool
}
// rootVerdict is the judgement on one machine, as the root-free verb answers it.
type rootVerdict struct {
Machine string `json:"machine"`
Free bool `json:"free"`
Why string `json:"why"`
Judged time.Time `json:"judged"`
// Quiet is a machine not free only because its setuid search still runs, within searchQuietFor: the
// self-check raises nothing for it then (ADR 0266's quiet window). It is never free for it.
Quiet bool `json:"quiet,omitempty"`
}
// judgeRoot is the one judgement: free only when nothing failed to read, an agent account is named and judged
// confined, and execute is not served.
func judgeRoot(f rootFacts, now time.Time) rootVerdict {
v := rootVerdict{Machine: f.Machine, Judged: now.UTC()}
var not []string
if len(f.Unread) > 0 {
not = append(not, "not measured: "+strings.Join(f.Unread, "; "))
}
switch {
case f.ConfinedWhy == "":
// Not read (said above), or nothing said of it: never a pass.
if len(f.Unread) == 0 {
not = append(not, "whether agents there can become root was not judged")
}
case !f.AgentNamed:
not = append(not, "agents run as the operator's account there, which may become root ("+f.ConfinedWhy+")")
case !f.Confined:
not = append(not, f.ConfinedWhy)
}
if f.Execute {
not = append(not, "the login shell runs any command an agent gives it as the machine's runtime account, "+
"which can become root ("+orNoneKnown(f.ExecuteWhy)+")")
}
if len(not) > 0 {
v.Why = strings.Join(not, "; ")
// The one failure is the agent account not judged yet, because its search runs.
v.Quiet = len(not) == 1 && f.SearchPending && f.AgentNamed && !f.Confined && len(f.Unread) == 0 && !f.Execute
return v
}
v.Free = true
v.Why = f.ConfinedWhy + "; the login shell's execute is not served there"
return v
}
// rootReader reads the facts of machines live: the catalogue's placements, the node-engine's verdicts and the
// bus's discovery, each once per reader.
type rootReader struct {
entries []inventory.Entry
read error
heard map[string]map[string]map[string]bool
asked error
// confined is agentConfined; settings the login shell holder's settings on a machine. Replaceable in a test.
confined func(ctx context.Context, node string, now time.Time) (named, confined bool, why string, err error)
// quiet says the one thing keeping a machine's agent account unjudged is the node-engine's setuid
// search, within searchQuietFor (ADR 0266, searchStillRunning). Read by the self-check alone, to raise nothing
// then; nil reads no quiet. It never makes a machine root-free.
quiet func(ctx context.Context, node string, now time.Time) bool
settings func(ctx context.Context, node, module string) ([]catalogue.Layer, error)
}
func newRootReader(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn) *rootReader {
r := &rootReader{}
r.entries, r.read = inv.Catalogued(ctx)
if conn == nil {
r.asked = fmt.Errorf("this process holds no connection to the bus")
} else {
r.heard, r.asked = discoverSeatVerbs(ctx, conn)
}
r.confined = func(ctx context.Context, node string, now time.Time) (bool, bool, string, error) {
return agentConfined(ctx, inv, node, now)
}
r.settings = inv.SettingsFor
return r
}
// facts reads one machine, at now.
func (r *rootReader) facts(ctx context.Context, machine string, now time.Time) rootFacts {
f := rootFacts{Machine: machine}
if r.read != nil {
f.Unread = append(f.Unread, "the catalogue's placements could not be read: "+r.read.Error())
}
named, confined, why, err := r.confined(ctx, machine, now)
if err != nil {
f.Unread = append(f.Unread, "the account agents run as could not be read: "+err.Error())
} else {
f.AgentNamed, f.Confined, f.ConfinedWhy = named, confined, why
}
f.Execute, f.ExecuteWhy = r.executeServed(ctx, machine)
if r.quiet != nil && f.AgentNamed && !f.Confined {
f.SearchPending = r.quiet(ctx, machine, now)
}
return f
}
// executeServed is whether the login shell's execute is served on a machine, failing closed: the bus not
// asked, the placements not read, or a holder's setting not read, is served.
func (r *rootReader) executeServed(ctx context.Context, machine string) (bool, string) {
heard := r.heard[loginShellSeat][loginShellVerb][machine]
asked := r.asked == nil
var whys []string
served := false
holders := 0
for _, e := range r.entries {
if !e.Manifest.ClaimsSeat(loginShellSeat) || !slices.Contains(e.On, machine) {
continue
}
holders++
var setting *any
if _, declared := e.Manifest.Settings[loginShellVerb]; declared {
layers, err := r.settings(ctx, machine, e.Manifest.Module)
if err != nil {
served = true
whys = append(whys, e.Manifest.Module+"'s setting there could not be read: "+err.Error())
continue
}
for _, s := range catalogue.Effective(e.Manifest, layers) {
if s.Key == loginShellVerb {
v := s.Value
setting = &v
}
}
}
if s, why := loginShellServed(e.Manifest.Module, claimServes(e.Manifest, loginShellSeat, loginShellVerb),
setting, heard, asked); s {
served = true
whys = append(whys, why)
}
}
if holders == 0 {
// Nobody is assigned to serve it; the bus must still hear nobody answering it.
if s, why := loginShellServed("no holder", false, nil, heard, asked); s {
served = true
whys = append(whys, why)
}
}
if r.read != nil {
served = true
whys = append(whys, "who holds the login shell there could not be read")
}
return served, strings.Join(whys, "; ")
}
// claimServes says whether a manifest's claim of a seat names a verb among those it serves.
func claimServes(m catalogue.Manifest, seat, verb string) bool {
for _, c := range m.Claims {
if c.Name == seat && slices.Contains(c.Serves, verb) {
return true
}
}
return false
}
// judgeRootFree is the root-free verb's answer: each named machine judged now. It never fails: what could not
// be read is a machine not free, saying so.
func judgeRootFree(ctx context.Context, r *rootReader, machines []string, now time.Time) []rootVerdict {
out := make([]rootVerdict, 0, len(machines))
for _, m := range machines {
out = append(out, judgeRoot(r.facts(ctx, m, now), now))
}
return out
}
// trustedMachines are the machines where the router or a module of its own account runs, each with those
// modules.
func trustedMachines(entries []inventory.Entry) map[string][]string {
trusted := map[string][]string{}
for _, e := range entries {
for _, node := range e.On {
if e.Manifest.RunsAs != "" || e.Manifest.ClaimsSeat(routerSeat) {
trusted[node] = append(trusted[node], e.Manifest.Module)
}
}
}
return trusted
}
// agentRootObservation is the condition of a trusted party's machine that is not root-free.
func agentRootObservation(v rootVerdict, trusted []string) conditions.Observation {
trusted = append([]string(nil), trusted...)
sort.Strings(trusted)
return conditions.Observation{Scope: conditions.ScopeMachine, ID: v.Machine, Token: kindRootNotFree,
Machine: v.Machine, Kind: kindRootNotFree, Severity: conditions.Urgent,
Summary: fmt.Sprintf("%s is not root-free, where %s run: until it is, the router approves nothing proven "+
"there (novox/hq ADR 0259 §8): %s", v.Machine, strings.Join(trusted, ", "), v.Why),
Headline: "Phone answers held on " + v.Machine,
Needs: "give the programs working for you on " + v.Machine + " an account that cannot become root.",
Explanation: "The modules that prove your answers from your phone run on " + v.Machine + ", and the mesh " +
"cannot show that a program working for you there is unable to become root or to act as you. Until " +
"it can, answers from your phone can only acknowledge.",
Resolved: "Answers from your phone can approve again on " + v.Machine}
}
// probeAgentRoot is the probe: every trusted party's machine, judged by the one judgement.
func probeAgentRoot(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
var conn *nats.Conn
if d.js != nil {
conn = d.js.Conn()
}
inv := d.open.inventory
r := newRootReader(ctx, inv, conn)
if r.read != nil {
return nil, r.read
}
// The self-check alone reads ADR 0266's quiet window: nothing raised while a machine's setuid search
// runs, within searchQuietFor. The root-free verb never reads it, so the machine still answers not free.
r.quiet = func(ctx context.Context, node string, now time.Time) bool {
n, err := inv.NodeByName(ctx, node)
if err != nil || n.AgentAccount == "" {
return false
}
h, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false
}
quiet, err := searchStillRunning(ctx, inv, node, n.AgentAccount, h, had, now)
return err == nil && quiet
}
return rootObservations(ctx, r, trustedMachines(r.entries), time.Now()), nil
}
// rootObservations judges the machines and says each that fails.
func rootObservations(ctx context.Context, r *rootReader, trusted map[string][]string, now time.Time) []conditions.Observation {
var machines []string
for m := range trusted {
machines = append(machines, m)
}
sort.Strings(machines)
var out []conditions.Observation
for _, v := range judgeRootFree(ctx, r, machines, now) {
if !v.Free && !v.Quiet {
out = append(out, agentRootObservation(v, trusted[v.Machine]))
}
}
return out
}
// rootClock is the clock the root-free verb judges by.
var rootClock = time.Now
// rootFreeAnswer is the root-free verb: the named machines, each judged now by the serving controller. Only it
// answers: a process that is not serving says so, and a caller reads that as no machine free.
func rootFreeAnswer(ctx context.Context, machines string, now time.Time) (any, error) {
d := doctorFrom
if d == nil || d.open == nil || d.open.inventory == nil {
return nil, fmt.Errorf("this controller is not serving, so it judges no machine root-free: ask again, and " +
"the serving controller answers")
}
var names []string
for _, m := range strings.Split(machines, ",") {
if m = strings.TrimSpace(m); m != "" && !slices.Contains(names, m) {
names = append(names, m)
}
}
if len(names) == 0 {
return nil, fmt.Errorf("root-free judges the machines named, and none was")
}
var conn *nats.Conn
if d.js != nil {
conn = d.js.Conn()
}
return map[string]any{"machines": judgeRootFree(ctx, newRootReader(ctx, d.open.inventory, conn), names, now)}, nil
}
// rootFreeNow is the machines judged root-free, for composing a push's memberships: only those that pass.
func rootFreeNow(ctx context.Context, r *rootReader, machines []string, now time.Time) map[string]bool {
free := map[string]bool{}
for _, v := range judgeRootFree(ctx, r, machines, now) {
if v.Free {
free[v.Machine] = true
}
}
return free
}
@@ -0,0 +1,265 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
var rootNow = time.Date(2026, 10, 9, 12, 0, 0, 0, time.UTC)
// A machine is root-free only on a positive measure of each thing (the review of 2026-10-09, H2/H3): every
// read succeeded, an agent account is named and judged confined by the node-engine, execute is not served.
func TestRootFreeIsAPositiveMeasureAndNothingElse(t *testing.T) {
pass := rootFacts{Machine: "anchor", AgentNamed: true, Confined: true,
ConfinedWhy: "the agent account agents cannot become root without a person (judged 2026-10-09 12:00)"}
if v := judgeRoot(pass, rootNow); !v.Free || v.Machine != "anchor" || !v.Judged.Equal(rootNow) {
t.Fatalf("the one pass: %+v", v)
}
fails := map[string]func(*rootFacts){
"a read failed": func(f *rootFacts) { f.Unread = []string{"the store did not answer"} },
"no agent account named": func(f *rootFacts) { f.AgentNamed, f.Confined = false, false },
"not confined": func(f *rootFacts) { f.Confined = false },
"nothing said of it": func(f *rootFacts) { f.ConfinedWhy = "" },
"execute served": func(f *rootFacts) { f.Execute, f.ExecuteWhy = true, "the bus hears execute answered there" },
"confined, but agent read": func(f *rootFacts) { f.Unread, f.ConfinedWhy = []string{"x"}, "" },
}
for name, mutate := range fails {
f := pass
mutate(&f)
if v := judgeRoot(f, rootNow); v.Free || v.Why == "" {
t.Errorf("%s: judged %+v", name, v)
}
}
}
// The reader fails closed on every case the review named: the agent account read only where the coding-agent
// module runs (old :225), no account to measure taken for a pass (old :246), and a measure that did not answer
// taken for "not root" (old :114, :123).
func TestTheRootReaderFailsClosed(t *testing.T) {
shell := catalogue.Manifest{Module: "zsh", Claims: []catalogue.Claim{{Name: loginShellSeat, Serves: []string{"execute"}}},
Settings: map[string]catalogue.SettingDeclaration{"execute": {Default: "withhold"}}}
reader := func() *rootReader {
return &rootReader{
entries: []inventory.Entry{{Manifest: shell, On: []string{"anchor"}}},
heard: map[string]map[string]map[string]bool{},
confined: func(context.Context, string, time.Time) (bool, bool, string, error) {
return true, true, "the agent account agents cannot become root without a person", nil
},
settings: func(context.Context, string, string) ([]catalogue.Layer, error) { return nil, nil },
}
}
ctx := context.Background()
if v := judgeRootFree(ctx, reader(), []string{"anchor"}, rootNow)[0]; !v.Free {
t.Fatalf("the control: agents confined, execute withheld and unheard, everything read: %+v", v)
}
cases := map[string]func(*rootReader){
"no agent account named, no coding-agent module there": func(r *rootReader) {
r.confined = func(context.Context, string, time.Time) (bool, bool, string, error) {
return false, false, "anchor names no agent account: agents run as the operator account (ops)", nil
}
},
"the node-engine's verdict not read": func(r *rootReader) {
r.confined = func(context.Context, string, time.Time) (bool, bool, string, error) {
return false, false, "", errors.New("the store did not answer")
}
},
"a stale verdict": func(r *rootReader) {
r.confined = func(context.Context, string, time.Time) (bool, bool, string, error) {
return true, false, "the agent account agents is not judged: the machine's newest statement was heard at …", nil
}
},
"the bus not asked": func(r *rootReader) { r.asked = errors.New("no bus") },
"the placements not read": func(r *rootReader) { r.read = errors.New("no store") },
"the holder's setting not read": func(r *rootReader) {
r.settings = func(context.Context, string, string) ([]catalogue.Layer, error) { return nil, errors.New("no store") }
},
"execute heard on the bus": func(r *rootReader) {
r.heard = map[string]map[string]map[string]bool{loginShellSeat: {"execute": {"anchor": true}}}
},
"a holder that serves execute": func(r *rootReader) {
r.entries[0].Manifest.Settings = nil
},
}
for name, mutate := range cases {
r := reader()
mutate(r)
if v := judgeRootFree(ctx, r, []string{"anchor"}, rootNow)[0]; v.Free {
t.Errorf("%s: judged free: %+v", name, v)
}
}
// A machine with no login shell holder at all: still the bus must hear none.
r := reader()
r.entries = nil
r.heard = map[string]map[string]map[string]bool{loginShellSeat: {"execute": {"anchor": true}}}
if v := judgeRootFree(ctx, r, []string{"anchor"}, rootNow)[0]; v.Free {
t.Errorf("execute answered by a module nobody assigned: %+v", v)
}
}
// The probe says agent-root, by the same judgement, on each machine where the router or a module of its own
// account runs and that is not root-free; urgent and in plain words; nothing where every one is free.
func TestTheProbeSaysEachMachineThatIsNotRootFree(t *testing.T) {
entries := []inventory.Entry{
{Manifest: catalogue.Manifest{Module: "telegram", RunsAs: "telegram"}, On: []string{"anchor"}},
{Manifest: catalogue.Manifest{Module: "messenger", RunsAs: "messenger",
Claims: []catalogue.Claim{{Name: routerSeat}}}, On: []string{"anchor"}},
{Manifest: catalogue.Manifest{Module: "xorg", Claims: []catalogue.Claim{{Name: catalogue.DisplayServerSeat}}},
On: []string{"laptop"}},
}
trusted := trustedMachines(entries)
if len(trusted["anchor"]) != 2 || len(trusted["laptop"]) != 0 {
t.Fatalf("trusted %v", trusted)
}
r := &rootReader{entries: entries, heard: map[string]map[string]map[string]bool{},
confined: func(_ context.Context, node string, _ time.Time) (bool, bool, string, error) {
return false, false, node + " names no agent account: agents run as the operator account (ops)", nil
},
settings: func(context.Context, string, string) ([]catalogue.Layer, error) { return nil, nil }}
got := rootObservations(context.Background(), r, trusted, rootNow)
if len(got) != 1 || got[0].Machine != "anchor" || got[0].Kind != kindRootNotFree || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "messenger, telegram") || !strings.Contains(got[0].Summary, "names no agent account") {
t.Fatalf("said %+v", got)
}
o := got[0]
if why, ok := conditions.PlainWords(conditions.Words{Headline: o.Headline, Explanation: o.Explanation,
Needs: o.Needs, Resolved: o.Resolved}, o.Machine); !ok {
t.Errorf("not plain: %s", why)
}
r.confined = func(context.Context, string, time.Time) (bool, bool, string, error) {
return true, true, "confined", nil
}
if got := rootObservations(context.Background(), r, trusted, rootNow); len(got) != 0 {
t.Errorf("said of a free machine: %+v", got)
}
}
// novox/hq ADR 0268: the login shell counts only where its execute is served, and fails closed.
func TestTheLoginShellCountsOnlyWhereExecuteIsServed(t *testing.T) {
val := func(v any) *any { return &v }
cases := []struct {
name string
claims bool
setting *any
heard, asked bool
served bool
saysInTheWhys string
}{
{"withheld by the setting and silent on the bus", true, val("withhold"), false, true, false, ""},
{"withheld by the setting, the machine not yet pushed", true, val("withhold"), true, true, true, "the bus hears"},
{"the setting serves", true, val("serve"), false, true, true, "setting there is serve"},
{"a wrong value withholds, as the holder does", true, val("Serve"), false, true, false, ""},
{"the setting withholds, the bus could not be asked", true, val("withhold"), false, false, true, "could not be asked"},
{"a holder with no such setting that claims execute", true, nil, false, true, true, "claims execute"},
{"a holder with no such setting that does not claim it, heard all the same", false, nil, true, true, true, "the bus hears"},
{"a holder with no such setting that does not claim it, silent", false, nil, false, true, false, ""},
}
for _, c := range cases {
served, why := loginShellServed("zsh", c.claims, c.setting, c.heard, c.asked)
if served != c.served || (c.saysInTheWhys != "" && !strings.Contains(why, c.saysInTheWhys)) {
t.Errorf("%s: served %v (%q), want %v saying %q", c.name, served, why, c.served, c.saysInTheWhys)
}
}
}
// The root-free verb is the serving controller's alone, answered in its process and never as a command; a
// controller not serving answers an error, which the router reads as no machine free.
func TestRootFreeIsAnsweredOnlyByTheServingController(t *testing.T) {
was := doctorFrom
doctorFrom = nil
t.Cleanup(func() { doctorFrom = was })
if _, err := rootFreeAnswer(context.Background(), "anchor", rootNow); err == nil {
t.Error("a controller not serving judged a machine")
}
if !inProcess["root-free"] {
t.Error("root-free is not answered in the serving process")
}
if _, err := argvFor("root-free", map[string]any{"machines": "anchor"}); err == nil {
t.Error("root-free ran as a command")
}
// The router names its machines as a list (its contract with this verb); one text separated by commas is
// the same; anything else in the list is refused.
for _, given := range []any{[]any{"anchor", "relay"}, "anchor, relay"} {
a, err := readArguments("root-free", map[string]any{"machines": given})
if err != nil || a.given["machines"] != "anchor,relay" && a.given["machines"] != "anchor, relay" {
t.Errorf("root-free given %v read %v (%v)", given, a, err)
}
}
if _, err := readArguments("root-free", map[string]any{"machines": []any{"anchor", 7}}); err == nil {
t.Error("root-free took a number for a machine")
}
if _, err := readArguments("status", map[string]any{"machines": []any{"anchor"}}); err == nil {
t.Error("a verb that takes no list took one")
}
}
// The confirmation review of 2026-10-09: ADR 0266's quiet window (#175) keeps the self-check from raising
// agent-can-become-root while the node-engine's setuid search runs and no complete one judges. It must not make root-free answer free:
// root-free needs a complete, fresh verdict. A verdict still waiting for the search is "not judged" to
// agentConfined, so the machine is not root-free, whatever the quiet says — and the same statement, complete
// and healthy, is the control.
func TestAMachineWaitingForItsFirstSetuidSearchIsNotRootFree(t *testing.T) {
now := rootNow
statement := func(state, reason string) inventory.NodeHealth {
return inventory.NodeHealth{Node: "anchor", Contract: link.RootContract, SaidAt: now, HeardAt: now,
Resources: []inventory.ResourceHealth{{Module: "claude-code", Resource: "agent", Kind: link.KindAccount,
Target: "agents", State: state, Reason: reason, Root: link.RootNever}}}
}
judged := func(h inventory.NodeHealth) rootVerdict {
confined, why := judgedConfined("agents", h, true, now)
return judgeRoot(rootFacts{Machine: "anchor", AgentNamed: true, Confined: confined, ConfinedWhy: why}, now)
}
if v := judged(statement(link.StateHealthy, "")); !v.Free {
t.Fatalf("the control: a complete healthy verdict, fresh: %+v", v)
}
pending := statement(link.StateUnknown, link.ReasonRootPending+": the search runs")
if rootVerdictKind("agents", pending, true, now) != verdictPending {
t.Fatal("the statement is not one the quiet window counts as waiting for the search")
}
if v := judged(pending); v.Free {
t.Errorf("a machine whose setuid search is pending was judged root-free: %+v", v)
}
}
// The confirmation review of 2026-10-09, on #154 beside ADR 0266: D-root keeps ADR 0266's quiet window — nothing
// raised while the one thing unjudged is the setuid search, within searchQuietFor — while the root-free verb
// still answers the machine not free; and D-root's condition has a key of its own, apart from DA's.
func TestRootNotFreeIsQuietWhileTheFirstSearchRunsAndKeyedApartFromDA(t *testing.T) {
entries := []inventory.Entry{{Manifest: catalogue.Manifest{Module: "telegram", RunsAs: "telegram"}, On: []string{"anchor"}}}
r := &rootReader{entries: entries, heard: map[string]map[string]map[string]bool{},
confined: func(context.Context, string, time.Time) (bool, bool, string, error) {
return true, false, "the agent account agents is not judged: the search for setuid programs runs", nil
},
settings: func(context.Context, string, string) ([]catalogue.Layer, error) { return nil, nil },
quiet: func(context.Context, string, time.Time) bool { return true }}
trusted := trustedMachines(entries)
if got := rootObservations(context.Background(), r, trusted, rootNow); len(got) != 0 {
t.Errorf("raised while the search runs: %+v", got)
}
if v := judgeRootFree(context.Background(), r, []string{"anchor"}, rootNow)[0]; v.Free || !v.Quiet {
t.Errorf("root-free while the search runs: %+v", v)
}
// Quiet hides nothing else: the login shell served as well is said.
r.heard = map[string]map[string]map[string]bool{loginShellSeat: {"execute": {"anchor": true}}}
if got := rootObservations(context.Background(), r, trusted, rootNow); len(got) != 1 {
t.Errorf("a second failure was kept quiet: %+v", got)
}
// Past searchQuietFor, said.
r.heard, r.quiet = map[string]map[string]map[string]bool{}, func(context.Context, string, time.Time) bool { return false }
got := rootObservations(context.Background(), r, trusted, rootNow)
if len(got) != 1 {
t.Fatalf("a search past searchQuietFor was not said: %+v", got)
}
// One key per judgement: DA's is machine.<m>.agent-root, this one its own.
da := conditions.Observation{Scope: conditions.ScopeMachine, ID: "anchor", Token: "agent-root", Machine: "anchor"}
if got[0].Key() == da.Key() {
t.Errorf("D-root and DA share the key %s", da.Key())
}
}
+98 -31
View File
@@ -252,6 +252,10 @@ func askEveryResolver(ctx context.Context, resolvers map[string]string, places [
v.wrong[0], andMore(len(v.wrong)-1))
} else {
// Nothing but silence: held for the next run, which raises it if the resolver is still silent.
// Refused on every try too: a few hundred milliseconds of refusals is a resolver restarting as
// well as one that stopped, and the two looks a finding needs are this run and the next
// (novox/hq issue 348). The machine's own node-engine is the fast detector: its names check
// raised the control node's resolver within a minute on 2026-10-09.
o.Confirm = true
o.Summary = fmt.Sprintf("the mesh's resolver on %s does not answer: %d of the %d question(s) about the "+
"machines' names went unanswered, each asked %d times — the first, %s", node, len(v.unanswered), v.asked,
@@ -637,31 +641,8 @@ const (
// discoverHolders asks the bus's discovery who serves what, and answers seat → machine for every
// endpoint a seat's verb is served on.
func discoverHolders(ctx context.Context, conn *nats.Conn) (map[string]map[string]bool, error) {
inbox := conn.NewRespInbox()
sub, err := conn.SubscribeSync(inbox)
if err != nil {
return nil, err
}
defer func() { _ = sub.Unsubscribe() }()
if err := conn.PublishRequest("$SRV.INFO", inbox, nil); err != nil {
return nil, fmt.Errorf("asking the bus who serves what: %w", err)
}
out := map[string]map[string]bool{}
deadline := time.Now().Add(discoveryPatience)
for time.Now().Before(deadline) {
wait, cancel := context.WithTimeout(ctx, discoveryQuiet)
msg, err := sub.NextMsgWithContext(wait)
cancel()
if err != nil {
if ctx.Err() != nil {
return nil, ctx.Err()
}
break
}
var info micro.Info
if json.Unmarshal(msg.Data, &info) != nil {
continue
}
err := discoverServices(ctx, conn, func(info micro.Info) {
for _, e := range info.Endpoints {
seat, node := e.Metadata["seat"], e.Metadata["node"]
if seat == "" {
@@ -680,8 +661,69 @@ func discoverHolders(ctx context.Context, conn *nats.Conn) (map[string]map[strin
out[info.Name] = map[string]bool{}
}
out[info.Name][info.ID] = true
})
return out, err
}
// discoverSeatVerbs asks the bus's discovery the same, one level finer: seat → verb → machine, for every
// seat verb answered (an endpoint's `tool` is its verb). A seat held where a verb is withheld (novox/hq ADR
// 0268) shows the seat and not that verb.
func discoverSeatVerbs(ctx context.Context, conn *nats.Conn) (map[string]map[string]map[string]bool, error) {
out := map[string]map[string]map[string]bool{}
err := discoverServices(ctx, conn, func(info micro.Info) {
for _, e := range info.Endpoints {
seat, verb, node := e.Metadata["seat"], e.Metadata["tool"], e.Metadata["node"]
if seat == "" || verb == "" {
continue
}
if node == "" {
node = info.ID
}
if out[seat] == nil {
out[seat] = map[string]map[string]bool{}
}
if out[seat][verb] == nil {
out[seat][verb] = map[string]bool{}
}
out[seat][verb][node] = true
}
})
return out, err
}
// discoverServices asks the bus's discovery once and hands every service's answer to visit, waiting
// discoveryQuiet after the last and discoveryPatience at the most.
func discoverServices(ctx context.Context, conn *nats.Conn, visit func(micro.Info)) error {
if conn == nil {
return errors.New("the controller holds no connection to the bus")
}
return out, nil
inbox := conn.NewRespInbox()
sub, err := conn.SubscribeSync(inbox)
if err != nil {
return err
}
defer func() { _ = sub.Unsubscribe() }()
if err := conn.PublishRequest("$SRV.INFO", inbox, nil); err != nil {
return fmt.Errorf("asking the bus who serves what: %w", err)
}
deadline := time.Now().Add(discoveryPatience)
for time.Now().Before(deadline) {
wait, cancel := context.WithTimeout(ctx, discoveryQuiet)
msg, err := sub.NextMsgWithContext(wait)
cancel()
if err != nil {
if ctx.Err() != nil {
return ctx.Err()
}
break
}
var info micro.Info
if json.Unmarshal(msg.Data, &info) != nil {
continue
}
visit(info)
}
return nil
}
// probeArchives is D4: every archive the mesh keeps is held by its manifest in the artifact store.
@@ -875,6 +917,16 @@ func streamDiffers(want broker.Stream, have nats.StreamConfig) string {
if perSubject != 0 && have.MaxMsgsPerSubject != perSubject {
differs = append(differs, fmt.Sprintf("keeps %d per subject, defined %d", have.MaxMsgsPerSubject, perSubject))
}
if want.MaxBytes > 0 && have.MaxBytes != want.MaxBytes {
differs = append(differs, fmt.Sprintf("holds up to %d bytes, defined %d", have.MaxBytes, want.MaxBytes))
}
if want.DuplicatesSeconds > 0 && have.Duplicates != time.Duration(want.DuplicatesSeconds)*time.Second {
differs = append(differs, fmt.Sprintf("keeps one of a message id for %s, defined %s", have.Duplicates,
time.Duration(want.DuplicatesSeconds)*time.Second))
}
if want.DiscardNew && have.Discard != nats.DiscardNew {
differs = append(differs, "drops what it holds when full, defined to refuse what comes next")
}
return strings.Join(differs, "; ")
}
@@ -1030,12 +1082,7 @@ func probeCoreBuilds(ctx context.Context, d *doctor) ([]conditions.Observation,
return nil, err
}
hostVersions := deliveredVersions(shelf[hostModule])
rolling := map[string]bool{}
for _, p := range plans {
for m := range p.Modules {
rolling[m] = true
}
}
rolling := rollingModules(plans)
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, err
@@ -1093,6 +1140,26 @@ func probeCoreBuilds(ctx context.Context, d *doctor) ([]conditions.Observation,
return out, nil
}
// rollingModules is every module an open plan is rolling out: those it keeps a record of, and those its
// tiers name. **A release keeps no record per module** — its walk is per machine, its modules only in its
// tier — so reading the records alone, D10 said "no plan is rolling them out" about the node-engine while
// a release walked it, and the gate on that release's first machine waited on what its own send causes
// (novox/hq issue 348). Pure.
func rollingModules(plans []inventory.Plan) map[string]bool {
rolling := map[string]bool{}
for _, p := range plans {
for m := range p.Modules {
rolling[m] = true
}
for _, tier := range p.Tiers {
for _, m := range tier {
rolling[m] = true
}
}
}
return rolling
}
// deliveredVersions are the versions a module's registered build is delivered as: the last element
// of every resource path under a `versions/` directory, which registration filled from the artifact's
// digest (catalogue `${version}`). The node-engine names itself by that directory.
+793
View File
@@ -0,0 +1,793 @@
package main
// A trusted setting proposed through a verb and set only on the operator's warrant (novox/hq ADR 0277).
//
// mesh-controller settings propose <module> <values.json | {…}> [--node <node>] [--replace]
// mesh-controller settings propose <module> --clear [--node <node>]
// mesh-controller settings proposals [<id>]
//
// Whoever the bus admits may PROPOSE a settings layer — the keys issue 339 made the terminal's (`places`,
// `accesses`, what a provider serves, what a trusted file asks for) among them. A proposal changes nothing: it is
// kept in the controller's own asks (broker.AskedBucket, written by the controller alone) and asked of the
// operator through the operator channel as an ask whose two answers, Approve and Decline, are both at the level
// approve, so only a channel that proves who answered (Telegram, today) carries either. The serving controller
// acts on the warrant as on any other of its asks (asker.Decided): once, for the ask it holds, the option it
// offered, and only when the act about to be performed — the module, the machine, the digest of the exact values,
// the layer they replace, whether a removal is meant — is the one the option bound when the operator was shown
// it. Then it sets the layer as `settings set` at the terminal does, with the same judgement, keeps who approved
// it beside the layer (`settings` says it back), and records the warrant in the hand-act log on the bus, where a
// person's decisions are read; the router edits the ask on every channel to its outcome. No seat event of its
// own: nothing consumes one, and the installer's first user list in the node-engine's repository names every
// controller event, so one would cost a node-engine change for a fact without a reader. The push afterwards is a
// separate act, as it is for a layer set at the terminal.
//
// **What the operator reads** is the module, the machine, each key with its exact new value and, for a changed
// key, the value it replaces. A value that may not leave the mesh (an address, a path, a secret's shape: the
// router's content rule, outward.Check) is shown with that part replaced by ‹address›, ‹path› or ‹withheld›, the
// ask says so, and the whole is read with `settings proposals <id>` — whose fingerprint must be the one on the
// phone. Fail closed: a proposal nothing can carry to the operator is refused at once, in words, and never left
// waiting for an answer that cannot come.
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"regexp"
"sort"
"strconv"
"strings"
"time"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/outward"
)
// settingsProposal is one proposed change to a module's settings layer, as the controller's ask keeps it
// (asked.Proposal). Every field the ask is composed from is here, so the serving controller composes the same ask
// the proposer did, and the warrant's digest holds to it.
type settingsProposal struct {
Module string `json:"module"`
// Node is the machine, or empty for the whole mesh.
Node string `json:"node,omitempty"`
// Values is the layer proposed, whole; Clear says the layer is removed instead.
Values map[string]any `json:"values,omitempty"`
Clear bool `json:"clear,omitempty"`
// Replace says the keys the layer had and Values do not name are meant to go (novox/hq ADR 0217).
Replace bool `json:"replace,omitempty"`
// Before is the layer as it stood when the proposal was made, and HadLayer whether there was one: what the
// operator was shown the change against, and what must still stand when the warrant is acted on.
Before map[string]any `json:"before,omitempty"`
HadLayer bool `json:"had-layer"`
// From is who proposed it, as the bus named the caller; At is when.
From string `json:"from"`
At time.Time `json:"at"`
// Digest is the digest of Values (layerDigest), BeforeDigest of Before.
Digest string `json:"digest"`
BeforeDigest string `json:"before-digest"`
}
// proposalVerb is the verb a proposal's answers bind: the controller's own settings, performed by itself.
const proposalVerb = askerName + ".settings"
// The two answers, both at the level approve.
const (
answerApprove = "approve"
answerDecline = "decline"
)
// layerDigest is the digest a proposal binds: SHA-256 over the layer's canonical JSON (Go sorts a map's keys), an
// absent layer and an empty one alike.
func layerDigest(values map[string]any) string {
if values == nil {
values = map[string]any{}
}
raw, _ := json.Marshal(values)
sum := sha256.Sum256(raw)
return "sha256:" + hex.EncodeToString(sum[:])
}
// fingerprint is a digest as the operator is shown it: its first 24 hexadecimal digits in groups of four, so no
// word of it is long enough for the router to take for a secret.
func fingerprint(digest string) string {
hexed := strings.TrimPrefix(digest, "sha256:")
if len(hexed) < 24 {
return hexed
}
var groups []string
for i := 0; i < 24; i += 4 {
groups = append(groups, hexed[i:i+4])
}
return strings.Join(groups, " ")
}
// where is the layer in words: "shanks" or "the whole mesh".
func (p settingsProposal) where() string {
if p.Node == "" {
return "the whole mesh"
}
return p.Node
}
// about is what the ask is about, a key without spaces: the module's layer on the machine, or on the mesh.
func (p settingsProposal) about() string {
if p.Node == "" {
return "settings." + p.Module + ".mesh"
}
return "settings." + p.Module + "." + p.Node
}
// actions are the proposal's two answers as the controller keeps them (asked.Actions): each binds the exact act
// through boundAct — the verb, the machine, the level and every argument, the values' digest among them.
func (p settingsProposal) actions(id string) []conditions.Action {
args := func(answer string) map[string]string {
return map[string]string{"proposal": id, "answer": answer, "module": p.Module, "node": p.Node,
"values": p.Digest, "before": p.BeforeDigest, "had-layer": strconv.FormatBool(p.HadLayer),
"replace": strconv.FormatBool(p.Replace), "clear": strconv.FormatBool(p.Clear), "from": p.From,
"at": p.At.UTC().Format(time.RFC3339Nano)}
}
return []conditions.Action{
{Label: "Approve", Verb: proposalVerb, Machine: p.Node, Level: conditions.LevelApprove, Arguments: args(answerApprove)},
{Label: "Decline", Verb: proposalVerb, Machine: p.Node, Level: conditions.LevelApprove, Arguments: args(answerDecline)},
}
}
// ask is the proposal's ask, composed from it alone, as the router is sent it: the headline, the explanation
// with the change shown under the content rule, and the two options with their bound acts.
func (p settingsProposal) ask(id string, machines []string) (asks.Ask, map[string]int) {
verb := "Set"
if p.Clear {
verb = "Clear"
}
headline := fmt.Sprintf("%s %s on %s?", verb, p.Module, p.where())
if len([]rune(headline)) > asks.HeadlineLength {
headline = fmt.Sprintf("%s settings on %s?", verb, p.where())
}
if len([]rune(headline)) > asks.HeadlineLength {
headline = verb + " settings?"
}
shown, whole := p.change(machines)
var b strings.Builder
if p.Clear {
fmt.Fprintf(&b, "Clear the settings of %s on %s, back to what the module says?\n\n", p.Module, p.where())
} else {
fmt.Fprintf(&b, "Set the settings of %s on %s to these values?\n\n", p.Module, p.where())
}
fmt.Fprintf(&b, "Proposed by %s, at %s. ", sayable(p.From, machines), p.At.Local().Format("15:04 on 2 Jan"))
switch {
case p.Clear && p.HadLayer:
b.WriteString("The layer it removes:\n\n")
case p.Clear:
b.WriteString("There is no layer to remove; approving changes nothing.")
case !p.HadLayer:
b.WriteString("There is no layer yet; this is the whole of it:\n\n")
default:
b.WriteString("The layer is replaced whole; what changes against it:\n\n")
}
b.WriteString(shown)
fmt.Fprintf(&b, "\n\nFingerprint %s.", fingerprint(p.Digest))
if !whole {
b.WriteString(" Parts shown as ‹address›, ‹path› or ‹withheld› may not leave the mesh: read it whole, with " +
"this fingerprint, through the mesh MCP server (mesh-controller.settings, proposal " + id + ") or with " +
"mesh-cli settings proposals " + id + ".")
}
if p.Clear {
b.WriteString(" Approved, the layer is removed at once and the machine takes it at its next push.")
} else {
b.WriteString(" Approved, the layer is set at once and the machine takes it at its next push.")
}
q := asks.Ask{ID: id, Headline: headline, Explanation: b.String(), Who: asks.Operator,
Expires: p.At.Add(askApproveFor), OnExpiry: "the proposal is discarded; nothing changes",
About: p.about()}
options := map[string]int{}
for i, act := range p.actions(id) {
binds, _ := asks.ActDigest(boundAct(act))
oid := optionID(act.Label)
options[oid] = i
does := "the settings are set; nothing is pushed yet"
if p.Clear {
does = "the layer is removed; nothing is pushed yet"
}
if act.Arguments["answer"] == answerDecline {
does = "nothing changes; the proposal is discarded"
}
q.Options = append(q.Options, asks.Option{ID: oid, Label: act.Label, Does: does, Level: asks.Level(act.Level),
Binds: binds})
}
return q, options
}
// shownMost is the most of a change the phone is shown, in bytes; the rest is read whole with `settings proposals`.
const shownMost = 1400
// change is what changes, line by line, each value shown under the content rule, and whether every value was
// shown whole: "+ key: value" added, "~ key: value (was: old)" changed, "- key (was: old)" removed, and the
// count of keys unchanged.
func (p settingsProposal) change(machines []string) (string, bool) {
whole := true
say := func(v any) string {
s, w := sayableValue(v, machines)
whole = whole && w
return s
}
var lines []string
if p.Clear {
was := leaves(p.Before, "")
for _, k := range layerKeys(was) {
lines = append(lines, fmt.Sprintf("- %s: %s", k, say(was[k])))
}
} else {
added, changed, removed := settingsChange(p.Before, p.Values)
was, now := leaves(p.Before, ""), leaves(p.Values, "")
for _, k := range added {
lines = append(lines, fmt.Sprintf("+ %s: %s", k, say(now[k])))
}
for _, k := range changed {
lines = append(lines, fmt.Sprintf("~ %s: %s (was: %s)", k, say(now[k]), say(was[k])))
}
for _, k := range removed {
lines = append(lines, fmt.Sprintf("- %s (was: %s)", k, say(was[k])))
}
unchanged := len(now) - len(added) - len(changed)
switch {
case len(lines) == 0 && unchanged > 0:
lines = append(lines, fmt.Sprintf("= nothing changes: the %d key(s) are as they stand", unchanged))
case unchanged > 0:
lines = append(lines, fmt.Sprintf("= %d key(s) unchanged", unchanged))
}
}
out := strings.Join(lines, "\n")
if len(out) > shownMost {
cut := shownMost
for cut > 0 && out[cut] != '\n' {
cut--
}
out = out[:cut] + fmt.Sprintf("\n… NOT SHOWN IN FULL here: %d more bytes", len(strings.Join(lines, "\n"))-cut)
whole = false
}
return out, whole
}
func layerKeys(m map[string]any) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
// The shapes a value is shown without, in place: an address with what follows it up to a separator, and a
// path. Replaced in place rather than word by word, so "recalbox=smb://host/share@/mnt/recalbox" is shown as
// "recalbox=‹address›@‹path›" and keeps its shape.
var (
shownURL = regexp.MustCompile(`(?i)\b[a-z][a-z0-9+.-]*://[^\s@"',;)\]}]*`)
shownIPv4 = regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}(:\d+)?\b`)
shownEmail = regexp.MustCompile(`[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)*\.[A-Za-z]{2,}`)
shownPath = regexp.MustCompile(`(^|[\s=@,;:"'(\[{])((?:~|\.{1,2})?/[^\s"',;:)\]}]*)`)
)
// Markers for what is not shown.
const (
markAddress = "‹address›"
markPath = "‹path›"
markWithheld = "‹withheld›"
)
// sayableValue is a value as the phone is shown it, and whether it was shown whole. A string is shown bare; any
// other value as JSON.
func sayableValue(v any, machines []string) (string, bool) {
text, ok := v.(string)
if !ok {
raw, err := json.Marshal(v)
if err != nil {
return markWithheld, false
}
text = string(raw)
}
if text == "" {
return `""`, true
}
out := sayable(text, machines)
return out, out == text
}
// sayable is a text with what may not leave the mesh replaced in place by a marker; what the markers cannot make
// pass is withheld whole.
func sayable(text string, machines []string) string {
if _, ok := outward.Check(text, machines...); ok {
return text
}
out := shownURL.ReplaceAllString(text, markAddress)
out = shownEmail.ReplaceAllString(out, markAddress)
out = shownIPv4.ReplaceAllString(out, markAddress)
out = shownPath.ReplaceAllString(out, "${1}"+markPath)
// Then word by word, as the router reads them: a host name, a path the shapes above missed, a secret's shape.
words := strings.FieldsFunc(out, func(r rune) bool {
return r == ' ' || r == '\t' || r == '\n' || strings.ContainsRune("\"'`()[]{}<>,;|", r)
})
for _, w := range words {
if refusal, ok := outward.Check(w, machines...); !ok {
mark := markWithheld
switch refusal.Class {
case "address":
mark = markAddress
case "path":
mark = markPath
}
out = strings.ReplaceAll(out, w, mark)
}
}
if _, ok := outward.Check(out, machines...); ok {
return out
}
out = strings.ReplaceAll(outward.Scrub(out, markWithheld, machines...), "(withheld)", markWithheld)
if _, ok := outward.Check(out, machines...); ok {
return out
}
return markWithheld
}
// ---- proposing ---------------------------------------------------------------------------------------
// proposer is the propose command's reaches, given so a test needs no store and no bus.
type proposer struct {
// layer reads a module's layer as it stands.
layer func(ctx context.Context, node, module string) (map[string]any, bool, error)
// judge judges the layer as SetSettings would, keeping nothing.
judge func(ctx context.Context, node, module string, values map[string]any) error
// machines are the mesh's machine names: allowed in the ask's words.
machines func(ctx context.Context) ([]string, error)
store askedStore
publish func(ctx context.Context, subject string, body []byte, id string) error
// routerHere and grantHeld are the asker's own judgements of whether an ask can be carried; nil is yes.
routerHere func(ctx context.Context) (bool, error)
grantHeld func(ctx context.Context) (bool, string, error)
// routerRecord reads the router's record of an ask: its state, or "" for none.
routerRecord func(ctx context.Context, id string) (string, error)
now func() time.Time
caller string
// waitFor and waitEvery bound how long propose waits for the router's word on the new ask.
waitFor time.Duration
waitEvery time.Duration
}
// proposeInput is what is proposed.
type proposeInput struct {
module string
node string
values map[string]any
clear bool
replace bool
}
// atTheTerminalInstead names the other way, said whenever a proposal is refused for want of a channel.
func atTheTerminalInstead(in proposeInput) string {
if in.clear {
return "the operator may clear it at the controller's terminal instead: mesh-cli settings clear " + in.module + nodeFlag(in.node)
}
return "the operator may set it at the controller's terminal instead: mesh-cli settings set " + in.module + " '{…}'" + nodeFlag(in.node)
}
// propose keeps a proposal in the controller's asks and asks the operator; it answers the words said to the
// caller. Nothing is set here.
func (pr proposer) propose(ctx context.Context, in proposeInput) (string, error) {
refuse := func(format string, args ...any) (string, error) {
return "", fmt.Errorf(format+". Nothing was proposed", args...)
}
if in.module == "" {
return refuse("a proposal names a module")
}
if !in.clear && in.values == nil {
return refuse("a proposal gives the values, or says --clear")
}
now := pr.now()
before, had, err := pr.layer(ctx, in.node, in.module)
if err != nil {
return "", err
}
p := settingsProposal{Module: in.module, Node: in.node, Values: in.values, Clear: in.clear, Replace: in.replace,
Before: before, HadLayer: had, From: pr.caller, At: now, BeforeDigest: layerDigest(before)}
if in.clear {
// A clear binds the layer it removes: its digest is the fingerprint the operator reads.
p.Values, p.Digest = nil, layerDigest(before)
} else {
// Judged now, as SetSettings judges, so the operator is never asked about a layer the mesh would refuse —
// and judged again when the warrant is acted on.
if err := pr.judge(ctx, in.node, in.module, in.values); err != nil {
return refuse("%v", err)
}
_, _, removed := settingsChange(before, in.values)
if len(removed) > 0 && !in.replace {
return refuse("%s on %s: this layer would no longer set %s. A layer is replaced whole; read it with "+
"`settings show %s%s` and include what should stay, or add --replace if the removal is meant "+
"(novox/hq ADR 0217)", in.module, p.where(), strings.Join(removed, ", "), in.module, nodeFlag(in.node))
}
p.Digest = layerDigest(in.values)
}
var machines []string
if pr.machines != nil {
if machines, err = pr.machines(ctx); err != nil {
return "", err
}
}
id := newProposalID()
q, options := p.ask(id, machines)
if err := q.Check(now); err != nil {
return refuse("%v", err)
}
words := []string{q.Headline, q.Explanation, q.OnExpiry}
for _, o := range q.Options {
words = append(words, o.Label, o.Does)
}
if refusal, ok := outward.Check(strings.Join(words, "\n"), machines...); !ok {
return refuse("the ask's words would carry %s, which may not leave the mesh, and the change could not be "+
"shown without it; %s", refusal, atTheTerminalInstead(in))
}
// Fail closed, before anything is kept: no router, no grant, no channel means no ask.
if pr.routerHere != nil {
here, err := pr.routerHere(ctx)
if err != nil {
return "", err
}
if !here {
return refuse("no router takes the controller's asks (no module holding the operator channel is assigned), "+
"so the operator cannot be asked; %s", atTheTerminalInstead(in))
}
}
if pr.grantHeld != nil {
held, why, err := pr.grantHeld(ctx)
if err != nil {
return "", err
}
if !held {
return refuse("the operator cannot be asked yet: %s; %s", why, atTheTerminalInstead(in))
}
}
all, err := pr.store.All(ctx)
if err != nil {
return "", err
}
open := 0
for _, r := range all {
if r.State != askOpen || !now.Before(r.Ask.Expires) {
continue
}
if r.Proposal != nil && r.Proposal.Module == p.Module && r.Proposal.Node == p.Node && r.Proposal.Digest == p.Digest &&
r.Proposal.Clear == p.Clear {
return refuse("the same change is already proposed as %s and waits for the operator's answer until %s; "+
"`settings proposals` lists it", r.ID, r.Ask.Expires.Local().Format("15:04"))
}
open++
}
if open >= askMostOpen {
return refuse("%d questions already wait for the operator's answer, and the operator is asked at most %d at "+
"once; `settings proposals` lists the proposals among them", open, askMostOpen)
}
// Kept before it is published, as the asker keeps every ask, so a warrant always finds it.
if err := pr.store.Create(ctx, asked{ID: id, Condition: q.About, Ask: q, Actions: p.actions(id), Options: options,
State: askOpen, Opened: now, Proposal: &p}); err != nil {
return "", fmt.Errorf("the proposal could not be kept in the controller's asks, so the operator was not asked: %w", err)
}
body, err := json.Marshal(q)
if err != nil {
return "", err
}
if err := pr.publish(ctx, asks.AskSubject(askerName), body, "ask."+id); err != nil {
_, _ = pr.store.Change(ctx, id, func(x *asked) bool {
if x.State != askOpen || x.Acted != "" {
return false
}
x.State, x.Ended, x.Acted = askUnsent, pr.now(), "nothing: it could not be published: "+err.Error()
return true
})
return "", fmt.Errorf("the operator could not be asked (%v); the proposal %s is kept and will never become "+
"active. Propose it again, or %s", err, id, atTheTerminalInstead(in))
}
// The router's word, before answering: it refuses at once an ask no channel can carry (the serving controller
// hears that and ends the ask here), and records one it took.
taken := "the router has not said yet whether it took the ask; `settings proposals` shows where it stands"
for deadline := time.Now().Add(pr.waitFor); time.Now().Before(deadline); {
if r, err := pr.store.Get(ctx, id); err == nil && r != nil && r.State != askOpen {
why := r.Acted
if r.Warrant != nil && r.Warrant.Words != "" {
why = r.Warrant.Words
}
return "", fmt.Errorf("the router did not ask the operator: the ask %s %s (%s). Nothing changes; %s",
id, r.State, why, atTheTerminalInstead(in))
}
if pr.routerRecord != nil {
if state, err := pr.routerRecord(ctx, id); err == nil && state != "" {
taken = "the router took the ask and shows it on the channels that can carry it"
break
}
}
time.Sleep(pr.waitEvery)
}
var b strings.Builder
fmt.Fprintf(&b, "proposal %s: %s\n", id, q.Headline)
fmt.Fprintf(&b, " %s\n", taken)
fmt.Fprintf(&b, " the operator is asked on a channel that proves who answers, and reads the change with fingerprint %s\n",
fingerprint(p.Digest))
fmt.Fprintf(&b, " until %s nothing changes: the layer is set only on Approve, and discarded on Decline or at expiry\n",
q.Expires.Local().Format("2006-01-02 15:04"))
fmt.Fprintf(&b, " `settings proposals %s` shows it whole; once approved, `push %s` sends it", id, pushWord(p.Node))
return b.String(), nil
}
func pushWord(node string) string {
if node == "" {
return "--behind"
}
return node
}
func newProposalID() string {
return "s" + strings.TrimPrefix(newAskID(), "c")
}
// How long propose waits for the router's word on a new ask, and how often it looks.
const (
routerAnswersWithin = 8 * time.Second
routerAnswersEvery = 250 * time.Millisecond
)
// proposeCommand is `settings propose`, on this controller's stores and bus.
func proposeCommand(ctx context.Context, module, node, valuesArg string, clear, replace bool) error {
var values map[string]any
if !clear {
if valuesArg == "" {
return errors.New("settings propose <module> <settings.json | {…}> [--node <node>] [--replace], or settings propose <module> --clear [--node <node>]")
}
var raw []byte
var err error
if strings.HasPrefix(strings.TrimSpace(valuesArg), "{") {
raw = []byte(valuesArg)
} else if raw, err = os.ReadFile(valuesArg); err != nil {
return err
}
if err := json.Unmarshal(raw, &values); err != nil {
return fmt.Errorf("%s is not a settings file: %w", valuesArg, err)
}
} else if valuesArg != "" {
return errors.New("settings propose: --clear takes no values")
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
js, err := aBus()
if err != nil {
return err
}
defer js.Close()
conn := js.Conn()
pr := proposer{
layer: open.inventory.Layer,
judge: open.inventory.JudgeSettings,
machines: func(ctx context.Context) ([]string, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, err
}
var names []string
for _, n := range nodes {
names = append(names, n.Name)
}
return names, nil
},
store: busAsked{conn: conn},
publish: func(ctx context.Context, subject string, body []byte, id string) error {
_, err := js.Context().Publish(subject, body, nats.MsgId(id), nats.Context(ctx))
return err
},
routerHere: routerHereIn(open.inventory),
grantHeld: grantHeldIn(open),
routerRecord: func(ctx context.Context, id string) (string, error) {
bucket, err := asksRecords(ctx, open.inventory)
if err != nil || bucket == "" {
return "", err
}
state, _, err := readRouterRecord(ctx, conn, bucket, askerName, id)
return state, err
},
now: time.Now,
caller: link.Caller(),
waitFor: routerAnswersWithin, waitEvery: routerAnswersEvery,
}
words, err := pr.propose(ctx, proposeInput{module: module, node: node, values: values, clear: clear, replace: replace})
if err != nil {
return err
}
fmt.Println(words)
return nil
}
// ---- listing ---------------------------------------------------------------------------------------
// proposalsCommand is `settings proposals [<id>]`: every proposal, newest first, and where each stands; with an
// id, the proposal whole — its values, the layer it was shown against, and its digest.
func proposalsCommand(ctx context.Context, id string) error {
return onTheBus(func(conn *nats.Conn) error {
all, err := busAsked{conn: conn}.All(ctx)
if err != nil {
return err
}
var proposals []asked
for _, r := range all {
if r.Proposal != nil {
proposals = append(proposals, r)
}
}
sort.Slice(proposals, func(i, j int) bool { return proposals[i].Opened.After(proposals[j].Opened) })
if id != "" {
for _, r := range proposals {
if r.ID == id {
fmt.Print(describeProposal(r, time.Now()))
return nil
}
}
return fmt.Errorf("no proposal %s is kept", id)
}
if len(proposals) == 0 {
fmt.Println("no settings have been proposed")
return nil
}
for _, r := range proposals {
fmt.Print(proposalLine(r, time.Now()))
}
return nil
})
}
// proposalState is where a proposal stands, in a word or two.
func proposalState(r asked, now time.Time) string {
switch {
case r.State == askOpen && now.Before(r.Ask.Expires):
return "waiting for the operator until " + r.Ask.Expires.Local().Format("2006-01-02 15:04")
case r.State == askOpen:
return "expired unanswered; discarded"
case r.Acted != "":
return r.State + ": " + r.Acted
}
return r.State
}
func proposalLine(r asked, now time.Time) string {
p := *r.Proposal
what := "set"
if p.Clear {
what = "clear"
}
keys := strings.Join(layerKeys(p.Values), ", ")
if p.Clear {
keys = "the whole layer"
}
return fmt.Sprintf("%s %s %s on %s (%s)\n by %s at %s; fingerprint %s\n %s\n", r.ID, what, p.Module, p.where(),
keys, p.From, p.At.Local().Format("2006-01-02 15:04"), fingerprint(p.Digest), proposalState(r, now))
}
func describeProposal(r asked, now time.Time) string {
p := *r.Proposal
var b strings.Builder
b.WriteString(proposalLine(r, now))
fmt.Fprintf(&b, " digest %s; the layer it was shown against %s\n", p.Digest, p.BeforeDigest)
shownValues, _ := json.MarshalIndent(p.Values, " ", " ")
if p.Clear {
fmt.Fprintf(&b, " clears the layer\n")
} else {
fmt.Fprintf(&b, " values:\n %s\n", shownValues)
}
if p.HadLayer {
shownBefore, _ := json.MarshalIndent(p.Before, " ", " ")
fmt.Fprintf(&b, " the layer as it stood:\n %s\n", shownBefore)
} else {
b.WriteString(" there was no layer\n")
}
if r.Warrant != nil && r.Warrant.By != nil {
fmt.Fprintf(&b, " answered: %s, through %s, at %s\n", r.Warrant.Says(), viaWords(*r.Warrant),
r.Warrant.At.Local().Format("2006-01-02 15:04"))
}
return b.String()
}
// ---- acting on the warrant --------------------------------------------------------------------------
// setOnWarrant performs an approved proposal: the layer set or cleared as `settings set` and `settings clear` at the
// terminal do, with who approved it kept beside the layer. It answers the words of what changed.
type setOnWarrant func(ctx context.Context, p settingsProposal, askID, setBy string) (string, error)
// decideProposal is what the asker does with a warrant for a proposal (asker.Decided): nothing on Decline, and on
// Approve the act only when it is the one the option bound — the digest of the exact values kept here is the one
// in the act, and so the one the ask's option bound and the warrant's ask digest covers.
func (a *asker) decideProposal(ctx context.Context, r asked, act conditions.Action, w asks.Warrant) (string, error) {
p := *r.Proposal
if act.Arguments["answer"] != answerApprove {
return "nothing: the operator declined; the layer is unchanged", nil
}
if a.setLayer == nil {
return "", errors.New("this controller cannot set a layer on a warrant")
}
// The record's own proposal against the act the option bound: the act is composed again from the record —
// its module, machine, values (digested again), the layer they replace, whether a removal is meant, a clear,
// who proposed it and when — and must digest to what the option bound when the operator was shown it. A
// record changed after the ask, in any field, is refused and nothing is set.
again := p
again.Digest, again.BeforeDigest = layerDigest(p.Values), layerDigest(p.Before)
if p.Clear {
again.Digest = layerDigest(p.Before)
}
recomposed := again.actions(r.ID)
chosen := -1
for i, candidate := range recomposed {
if candidate.Arguments["answer"] == act.Arguments["answer"] {
chosen = i
}
}
option, offered := r.Ask.Option(w.Option)
if chosen < 0 || !offered {
return "", fmt.Errorf("the proposal %s offers no answer %q: nothing is set", r.ID, act.Arguments["answer"])
}
if err := option.Performs(boundAct(recomposed[chosen])); err != nil {
return "", fmt.Errorf("the proposal kept for %s is not the one the operator was shown: %v", r.ID, err)
}
setBy := fmt.Sprintf("approved by %s via %s at %s (ask %s, proposed by %s)", byWords(w), viaWords(w),
w.At.Local().Format("2006-01-02 15:04"), r.ID, p.From)
return a.setLayer(ctx, p, r.ID, setBy)
}
// setLayerIn is setOnWarrant on this controller's stores: the layer must still be the one the operator was shown
// the change against (to-be 46 §10, step 7), then it is set with the same judgement as at the terminal.
func setLayerIn(open *stores) setOnWarrant {
return func(ctx context.Context, p settingsProposal, askID, setBy string) (string, error) {
inv := open.inventory
before, had, err := inv.Layer(ctx, p.Node, p.Module)
if err != nil {
return "", err
}
if layerDigest(before) != p.BeforeDigest || had != p.HadLayer {
return "", fmt.Errorf("the layer of %s on %s changed since the operator was shown the change: it is not set; "+
"propose it again", p.Module, p.where())
}
var changed string
if p.Clear {
if err := inv.ClearSettingsBy(ctx, p.Node, p.Module, setBy); err != nil {
return "", err
}
changed = "the layer is removed"
} else {
added, altered, removed := settingsChange(before, p.Values)
if len(removed) > 0 && !p.Replace {
return "", fmt.Errorf("the layer would no longer set %s, and the removal was not meant: nothing is set",
strings.Join(removed, ", "))
}
if err := inv.SetSettingsBy(ctx, p.Node, p.Module, p.Values, setBy); err != nil {
return "", err
}
var parts []string
for _, k := range added {
parts = append(parts, "+ "+k)
}
for _, k := range altered {
parts = append(parts, "~ "+k)
}
for _, k := range removed {
parts = append(parts, "- "+k)
}
changed = strings.Join(parts, ", ")
if changed == "" {
changed = "nothing changed"
}
}
return changed + " (ask " + askID + ")", nil
}
}
+746
View File
@@ -0,0 +1,746 @@
package main
import (
"context"
"encoding/json"
"errors"
"io"
"os"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/outward"
)
// novox/hq ADR 0277: a trusted setting is proposed through a verb by anyone the bus admits, asked of the
// operator at the level approve with the exact change, and set only on the operator's warrant — once, for the
// exact values the option bound; declined, expired or refused, it is discarded; nothing is asked when nothing
// can carry the ask.
// aProposer is the propose path with its reaches faked: a layer as it stands, a judge that records what it
// judged, a memory store, and what was published.
type proposerRig struct {
pr proposer
store memAskedStore
sent []published
judged []map[string]any
before map[string]any
had bool
refusedBy string
now time.Time
}
func newProposerRig(t *testing.T) *proposerRig {
r := &proposerRig{store: memAskedStore{}, now: time.Date(2026, 10, 10, 14, 5, 0, 0, time.UTC)}
r.pr = proposer{
layer: func(_ context.Context, node, module string) (map[string]any, bool, error) {
return r.before, r.had, nil
},
judge: func(_ context.Context, node, module string, values map[string]any) error {
r.judged = append(r.judged, values)
if r.refusedBy != "" {
return errors.New(r.refusedBy)
}
return nil
},
machines: func(context.Context) ([]string, error) { return []string{"anchor", "laptop", "shanks"}, nil },
store: r.store,
publish: func(_ context.Context, subject string, body []byte, id string) error {
r.sent = append(r.sent, published{subject, id, body})
return nil
},
now: func() time.Time { return r.now },
caller: "g14/claude-code, through the mesh-controller seat",
waitFor: time.Millisecond,
waitEvery: time.Millisecond,
}
return r
}
func (r *proposerRig) askSent(t *testing.T) asks.Ask {
t.Helper()
if len(r.sent) != 1 || r.sent[0].subject != asks.AskSubject("mesh-controller") {
t.Fatalf("published %+v", r.sent)
}
var q asks.Ask
if err := json.Unmarshal(r.sent[0].body, &q); err != nil {
t.Fatal(err)
}
return q
}
func (r *proposerRig) theProposal(t *testing.T) asked {
t.Helper()
for _, a := range r.store {
if a.Proposal != nil {
return a
}
}
t.Fatal("no proposal is kept")
return asked{}
}
var mountsSources = map[string]any{"sources": "recalbox=smb://nas.lan/recalbox@/mnt/recalbox:ro", "shares": "library=/mnt/library"}
// The ask: at the level approve on both answers, each binding the proposal's act, with every key and its exact new
// value as far as the content rule lets it leave the mesh, the layer it was shown against, and nothing set.
func TestAProposalAsksAtTheLevelApproveWithTheExactChange(t *testing.T) {
r := newProposerRig(t)
r.before, r.had = map[string]any{"shares": "none", "old": "x"}, true
words, err := r.pr.propose(context.Background(), proposeInput{module: "mounts", node: "shanks", values: mountsSources, replace: true})
if err != nil {
t.Fatal(err)
}
q := r.askSent(t)
if err := q.Check(r.now); err != nil {
t.Fatalf("the ask is refused: %v", err)
}
if q.Headline != "Set mounts on shanks?" || q.About != "settings.mounts.shanks" || q.Who != asks.Operator ||
!q.Expires.Equal(r.now.Add(askApproveFor)) {
t.Errorf("the ask: %+v", q)
}
if len(q.Options) != 2 {
t.Fatalf("options %+v", q.Options)
}
for _, o := range q.Options {
if o.Level != asks.Approve || !strings.HasPrefix(o.Binds, "sha256:") {
t.Errorf("the option %s is %s and binds %q", o.ID, o.Level, o.Binds)
}
}
if q.Options[0].Binds == q.Options[1].Binds {
t.Error("Approve and Decline bind the same act")
}
for _, line := range []string{
"Set the settings of mounts on shanks to these values?",
"Proposed by g14/claude-code, through the mesh-controller seat, at " + r.now.Local().Format("15:04 on 2 Jan") + ".",
"+ sources: recalbox=‹address›@‹path›:ro",
"~ shares: library=‹path› (was: none)",
"- old (was: x)",
"Fingerprint " + fingerprint(layerDigest(mountsSources)) + ".",
"read it whole, with this fingerprint",
} {
if !strings.Contains(q.Explanation, line) {
t.Errorf("the explanation lacks %q:\n%s", line, q.Explanation)
}
}
for _, leak := range []string{"nas.lan", "/mnt/recalbox", "/mnt/library", "smb://"} {
if strings.Contains(q.Explanation, leak) {
t.Errorf("the explanation carries %q, which may not leave the mesh", leak)
}
}
all := []string{q.Headline, q.Explanation, q.OnExpiry}
for _, o := range q.Options {
all = append(all, o.Label, o.Does)
}
if refusal, ok := outward.Check(strings.Join(all, "\n"), "anchor", "laptop", "shanks"); !ok {
t.Errorf("the router would refuse the ask: %s", refusal)
}
// Kept as the controller's own ask, about no condition, with the proposal whole and its digests.
kept := r.theProposal(t)
p := kept.Proposal
if kept.State != askOpen || kept.Ask.Digest() != q.Digest() || p.Module != "mounts" || p.Node != "shanks" ||
p.Digest != layerDigest(mountsSources) || p.BeforeDigest != layerDigest(r.before) || !p.Replace || !p.HadLayer ||
p.From != r.pr.caller || kept.ofACondition() {
t.Errorf("kept %+v / %+v", kept, p)
}
// Each option binds exactly the act the controller will perform (boundAct over the kept action).
for i, act := range kept.Actions {
binds, _ := asks.ActDigest(boundAct(act))
if q.Options[i].Binds != binds || act.Arguments["values"] != p.Digest || act.Arguments["before"] != p.BeforeDigest ||
act.Verb != proposalVerb || act.Level != conditions.LevelApprove {
t.Errorf("the option %s does not bind the kept act: %+v", q.Options[i].ID, act)
}
}
if len(r.judged) != 1 || r.judged[0]["sources"] != mountsSources["sources"] {
t.Errorf("judged %v", r.judged)
}
if !strings.Contains(words, "nothing changes") || !strings.Contains(words, kept.ID) {
t.Errorf("the caller is told: %s", words)
}
}
// A layer for the whole mesh is proposed too.
func TestAMeshWideLayerIsProposed(t *testing.T) {
r := newProposerRig(t)
if _, err := r.pr.propose(context.Background(), proposeInput{module: "notes", values: map[string]any{"x": "1"}}); err != nil {
t.Fatal(err)
}
q := r.askSent(t)
if q.Headline != "Set notes on the whole mesh?" || q.About != "settings.notes.mesh" ||
!strings.Contains(q.Explanation, "Set the settings of notes on the whole mesh to these values?") {
t.Errorf("%+v", q)
}
}
// A proposal expired unanswered is kept so by the reconciling, and a warrant for it afterwards sets nothing.
func TestAnExpiredProposalIsKeptExpired(t *testing.T) {
r := newAskerRig(t)
calls := withSetLayer(r)
a := aProposalAsked(t, r, "s7", aProposal(r.now))
r.now = r.now.Add(askApproveFor + time.Minute)
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
if got := r.store["s7"]; got.State != string(asks.OutcomeExpired) || !strings.HasPrefix(got.Acted, "nothing") {
t.Errorf("kept as %+v", got)
}
answerWith(t, r, warrantOn(a, "approve", r.now.Add(-2*time.Minute)))
if len(*calls) != 0 {
t.Errorf("set after expiry: %+v", *calls)
}
}
// A clear is proposed too, and shows the layer it removes.
func TestAClearIsProposedAndShowsWhatItRemoves(t *testing.T) {
r := newProposerRig(t)
r.before, r.had = map[string]any{"places": map[string]any{"data": map[string]any{"path": "/srv/notes", "owner": "1001:1001"}}}, true
if _, err := r.pr.propose(context.Background(), proposeInput{module: "notes", node: "laptop", clear: true}); err != nil {
t.Fatal(err)
}
q := r.askSent(t)
if q.Headline != "Clear notes on laptop?" || !strings.Contains(q.Explanation, "- places.data.owner: 1001:1001") ||
!strings.Contains(q.Explanation, "- places.data.path: ‹path›") {
t.Errorf("%+v", q)
}
if p := r.theProposal(t).Proposal; !p.Clear || p.Digest != layerDigest(r.before) || len(r.judged) != 0 {
t.Errorf("a clear: %+v, judged %v", p, r.judged)
}
}
// What the phone is shown keeps a value's shape and passes the content rule: an address, a path, a secret's
// shape each replaced in place; a value every word of which may leave the mesh shown whole.
func TestAValueIsShownInItsShapeAndPassesTheContentRule(t *testing.T) {
for in, want := range map[any]string{
"recalbox=smb://nas.lan/recalbox@/mnt/recalbox:ro": "recalbox=‹address›@‹path›:ro",
"library=/mnt/library": "library=‹path›",
"none": "none",
"Inter 13": "Inter 13",
"10.77.0.9:53": "‹address›",
"jochen@example.com": "‹address›",
"nas.lan": "‹address›",
"anchor": "anchor",
"-----BEGIN CERTIFICATE-----": "‹withheld›",
42: "42",
true: "true",
} {
got, whole := sayableValue(in, []string{"anchor"})
if got != want {
t.Errorf("%v shown as %q, want %q", in, got, want)
}
if whole != (got == want && !strings.Contains(want, "‹")) {
t.Errorf("%v: whole %v", in, whole)
}
if _, ok := outward.Check(got, "anchor"); !ok {
t.Errorf("%v shown as %q, which the router refuses", in, got)
}
}
for _, v := range []any{[]any{"a", "/etc/x"}, map[string]any{"path": "/srv/x", "owner": "1001:1001"}} {
got, _ := sayableValue(v, nil)
if _, ok := outward.Check(got); !ok || strings.Contains(got, "/srv") || strings.Contains(got, "/etc") {
t.Errorf("%v shown as %q", v, got)
}
}
}
// A proposal that the mesh would refuse to set, or that would silently remove a key, is refused before anybody is
// asked (ADR 0217 holds for a proposal as for a set).
func TestAProposalTheMeshWouldRefuseIsNotAsked(t *testing.T) {
r := newProposerRig(t)
r.before, r.had = map[string]any{"a": 1, "b": 2}, true
_, err := r.pr.propose(context.Background(), proposeInput{module: "notes", node: "laptop", values: map[string]any{"a": 1}})
if err == nil || !strings.Contains(err.Error(), "would no longer set b") || !strings.Contains(err.Error(), "ADR 0217") {
t.Errorf("a silent removal: %v", err)
}
r.refusedBy = "refused: notes on laptop cannot compose"
_, err = r.pr.propose(context.Background(), proposeInput{module: "notes", node: "laptop", values: map[string]any{"a": 1, "b": 3}})
if err == nil || !strings.Contains(err.Error(), "cannot compose") {
t.Errorf("a layer the mesh refuses: %v", err)
}
if len(r.sent) != 0 || len(r.store) != 0 {
t.Errorf("asked anyway: %+v %+v", r.sent, r.store)
}
}
// Fail closed: no router, no grant, a publish that fails, or the router's refusal each leave nothing waiting for
// an answer that cannot come, and name the terminal's line.
func TestAProposalFailsClosedWhenNothingCanCarryIt(t *testing.T) {
r := newProposerRig(t)
in := proposeInput{module: "mounts", node: "shanks", values: mountsSources}
r.pr.routerHere = func(context.Context) (bool, error) { return false, nil }
if _, err := r.pr.propose(context.Background(), in); err == nil || !strings.Contains(err.Error(), "no router") ||
!strings.Contains(err.Error(), "mesh-cli settings set mounts") {
t.Errorf("without a router: %v", err)
}
r.pr.routerHere = nil
r.pr.grantHeld = func(context.Context) (bool, string, error) { return false, "the bus's user list is behind", nil }
if _, err := r.pr.propose(context.Background(), in); err == nil || !strings.Contains(err.Error(), "user list is behind") {
t.Errorf("without the grant: %v", err)
}
if len(r.sent) != 0 || len(r.store) != 0 {
t.Fatalf("asked anyway: %+v %+v", r.sent, r.store)
}
r.pr.grantHeld = nil
r.pr.publish = func(context.Context, string, []byte, string) error { return errors.New("the bus is away") }
if _, err := r.pr.propose(context.Background(), in); err == nil || !strings.Contains(err.Error(), "the bus is away") ||
!strings.Contains(err.Error(), "never become active") {
t.Errorf("a publish that fails: %v", err)
}
if kept := r.theProposal(t); kept.State != askUnsent {
t.Errorf("an unpublished proposal is %s", kept.State)
}
// The router's refusal, heard by the serving controller and kept here, is said to the caller.
r = newProposerRig(t)
r.pr.publish = func(_ context.Context, _ string, _ []byte, id string) error {
ask := strings.TrimPrefix(id, "ask.")
r.store[ask] = func() asked {
a := r.store[ask]
a.State, a.Acted = string(asks.OutcomeRefused), "nothing: the ask refused: no channel can carry any of its answers now"
return a
}()
return nil
}
if _, err := r.pr.propose(context.Background(), in); err == nil || !strings.Contains(err.Error(), "did not ask the operator") ||
!strings.Contains(err.Error(), "no channel can carry") {
t.Errorf("the router's refusal: %v", err)
}
}
// The bounds: at most three asks open for the controller, and the same change not proposed twice.
func TestAProposalIsBounded(t *testing.T) {
r := newProposerRig(t)
in := proposeInput{module: "mounts", node: "shanks", values: mountsSources}
if _, err := r.pr.propose(context.Background(), in); err != nil {
t.Fatal(err)
}
if _, err := r.pr.propose(context.Background(), in); err == nil || !strings.Contains(err.Error(), "already proposed") {
t.Errorf("the same change twice: %v", err)
}
for _, node := range []string{"laptop", "anchor"} {
if _, err := r.pr.propose(context.Background(), proposeInput{module: "mounts", node: node, values: mountsSources}); err != nil {
t.Fatal(err)
}
}
_, err := r.pr.propose(context.Background(), proposeInput{module: "notes", node: "laptop", values: map[string]any{"x": 1}})
if err == nil || !strings.Contains(err.Error(), "3 questions already wait") {
t.Errorf("a fourth: %v", err)
}
if len(r.sent) != 3 {
t.Errorf("published %d", len(r.sent))
}
}
// ---- the warrant ------------------------------------------------------------------------------------
// aProposalAsked keeps a proposal's ask in the asker rig's store, as propose keeps it.
func aProposalAsked(t *testing.T, r *askerRig, id string, p settingsProposal) asked {
t.Helper()
q, options := p.ask(id, nil)
if err := q.Check(r.now); err != nil {
t.Fatal(err)
}
a := asked{ID: id, Condition: q.About, Ask: q, Actions: p.actions(id), Options: options, State: askOpen, Opened: r.now, Proposal: &p}
r.store[id] = a
return a
}
func aProposal(now time.Time) settingsProposal {
before := map[string]any{"shares": "none"}
return settingsProposal{Module: "mounts", Node: "shanks", Values: mountsSources, Replace: true, Before: before, HadLayer: true,
From: "g14/claude-code", At: now, Digest: layerDigest(mountsSources), BeforeDigest: layerDigest(before)}
}
// warrantOn is the router's warrant for a kept ask, choosing an option by id.
func warrantOn(a asked, option string, now time.Time) asks.Warrant {
o, _ := a.Ask.Option(option)
return asks.Warrant{Ask: a.ID, Asker: "mesh-controller", About: a.Ask.About, Outcome: asks.OutcomeChosen, Option: o.ID,
Label: o.Label, Level: o.Level, Channel: "telegram", Proofs: []string{"P1"}, At: now, AskDigest: a.Ask.Digest(),
By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
}
type setCall struct {
p settingsProposal
ask string
setBy string
}
func withSetLayer(r *askerRig) *[]setCall {
var calls []setCall
r.a.setLayer = func(_ context.Context, p settingsProposal, askID, setBy string) (string, error) {
calls = append(calls, setCall{p, askID, setBy})
return "+ sources, ~ shares", nil
}
return &calls
}
// On Approve the layer is set once, with who approved it and through which channel kept beside it, and the warrant
// is recorded as the operator's decision; heard again, nothing more happens.
func TestTheLayerIsSetOnceOnTheOperatorsApproval(t *testing.T) {
r := newAskerRig(t)
calls := withSetLayer(r)
a := aProposalAsked(t, r, "s1", aProposal(r.now))
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
if got := r.store["s1"]; got.State != askOpen {
t.Fatalf("the reconciling of conditions ended the proposal: %+v", got)
}
w := warrantOn(a, "approve", r.now.Add(time.Hour))
answerWith(t, r, w)
answerWith(t, r, w) // heard again, or replayed
if len(*calls) != 1 {
t.Fatalf("set %d time(s): %+v", len(*calls), *calls)
}
c := (*calls)[0]
if c.ask != "s1" || c.p.Digest != layerDigest(mountsSources) ||
!strings.HasPrefix(c.setBy, "approved by the operator, as telegram identity 42 via telegram (telegram), user id verified at ") ||
!strings.Contains(c.setBy, "(ask s1, proposed by g14/claude-code)") {
t.Errorf("set by %q for %+v", c.setBy, c.p)
}
if len(r.called)+len(r.silenced) != 0 {
t.Errorf("a verb was called: %v %v", r.called, r.silenced)
}
if len(r.acts) != 1 {
t.Fatalf("hand-acts %+v", r.acts)
}
act := r.acts[0]
if act.Verb != handActWarrant || act.By != "the operator, as telegram identity 42" || act.Ask != "s1" ||
!slices.Contains(act.Args, "answer=approve") || !slices.Contains(act.Args, "values="+layerDigest(mountsSources)) ||
!strings.HasPrefix(act.Outcome, "done") || !personsDecision(act) {
t.Errorf("the record: %+v", act)
}
if got := r.store["s1"]; got.State != string(asks.OutcomeChosen) || !strings.HasPrefix(got.Acted, "done") {
t.Errorf("kept as %+v", got)
}
}
// Decline, expiry, the router's refusal, a cancel: nothing is set, and the proposal is recorded as ended.
func TestDeclineExpiryAndRefusalDiscardAProposal(t *testing.T) {
for name, outcome := range map[string]asks.Outcome{"declined": asks.OutcomeChosen, "expired": asks.OutcomeExpired,
"refused": asks.OutcomeRefused, "cancelled": asks.OutcomeCancelled, "replaced": asks.OutcomeReplaced} {
t.Run(name, func(t *testing.T) {
r := newAskerRig(t)
calls := withSetLayer(r)
a := aProposalAsked(t, r, "s2", aProposal(r.now))
w := warrantOn(a, "decline", r.now.Add(time.Hour))
if outcome != asks.OutcomeChosen {
w = asks.Warrant{Ask: a.ID, Asker: "mesh-controller", Outcome: outcome, Words: "its time passed", At: r.now.Add(time.Hour)}
}
answerWith(t, r, w)
if len(*calls) != 0 {
t.Fatalf("set: %+v", *calls)
}
got := r.store["s2"]
if got.State != string(outcome) || got.Acted == "" || !strings.HasPrefix(got.Acted, "nothing") {
t.Errorf("kept as %+v", got)
}
if outcome == asks.OutcomeChosen && (len(r.acts) != 1 || !strings.Contains(r.acts[0].Outcome, "declined")) {
t.Errorf("a decline is a decision too: %+v", r.acts)
}
// An approval after it ended is refused.
answerWith(t, r, warrantOn(a, "approve", r.now.Add(2*time.Hour)))
if len(*calls) != 0 {
t.Fatalf("set after the end: %+v", *calls)
}
})
}
}
// The warrant binds the exact values: a record whose values changed after the ask, an action changed, a warrant for
// another ask's digest, at another level, or after expiry each set nothing.
func TestAWarrantSetsOnlyTheExactValuesTheOperatorWasShown(t *testing.T) {
cases := map[string]func(r *askerRig, a asked) asks.Warrant{
"the values changed in the record": func(r *askerRig, a asked) asks.Warrant {
a.Proposal.Values = map[string]any{"sources": "recalbox=smb://evil/recalbox@/mnt/recalbox", "shares": "library=/mnt/library"}
r.store[a.ID] = a
return warrantOn(a, "approve", r.now.Add(time.Hour))
},
"the layer it was shown against changed in the record": func(r *askerRig, a asked) asks.Warrant {
a.Proposal.Before = map[string]any{"shares": "other"}
r.store[a.ID] = a
return warrantOn(a, "approve", r.now.Add(time.Hour))
},
"the module changed in the record": func(r *askerRig, a asked) asks.Warrant {
a.Proposal.Module = "sshd"
r.store[a.ID] = a
return warrantOn(a, "approve", r.now.Add(time.Hour))
},
"the machine changed in the record": func(r *askerRig, a asked) asks.Warrant {
a.Proposal.Node = "anchor"
r.store[a.ID] = a
return warrantOn(a, "approve", r.now.Add(time.Hour))
},
"the removal became meant in the record": func(r *askerRig, a asked) asks.Warrant {
a.Proposal.Replace = !a.Proposal.Replace
r.store[a.ID] = a
return warrantOn(a, "approve", r.now.Add(time.Hour))
},
"the set became a clear in the record": func(r *askerRig, a asked) asks.Warrant {
a.Proposal.Clear = true
r.store[a.ID] = a
return warrantOn(a, "approve", r.now.Add(time.Hour))
},
"the action's digest was changed": func(r *askerRig, a asked) asks.Warrant {
a.Actions[0].Arguments["values"] = layerDigest(map[string]any{"sources": "x"})
r.store[a.ID] = a
return warrantOn(a, "approve", r.now.Add(time.Hour))
},
"another ask's digest": func(r *askerRig, a asked) asks.Warrant {
w := warrantOn(a, "approve", r.now.Add(time.Hour))
w.AskDigest = "sha256:0000"
return w
},
"at the level acknowledge": func(r *askerRig, a asked) asks.Warrant {
w := warrantOn(a, "approve", r.now.Add(time.Hour))
w.Level = asks.Acknowledge
return w
},
"after expiry": func(r *askerRig, a asked) asks.Warrant {
return warrantOn(a, "approve", r.now.Add(askApproveFor+time.Minute))
},
"nobody chose": func(r *askerRig, a asked) asks.Warrant {
w := warrantOn(a, "approve", r.now.Add(time.Hour))
w.By = nil
return w
},
"another asker's": func(r *askerRig, a asked) asks.Warrant {
w := warrantOn(a, "approve", r.now.Add(time.Hour))
w.Asker = "claude-code"
return w
},
}
for name, tamper := range cases {
t.Run(name, func(t *testing.T) {
r := newAskerRig(t)
calls := withSetLayer(r)
a := aProposalAsked(t, r, "s3", aProposal(r.now))
answerWith(t, r, tamper(r, a))
if len(*calls) != 0 {
t.Fatalf("set: %+v", *calls)
}
if got := r.store["s3"]; strings.HasPrefix(got.Acted, "done") {
t.Errorf("kept as done: %+v", got)
}
})
}
}
// A proposal counts toward what the controller holds open, so a fourth ask is not attempted while three are.
func TestOpenProposalsCountTowardTheAsksHeldOpen(t *testing.T) {
r := newAskerRig(t)
for _, id := range []string{"s4", "s5", "s6"} {
aProposalAsked(t, r, id, aProposal(r.now))
}
r.open = []conditions.Condition{heldCondition()}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
if len(r.sent) != 0 {
t.Errorf("asked beyond the bound: %d", len(r.sent))
}
for _, id := range []string{"s4", "s5", "s6"} {
if r.store[id].State != askOpen {
t.Errorf("%s was ended by the reconciling of conditions: %+v", id, r.store[id])
}
}
}
// ---- the verb ---------------------------------------------------------------------------------------
func TestTheSettingsVerbComposesProposeAndProposals(t *testing.T) {
for name, c := range map[string]struct {
args map[string]any
want string
}{
"propose on a machine": {map[string]any{"module": "mounts", "node": "shanks", "values": `{"sources":"x"}`, "propose": "true"},
"settings propose mounts {\"sources\":\"x\"} --node shanks"},
"propose with replace": {map[string]any{"module": "mounts", "values": `{"a":1}`, "propose": "true", "replace": "true"},
"settings propose mounts {\"a\":1} --replace"},
"propose a clear": {map[string]any{"module": "mounts", "node": "shanks", "propose": "true", "clear": "true"},
"settings propose mounts --clear --node shanks"},
"the proposals": {map[string]any{"proposals": "true"}, "settings proposals"},
"one proposal": {map[string]any{"proposal": "s1"}, "settings proposals s1"},
} {
argv, err := argvFor("settings", c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%s: %v %v", name, argv, err)
}
}
for name, args := range map[string]map[string]any{
"propose without a module": {"values": `{"a":1}`, "propose": "true"},
"propose without values": {"module": "mounts", "propose": "true"},
"propose values and clear": {"module": "mounts", "values": `{"a":1}`, "clear": "true", "propose": "true"},
"proposals with a module": {"proposals": "true", "module": "mounts"},
"proposals and a proposal": {"proposals": "true", "proposal": "s1"},
"a proposal with values": {"proposal": "s1", "values": `{"a":1}`},
"proposals with a listing": {"proposals": "true", "list": "preferences"},
} {
if _, err := argvFor("settings", args); err == nil {
t.Errorf("%s was composed", name)
}
}
// The generic command verb proposes nothing (it is the settings verb's), and lists proposals (a read).
if err := refusedAsTheGenericCommand([]string{"settings", "propose", "mounts", "{}", "--node", "shanks"}); err == nil {
t.Error("the generic command proposed")
}
if err := refusedAsTheGenericCommand([]string{"settings", "proposals"}); err != nil {
t.Errorf("the generic command may not list proposals: %v", err)
}
}
// ---- on the real stores -----------------------------------------------------------------------------
// setLayerIn sets the layer as the terminal does — judged, the removal meant, the history kept — with who approved it
// beside it; and refuses once the layer is no longer the one the operator was shown the change against.
func TestSetOnAWarrantJudgesTheLayerAndKeepsWhoApprovedIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "notes", Version: "1",
// y is a preference with a default, so a layer may leave it out; x is the operator's own (ADR 0262).
Settings: map[string]catalogue.SettingDeclaration{"y": {Kind: "preference", Default: "10", Why: "a size"}},
Resources: []map[string]any{{"id": "data", "type": "directory", "mode": "0755"},
// A trusted file (unmarked), so its keys are the terminal's — and now the warrant's (novox/hq ADR 0277).
{"id": "rc", "type": "file", "path": "/etc/notes.conf", "mode": "0644", "content": "x = ${setting:x}\ny = ${setting:y}\n"}}})
if _, err := assign(ctx, open, "laptop", "notes"); err != nil {
t.Fatal(err)
}
set := setLayerIn(open)
now := time.Now()
first := map[string]any{"x": "1", "y": "2"}
p := settingsProposal{Module: "notes", Node: "laptop", Values: first, From: "g14/claude-code", At: now,
Digest: layerDigest(first), BeforeDigest: layerDigest(nil)}
changed, err := set(ctx, p, "s9", "approved by the operator, as telegram identity 42 via telegram at 14:05 (ask s9)")
if err != nil || !strings.Contains(changed, "+ x") {
t.Fatalf("%q %v", changed, err)
}
layer, has, err := open.inventory.Layer(ctx, "laptop", "notes")
if err != nil || !has || layer["x"] != "1" {
t.Fatalf("the layer: %v %v %v", layer, has, err)
}
setBy, _, has, err := open.inventory.LayerOrigin(ctx, "laptop", "notes")
if err != nil || !has || !strings.HasPrefix(setBy, "approved by the operator, as telegram identity 42 via telegram") {
t.Fatalf("who set it: %q %v", setBy, err)
}
// Shown by `settings show`, at the terminal and through the verb.
out := captureStdout(t, func() {
if err := atTheTerminal(t, "settings", "show", "notes", "--node", "laptop"); err != nil {
t.Fatal(err)
}
})
if !strings.Contains(out, "approved by the operator, as telegram identity 42 via telegram") {
t.Errorf("settings show says:\n%s", out)
}
// The same proposal again: the layer is no longer the one the operator was shown it against.
if _, err := set(ctx, p, "s9", "approved …"); err == nil || !strings.Contains(err.Error(), "changed since the operator was shown") {
t.Errorf("a stale proposal: %v", err)
}
// A removal not meant is refused; one meant is taken, and the history says who had set the layer.
second := map[string]any{"x": "1"}
q := settingsProposal{Module: "notes", Node: "laptop", Values: second, Before: first, HadLayer: true,
From: "g14/claude-code", At: now, Digest: layerDigest(second), BeforeDigest: layerDigest(first)}
if _, err := set(ctx, q, "s10", "approved …"); err == nil || !strings.Contains(err.Error(), "removal was not meant") {
t.Errorf("a silent removal: %v", err)
}
// A layer the mesh refuses is refused here too, with the same words as at the terminal.
bad := q
bad.Values, bad.Digest = map[string]any{"x": "a\nb", "y": "2"}, layerDigest(map[string]any{"x": "a\nb", "y": "2"})
if _, err := set(ctx, bad, "s11", "approved …"); err == nil || !strings.Contains(err.Error(), "line break") {
t.Errorf("a line break on a warrant: %v", err)
}
if layer, _, _ := open.inventory.Layer(ctx, "laptop", "notes"); layer["y"] != "2" {
t.Fatalf("a refused act changed the layer: %v", layer)
}
q.Replace = true
if _, err := set(ctx, q, "s10", "approved later"); err != nil {
t.Fatal(err)
}
past, err := open.inventory.SettingsHistory(ctx, "laptop", "notes")
if err != nil || len(past) != 1 || !strings.HasPrefix(past[0].SetBy, "approved by the operator") {
t.Errorf("the history: %+v %v", past, err)
}
// And a clear, keeping who cleared it with the copy.
c := settingsProposal{Module: "notes", Node: "laptop", Clear: true, Before: second, HadLayer: true, From: "x", At: now,
Digest: layerDigest(second), BeforeDigest: layerDigest(second)}
if _, err := set(ctx, c, "s12", "approved by the operator at 15:00"); err != nil {
t.Fatal(err)
}
if _, has, _ := open.inventory.Layer(ctx, "laptop", "notes"); has {
t.Error("the layer was not cleared")
}
past, _ = open.inventory.SettingsHistory(ctx, "laptop", "notes")
if len(past) != 2 || !strings.Contains(past[0].SetBy, "cleared approved by the operator at 15:00") {
t.Errorf("the history after a clear: %+v", past)
}
}
// A layer set at the terminal says so, and one set through a verb names the verb (novox/hq ADR 0277).
func TestALayerSaysWhoSetIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "notes", Version: "1",
Resources: []map[string]any{{"id": "rc", "type": "file", "path": "/etc/notes.conf", "mode": "0644", "trusted": false,
"content": "x = ${setting:x}\n"}}})
if _, err := assign(ctx, open, "laptop", "notes"); err != nil {
t.Fatal(err)
}
if err := atTheTerminal(t, "settings", "set", "notes", `{"x":"1"}`, "--node", "laptop"); err != nil {
t.Fatal(err)
}
setBy, _, _, _ := open.inventory.LayerOrigin(ctx, "laptop", "notes")
if !strings.HasPrefix(setBy, "set at the controller's terminal by ") {
t.Errorf("at the terminal: %q", setBy)
}
if err := throughVerb(t, "settings", map[string]any{"module": "notes", "node": "laptop", "values": `{"x":"2"}`}); err != nil {
t.Fatal(err)
}
setBy, _, _, _ = open.inventory.LayerOrigin(ctx, "laptop", "notes")
if !strings.HasPrefix(setBy, "set by ") || !strings.Contains(setBy, "through settings") {
t.Errorf("through the verb: %q", setBy)
}
}
// A trusted setting is still refused through the settings verb (novox/hq issue 339), and the refusal now names the
// proposal as the way.
func TestATrustedSettingThroughAVerbNamesTheProposal(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "notes", Version: "1",
Resources: []map[string]any{{"id": "data", "type": "directory", "mode": "0755"}}})
if _, err := assign(ctx, open, "laptop", "notes"); err != nil {
t.Fatal(err)
}
err := throughVerb(t, "settings", map[string]any{"module": "notes", "node": "laptop",
"values": `{"places":{"data":{"path":"/srv/notes","owner":"1001:1001"}}}`})
if err == nil || !strings.Contains(err.Error(), "issue 339") || !strings.Contains(err.Error(), "propose") {
t.Errorf("%v", err)
}
if _, has, _ := open.inventory.Layer(ctx, "laptop", "notes"); has {
t.Error("a layer was kept")
}
}
// captureStdout runs f and answers what it printed to standard output.
func captureStdout(t *testing.T, f func()) string {
t.Helper()
before := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stdout = w
done := make(chan string)
go func() {
var b strings.Builder
_, _ = io.Copy(&b, r)
done <- b.String()
}()
f()
os.Stdout = before
_ = w.Close()
return <-done
}
+49 -1
View File
@@ -65,6 +65,8 @@ func connectLink(ctx context.Context, inv *inventory.Inventory, enroller link.En
}
func serve(ctx context.Context) (err error) {
// Nothing this process does, or starts, is the operator at the terminal (novox/hq ADR 0266).
markServed()
// The one process whose log is read over time, so the one that says each change to a node's
// unmet seat dependencies once (novox/hq ADR 0207).
logUnheldChanges = true
@@ -74,6 +76,21 @@ func serve(ctx context.Context) (err error) {
}
defer open.Close()
inv := open.inventory
// **What this build reads of the store's schema, on record** (novox/hq issue 352): the highest migration
// it carries, by its version, so a gate that would put this build back later knows it reads the store
// as it is then. A build that does not know its version records nothing, and is never put back.
if build := runningBuild(); build != "" {
if reach, err := schemaReach(); err != nil {
fmt.Printf("what this build reads of the store's schema is not recorded: %v\n", err)
} else if err := inv.RecordSchemaReach(ctx, build, reach); err != nil {
fmt.Printf("what this build (%s) reads of the store's schema is not recorded: %v\n", build, err)
} else {
fmt.Printf("this build (%s) reads the store's schema up to migration %04d; recorded\n", build, reach)
}
} else {
fmt.Printf("this process was not told which build it is (%s), so what it reads of the store's schema is not "+
"recorded, and a gate will never put it back\n", RunningBuildVar)
}
ident, err := openIdentity(ctx)
if err != nil {
@@ -165,6 +182,9 @@ func serve(ctx context.Context) (err error) {
// Open plans move on a timer as well as on outcomes (novox/hq ADR 0162): a tier waiting for
// machines to report moves when they have, and a plan left by a replaced controller resumes.
go planTicker(ctx, open)
// And the merge window (novox/hq ADR 0276): a batch is cut into its walk when the window closes, which no
// merge or outcome says.
go batchCutter(ctx, open)
// And the pending assignments settled on a tick of their own (novox/hq ADR 0261): made once their module
// is registered, ended with why when its build will not register it, raised and cleared as conditions.
// Never by a read.
@@ -219,6 +239,10 @@ func serve(ctx context.Context) (err error) {
}
// The hand-act log is counted for `status` on this connection rather than a new one a minute.
handActConn = bus.Conn
// And everything else this controller does on the bus for a moment (novox/hq issue 327).
servingBus.Store(server.JetStream())
// No longer serving: nothing is lent, and dead-letters says it is not read here (novox/hq issue 330).
defer servingBus.Store(nil)
// And says when it replaced a value given by hand (novox/hq ADR 0228).
givenEvents = bus
// And a pull request's merge check, asked when the forge announces its head and said when judged
@@ -260,6 +284,12 @@ func serve(ctx context.Context) (err error) {
return err
}
defer stopServing()
// And the operator's command on every node, asked through each node's engine (novox/hq ADR 0272).
stopCLI, err := bus.ServeCLI(answerMeshCLI(open.inventory), log.New(os.Stdout, "", log.LstdFlags))
if err != nil {
return err
}
defer stopCLI()
// And says so on the bus (novox/hq ADR 0197): what it serves, as the NATS services protocol asks.
stopAnnouncing, err := bus.Announce(seatAnnouncement(handlers), log.New(os.Stdout, "", log.LstdFlags))
if err != nil {
@@ -1238,11 +1268,14 @@ func issueMemberships(ctx context.Context, open *stores, server *link.Server, se
if err != nil {
return err
}
where := broker.PlacementsOf(records, records.Interchangeable)
bus, ok := server.Bus().(link.OverNATS)
if !ok {
return nil
}
// Which machines are root-free now (novox/hq ADR 0259 §8): a channel's verified sender is composed for the
// router only from one, beside a router on one. Judged once per push, by the root-free verb's judgement.
records.RootFree = rootFreeNow(ctx, newRootReader(ctx, open.inventory, bus.Conn), records.Nodes, time.Now())
where := broker.PlacementsOf(records, records.Interchangeable)
// **Every declared state's bucket, before the memberships that name it** (novox/hq ADR 0201). The
// raise at start asserts them too, but a module registered and assigned since would otherwise have
// its bucket only after the control plane next restarts — found the first time a module declared
@@ -1583,3 +1616,18 @@ func reportUnheldPushed(w io.Writer, named bool, asked []string, unheld map[stri
fmt.Fprintf(w, "%s: %d unmet seat dependenc(ies) — see `status`\n", node, len(lines))
}
}
// schemaReach is the highest migration this build carries for the inventory's store (novox/hq issue 352).
func schemaReach() (int, error) {
migrations, err := inventory.Migrations()
if err != nil {
return 0, err
}
reach := 0
for _, m := range migrations {
if m.Number > reach {
reach = m.Number
}
}
return reach, nil
}
+1 -1
View File
@@ -26,7 +26,7 @@ func TestAPushSaysWhatItRecreates(t *testing.T) {
aContainerBuild(t, "postgres", "c1111111", "", start.Add(time.Second),
map[string][2]string{"server": {image("e"), ""}}),
} {
if _, _, err := takeIn(ctx, inv, b); err != nil {
if _, _, err := takeIn(ctx, inv, asTheOperator(t, inv, b)); err != nil {
t.Fatal(err)
}
}
+12 -12
View File
@@ -6,6 +6,7 @@ import (
"errors"
"flag"
"fmt"
"io"
"os"
"sort"
"strings"
@@ -37,22 +38,21 @@ const (
killAnswer = 75 * time.Second
)
// dialTheBus opens the controller's own connection, for a command that reads or changes the queue.
// dialTheBus is the controller's connection, for a command that reads or changes the queue: the serving
// controller's own, lent, when this process is it (novox/hq issue 327).
func dialTheBus() (*broker.JetStream, error) {
address, err := broker.BusAddress()
if err != nil {
return nil, err
}
js, err := broker.Dial(address)
if err != nil {
return nil, fmt.Errorf("cannot reach the bus: %w", err)
}
return js, nil
return aBus()
}
// queueCommand prints every ask in the build seat's work queue.
func queueCommand(ctx context.Context, args []string) error {
return listQueue(ctx, args, os.Stdout)
}
// listQueue is `queue`: the build queue, as a person reads it or as JSON.
func listQueue(ctx context.Context, args []string, w io.Writer) error {
set := flag.NewFlagSet("queue", flag.ContinueOnError)
usageTo(set, w)
asJSON := set.Bool("json", false, "the queue as JSON")
if _, err := parseAround(set, args); err != nil {
return err
@@ -72,10 +72,10 @@ func queueCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
fmt.Println(string(body))
fmt.Fprintln(w, string(body))
return nil
}
fmt.Print(queueText(q, time.Now()))
fmt.Fprint(w, queueText(q, time.Now()))
return nil
}
+11 -4
View File
@@ -135,12 +135,19 @@ func TestRetryRefusesWhatItCannotResume(t *testing.T) {
if err := retryRefusal(stopped, nil); err == nil || !strings.Contains(err.Error(), "nothing in tier 0") {
t.Errorf("a plan with nothing failed to build was retried: %v", err)
}
// Another branch's newer plan, an older one, and a failed one do not supersede it.
// A failed or done plan does not supersede it.
if err := retryRefusal(failed, []inventory.Plan{plan("plan-4", inventory.PlanFailed, at.Add(time.Hour)),
plan("plan-5", inventory.PlanDone, at.Add(time.Hour))}); err != nil {
t.Errorf("refused for a plan that does not supersede it: %v", err)
}
// Another branch's open walk, or an older one, does not supersede it, and is one walk open: one at a time
// (novox/hq ADR 0276).
other := plan("plan-3", inventory.PlanBuilding, at.Add(time.Hour))
other.Branch = "release"
if err := retryRefusal(failed, []inventory.Plan{other, plan("plan-0", inventory.PlanBuilding, at.Add(-time.Hour)),
plan("plan-4", inventory.PlanFailed, at.Add(time.Hour))}); err != nil {
t.Errorf("refused for a plan that does not supersede it: %v", err)
for _, open := range []inventory.Plan{other, plan("plan-0", inventory.PlanBuilding, at.Add(-time.Hour))} {
if err := retryRefusal(failed, []inventory.Plan{open}); err == nil || !strings.Contains(err.Error(), "one walk at a time") {
t.Errorf("retried beside the open walk %s: %v", open.ID, err)
}
}
}
+173
View File
@@ -0,0 +1,173 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// D15: no client of the bus reconnects in a loop (novox/hq issue 327).
//
// The server's connection total was the one number that would show a client reconnecting, and every verb
// the controller served opened and closed a connection of its own, hundreds an hour, so a loop was
// invisible in it. The bus's own module reads the server's record of closed connections
// (`nats_closed_connections`); this asks it every run, and says each user whose connections were dropped —
// closed by anything but the client itself: a read or write error, a stale connection, a slow consumer, a
// refused login — more often than reconnectBound in the last hour.
const (
probeReconnectsID = "D15"
kindBusReconnects = "bus-reconnects"
// reconnectBound is how many dropped connections in an hour one user may have before it is said:
// a client that loses its connection every five minutes. Provisional.
reconnectBound = 12
// busModule is the module that is the bus, and closedTool its tool that reads closed connections.
busModule = "nats"
closedTool = "nats_closed_connections"
closedAsk = 20 * time.Second
)
// closedConnections is what the bus's module answers.
type closedConnections struct {
Hours float64 `json:"hours"`
Reaches bool `json:"reaches"`
Users []struct {
User string `json:"user"`
Closed int `json:"closed"`
Dropped int `json:"dropped"`
DroppedPerHour float64 `json:"dropped_per_hour"`
Names []struct {
Name string `json:"name"`
Closed int `json:"closed"`
Dropped int `json:"dropped"`
Reasons map[string]int `json:"reasons"`
} `json:"names"`
} `json:"users"`
}
// probeReconnects is D15.
func probeReconnects(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
if d.js == nil {
return nil, fmt.Errorf("no bus to ask the bus's module over")
}
on, err := d.open.inventory.Running(ctx, busModule)
if err != nil {
return nil, err
}
if len(on) == 0 {
return nil, nil // no bus module assigned: a mesh whose bus is not the mesh's module
}
return reconnectsOn(ctx, d.js.Conn(), on[0])
}
// reconnectsOn asks the bus module on its machine and says who reconnects in a loop.
func reconnectsOn(ctx context.Context, conn *nats.Conn, node string) ([]conditions.Observation, error) {
read, err := askClosed(ctx, conn, node)
if isNothingServes(err) {
// A bus module older than its tool has nothing it can say, which is not a failure of the probe.
return nil, nil
}
if err != nil {
return nil, err
}
return reconnecting(read), nil
}
// askClosed asks the bus's module, on its machine, who closed connections in the last hour.
func askClosed(ctx context.Context, conn *nats.Conn, node string) (closedConnections, error) {
var read closedConnections
answer, err := link.AskModuleToolOn(ctx, conn, busModule, closedTool, node, map[string]any{"hours": 1}, closedAsk)
if err != nil {
return read, err
}
if answer.Error != "" {
return read, fmt.Errorf("%s on %s answered %s with an error: %s", busModule, node, closedTool, answer.Error)
}
if err := unmarshalAnswer(answer, &read); err != nil {
return read, fmt.Errorf("%s on %s answered %s with something unreadable: %w", busModule, node, closedTool, err)
}
return read, nil
}
// reconnecting is one observation per user whose connections were dropped more than reconnectBound times
// an hour.
func reconnecting(read closedConnections) []conditions.Observation {
hours := read.Hours
if hours <= 0 {
hours = 1
}
var out []conditions.Observation
for _, u := range read.Users {
if u.User == "" || strings.HasPrefix(u.User, "(") {
// Refused before it logged in: no client of the mesh's, so nobody's reconnect loop. A login
// refused again and again is a question of its own, not this probe's.
continue
}
perHour := float64(u.Dropped) / hours
if perHour <= reconnectBound {
continue
}
reasons := map[string]int{}
var names []string
for _, n := range u.Names {
if n.Dropped == 0 {
continue
}
names = append(names, fmt.Sprintf("%q ×%d", n.Name, n.Dropped))
for r, c := range n.Reasons {
if r != "Client Closed" {
reasons[r] += c
}
}
}
var why []string
for r, c := range reasons {
why = append(why, fmt.Sprintf("%s ×%d", r, c))
}
sort.Strings(why)
partial := ""
if !read.Reaches {
partial = " (at least: the server's record of closed connections does not reach back the whole hour)"
}
who, machine := busUserWords(u.User)
out = append(out, conditions.Observation{Scope: conditions.ScopeBus, ID: u.User, Kind: kindBusReconnects,
Machine: machine, Severity: conditions.Warning,
Summary: fmt.Sprintf("the bus dropped %s's connection %d times in the last hour%s, more than %d: a "+
"client reconnecting in a loop", u.User, u.Dropped, partial, reconnectBound),
Said: fmt.Sprintf("%d of %d closed connections dropped in %.0f h; by name %s; why %s", u.Dropped,
u.Closed, hours, strings.Join(names, ", "), strings.Join(why, ", ")),
Headline: clip(conditions.Capital(who)+" keeps losing the bus", 60),
Explanation: conditions.Capital(fmt.Sprintf("%s lost its connection to the bus %d times in the last hour "+
"and connected again each time. While it reconnects, what it says and what it is asked waits.", who,
u.Dropped)),
Resolved: "Resolved: " + who + " stays connected"})
}
return out
}
// busUserWords is a bus user as the operator says it, and the machine it is on: `node.<machine>` the
// node-engine, `<machine>.node-tools` the tool runner, `controller` the controller, `<machine>.<module>` a
// module.
func busUserWords(user string) (string, string) {
switch {
case user == "controller":
return "the controller", ""
case strings.HasPrefix(user, "node."):
m := strings.TrimPrefix(user, "node.")
return "the node-engine on " + m, m
}
if m, module, ok := strings.Cut(user, "."); ok {
if module == "node-tools" {
return "the tool runner on " + m, m
}
return module + " on " + m, m
}
return "a client of the bus", ""
}
+189
View File
@@ -0,0 +1,189 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// The serving controller's own connection serves what it does for a moment: no connection is opened,
// and closing what it was lent leaves the serving one open (novox/hq issue 327).
func TestTheServingControllerLendsItsOwnConnection(t *testing.T) {
bus := testbus.Start(t)
serving, err := broker.Dial(bus.ClientURL())
if err != nil {
t.Fatal(err)
}
defer serving.Close()
if err := serving.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
was := servingBus.Load()
servingBus.Store(serving)
defer servingBus.Store(was)
t.Setenv(broker.NATSVar, bus.ClientURL())
before, _ := bus.Varz(nil)
lent, err := aBus()
if err != nil {
t.Fatal(err)
}
lent.Close()
if !serving.Conn().IsConnected() {
t.Fatal("closing a lent connection closed the serving controller's")
}
if err := onTheBus(func(*nats.Conn) error { return nil }); err != nil {
t.Fatal(err)
}
handlers, _, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
answer, err := handlers["hand-acts"](context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if a := answer.(verbAnswer); !a.OK || a.Answer == nil {
t.Fatalf("hand-acts answered %+v", a)
}
_, _ = handlers["queue"](context.Background(), json.RawMessage(`{}`))
after, _ := bus.Varz(nil)
if opened := after.TotalConnections - before.TotalConnections; opened != 0 {
t.Fatalf("the serving controller opened %d connection(s) of its own", opened)
}
}
// A verb that still runs as a process of its own says which in the bus's list of connections.
func TestAVerbsOwnProcessNamesItsConnection(t *testing.T) {
bus := testbus.Start(t)
setup, err := broker.Dial(bus.ClientURL())
if err != nil {
t.Fatal(err)
}
if err := setup.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
setup.Close()
asAProcess(t, bus.ClientURL())
handlers, _, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
// A silence acts, so it is still its own process; it dials, and finds no such condition.
_, _ = handlers["conditions"](context.Background(),
json.RawMessage(`{"silence":"bus.nothing.here","for":"1h","why":"a test"}`))
names := closedNames(t, bus)
found := false
for _, n := range names {
found = found || n == "mesh-controller verb conditions"
}
if !found {
t.Fatalf("the verb's connection was named %q", names)
}
}
func TestEachProcessNamesItsConnectionsForWhatItIs(t *testing.T) {
for _, c := range []struct{ command, verb, want string }{
{"serve", "", "mesh-controller serving"},
{"conditions", "conditions", "mesh-controller verb conditions"},
{"delivery", "delivery-check", "mesh-controller verb delivery-check"},
{"push", "", "mesh-controller command push"},
} {
if got := connectionName(c.command, c.verb); got != c.want {
t.Errorf("%s/%s: %q", c.command, c.verb, got)
}
}
}
// D15: a user whose connections are dropped more than twelve times an hour is said, in plain words; one
// whose client closes its own short connections is not.
func TestAClientReconnectingInALoopIsSaid(t *testing.T) {
var read closedConnections
if err := json.Unmarshal([]byte(`{"hours":1,"reaches":true,"users":[
{"user":"ace.node-tools","closed":40,"dropped":40,"dropped_per_hour":40,"names":[
{"name":"ace.node-tools","closed":40,"dropped":40,"reasons":{"Stale Connection":30,"Read Error":10}}]},
{"user":"controller","closed":300,"dropped":0,"names":[
{"name":"mesh-controller verb delivery-check","closed":300,"dropped":0,"reasons":{"Client Closed":300}}]},
{"user":"(no user: refused before it logged in)","closed":90,"dropped":90,"names":[{"name":"","closed":90,
"dropped":90,"reasons":{"Authentication Failure":90}}]},
{"user":"node.anchor","closed":5,"dropped":5,"names":[{"name":"mesh-host/anchor","closed":5,"dropped":5,
"reasons":{"Read Error":5}}]}]}`), &read); err != nil {
t.Fatal(err)
}
found := reconnecting(read)
if len(found) != 1 {
t.Fatalf("%+v", found)
}
o := found[0]
if o.ID != "ace.node-tools" || o.Machine != "ace" || o.Kind != kindBusReconnects ||
o.Headline != "The tool runner on ace keeps losing the bus" || !strings.Contains(o.Said, "Stale Connection ×30") {
t.Fatalf("%+v", o)
}
if why, ok := conditions.PlainWords(conditions.Words{Headline: o.Headline, Explanation: o.Explanation,
Resolved: o.Resolved}, "ace"); !ok {
t.Fatalf("not plain: %s", why)
}
}
// The bus's module is asked on its machine, over the bus.
func TestTheBusModuleIsAskedWhoClosedConnections(t *testing.T) {
bus := testbus.Start(t)
conn, err := nats.Connect(bus.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
sub, err := conn.Subscribe(link.ModuleToolOn("nats", "nats_closed_connections", "anchor"), func(m *nats.Msg) {
_ = m.Respond([]byte(`{"result":{"hours":1,"reaches":true,"users":[{"user":"controller","closed":2,"dropped":0}]}}`))
})
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
read, err := askClosed(context.Background(), conn, "anchor")
if err != nil {
t.Fatal(err)
}
if len(read.Users) != 1 || read.Users[0].Closed != 2 {
t.Fatalf("%+v", read)
}
}
// A bus module older than the tool answers nothing to the question: the probe passes over it quietly.
func TestAnOlderBusModuleIsPassedOverQuietly(t *testing.T) {
bus := testbus.Start(t)
conn, err := nats.Connect(bus.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
found, err := reconnectsOn(context.Background(), conn, "anchor")
if err != nil || len(found) != 0 {
t.Fatalf("a bus module without the tool: %v %v", found, err)
}
}
// A verb answered in the serving controller says its flag errors in its answer, not in the controller's log.
func TestAFlagErrorIsSaidInTheAnswer(t *testing.T) {
bus := testbus.Start(t)
serving, err := broker.Dial(bus.ClientURL())
if err != nil {
t.Fatal(err)
}
defer serving.Close()
was := servingBus.Load()
servingBus.Store(serving)
defer servingBus.Store(was)
answer, read := readHere(context.Background(), []string{"hand-acts", "--bogus"})
if !read || answer.OK || !strings.Contains(answer.Output, "flag provided but not defined") {
t.Fatalf("answered %+v", answer)
}
}
+1 -1
View File
@@ -71,7 +71,7 @@ func TestARecordedBuildIsCarriedOnlyByAPersonsPush(t *testing.T) {
aContainerBuild(t, "mailu", "c1111111", "", start.Add(time.Second),
map[string][2]string{"smtp": {mailImage, ""}, "imap": {mailImage, ""}}),
} {
if _, _, err := takeIn(ctx, inv, b); err != nil {
if _, _, err := takeIn(ctx, inv, asTheOperator(t, inv, b)); err != nil {
t.Fatal(err)
}
}
+114
View File
@@ -0,0 +1,114 @@
package main
// The rehearsal of the operator's answers — not a drill, which in the glossary is something broken on purpose (novox/hq ADR 0259, the live acceptance after rollout): an ask the
// operator starts at the controller's terminal, answered on the phone, whose approval changes nothing and is
// recorded as a person's decision like any other.
//
// mesh-controller rehearse [--for 15m]
//
// It asks with two answers, Approve and Decline, each bound to the rehearsal's own act and **both at the level
// approve** (the review of 2026-10-09, M1: an acknowledgement never shares an ask with an approval), so only a
// channel that proves who answered carries either — the rehearsal is of exactly that. The serving controller acts on the warrant
// as on any other: it claims the ask once, checks the act is the one bound, performs nothing, and records the
// hand-act `warrant` with who answered, through which channel, and the proofs. `hand-acts` then shows it.
//
// **The terminal's alone** (startedAtTheTerminal): a command a verb runs, an ordinary mesh-cli line and anything the
// serving controller started are refused, so no agent starts a rehearsal — a
// rehearsal is a question the operator expects, and one an agent could start would teach them to approve what they
// did not ask for.
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"time"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/conditions"
)
// rehearsalVerb is the act a rehearsal's answers bind: nothing is called.
const rehearsalVerb = "rehearsal"
// rehearsalActions are the rehearsal's two answers.
func rehearsalActions() []conditions.Action {
return []conditions.Action{
{Label: "Approve", Verb: rehearsalVerb, Level: conditions.LevelApprove, Arguments: map[string]string{"rehearsal": "approve"}},
{Label: "Decline", Verb: rehearsalVerb, Level: conditions.LevelApprove, Arguments: map[string]string{"rehearsal": "decline"}},
}
}
// rehearsalAsk is the rehearsal's ask, as the router is sent it.
func rehearsalAsk(id string, now time.Time, lasts time.Duration) (asks.Ask, map[string]int) {
q := asks.Ask{ID: id, Headline: "Rehearsal: approve this test question?", Who: asks.Operator,
Explanation: "Needs you: approve or decline. You started this rehearsal at the controller's terminal. Approving " +
"changes nothing on the mesh; it is recorded as your decision, so you can check the record.",
OnExpiry: "nothing is done", Expires: now.Add(lasts), About: "rehearsal." + id}
options := map[string]int{}
for i, act := range rehearsalActions() {
binds, _ := asks.ActDigest(boundAct(act))
oid := optionID(act.Label)
options[oid] = i
q.Options = append(q.Options, asks.Option{ID: oid, Label: act.Label, Does: doesRehearsal(act), Level: asks.Level(act.Level),
Binds: binds})
}
return q, options
}
func doesRehearsal(act conditions.Action) string {
if act.Arguments["rehearsal"] == "approve" {
return "nothing changes; your approval is recorded"
}
return "nothing changes; your answer is recorded"
}
func rehearseCommand(ctx context.Context, args []string) error {
// The terminal as main judges it (startedAtTheTerminal): not a verb, not the serving controller or anything it
// started, and a mesh-cli line only when it is the control-node's operator's (novox/hq ADR 0272 §4).
if !startedAtTheTerminal() {
return errors.New("rehearse is the controller's terminal's alone: a verb, a mesh-cli line from anybody but " +
"the control-node's operator, or a process the serving controller started may not start one, so no agent " +
"asks the operator a question they did not start (novox/hq ADR 0259)")
}
set := flag.NewFlagSet("rehearse", flag.ContinueOnError)
lasts := set.Duration("for", 15*time.Minute, "how long the question waits for an answer")
if err := set.Parse(args); err != nil {
return err
}
if *lasts < time.Minute || *lasts > askApproveFor {
return fmt.Errorf("a rehearsal waits between a minute and %s", askApproveFor)
}
js, err := aBus()
if err != nil {
return err
}
defer js.Close()
now := time.Now()
id := newAskID()
q, options := rehearsalAsk(id, now, *lasts)
if err := q.Check(now); err != nil {
return err
}
store := busAsked{conn: js.Conn()}
// Kept before it is published, as the asker keeps every ask, so a warrant always finds it.
if err := store.Create(ctx, asked{ID: id, Condition: q.About, Ask: q, Actions: rehearsalActions(), Options: options,
State: askOpen, Opened: now, Rehearsal: true}); err != nil {
return fmt.Errorf("the rehearsal could not be kept in the controller's asks: %w", err)
}
body, err := json.Marshal(q)
if err != nil {
return err
}
if _, err := js.Context().Publish(asks.AskSubject(askerName), body, nats.MsgId("ask."+id), nats.Context(ctx)); err != nil {
return fmt.Errorf("the rehearsal could not be asked: %w", err)
}
fmt.Printf("rehearsal %s asked: answer it on your phone before %s. Then `mesh-controller hand-acts` shows the "+
"answer as a warrant, with who answered, through which channel and the proofs; nothing else changes.\n",
id, q.Expires.Local().Format("15:04"))
return nil
}
+76
View File
@@ -0,0 +1,76 @@
package main
import (
"context"
"github.com/novox/mesh-controller/internal/link"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
)
// A rehearsal (the live acceptance of novox/hq ADR 0259): its approval is a warrant like any other — claimed once,
// its act checked against what the option bound, recorded as the operator's decision with who, how and the
// proofs — and it performs nothing. The reconciling of conditions leaves it open.
func TestARehearsalsApprovalIsRecordedAndPerformsNothing(t *testing.T) {
r := newAskerRig(t)
q, options := rehearsalAsk("crehearsal", r.now, askerRehearsalFor)
if err := q.Check(r.now); err != nil {
t.Fatalf("the rehearsal's ask is refused: %v", err)
}
r.store["crehearsal"] = asked{ID: "crehearsal", Condition: q.About, Ask: q, Actions: rehearsalActions(), Options: options,
State: askOpen, Opened: r.now, Rehearsal: true}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
if got := r.store["crehearsal"]; got.State != askOpen {
t.Fatalf("the reconciling of conditions ended the rehearsal: %+v", got)
}
approve, _ := q.Option("approve")
w := asks.Warrant{Ask: "crehearsal", Asker: "mesh-controller", Outcome: asks.OutcomeChosen, Option: approve.ID,
Label: approve.Label, Level: approve.Level, Channel: "telegram", Proofs: []string{"P1"}, At: r.now,
AskDigest: q.Digest(), By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
answerWith(t, r, w)
answerWith(t, r, w) // heard again
if len(r.called)+len(r.silenced) != 0 {
t.Errorf("a rehearsal performed something: %v %v", r.called, r.silenced)
}
if len(r.acts) != 1 {
t.Fatalf("hand-acts %+v", r.acts)
}
act := r.acts[0]
if act.Verb != handActWarrant || act.By != "the operator, as telegram identity 42" || act.Ask != "crehearsal" ||
strings.Join(act.Args, " ") != "rehearsal rehearsal=approve" || act.Outcome != "done" || strings.Join(act.Proofs, ",") != "P1" {
t.Errorf("the rehearsal's record: %+v", act)
}
if !personsDecision(act) {
t.Error("a rehearsal's answer counts as a repair")
}
}
// Only the terminal starts a rehearsal: a verb's process is refused before anything is asked.
func TestARehearsalIsTheTerminalsAlone(t *testing.T) {
t.Setenv(verbVar, "mesh-controller.command")
if err := rehearseCommand(context.Background(), nil); err == nil || !strings.Contains(err.Error(), "terminal") {
t.Fatalf("a verb started a rehearsal: %v", err)
}
// Nor a mesh-cli line from anybody but the control-node's operator (hq ADR 0272 §4): run without a verb,
// naming its caller, and without the terminal's mark — and nor anything the serving controller started.
for name, env := range map[string]map[string]string{
"an ordinary mesh-cli line": {verbVar: "", link.CallerVar: "laptop/agent"},
"a process the serving controller ran": {verbVar: "", servedVar: "1"},
} {
t.Run(name, func(t *testing.T) {
for k, v := range env {
t.Setenv(k, v)
}
if err := rehearseCommand(context.Background(), nil); err == nil || !strings.Contains(err.Error(), "terminal") {
t.Fatalf("%s started a rehearsal: %v", name, err)
}
})
}
}
// askerRehearsalFor is how long the test's rehearsal waits.
const askerRehearsalFor = 15 * time.Minute
+47 -7
View File
@@ -180,12 +180,35 @@ func (f moveFacts) moves(node string, modules []string, sent map[string]string,
return out
}
// walkedBy is the open plan that has started walking a module's build — sent it to a first machine,
// not yet passed — other than to this machine; empty when none does.
func (f moveFacts) walkedBy(module, node string) string {
// walkedBy is the open plan that has started walking a module's build — sent it to a first machine, not
// yet passed — and whose walk this move would cross; empty when none does. A move of the same build to a
// machine that plan already sent it is not a crossing: the build is there. A move of **another** build of
// the module to that machine is (novox/hq issue 352): on 2026-10-09 a merge's plan sent the control node a
// newer controller while a release's gate was judging the controller there, the release's gate read the
// machine's report against the newer send, failed three builds and put them back under the new plan's
// feet. A release keeps no module records: its open gate's carried moves are its walk.
func (f moveFacts) walkedBy(module, node, to string) string {
for _, p := range f.plans {
if !p.Open() {
continue
}
if p.Release != nil {
g := p.Release.Gate
if g == nil || g.Verdict != "" {
continue
}
for _, c := range g.Carried {
if c.Module == module && !(slices.Contains(g.Machines, node) && sameCommit(c.To, to)) {
return p.ID
}
}
continue
}
s, holds := p.Modules[module]
if !p.Open() || !holds || s == nil || s.FirstAt == nil || s.SentAt != nil || slices.Contains(s.First, node) {
if !holds || s == nil || s.FirstAt == nil || s.SentAt != nil {
continue
}
if slices.Contains(s.First, node) && sameCommit(s.Commit, to) {
continue
}
if s.Gate != nil && s.Gate.Verdict == inventory.GatePassed {
@@ -277,11 +300,19 @@ func gatedSend(ctx context.Context, open *stores, node string, owns []inventory.
if own(mv.Module) {
continue
}
if id := f.walkedBy(mv.Module, node); id != "" {
if id := f.walkedBy(mv.Module, node, mv.To); id != "" {
return nil, nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
mv.Module, short(mv.To), node, id)
}
}
// **And the plan's own modules wait too** (novox/hq issue 352): a walk of the same module by another
// plan, or a release, on this machine is not crossed with a newer build; this send waits for its gate.
for _, o := range owns {
if id := f.walkedBy(o.Module, node, o.To); id != "" {
return nil, nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
o.Module, short(o.To), node, id)
}
}
for _, o := range owns {
i := slices.IndexFunc(moves, func(mv inventory.CarriedMove) bool { return mv.Module == o.Module })
switch {
@@ -526,7 +557,7 @@ func waitingMoves(ctx context.Context, open *stores, all bool) (map[string][]inv
return nil, err
}
for _, mv := range moves {
if f.walkedBy(mv.Module, n.Name) == "" {
if f.walkedBy(mv.Module, n.Name, mv.To) == "" {
out[n.Name] = append(out[n.Name], mv)
}
}
@@ -658,7 +689,7 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
r.Next++
continue
}
r.Gate = &inventory.PlanGate{Machines: sent, Since: &now, Carried: moves}
r.Gate = &inventory.PlanGate{Machines: sent, Since: &now, Carried: moves, Sent: sentNow(ctx, open.inventory, sent)}
p.Note = fmt.Sprintf("sent %s %d build(s) that waited for a gate; judging them there", node, len(moves))
if said := recreationsSaid(moves); said != "" {
p.Note += "; " + said
@@ -693,6 +724,15 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
r.Next++
r.Gate = nil
return true, nil
case inventory.GateSuperseded:
// Another send moved a judged module on the judged machine (novox/hq issue 352): no verdict on what
// was carried, nothing put back, and this release ends; the builds still waiting are released again
// by the next pass, judged afresh.
p.State = inventory.PlanSuperseded
p.Note = fmt.Sprintf("superseded on %s: %s — nothing judged, nothing put back; what still waits is released again",
strings.Join(g.Machines, ", "), g.Why)
fmt.Printf("%s: %s\n", p.ID, p.Note)
return true, nil
}
p.Note = ""
batched, back := batchingRollbacks(ctx)
+276 -96
View File
@@ -138,9 +138,11 @@ func reachableFrom(moved []string, edges []inventory.Edge) []string {
grew = false
for _, e := range edges {
// Built-by and worker-of order a plan; neither widens it. A new build machine changes
// nothing it builds, and a new controller changes nothing about the holder it orders —
// what packages the controller's source is already a code edge.
if e.Kind == inventory.EdgeBuiltBy || e.Kind == inventory.EdgeWorkerOf {
// nothing it builds, and a new controller changes nothing about the holder it orders. A
// packages edge, read from a record made before novox/hq ADR 0267, widens nothing either:
// a shared file moves each module whose build source holds it, directly.
if e.Kind == inventory.EdgeBuiltBy || e.Kind == inventory.EdgeWorkerOf || e.Kind == inventory.EdgePackages ||
e.Kind == edgeGroupOrder {
continue
}
if in[e.To] && !in[e.From] {
@@ -157,7 +159,9 @@ func reachableFrom(moved []string, edges []inventory.Edge) []string {
return out
}
// hasCycle says whether the tiers' last tier holds modules that still depend on each other.
// hasCycle says whether the tiers' last tier holds modules that still depend on each other. A packages
// edge orders nothing (tiersOf), so a module and what packages its source share a tier by rule: that is
// no cycle, and saying one was is a false report in every such plan's log.
func hasCycle(tiers [][]string, edges []inventory.Edge) bool {
if len(tiers) == 0 {
return false
@@ -167,6 +171,9 @@ func hasCycle(tiers [][]string, edges []inventory.Edge) bool {
last[m] = true
}
for _, e := range edges {
if e.Kind == inventory.EdgePackages {
continue
}
if last[e.From] && last[e.To] {
return true
}
@@ -174,9 +181,18 @@ func hasCycle(tiers [][]string, edges []inventory.Edge) bool {
return false
}
// planFor is the plan a merge produces: the moved modules and everything reachable from them,
// planOfMerge is the plan one merge produces: the moved modules and everything reachable from them,
// tiered, with the merge it answers.
func planOfMerge(m link.SourceMoved, moved []string, edges []inventory.Edge) inventory.Plan {
p := planOfMoves(moved, edges)
merged, _ := time.Parse(time.RFC3339Nano, m.MergedAt)
p.Repository, p.Branch, p.Commit, p.Merged = m.Owner+"/"+m.Repo, m.Base, m.Commit, merged.UTC()
return p
}
// planOfMoves is the walk of a set of moved modules (novox/hq ADR 0162, 0276): they and everything reachable
// from them, tiered along the graph, with no merge named yet.
func planOfMoves(moved []string, edges []inventory.Edge) inventory.Plan {
set := reachableFrom(moved, edges)
tiers := tiersOf(set, edges)
modules := map[string]*inventory.PlanModule{}
@@ -184,71 +200,14 @@ func planOfMerge(m link.SourceMoved, moved []string, edges []inventory.Edge) inv
modules[name] = &inventory.PlanModule{}
}
return inventory.Plan{
ID: fmt.Sprintf("plan-%d", time.Now().UnixNano()),
Repository: m.Owner + "/" + m.Repo,
Branch: m.Base,
Commit: m.Commit,
Created: time.Now().UTC(),
State: inventory.PlanBuilding,
Tiers: tiers,
Modules: modules,
ID: fmt.Sprintf("plan-%d", time.Now().UnixNano()),
Created: time.Now().UTC(),
State: inventory.PlanBuilding,
Tiers: tiers,
Modules: modules,
}
}
// supersededBy is what a newer plan takes over from the open plans it supersedes (novox/hq issue
// 254, ADR 0218): the modules they had not finished, and those plans closed as superseded.
//
// **A merge looked at no plan but its own.** Two merges of one repository a few minutes apart were
// two open plans asking for the same modules, each sending machines what it built; and a plan that
// would never move again — waiting on a report that could not come, at 97b1b2b — stayed open for
// ever beside the newer ones, read as work in progress by everyone who looked. The newer merge is the
// newer intent for that repository and branch, so its plan takes over: every open plan of the same
// repository and branch **created before it** — by the time the plans were made, never by comparing
// commits, which have no order of their own — gives up the modules it had not built, and those are
// planned again in the newer plan beside what the newer merge moved.
//
// "Not built" is a module not yet asked, or asked and not answered; **and a module built and not
// yet sent to its machines**, where its policy rolls it out: closed, the older plan would never send
// it, and the catalogue announces no move for a rebuild (issue 189), so the newer plan builds and
// sends it. A build the older plan asked still finishes and registers as any build does — ordered by
// when it was asked (issue 219), so the newer plan's ask, made later, is the one that stands.
//
// A plan with no branch recorded is from before branches were kept, and is superseded by the next
// plan of its repository: what it had not built is folded in, so nothing is lost by it.
func supersededBy(newer inventory.Plan, open []inventory.Plan, rollsOut func(string) bool) ([]string, []inventory.Plan) {
folded := map[string]bool{}
var closed []inventory.Plan
for _, old := range open {
if old.ID == newer.ID || !old.Open() || !strings.EqualFold(old.Repository, newer.Repository) ||
(old.Branch != "" && old.Branch != newer.Branch) || !old.Created.Before(newer.Created) {
continue
}
var took []string
for name, s := range old.Modules {
if s != nil && s.State == planDeleted {
continue // deleted at its source: nothing to plan again
}
if s == nil || s.State != "built" || (s.SentAt == nil && rollsOut(name)) {
folded[name] = true
took = append(took, name)
}
}
sort.Strings(took)
old.State = inventory.PlanSuperseded
old.Note = fmt.Sprintf("superseded at tier %d by %s (%s at %s)", old.Tier, newer.ID, newer.Repository, short(newer.Commit))
if len(took) > 0 {
old.Note += "; " + strings.Join(took, ", ") + " planned there again"
}
closed = append(closed, old)
}
out := make([]string, 0, len(folded))
for name := range folded {
out = append(out, name)
}
sort.Strings(out)
return out, closed
}
// gates is what the next tier needs running from this one: a module of the tier that a later
// tier is built by — the runtime dependency — and whose policy rolls it out, must be applied by
// the machines running it before the next tier is asked. A base an image stands on need only be
@@ -276,8 +235,11 @@ func gates(p inventory.Plan, edges []inventory.Edge, rollsOut func(string) bool)
seen := map[string]bool{}
var out []string
for _, e := range edges {
if later[e.From] && inTier[e.To] && e.Kind == inventory.EdgeBuiltBy && !seen[e.To] && rollsOut(e.To) &&
!isBaseOf(e.From, e.To, edges, all) {
// A delivery group's order inside a walk (novox/hq ADR 0276 decision 5) gates as built-by does: the
// member before is running on its machines before the member after is asked.
ordered := e.Kind == edgeGroupOrder ||
e.Kind == inventory.EdgeBuiltBy && !isBaseOf(e.From, e.To, edges, all)
if later[e.From] && inTier[e.To] && ordered && !seen[e.To] && rollsOut(e.To) {
seen[e.To] = true
out = append(out, e.To)
}
@@ -368,8 +330,15 @@ func askModule(ctx context.Context, p *inventory.Plan, name string, byName map[s
}
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
fmt.Printf(" tier %d: ", p.Tier)
// The branch it follows, never a commit a build once named (novox/hq 04-ISSUES/215).
id, err := askABuild(ctx, source, e.Source.Path, followedBranch(e.Source.Ref))
// The branch it follows, never a commit a build once named (novox/hq 04-ISSUES/215): the branch contains every
// commit a batch's walk carries — but for a merge walked alone after a failed walk (novox/hq ADR 0276
// decision 3), built on its own commit to find which merge brought the failure.
ref := followedBranch(e.Source.Ref)
carried := p.CommitOn(e.Source.Repository, ref)
if p.Delivery != nil && p.Delivery.Alone && carried != "" {
ref = carried
}
id, err := askABuild(ctx, source, e.Source.Path, ref)
if err != nil {
state.State = "failed"
state.Why = err.Error()
@@ -377,8 +346,14 @@ func askModule(ctx context.Context, p *inventory.Plan, name string, byName map[s
p.Note = fmt.Sprintf("%s could not be asked for: %v", name, err)
return
}
// The commit of the module's own repository the walk carries (novox/hq ADR 0276): a batch's walk carries
// one per repository.
commit := carried
if commit == "" {
commit = p.Commit
}
recordAsked(ctx, inventory.BuildRequest{ID: id, Repository: e.Source.Repository, Seat: e.Source.Seat,
Path: e.Source.Path, Ref: followedBranch(e.Source.Ref), Commit: p.Commit, For: "plan"})
Path: e.Source.Path, Ref: followedBranch(e.Source.Ref), Commit: commit, For: "plan"})
state.State = "asked"
state.AskedAt = &now
state.Build = id
@@ -471,6 +446,47 @@ func planBuilt(ctx context.Context, open *stores, module, commit, failed string,
advanceHeld(ctx, open)
}
// startedBeside is the id of a started walk open beside this one — a merge's walk past its wait, not the
// backlog's — or empty: one walk at a time (novox/hq ADR 0276).
func startedBeside(ctx context.Context, inv *inventory.Inventory, id string, created time.Time) (string, error) {
plans, err := inv.OpenPlans(ctx)
if err != nil {
return "", err
}
for _, q := range plans {
if q.ID == id || q.Release != nil || q.Waiting() {
continue
}
// Started: a tier asked, or on its way (let go, or waiting for nobody) before this one.
if q.Tier > 0 || askedAny(q) || q.Created.Before(created) {
return q.ID, nil
}
}
return "", nil
}
// deferredNote begins the note of a walk deferred behind another.
const deferredNote = "let go; starts once "
// deferred says a walk has its word and has not started: let go while another walk was started, it waits for
// that one to end (startedBeside), and its note says so. Read as a wait, not as a tier running late: `plans`
// and `status` say its note, and S3 leaves it out. A let-go walk whose first ask failed carries that error as
// its note instead, and is watched as before.
func deferred(p inventory.Plan) bool {
return p.Open() && p.Release == nil && p.Tier == 0 && !askedAny(p) && p.Delivery != nil &&
p.Delivery.Awaits != "" && p.Delivery.Go != nil && strings.HasPrefix(p.Note, deferredNote)
}
// askedAny says a plan asked any module.
func askedAny(p inventory.Plan) bool {
for _, s := range p.Modules {
if s != nil && s.State != "" {
return true
}
}
return false
}
// advancePlans moves every open plan as far as the facts allow: a tier whose modules are all built
// and whose gates are applied gives way to the next; the last tier done is the plan done. Called
// after every outcome and on a timer, so a plan waiting on a machine's report moves when it comes.
@@ -497,6 +513,12 @@ func advancePlans(ctx context.Context, open *stores) {
// advanceHeld is advancePlans for a caller already holding the plans.
func advanceHeld(ctx context.Context, open *stores) {
inv := open.inventory
// A walk that ended lets the next batch be cut at once, not at the cutter's next look (novox/hq ADR 0276).
defer func() {
if err := cutBatchesHeld(ctx, open, time.Now().UTC()); err != nil {
fmt.Printf("batches: %v\n", err)
}
}()
plans, err := inv.OpenPlans(ctx)
if err != nil {
fmt.Printf("plans: cannot read them: %v\n", err)
@@ -528,6 +550,10 @@ func advanceHeld(ctx context.Context, open *stores) {
// the state is left as it was and the step is tried again on the next tick. Said and
// kept when it is new: the same refusal on every tick is one fact, not one per tick.
p.Note = "tier " + fmt.Sprint(p.Tier) + ": " + err.Error() + " — tried again"
// A refusal for the bus's planned step is a person's to end: S17 says it from its first moment.
if refusedForTheBus(err) {
busRefusedFirst.mark(p.ID, time.Now())
}
if planSnapshot(*p) != before {
fmt.Printf("%s: %v\n", p.ID, err)
if err := inv.SavePlan(ctx, p); err != nil {
@@ -593,6 +619,18 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
}
}
if unasked == len(tier) {
// **One walk at a time** (novox/hq ADR 0276): a walk about to ask its first tier while another started
// walk is open waits for that one to end, let go or not, and says so.
if p.Tier == 0 {
if behind, err := startedBeside(ctx, inv, p.ID, p.Created); err != nil {
return false, err
} else if behind != "" {
note := fmt.Sprintf("%s%s ended — one walk at a time", deferredNote, behind)
changed := p.Note != note
p.Note = note
return changed, nil
}
}
if err := askTier(ctx, inv, p); err != nil {
return false, err
}
@@ -701,7 +739,6 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
}
}
now := time.Now().UTC()
step := nextRollout(*state, running, policy.Together, reports, now, planWaitBound)
// **Sent with others, judged with them** (issue 281): the gate of the send that carried it is its
// verdict on its first machine. A failure there stopped the plan already.
if state.GatedBy != "" {
@@ -715,6 +752,9 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
}
passedWith(m, state, state.GatedBy, lead.Gate)
}
// Read after its pass is taken over from the send that carried it, so a passed gate is never judged
// again from the first machine's later reports (novox/hq issue 335).
step := nextRollout(*state, running, policy.Together, reports, now, planWaitBound)
switch {
case step.failed != "":
// The first machine refused or failed what it was sent, or never said: the gate failed, and
@@ -748,6 +788,15 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
case inventory.GateFailed:
failFirstSend(ctx, open, p, m, state, state.Gate.Machines, state.Gate.Why, step.rest)
return true, nil
case inventory.GateSuperseded:
// Another send moved this module on its first machine (novox/hq issue 352): the build is not
// judged, not marked, not put back; the plan ends here, said, and a newer plan carries on.
state.Why = "superseded: " + state.Gate.Why
p.State = inventory.PlanSuperseded
p.Note = fmt.Sprintf("%s's judging on %s was superseded: %s", m, strings.Join(state.Gate.Machines, ", "),
state.Gate.Why)
fmt.Printf("%s: %s\n", p.ID, p.Note)
return true, nil
case inventory.GatePassed:
if !state.Gate.Kept {
gatePassed(ctx, open, p, m, state)
@@ -919,6 +968,9 @@ func firstSend(ctx context.Context, open *stores, p *inventory.Plan, node string
}
now := time.Now().UTC()
lead := modules[0]
// What each machine was just sent, kept on the gate (novox/hq issue 352): its report is held against
// this send, whatever it is sent after.
sentWhat := sentNow(ctx, inv, sent)
for _, m := range modules {
s := p.Modules[m]
// What the first machine ran before: what a failed gate puts back (ADR 0236).
@@ -927,7 +979,7 @@ func firstSend(ctx context.Context, open *stores, p *inventory.Plan, node string
}
s.First, s.FirstAt = sent, &now
s.Gate = &inventory.PlanGate{Component: coreComponent(m), Machines: firstRunning(sent, runningOf[m]),
From: s.Previous, To: s.Commit, Since: &now}
From: s.Previous, To: s.Commit, Since: &now, Sent: sentWhat}
s.GatedBy = ""
if m == lead {
s.Gate.Carried = carried
@@ -974,6 +1026,19 @@ func failFirstSend(ctx context.Context, open *stores, p *inventory.Plan, module
fmt.Printf("%s: %s\n", p.ID, p.Note)
}()
g := state.Gate
if g != nil && g.Verdict == inventory.GatePassed {
// **A guard: a build that passed its gate is never put back for what came after** (novox/hq issue 335).
// Not reached while nextRollout answers a passed gate with the rest to send, and advanceOnce reads it
// after a carried module takes over its lead's pass; it is here so that a path added later cannot
// overturn a verdict. If it is reached, the plan stops and says why, and the build is not marked failed.
// The module is left a stopped rollout (its Why said, sent first and not to the rest), which
// `plans retry` takes: it sends the first machine again, and the passed gate then sends the rest.
state.Why = "passed its gate; its walk then stopped: " + why
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("%s passed its gate on %s in tier %d (%s) and is kept; its walk stopped after: %s",
module, strings.Join(g.Machines, ", "), p.Tier, g.Why, why)
return
}
if g != nil && slices.Contains(g.Returned, module) {
// Put back at once when it broke: its rollback was made then, and is not made again.
state.Why = "put back when it broke; its send failed: " + g.Why
@@ -1061,11 +1126,27 @@ func nextRollout(s inventory.PlanModule, running []string, together bool, report
rest = append(rest, n)
}
}
// **A passed gate is the first machine's verdict, and its later reports are not** (novox/hq issue 335).
// Once the build passed there, what that machine reports next is about whatever it was sent after —
// another walk's send, a push — and says nothing of this build. On 2026-10-08 a build passed on the
// laptop, its send to the rest waited on another walk, that walk sent the laptop a new declaration it did
// not report for half an hour, and the plan read the silence as the first machine never applying the
// passed build: it marked it failed at its gate and put it back. The rest are sent, as the pass said.
if s.Gate != nil && s.Gate.Verdict == inventory.GatePassed {
return rolloutStep{send: rest}
}
var waiting, failed []string
for _, n := range s.First {
r, said := byNode[n]
// Only a report about what it was last sent says anything about this build.
if !said || r.At == nil || !r.Current {
// Only a report about what this plan sent it — or what it was sent after that — says anything about
// this build (novox/hq issue 352); a plan from before sends were kept on the gate reads the report
// against the send made last, as before.
reported := r.Current
if s.Gate != nil && s.Gate.Sent != nil {
sent, kept := s.Gate.Sent[n]
reported = kept && sent.ReportsOn(r)
}
if !said || r.At == nil || !reported {
waiting = append(waiting, n)
continue
}
@@ -1153,24 +1234,38 @@ func inTierSince(p inventory.Plan) time.Time {
// planLineWith is planLine knowing whether the build seat is paused (novox/hq ADR 0219): a plan
// waiting on builds nobody will take until a person resumes the seat says so, and is not late. bound is
// the plan's tier bound, the one its stalled condition is raised at (tierBounds): LATE is that condition
// said on the line (novox/hq issue 296).
// said on the line (novox/hq issue 296). The machines running what it moves are not named: planLineOn.
func planLineWith(p inventory.Plan, now time.Time, pause pauseView, bound time.Duration) string {
return planLineOn(p, now, pause, bound, nil)
}
// planLineOn is planLineWith naming the plan by what it moves and where (planHeadline), given what runs each
// module; nil names no machine.
func planLineOn(p inventory.Plan, now time.Time, pause pauseView, bound time.Duration, running func(string) []string) string {
name := planHeadline(p, running)
where := fmt.Sprintf("tier %d of %d", min(p.Tier+1, len(p.Tiers)), len(p.Tiers))
switch p.State {
case inventory.PlanAssembling, inventory.PlanQueued:
// A batch not yet a walk (novox/hq ADR 0276): what it holds and how long is left.
return batchWords(p, now)
case inventory.PlanDone:
return fmt.Sprintf("%s %s done, %d tier(s)", p.Repository, short(p.Commit), len(p.Tiers))
return fmt.Sprintf("%s · done, %d tier(s)", name, len(p.Tiers))
case inventory.PlanFailed:
return fmt.Sprintf("%s %s FAILED at %s: %s", p.Repository, short(p.Commit), where, p.Note)
return fmt.Sprintf("%s · FAILED at %s: %s", name, where, p.Note)
case inventory.PlanSuperseded:
return fmt.Sprintf("%s %s %s", p.Repository, short(p.Commit), p.Note)
return fmt.Sprintf("%s · %s", name, p.Note)
}
since := now.Sub(inTierSince(p)).Round(time.Second)
if p.Waiting() {
// Waiting for its delivery's word is no lateness of the walk's (novox/hq ADR 0239).
return fmt.Sprintf("%s %s %s, %s, for %s", p.Repository, short(p.Commit), where, waitingNote(p), since)
return fmt.Sprintf("%s · %s, %s, for %s", name, where, waitingNote(p), since)
}
if deferred(p) {
// Nor is waiting for the walk before it to end (one walk at a time, novox/hq ADR 0276).
return fmt.Sprintf("%s · %s, %s, for %s", name, where, p.Note, since)
}
if waiting, paused := pausedWaiting(p, pause, now); paused {
return fmt.Sprintf("%s %s %s, %s", p.Repository, short(p.Commit), where, waiting)
return fmt.Sprintf("%s · %s, %s", name, where, waiting)
}
late := ""
if since > bound {
@@ -1180,7 +1275,53 @@ func planLineWith(p inventory.Plan, now time.Time, pause pauseView, bound time.D
if p.State == inventory.PlanRolling {
what = p.Note
}
return fmt.Sprintf("%s %s %s, %s for %s%s", p.Repository, short(p.Commit), where, what, since, late)
return fmt.Sprintf("%s · %s, %s for %s%s", name, where, what, since, late)
}
// planHeadline names a plan as a person knows it (asked by the operator, 2026-10-10: a plan called by its
// repository alone said nothing of what it delivers): each repository as its pull request and title, what the
// plan moves, and the machines running that — "mesh-catalog #175 a tap shows its outcome · messenger,
// telegram → novox". A record naming no pull request is named by its commit, as before; one naming no moves
// says none; running nil names no machine.
func planHeadline(p inventory.Plan, running func(string) []string) string {
var repos []string
moves := map[string]bool{}
for _, c := range p.Carried() {
seg := repoName(c.Repository) + " " + short(c.Commit)
if p.Delivery != nil {
for _, m := range p.Delivery.Merges {
if !strings.EqualFold(m.Repository, c.Repository) {
continue
}
for _, n := range m.Moves {
moves[n] = true
}
if m.Commit == c.Commit && m.Number > 0 {
seg = repoName(c.Repository) + " " + pullWords(m)
}
}
}
repos = append(repos, seg)
}
out := strings.Join(repos, " + ")
if len(moves) == 0 {
return out
}
names := sortedKeysOf(boolsToStrings(moves))
out += " · " + strings.Join(names, ", ")
if running == nil {
return out
}
nodes := map[string]bool{}
for _, n := range names {
for _, node := range running(n) {
nodes[node] = true
}
}
if len(nodes) > 0 {
out += " → " + strings.Join(sortedKeysOf(boolsToStrings(nodes)), ", ")
}
return out
}
// planFailedBuild marks the module a failed build was for when the result names no module: by the
@@ -1322,8 +1463,11 @@ func plansCommand(ctx context.Context, args []string) error {
return err
}
bounds := readTierBounds(ctx, inv, now)
fmt.Printf("%s — %s\n", p.ID, planLineWith(p, now, buildSeatPause(ctx, inv, []inventory.Plan{p}),
bounds.of(p.Repository)))
fmt.Printf("%s — %s\n", p.ID, planLineOn(p, now, buildSeatPause(ctx, inv, []inventory.Plan{p}),
bounds.of(p.Repository), func(module string) []string {
on, _ := inv.Running(ctx, module)
return on
}))
if r := p.Release; r != nil {
// A release plan's walk (ADR 0236): machines done, the one judged, those to come.
fmt.Printf(" machines in order: %s; done: %s; skipped: %s\n", strings.Join(r.Order, ", "),
@@ -1447,14 +1591,34 @@ func plansCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
if len(plans) == 0 {
// **The batch being assembled first** (novox/hq ADR 0276 decision 7): what it holds, how long is left,
// and that its plan is not yet calculated.
batches, err := inv.Batches(ctx)
if err != nil {
return err
}
if len(plans) == 0 && len(batches) == 0 {
fmt.Println("no merge has produced a plan yet")
return nil
}
for _, b := range batches {
fmt.Printf("%-28s %s\n", b.ID, batchWords(b, now))
// One line per repository: its pull request, what it answers, what it moves.
for _, line := range batchLines(b) {
fmt.Printf("%-28s %s\n", "", line)
}
}
pause := buildSeatPause(ctx, inv, plans)
bounds := readTierBounds(ctx, inv, now)
running := func(module string) []string {
on, _ := inv.Running(ctx, module)
return on
}
for _, p := range plans {
fmt.Printf("%-28s %s\n", p.ID, planLineWith(p, now, pause, bounds.of(p.Repository)))
if p.Batch() {
continue
}
fmt.Printf("%-28s %s\n", p.ID, planLineOn(p, now, pause, bounds.of(p.Repository), running))
}
return nil
}
@@ -1475,11 +1639,12 @@ func planWhatIf(ctx context.Context, inv *inventory.Inventory, repository string
if err != nil {
return err
}
read, err := inv.ReadRepositories(ctx)
read, err := readForPlanning(ctx, inv)
if err != nil {
return err
}
var from, packaging []inventory.Entry
deleted := map[string]bool{}
named := map[string]bool{}
for _, name := range modules {
named[name] = true
@@ -1489,6 +1654,9 @@ func planWhatIf(ctx context.Context, inv *inventory.Inventory, repository string
// request's check ask it (novox/hq ADR 0238).
r := reachOfMerge(m, entries, read, nil)
from, packaging = append(append([]inventory.Entry{}, r.Touched...), r.Deleted...), r.Packaging
for _, e := range r.Deleted {
deleted[e.Manifest.Module] = true
}
} else {
for _, e := range entries {
switch {
@@ -1515,6 +1683,18 @@ func planWhatIf(ctx context.Context, inv *inventory.Inventory, repository string
}
p := planOfMerge(m, names, edges)
fmt.Printf("a merge of %s would build %d module(s) in %d tier(s):\n", repository, len(p.Modules), len(p.Tiers))
why := map[string]string{}
for _, e := range packaging {
why[e.Manifest.Module] = whyMoved(e, read[e.Manifest.Module], m)
}
if len(named) == 0 {
for _, e := range from {
why[e.Manifest.Module] = whyMoved(e, read[e.Manifest.Module], m)
if deleted[e.Manifest.Module] {
why[e.Manifest.Module] = "its manifest is removed: deleted at its source, forgotten where nothing holds it, not built"
}
}
}
rolls := map[string]string{}
for i, tier := range p.Tiers {
fmt.Printf(" tier %d\n", i)
@@ -1532,19 +1712,19 @@ func planWhatIf(ctx context.Context, inv *inventory.Inventory, repository string
rolls[name] = how
}
fmt.Printf(" %-22s %s\n", name, how)
reason := why[name]
switch {
case reason == "" && len(named) > 0 && named[name]:
reason = "named"
case reason == "":
reason = "it stands on a module the merge moves"
}
fmt.Printf(" %-22s why: %s\n", "", reason)
}
}
if hasCycle(p.Tiers, edges) {
fmt.Println(" the last tier depends on itself and would be built together, in no order")
}
if len(packaging) > 0 {
var also []string
for _, e := range packaging {
also = append(also, e.Manifest.Module)
}
fmt.Printf(" %s package source from %s, so they are rebuilt without their own source moving\n",
strings.Join(also, ", "), repository)
}
return nil
}
+16 -10
View File
@@ -27,6 +27,8 @@ func TestAMergeIsPlannedInTiersAlongTheThreeKindsOfDependency(t *testing.T) {
{From: "mesh-controller", To: "builder", Kind: inventory.EdgeBuiltBy},
{From: "mesh-tools", To: "builder", Kind: inventory.EdgeBuiltBy},
{From: "builder", To: "mesh-tools", Kind: inventory.EdgeStandsOn},
// the build seat's holder follows the controller that defines its worker (hq issue 206)
{From: "builder", To: "mesh-controller", Kind: inventory.EdgeWorkerOf},
{From: "unrelated", To: "alpine", Kind: inventory.EdgeStandsOn},
}
// The runtime image moved: everything on it, and what is built by what is on it.
@@ -57,17 +59,21 @@ func TestAMergeIsPlannedInTiersAlongTheThreeKindsOfDependency(t *testing.T) {
t.Fatalf("no cycle here: %v", tiers)
}
// The controller alone moved: the proxy with it, nothing else.
small := reachableFrom([]string{"mesh-controller"}, edges)
if len(small) != 3 {
t.Fatalf("a controller merge rebuilds the controller and what packages it: %v", small)
// The controller alone moved: the controller alone — what packages its repository moves only when the
// change is in its own build source (novox/hq ADR 0267), so a packages edge widens nothing.
if alone := reachableFrom([]string{"mesh-controller"}, edges); len(alone) != 1 {
t.Fatalf("a controller merge rebuilds the controller alone: %v", alone)
}
// The builder packages the controller's source (same tier by that edge) and the controller is
// built by the builder (next tier by that one): the builder first, then the controller and the
// proxy together — a code dependency in one tier, a runtime dependency across tiers.
// A change to a package all three build from moves all three. The builder holds
// the build seat, whose worker the controller defines, so it follows the controller (worker-of,
// novox/hq issue 206), and the controller's built-by edge to it yields: the controller is built by the
// build machine that is running. The proxy is built by the new builder: the controller, the builder,
// the proxy — the live plan of every controller merge. (This read "the builder, then the controller
// and the proxy together" before issue 206, and the fixture had no worker-of edge.)
small := reachableFrom([]string{"mesh-controller", "builder", "route-proxy"}, edges)
smallTiers := tiersOf(small, edges)
if len(smallTiers) != 2 || smallTiers[0][0] != "builder" || len(smallTiers[1]) != 2 {
t.Fatalf("the builder, then the controller and the proxy together: %v", smallTiers)
if got := tiered(smallTiers); got != "mesh-controller | builder | route-proxy" {
t.Fatalf("the controller, then the builder, then the proxy: %v", smallTiers)
}
// The builder alone moved: the builder, and nothing it builds.
if only := reachableFrom([]string{"builder"}, edges); len(only) != 1 {
@@ -238,7 +244,7 @@ func TestAChangeToTheBuildAgentRebuildsTheBuildAgentAlone(t *testing.T) {
} {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "abc", Paths: paths,
ModuleDirs: []string{"modules/build-agent"}, ModuleDirsSaid: true}
touched := whatTheMergeTouched(entries, entries, m)
touched := whatTheMergeTouched(entries, entries, m, nil)
if len(touched) != 1 || touched[0].Manifest.Module != "build-agent" {
t.Fatalf("%v touched %v", paths, touched)
}
+106
View File
@@ -0,0 +1,106 @@
package main
import (
"context"
"encoding/json"
"os/exec"
"path/filepath"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/testbus"
)
// novox/hq issue 327, replayed with only what the controller had before its fix, so it can be laid over
// the older commit. On 2026-10-08 the bus's connection total rose by 5.2 a minute while the same 16
// connections stayed open, and three calls of `conditions` in one second added three: each verb ran as a
// process of the controller's own binary, which dialled the bus, logged in and left. The operator's
// channel reads `conditions` at least once a minute. A verb that only reads, served by the serving
// controller, opens no connection of its own.
func TestReplay327(t *testing.T) {
bus := testbus.Start(t)
serving, err := broker.Dial(bus.ClientURL())
if err != nil {
t.Fatal(err)
}
defer serving.Close()
if err := serving.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
keeper, err := keeperOn(context.Background(), serving.Conn())
if err != nil {
t.Fatal(err)
}
// The serving controller: its keeper and its connection, as serve sets them.
keptBefore, connBefore := conditionsFrom, handActConn
conditionsFrom, handActConn = keeper, serving.Conn()
defer func() { conditionsFrom, handActConn = keptBefore, connBefore }()
// A verb that runs as a process of its own runs this controller's binary, on this bus.
asAProcess(t, bus.ClientURL())
handlers, _, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
accepted := func() uint64 {
v, err := bus.Varz(nil)
if err != nil {
t.Fatal(err)
}
return v.TotalConnections
}
before := accepted()
for i := 0; i < 3; i++ {
answer, err := handlers["conditions"](context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if a, ok := answer.(verbAnswer); !ok || !a.OK {
t.Fatalf("conditions answered %+v", answer)
}
}
if opened := accepted() - before; opened != 0 {
t.Fatalf("three conditions calls opened %d connection(s) to the bus; the serving controller is on it already",
opened)
}
}
// asAProcess makes a verb that runs as a process of its own run this package's binary, on the bus at url:
// built into the test's own directory, which goes with the test.
func asAProcess(t *testing.T, url string) {
t.Helper()
path := filepath.Join(t.TempDir(), "mesh-controller")
if out, err := exec.Command("go", "build", "-o", path, ".").CombinedOutput(); err != nil {
t.Fatalf("the controller could not be built to run a verb as its own process: %v: %s", err, out)
}
was := ownImage
ownImage = func() string { return path }
t.Cleanup(func() { ownImage = was })
t.Setenv(broker.NATSVar, url)
t.Setenv(broker.CertificateVar, "")
}
// closedNames are the names of the connections the bus saw closed.
func closedNames(t *testing.T, bus *server.Server) []string {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
var names []string
for time.Now().Before(deadline) {
connz, err := bus.Connz(&server.ConnzOptions{State: server.ConnClosed})
if err != nil {
t.Fatal(err)
}
names = names[:0]
for _, c := range connz.Conns {
names = append(names, c.Name)
}
if len(names) > 0 {
return names
}
time.Sleep(50 * time.Millisecond)
}
return names
}
+132
View File
@@ -0,0 +1,132 @@
package main
import (
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// TestReplay335 replays novox/hq issue 335 (2026-10-08): dunst's new build passed its gate on the laptop
// (healthy 3 times over 2m5s); its send to the rest was refused while another walk's build waited on the
// workstation; that walk then sent the laptop a declaration the laptop did not report on; and thirty minutes
// after the first send the plan read the laptop's silence as the passed build never applied, marked it failed
// at its gate with the pass's own words, and put it back. Written with only what the controller had before
// its fix, so it is laid over the commit before.
func TestReplay335(t *testing.T) {
t.Run("the first machine's later reports", testAPassedGateIsNotJudgedAgainFromTheFirstMachinesLaterReports)
t.Run("a walk stopped after the pass", testAWalkStoppedAfterItsGatePassedKeepsThePass)
}
// novox/hq issue 335: a build that passed its gate on its first machine is not judged again from that
// machine's later reports. On 2026-10-08 the send to the rest waited on another walk, that walk sent the
// first machine a declaration it did not report for half an hour, and the plan failed the passed build at
// the wait's bound and put it back.
func testAPassedGateIsNotJudgedAgainFromTheFirstMachinesLaterReports(t *testing.T) {
sentAt := time.Date(2026, 10, 8, 16, 44, 45, 0, time.UTC)
judged := sentAt.Add(2 * time.Minute)
later := sentAt.Add(5 * time.Minute)
state := inventory.PlanModule{First: []string{"laptop"}, FirstAt: &sentAt,
Gate: &inventory.PlanGate{Machines: []string{"laptop"}, Verdict: inventory.GatePassed,
Why: "healthy 3 times over 2m5s", JudgedAt: &judged, Kept: true}}
running := []string{"laptop", "workstation"}
now := sentAt.Add(30*time.Minute + 9*time.Second)
for what, reports := range map[string][]inventory.Reported{
"no report about what it was sent since": {{Node: "laptop", At: &later, Outcome: inventory.OutcomeApplied, Current: false}},
"another send failed there since": {{Node: "laptop", At: &later, Outcome: inventory.OutcomeFailed, Current: true}},
"no report at all": nil,
} {
step := nextRollout(state, running, false, reports, now, 30*time.Minute)
if step.failed != "" || step.waiting != "" || !reflect.DeepEqual(step.send, []string{"workstation"}) {
t.Errorf("%s: %+v, want the rest sent as the pass said", what, step)
}
}
}
// novox/hq issue 335: whatever stops a walk after its gate passed, the passed build is not marked failed at
// its gate, nor put back.
func testAWalkStoppedAfterItsGatePassedKeepsThePass(t *testing.T) {
sentAt := time.Date(2026, 10, 8, 16, 44, 45, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"laptop"}, Verdict: inventory.GatePassed, Why: "healthy 3 times over 2m5s"}
state := &inventory.PlanModule{Build: "build-1", First: []string{"laptop"}, FirstAt: &sentAt, Gate: g}
p := &inventory.Plan{ID: "plan-1", Modules: map[string]*inventory.PlanModule{"dunst": state}}
// No stores: a walk that keeps the pass touches none, and one that reaches for them is putting it back.
defer func() {
if r := recover(); r != nil {
t.Fatalf("the passed build was taken to be failed and put back: %v", r)
}
}()
failFirstSend(t.Context(), nil, p, "dunst", state, []string{"laptop"}, "laptop did not report it applied within 30m0s",
[]string{"workstation"})
if g.Verdict != inventory.GatePassed || g.Rollback != "" {
t.Fatalf("the passed gate became %q, rollback %q", g.Verdict, g.Rollback)
}
if strings.Contains(p.Note, "failed its gate") || strings.Contains(p.Note, "put back") ||
!strings.Contains(p.Note, "did not report it applied") {
t.Fatalf("the note reads %q", p.Note)
}
}
// novox/hq issue 335 review: **a module carried by its lead's send takes over the lead's pass before its own
// step is read.** The state is the one a step leaves when the lead's gate passed and the step ended before the
// carried module's turn (an error read before it, kept with the plan): the lead passed, the carried module has
// no gate of its own yet. Since then another send reached the first machine and it has not reported on it, and
// the first send is older than the wait for a first machine's report. Read before the pass is taken over, the
// carried module's step said the first machine never applied it, and the passed build was put back.
func TestACarriedModuleTakesItsLeadsPassBeforeItsStepIsRead(t *testing.T) {
tm := aTierMesh(t, "m01", "m02")
ctx := t.Context()
inv := tm.open.inventory
advancePlans(ctx, tm.open)
if !reflect.DeepEqual(tm.sent, [][]string{{"anchor"}}) {
t.Fatalf("sent %v: the first machine once, for the tier", tm.sent)
}
p := tm.plan(t)
lead, carried := p.Modules["m01"], p.Modules["m02"]
if lead.Gate == nil || carried.GatedBy != "m01" {
t.Fatalf("m02 is not carried by m01's send: lead %+v, carried %+v", lead.Gate, carried)
}
// The lead passed; the carried module's turn did not come. The first send is past the wait's bound.
sent := time.Now().UTC().Add(-planWaitBound - time.Minute)
judged := sent.Add(2 * time.Minute)
lead.FirstAt, carried.FirstAt, lead.Gate.Since = &sent, &sent, &sent
lead.Gate.Verdict, lead.Gate.Why, lead.Gate.JudgedAt, lead.Gate.Kept = inventory.GatePassed,
"healthy 3 times over 2m5s", &judged, true
carried.Gate = nil
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
// Another walk's send reached anchor, which has not reported on it.
if err := inv.RecordSent(ctx, nodeID(t, tm.open, "anchor"), "d-anchor-elsewhere",
map[string]string{"m01": "c2", "m02": "c2"}); err != nil {
t.Fatal(err)
}
advancePlans(ctx, tm.open)
p = tm.plan(t)
if p.State == inventory.PlanFailed {
t.Fatalf("the plan failed after its gate passed: %s", p.Note)
}
if !reflect.DeepEqual(tm.sent, [][]string{{"anchor"}, {"laptop"}}) {
t.Fatalf("sent %v: the rest once, as the pass said", tm.sent)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
t.Fatal(err)
}
for _, m := range []string{"m01", "m02"} {
if current[m].Commit != "c2" {
t.Errorf("%s was put back to %s after its gate passed", m, current[m].Commit)
}
if failed, _ := inv.GateFailed(ctx, "build-"+m+"-2"); failed {
t.Errorf("%s's build was marked failed at its gate after the gate passed", m)
}
}
if g := p.Modules["m02"].Gate; g == nil || g.Verdict != inventory.GatePassed {
t.Errorf("m02 did not take over m01's pass: %+v", g)
}
}
+92
View File
@@ -0,0 +1,92 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// TestReplay336 replays novox/hq issue 336 (2026-10-08): a catalogue merge built a new bus build for the
// control-node; from then on every send to that machine was refused for the bus's planned step, which only a
// person starts, and for 28 minutes nothing but the walks' notes said so. The outcome asserted: the first
// watchdog tick after the first refusal opens a condition for the operator that names what waits and the verb
// that ends it, and it clears once the bus's machine runs the new build. Written with only what the controller
// had before its fix — the stores, register, assign, a plan saved and advanced, the watchdogs' gathering and
// seeing — so it is laid over the commit before.
func TestReplay336(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
bus := catalogue.Manifest{Module: "nats", Version: "2", Provides: []catalogue.Offer{{Name: "mesh-bus"}}}
if err := inv.RegisterModule(ctx, bus, inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/nats", BuiltFrom: "32307bd1", Head: "32307bd1"}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "engine", Version: "1"})
for _, m := range []string{"nats", "engine"} {
if _, err := inv.Assign(ctx, "anchor", m); err != nil {
t.Fatal(err)
}
}
// The control-node runs the bus build it was last sent; the mesh holds a newer one, which waits for its step.
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"nats": "88135ad0"}); err != nil {
t.Fatal(err)
}
// A walk of the engine, built, whose tier sends to the control-node.
now := time.Now().UTC()
plan := inventory.Plan{ID: "plan-engine", Repository: "novox/mesh-host", Commit: "e1e1e1e1", Created: now,
State: inventory.PlanBuilding, Tiers: [][]string{{"engine"}},
Modules: map[string]*inventory.PlanModule{"engine": {State: "built", BuiltAt: &now, Commit: "e1e1e1e1", Build: "b"}}}
if err := inv.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
advancePlans(ctx, open)
p, err := inv.PlanByID(ctx, "plan-engine")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(p.Note, errBusWaits.Error()) {
t.Fatalf("the walk's send to the bus's machine was not refused for the bus: %s %q", p.State, p.Note)
}
store := conditions.NewInMemory()
k := conditions.NewKeeper(ctx, conditions.Options{Store: store, History: store, Teller: &conditions.Told{}})
defer k.Close(context.Background())
w := &watchdogs{open: open, keeper: k, started: now.Add(-time.Hour)}
waiting := func() []conditions.Condition {
t.Helper()
w.see(ctx, w.gather(ctx))
all, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
var out []conditions.Condition
for _, c := range all {
if strings.HasPrefix(c.Key, "bus.") && c.Resolver == conditions.ResolverOperator {
out = append(out, c)
}
}
return out
}
got := waiting()
if len(got) != 1 {
t.Fatalf("the first tick after the refusal told the operator nothing about the bus's step: %d condition(s)", len(got))
}
for _, want := range []string{"anchor", "engine", "mesh-controller.bus", "upgrade", "32307bd1"} {
if !strings.Contains(got[0].Summary, want) {
t.Errorf("the condition does not say %q: %s", want, got[0].Summary)
}
}
// The step taken: the control-node is sent the new bus build, and the wait is over.
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor-2", map[string]string{"nats": "32307bd1"}); err != nil {
t.Fatal(err)
}
if got := waiting(); len(got) != 0 {
t.Fatalf("the bus's machine runs the new build, and the operator is still asked: %+v", got)
}
}
+2 -2
View File
@@ -539,8 +539,8 @@ func TestReplay301APersonsPushOfAHeldRecordedBuildIsNoRepair(t *testing.T) {
inv := open.inventory
start := time.Now().Add(-time.Hour)
image := "registry.invalid:5000/resolver/server@sha256:" + strings.Repeat("e", 64)
if _, _, err := takeIn(ctx, inv, aContainerBuild(t, "resolver", "c1111111", catalogue.PolicyRecord, start,
map[string][2]string{"server": {image, ""}})); err != nil {
if _, _, err := takeIn(ctx, inv, asTheOperator(t, inv, aContainerBuild(t, "resolver", "c1111111", catalogue.PolicyRecord, start,
map[string][2]string{"server": {image, ""}}))); err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
+4 -1
View File
@@ -131,7 +131,10 @@ func readinessOf(ctx context.Context, inv *inventory.Inventory) (broker.Readines
// **Dialled the way the mesh dials it** — credential and pin — because a bare connect to a
// bus that requires TLS and a user fails at the handshake, and the check then reported a
// standing server as absent (seen live, 2026-09-28).
if js, err := broker.Dial(address, nats.Timeout(5*time.Second)); err == nil {
// The serving controller's own connection answers it without a second (novox/hq issue 327).
if serving := servingBus.Load(); serving != nil && serving.Conn().IsConnected() {
state.ServerStanding = true
} else if js, err := broker.Dial(address, nats.Timeout(5*time.Second)); err == nil {
state.ServerStanding = true
js.Close()
}
+2 -2
View File
@@ -54,7 +54,7 @@ func TestARebuildOfAnUnchangedSourceKeepsItsArtifacts(t *testing.T) {
// Another module's merge rebuilt it: a new commit, a new image digest, the same source.
anImageBuild(t, "app", "", "c2bbbbbb", strings.Repeat("b", 64), "src1:same", start.Add(time.Minute)),
} {
if _, _, err := takeIn(ctx, inv, b); err != nil {
if _, _, err := takeIn(ctx, inv, asTheOperator(t, inv, b)); err != nil {
t.Fatalf("build %d: %v", i, err)
}
}
@@ -114,7 +114,7 @@ func TestABusRebuiltFromAnUnchangedSourceDemandsNoBusStep(t *testing.T) {
b.Manifest = manifest
return b
}
if _, _, err := takeIn(ctx, inv, bus("", "n1111111", strings.Repeat("a", 64), "src1:bus", start)); err != nil {
if _, _, err := takeIn(ctx, inv, asTheOperator(t, inv, bus("", "n1111111", strings.Repeat("a", 64), "src1:bus", start))); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "anchor", "nats"); err != nil {
+11
View File
@@ -26,6 +26,17 @@ import (
// shape fails the suite, naming its source. The keeper would say such a summary in words at run time;
// this is where the producer is made to say it rightly in the first place.
func TestMain(m *testing.M) {
// The process a mesh-cli test runs as a command line: it says the verb and the caller it was given, and
// ends (meshcli_test.go).
// The process a mesh-cli test runs as `secret accept`: it reads its value exactly as `secret accept` does
// (valueFor) and says whether it is the one the test gave, never the value (meshcli_stdin_test.go).
if want := os.Getenv(secretAcceptWants); want != "" {
os.Exit(readAsSecretAccept(want, os.Args[1:]))
}
if os.Getenv(echoEnvironment) != "" {
fmt.Printf("verb=%q caller=%q terminal=%v\n", os.Getenv("MESH_VERB"), os.Getenv("MESH_CALLER"), startedAtTheTerminal())
os.Exit(0)
}
conditions.Unsayable = func(o conditions.Observation, field string, r outward.Refusal) {
unsaid.note(o, field, r)
}
+6
View File
@@ -135,6 +135,12 @@ func handOver(ctx context.Context, seatName, to string, adding bool) error {
if !ok || nodeName == "" || module == "" {
return fmt.Errorf("the new holder is named <node>/<module>, not %q", to)
}
// A kinded bench is held once per kind, by the claims themselves (novox/hq ADR 0234 §2, ADR 0259): the
// record of who holds a seat has no kind, so a handover would name one holder for every kind.
if catalogue.KindedBenches[seatName] {
return fmt.Errorf("%s is a kinded bench: each kind is held by the module claiming it, and is not handed "+
"over by `seat` — assign the module that claims the kind, or unassign the one that does", seatName)
}
open, err := openStores(ctx)
if err != nil {
return err
+12
View File
@@ -1,6 +1,8 @@
package main
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
@@ -62,3 +64,13 @@ func TestAClaimOutsideTheSetIsShownNotHidden(t *testing.T) {
t.Fatalf("a claim outside the set was not shown: %+v", outside)
}
}
// A kinded bench is not handed over by `seat`: each kind is held by its claim (novox/hq ADR 0259).
func TestAKindedBenchIsNotHandedOver(t *testing.T) {
for _, bench := range []string{"channel", "intake"} {
err := handOver(context.Background(), bench, "anchor/telegram", false)
if err == nil || !strings.Contains(err.Error(), "is a kinded bench") {
t.Errorf("%s: %v", bench, err)
}
}
}
+377 -27
View File
@@ -7,6 +7,7 @@ import (
"errors"
"fmt"
"github.com/nats-io/nats.go/micro"
"io"
"os"
"os/exec"
"slices"
@@ -28,6 +29,9 @@ import (
// answer. It also means a refusal is the same refusal in the same words, because it is the same
// output.
// verbKey carries the verb a call is for, to the process it runs.
type verbKey struct{}
// verbAnswer is what a verb answers: what the command printed, whether it succeeded, and — where the
// command speaks JSON — the same as data.
type verbAnswer struct {
@@ -51,6 +55,9 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
return nil, err
}
argv, err := a.commandLine()
if err == nil {
err = terminalOnly(argv)
}
if len(a.misread) > 0 {
// The table and the command line disagree: the verb reads an argument no caller can see
// in its schema, so no caller could ever pass it.
@@ -107,6 +114,14 @@ func declaredArguments(v catalogue.Verb) (names []string, switches map[string]bo
return names, switches
}
// isList says a verb's argument is declared a list of text (catalogue's listed): given as a JSON array, it is
// read as its items joined by commas, as the same argument given as one text would be.
func isList(v catalogue.Verb, name string) bool {
props, _ := v.Input["properties"].(map[string]any)
p, _ := props[name].(map[string]any)
return p != nil && p["type"] == "array"
}
// readArguments refuses what the verb does not take, before anything is composed.
func readArguments(verb string, args map[string]any) (*verbArguments, error) {
v, known := controllerVerb(verb)
@@ -143,6 +158,19 @@ func readArguments(verb string, args map[string]any) (*verbArguments, error) {
return nil, fmt.Errorf("%s: %q is text, not true or false", verb, k)
}
value = fmt.Sprint(x)
case []any:
if !isList(v, k) {
return nil, fmt.Errorf("%s: %q is text, and was given a list", verb, k)
}
items := make([]string, 0, len(x))
for _, item := range x {
text, ok := item.(string)
if !ok || strings.TrimSpace(text) == "" || strings.Contains(text, ",") {
return nil, fmt.Errorf("%s: %q is a list of names, and holds %v", verb, k, item)
}
items = append(items, strings.TrimSpace(text))
}
value = strings.Join(items, ",")
default:
return nil, fmt.Errorf("%s: %q is text, and was given %T", verb, k, x)
}
@@ -222,6 +250,35 @@ func quoteAll(xs []string) string {
return strings.Join(q, ", ")
}
// refusedAsTheGenericCommand is what the generic `command` verb refuses of a command line, and so what every
// line asked through a verb's route refuses: the `command` verb's, and an ordinary line from mesh-cli (novox/hq ADR
// 0272 §4). One function, so the two routes cannot drift apart.
func refusedAsTheGenericCommand(argv []string) error {
if len(argv) == 0 {
return errors.New("command names no command")
}
// A layer is written through the settings verb, never the generic one (novox/hq issue 339): the
// settings verb is where what a verb may not set is refused, and one route is one set of words.
// The command refuses places and accesses through any verb as well; this says so before it runs.
if argv[0] == "settings" && slices.ContainsFunc(argv[1:], func(w string) bool { return w == "set" || w == "clear" || w == "propose" }) {
return &heldAtTheTerminal{msg: "settings are set, cleared and proposed through the settings verb, not the " +
"generic command; and places and accesses are set at the controller's terminal or on the operator's " +
"warrant (novox/hq issue 339, ADR 0277). Nothing was done"}
}
// The generic verb only reads (novox/hq ADR 0266): what writes has a named verb that composes its own
// line, or is the operator's at the controller's terminal.
if err := commandReads(argv); err != nil {
return err
}
// The generic verb is no way round the hand-act log (novox/hq to-be 45 §7): a repair through
// it says why, as it would through its own verb.
if repair := repairingCommand(argv); repair != "" && !slices.ContainsFunc(argv, isWhyFlag) {
return fmt.Errorf("%s is a repair done by hand, and says why: add --why <text> to the command "+
"line (recorded in the hand-act log). Nothing was done", repair)
}
return nil
}
// commandLine composes the command. Every argument it reads is one it uses: a branch that reads an
// argument and then drops it would pass it over, which is what the check after it exists to refuse.
func (a *verbArguments) commandLine() ([]string, error) {
@@ -238,18 +295,16 @@ func (a *verbArguments) commandLine() ([]string, error) {
if err != nil {
return nil, err
}
if len(argv) == 0 {
return nil, errors.New("command names no command")
}
// The generic verb is no way round the hand-act log (novox/hq to-be 45 §7): a repair through
// it says why, as it would through its own verb.
if repair := repairingCommand(argv); repair != "" && !slices.ContainsFunc(argv, isWhyFlag) {
return nil, fmt.Errorf("%s is a repair done by hand, and says why: add --why <text> to the command "+
"line (recorded in the hand-act log). Nothing was done", repair)
if err := refusedAsTheGenericCommand(argv); err != nil {
return nil, err
}
return argv, nil
case "tools":
return nil, errors.New("tools is answered from the records, not by a command")
case "dead-letters":
return nil, errors.New("dead-letters is answered by the serving controller, on its own connection, not by a command")
case "root-free":
return nil, errors.New("root-free is judged by the serving controller, on its own connection, not by a command")
case "status":
return []string{"status", "--json"}, nil
case "nodes":
@@ -481,6 +536,11 @@ func (a *verbArguments) commandLine() ([]string, error) {
argv = append(argv, "--condition", c)
}
return argv, nil
case "warranted":
if err := need("asker", "ask"); err != nil {
return nil, err
}
return []string{"hand-act", "warrant", "--asker", str("asker"), "--ask", str("ask")}, nil
case "hand-acts":
argv := []string{"hand-acts", "--json"}
if d := str("days"); d != "" {
@@ -707,6 +767,23 @@ func (a *verbArguments) commandLine() ([]string, error) {
argv = append(argv, "--probe", p)
}
return append(argv, "--json"), nil
case "give":
// The same line as secret-ask with the desk named (novox/hq ADR 0277): one path, one set of bounds.
if err := need("node", "module", "secret", "at"); err != nil {
return nil, err
}
return []string{"secret", "ask", str("node"), str("module"), str("secret"), "--at", str("at")}, nil
case "secret-ask":
// A module's own secret asked for, typed by the operator at the desk (novox/hq ADR 0277): never a value
// in the arguments. The desk is the module's machine unless at names another.
if err := need("node", "module", "secret"); err != nil {
return nil, err
}
argv := []string{"secret", "ask", str("node"), str("module"), str("secret")}
if at := str("at"); at != "" {
argv = append(argv, "--at", at)
}
return argv, nil
case "rotate":
if p := str("provision"); p != "" {
argv := []string{"rotate", p}
@@ -770,6 +847,21 @@ func (a *verbArguments) commandLine() ([]string, error) {
if str("module") == "" && on("clear") {
return nil, errors.New("settings: a module is needed to clear a layer; name it with module")
}
// The proposals, listed or one whole (novox/hq ADR 0277): a read, needing no module.
if on("proposals") || str("proposal") != "" {
if str("module") != "" || str("values") != "" || str("node") != "" || on("clear") || on("replace") ||
on("history") || str("list") != "" || on("propose") || (on("proposals") && str("proposal") != "") {
return nil, errors.New("settings: proposals lists what was proposed and proposal shows one; either takes nothing else")
}
argv := []string{"settings", "proposals"}
if id := str("proposal"); id != "" {
argv = append(argv, id)
}
return argv, nil
}
if str("module") == "" && on("propose") {
return nil, errors.New("settings: a module is needed to propose a layer; name it with module")
}
if list := str("list"); list != "" || str("module") == "" {
if list != "" && list != "preferences" {
return nil, fmt.Errorf("settings lists %q only; %q is not a listing", "preferences", list)
@@ -790,6 +882,22 @@ func (a *verbArguments) commandLine() ([]string, error) {
}
var argv []string
switch {
case on("propose"):
// A proposal: the values, or clear, put to the operator (novox/hq ADR 0277). Nothing is set here.
argv = []string{"settings", "propose", str("module")}
switch {
case on("clear") && str("values") != "":
return nil, errors.New("settings: a proposal gives values or says clear, not both")
case on("clear"):
argv = append(argv, "--clear")
case str("values") != "":
argv = append(argv, str("values"))
if on("replace") {
argv = append(argv, "--replace")
}
default:
return nil, errors.New("settings: a proposal gives the values, or says clear")
}
case on("clear"):
argv = []string{"settings", "clear", str("module")}
case str("values") != "":
@@ -875,6 +983,8 @@ func repairingCommand(argv []string) string {
return "plans " + argv[1]
case argv[0] == "broker" && len(argv) > 1 && argv[1] == "consumer-reset":
return "broker consumer-reset"
case argv[0] == "hand-act" && len(argv) > 1 && argv[1] == "warrant":
return "" // the router's record of a person's answer, never a repair (novox/hq ADR 0274)
case argv[0] == "hand-act" && len(argv) > 1 && argv[1] == "drill":
return "hand-act drill"
case argv[0] == "hand-act":
@@ -897,6 +1007,30 @@ func isWhyFlag(word string) bool {
return word == "--why" || word == "-why" || strings.HasPrefix(word, "--why=") || strings.HasPrefix(word, "-why=")
}
// commandEnvironment is the environment of a command line this controller runs for someone: its own — the
// stores' credentials, the bus, the broker, everything a command run from a shell beside it would have, because it
// is that — with who asked, and how it came.
//
// **terminal** is said, never inferred (novox/hq ADR 0272): only a line mesh-cli asked as the controller's terminal
// passes true. Its line has no `MESH_VERB`, not the served mark ADR 0266 puts on everything the serving controller
// starts, and the terminal's mark (cliTerminalVar), so startedAtTheTerminal reads yes. Every other line names its
// verb, and is stripped of the terminal's mark whatever this process's environment holds.
func commandEnvironment(caller, verb string, terminal bool) []string {
env := make([]string, 0, len(os.Environ())+3)
for _, kv := range os.Environ() {
if strings.HasPrefix(kv, verbVar+"=") || strings.HasPrefix(kv, link.CallerVar+"=") ||
strings.HasPrefix(kv, cliTerminalVar+"=") || (terminal && strings.HasPrefix(kv, servedVar+"=")) {
continue
}
env = append(env, kv)
}
env = append(env, link.CallerVar+"="+caller)
if terminal {
return append(env, cliTerminalVar+"=1")
}
return append(env, verbVar+"="+verb)
}
// runVerb runs this binary with the given command line and gathers what it said.
//
// **This binary is the image this process runs, never the file at the path it started from**
@@ -910,15 +1044,17 @@ func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
return verbAnswer{}, fmt.Errorf("%w: this controller is stopping and runs no new command", link.ErrHandingOver)
}
cmd := selfCommand(ctx, argv)
// The same environment: the stores' credentials, the bus, the broker — everything a command run
// from a shell in this container would have, because it is that.
cmd.Env = os.Environ()
// And who asked, so an act it does by hand is recorded as theirs (novox/hq to-be 45 §7).
caller := link.CallerIn(ctx)
if caller == "" {
caller = "a seat call whose caller the bus did not name"
}
cmd.Env = append(cmd.Env, link.CallerVar+"="+caller+", through the "+catalogue.ControllerSeatName+" seat")
// And which verb, so the connection it dials says so in the bus's list (novox/hq issue 327).
verb, _ := ctx.Value(verbKey{}).(string)
if verb == "" {
verb = argv[0]
}
cmd.Env = commandEnvironment(caller+", through the "+catalogue.ControllerSeatName+" seat", verb, false)
// Two buffers, one answer. What the command *says* is both streams, in the order a person at
// a shell would read them; what it *answers as data* is standard output alone — `status --json`
// prints its warnings beside the document, and a JSON parsed from the two together parsed
@@ -927,18 +1063,7 @@ func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
cmd.Stdout = &stdout
cmd.Stderr = &stderr
runErr := cmd.Run()
answer := verbAnswer{Output: stdout.String() + stderr.String(), OK: runErr == nil}
if jsonVerbs[argv[0]] && runErr == nil {
var parsed any
if json.Unmarshal(bytes.TrimSpace(stdout.Bytes()), &parsed) == nil {
answer.Answer = parsed
if overviewVerbs[argv[0]] {
// Once, as data: the same document again as text doubled an answer that already
// outgrew one message of the bus (novox/hq issue 314).
answer.Output = stderr.String() + "its answer, as data, is `answer`\n"
}
}
}
answer := answerOf(argv, stdout.Bytes(), stderr.String(), runErr == nil)
var exit *exec.ExitError
if runErr != nil && !errors.As(runErr, &exit) {
// Not the command refusing — the command not running at all, which is this process's fault.
@@ -953,6 +1078,66 @@ func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
return answer, nil
}
// answerOf is what a command said, as a verb answers it: both streams as text, and standard output as data
// where the command speaks JSON.
func answerOf(argv []string, stdout []byte, stderr string, ok bool) verbAnswer {
answer := verbAnswer{Output: string(stdout) + stderr, OK: ok}
if jsonVerbs[argv[0]] && ok {
var parsed any
if json.Unmarshal(bytes.TrimSpace(stdout), &parsed) == nil {
answer.Answer = parsed
if overviewVerbs[argv[0]] {
// Once, as data: the same document again as text doubled an answer that already
// outgrew one message of the bus (novox/hq issue 314).
answer.Output = stderr + "its answer, as data, is `answer`\n"
}
}
}
return answer
}
// readHere answers a verb that only reads the bus in the serving controller itself, on its own connection
// and keeper (novox/hq issue 327): the same command, writing to the answer rather than to a process's
// output, so the answer is the one the command prints. False for any other command line, which runs as a
// command of its own. Every `conditions` call — the operator's channel reads it at least once a minute —
// was a process that dialled the bus, logged in and left.
func readHere(ctx context.Context, argv []string) (verbAnswer, bool) {
var read func(context.Context, []string, io.Writer) error
args := argv[1:]
// Each where this process holds what it reads: the serving keeper, the hand-act log's connection, the
// serving connection.
switch argv[0] {
case "conditions":
sub := "list"
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
sub, args = args[0], args[1:]
}
if conditionsFrom != nil {
read = map[string]func(context.Context, []string, io.Writer) error{
"list": listConditions, "show": showCondition, "history": conditionHistory}[sub]
}
case "hand-acts":
if handActConn != nil || servingBus.Load() != nil {
read = listHandActs
}
case "queue":
if servingBus.Load() != nil {
read = listQueue
}
}
if read == nil {
return verbAnswer{}, false
}
var out bytes.Buffer
stderr := ""
err := read(ctx, args, &out)
if err != nil {
// As the command says it when it fails (main).
stderr = "mesh-controller: " + err.Error() + "\n"
}
return answerOf(argv, out.Bytes(), stderr, err == nil), true
}
// seatToolHandlers are the handlers for every verb the mesh-controller seat declares, from the
// store's row, so a verb the row does not carry is not served. A verb it carries that this binary
// cannot run is named at start and answers the reason when called — never a refusal to serve, which
@@ -982,6 +1167,12 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
if verb == "calls" {
return callsAnswer(link.Calls, a.given["call"])
}
if verb == "dead-letters" {
return deadLettersAnswer(ctx, a)
}
if verb == "root-free" {
return rootFreeAnswer(ctx, a.given["machines"], rootClock())
}
if verb == "doctor" {
// From the serving controller, which runs the self-check and hears the signals
// (novox/hq to-be 45 §4): the last verdict at once, or a run now.
@@ -999,7 +1190,8 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
}
continue
}
if _, err := argvFor(verb, sampleArguments(v)); err != nil {
var policy *heldAtTheTerminal
if _, err := argvFor(verb, sampleArguments(v)); err != nil && !errors.As(err, &policy) {
// **A row ahead of this binary is not a reason to go silent.**
//
// The row is the store's and a control plane follows it (novox/hq ADR 0154), so a verb
@@ -1033,6 +1225,7 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
if err != nil {
return nil, err
}
ctx = context.WithValue(ctx, verbKey{}, verb)
if verb == "status" && statusFrom != nil {
// At once, from the summary the serving controller keeps (novox/hq to-be 45 Phase 0).
return statusFrom.answer(ctx)
@@ -1041,6 +1234,9 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
// Whatever it did, `status` is composed again once it has.
defer statusFrom.nudge()
}
if answer, read := readHere(ctx, argv); read {
return answer, nil
}
if answersFirst(argv) {
// Before anything is sent: a push sends the bus's own machine first, and a broker
// reloading its user list forgets the answer it was about to permit (novox/hq issue 265).
@@ -1064,7 +1260,13 @@ func actsOnAPlan(args map[string]any) bool {
// inProcess are the verbs answered by this process rather than by a command it runs: `tools` from
// the records, `calls` from what this process served.
var inProcess = map[string]bool{"tools": true, "calls": true, "doctor": true}
var inProcess = map[string]bool{"tools": true, "calls": true, "doctor": true,
// What a consumer gave up on, read and changed on the serving controller's own connection (novox/hq
// issue 330).
"dead-letters": true,
// Whether a machine is root-free, judged live on the serving controller's store and connection (novox/hq ADR
// 0259 §8): the router asks it before an approval.
"root-free": true}
// answersFirst is a command line whose caller is answered before it runs: a push, by its verb or
// through `command`. A push sends the machine holding the bus first when its user list changed, the
@@ -1080,7 +1282,7 @@ func answersFirst(argv []string) bool {
// the reason said beside it.
func callsAnswer(log *link.CallLog, id string) (any, error) {
if id != "" {
c, ok, err := log.Get(id)
c, ok, err := log.Shown(id)
if err != nil {
return nil, fmt.Errorf("call %s is not in this controller's memory, and the calls kept on the "+
"bus could not be read: %w", id, err)
@@ -1253,3 +1455,151 @@ func seatAnnouncement(handlers map[string]link.ToolHandler) micro.Info {
Endpoints: endpoints,
}
}
// nodeReads are the `node` subcommands a verb may run: the ones that only read.
var nodeReads = map[string]bool{"list": true, "show": true}
// flagsOnly says a command line's rest names no subcommand: empty, or beginning with a flag. For a command
// with no subcommands every word is a flag, its value or a name it reads.
func flagsOnly(rest []string) bool { return len(rest) == 0 || strings.HasPrefix(rest[0], "-") }
// subIn says the rest begins with one of these subcommands.
func subIn(rest []string, subs ...string) bool {
return len(rest) > 0 && slices.Contains(subs, rest[0])
}
// commandReadForms are the command lines the generic `command` verb may run (novox/hq ADR 0266): **an allow
// list of the ones that only read**, judged command by command. Anything else — every command that writes a
// record, sends, builds, issues an account or a token, sets a key, accepts, rotates, recovers or exports a
// secret — is refused, and a command added later is refused until it is judged a read. Writing has its named
// verbs, which compose their own lines and are judged by terminalOnly; the rest is the operator's at the
// controller's terminal.
var commandReadForms = map[string]func(rest []string) bool{
"status": flagsOnly, "version": flagsOnly, "help": flagsOnly, "seats": flagsOnly, "healers": flagsOnly,
"hand-acts": flagsOnly, "durations": flagsOnly, "collection": flagsOnly, "images": flagsOnly,
"artifacts": flagsOnly, "data": flagsOnly, "builds": flagsOnly, "queue": flagsOnly,
// `plan <node>` previews a node's declaration; it sends nothing.
"plan": func([]string) bool { return true },
// `plans` lists and `plans <id>` shows one; `plans stop|close|go` acts.
// Judged on every word, not the first: a flag before the subcommand (`plans --json go <id>`) still acts.
"plans": func(r []string) bool {
return !slices.ContainsFunc(r, func(w string) bool { return slices.Contains(plansActs, w) })
},
// `doctor` answers the last run, `probes` and `signals` describe; `doctor run` runs.
"doctor": func(r []string) bool { return flagsOnly(r) || subIn(r, "probes", "signals") },
"conditions": func(r []string) bool { return flagsOnly(r) || subIn(r, "list", "show", "history") },
"node": func(r []string) bool { return subIn(r, "list", "show") },
"module": func(r []string) bool { return subIn(r, "list") },
"settings": func(r []string) bool { return subIn(r, "show", "preferences", "proposals") },
"retire": func(r []string) bool { return subIn(r, "list") },
"cleanup": func(r []string) bool { return subIn(r, "list") },
"delivery": func(r []string) bool { return subIn(r, "plan", "walks") },
// `bus` alone says the bus's step; `bus upgrade` takes one.
"bus": func(r []string) bool { return len(r) == 0 },
// `mirrors` lists; --record and --confirm keep a mirror.
"mirrors": func(r []string) bool {
return flagsOnly(r) && !slices.ContainsFunc(r, func(w string) bool {
return w == "--record" || w == "-record" || strings.HasPrefix(w, "--record=") || strings.HasPrefix(w, "-record=") ||
w == "--confirm" || w == "-confirm" || strings.HasPrefix(w, "--confirm=")
})
},
}
// plansActs are the `plans` subcommands that act on a walk; no other word of a plans line is one of them.
var plansActs = []string{"go", "stop", "close", "retry"}
// heldAtTheTerminal is a refusal of policy (novox/hq ADR 0266): the verb is known and served, and this line is
// the operator's at the controller's terminal. Never read as a verb this binary is behind on.
type heldAtTheTerminal struct{ msg string }
func (e *heldAtTheTerminal) Error() string { return e.msg }
func terminalRefusal(format string, args ...any) error {
return &heldAtTheTerminal{fmt.Sprintf(format, args...)}
}
// commandReads refuses a line the generic verb may not run, saying what it may.
func commandReads(argv []string) error {
if read, ok := commandReadForms[argv[0]]; ok && read(argv[1:]) {
return nil
}
return terminalRefusal("%q is not a reading command, and the generic command verb only reads (novox/hq ADR 0266): "+
"whoever may call a verb includes agents, and a line that writes, issues, sets a key or reveals a secret "+
"would be theirs to run. Use the named verb for it, or run it at the controller's terminal. The verb may "+
"run: %s. Nothing was done", strings.Join(argv, " "), commandReadNames())
}
func commandReadNames() string {
names := make([]string, 0, len(commandReadForms))
for n := range commandReadForms {
names = append(names, n)
}
sort.Strings(names)
return strings.Join(names, ", ") + " (each in its reading forms)"
}
// terminalOnlyCommands are the commands no verb runs, whatever composed them (novox/hq ADR 0266): they set
// the operator's key, issue a credential or a token that is answered to the caller, or accept, recover or
// export a secret. Their answers or effects hand whoever calls them what the runtime's account holds.
var terminalOnlyCommands = map[string]string{
"operator": "the operator's key and credential",
"identity": "the mesh's identity keys",
"token": "a token a machine joins with, answered to the caller",
"broker": "the bus's accounts",
"api": "the controller's API keys",
"licence": "the licences' secrets",
}
// givenAtTheDesk is exactly the line the `give` and `secret-ask` verbs compose, and nothing beside it: `secret ask
// <node> <module> <secret>`, with `--at <machine>` or no other word — no value, no file, no provider (novox/hq
// ADR 0277). The terminal's own `secret accept … --at-desk` is the terminal's.
func givenAtTheDesk(argv []string) bool {
if (len(argv) != 5 && len(argv) != 7) || argv[0] != "secret" || argv[1] != "ask" {
return false
}
for _, w := range argv[2:5] {
if w == "" || strings.HasPrefix(w, "-") {
return false
}
}
if len(argv) == 5 {
return true
}
return argv[5] == "--at" && argv[6] != "" && !strings.HasPrefix(argv[6], "-")
}
// terminalOnly refuses, through any verb, a command that is the operator's at the controller's terminal
// alone (novox/hq ADR 0266). **Every `node` subcommand that is not a read**: `node account` and
// `node agent-account` above all. Whoever may call a verb includes agents, and an agent that named itself
// the operator account, or cleared the agent account, would have the next send grant it root through the
// sudo module's rule. An allow list, so a subcommand added later is refused until it is judged a read.
func terminalOnly(argv []string) error {
if len(argv) == 0 {
return nil
}
if what, kept := terminalOnlyCommands[argv[0]]; kept {
return terminalRefusal("%s is run at the controller's terminal only, never through a verb: it holds %s, and "+
"whoever may call a verb includes agents (novox/hq ADR 0266). Nothing was done", argv[0], what)
}
// Of a secret's commands only rotation, which seals the new value to the machine that uses it, and the
// `give` verb's own line: an own secret typed by the operator into the desk's hidden prompt, sealed to this
// call and then to the module's machine, so no value travels in the verb or its answer (hq ADR 0259 §10).
if argv[0] == "secret" && (len(argv) < 2 || argv[1] != "rotate") && !givenAtTheDesk(argv) {
return terminalRefusal("secret %s is run at the controller's terminal only, never through a verb: accepting, "+
"recovering or exporting a secret hands it to whoever asks, and that includes agents (novox/hq ADR "+
"0266). Nothing was done", strings.Join(argv[1:], " "))
}
if argv[0] != "node" {
return nil
}
if len(argv) > 1 && nodeReads[argv[1]] {
return nil
}
sub := "node"
if len(argv) > 1 {
sub += " " + argv[1]
}
return terminalRefusal("%s is run at the controller's terminal only, never through a verb: a node's accounts "+
"decide who may become root on it (novox/hq ADR 0266). A verb may run node list and node show. "+
"Nothing was done", sub)
}
+1 -1
View File
@@ -271,11 +271,11 @@ var accountedFlags = map[string]map[string]string{
"builds": {"n": "=limit"},
"plans": {"n": "=limit", "what-if": "=repository"},
// The machine's tunnel key, named as the verb's other arguments are (novox/hq ADR 0169).
"token issue": {"overlay-key": "=overlay_key"},
"durations": {
"json": "set by the verb: the answer is data",
"all": "withheld: every measurement of a fortnight is more than a call should carry; `command` reaches it",
},
// The desk path of `secret accept` (novox/hq ADR 0259 §10): a value is never an argument of a call.
"hand-acts": {"json": "set by the verb: the answer is data"},
"conditions": {"json": "set by the verb: the answer is data"},
"retire": {"json": "set by the verb: the answer is data"},
+73 -13
View File
@@ -2,6 +2,7 @@ package main
import (
"context"
"errors"
"fmt"
"github.com/novox/mesh-controller/internal/link"
"strings"
@@ -108,11 +109,14 @@ func TestRotateTakesAProvisionOrAnOwnSecret(t *testing.T) {
}
}
// `token` is `token issue` at a shell, with the machine's tunnel key (novox/hq ADR 0169).
func TestTokenIssuesForAMachineAndItsTunnelKey(t *testing.T) {
argv, err := argvFor("token", map[string]any{"new": "laptop", "overlay_key": "k", "for": "2h"})
if err != nil || strings.Join(argv, " ") != "token issue --new laptop --overlay-key k --for 2h" {
t.Fatalf("token: %v %v", argv, err)
// `token` answers a joining token to its caller, and whoever may call a verb includes agents: it is the
// controller's terminal's alone (novox/hq ADR 0266), refused through the verb whatever it is given.
func TestTokenIsRefusedThroughAVerb(t *testing.T) {
for _, args := range []map[string]any{{"new": "laptop", "overlay_key": "k", "for": "2h"}, {"node": "ace"}} {
argv, err := argvFor("token", args)
if err == nil || !strings.Contains(err.Error(), "controller's terminal only") {
t.Fatalf("token %v: %v %v", args, argv, err)
}
}
}
@@ -237,20 +241,20 @@ func TestAJSONVerbsAnswerIsItsStandardOutput(t *testing.T) {
}
}
// `command` is the generic verb: the command line as given, split as a shell would, nothing added —
// so an operator's `node account g14 jochen` is one call through the console rather than a shell on
// the control node (novox/hq ADR 0154, ADR 0175).
// `command` is the generic verb: the command line as given, split as a shell would, nothing added
// (novox/hq ADR 0154, ADR 0175). It once carried an operator's `node account g14 jochen` too; a node's
// accounts are the controller's terminal's alone since ADR 0266 (TestNoVerbSetsANodesAccounts).
func TestCommandRunsTheLineAsGiven(t *testing.T) {
argv, err := argvFor("command", map[string]any{"command": "node account g14 jochen"})
if err != nil || strings.Join(argv, " ") != "node account g14 jochen" {
argv, err := argvFor("command", map[string]any{"command": "node show g14"})
if err != nil || strings.Join(argv, " ") != "node show g14" {
t.Fatalf("a plain line: %v %v", argv, err)
}
argv, err = argvFor("command", map[string]any{"command": `settings set dnsmasq '{"a": "b c"}' --node ace`})
argv, err = argvFor("command", map[string]any{"command": `settings show dnsmasq '{"a": "b c"}' --node ace`})
if err != nil || len(argv) != 6 || argv[3] != `{"a": "b c"}` {
t.Fatalf("a quoted word stays one word: %q %v", argv, err)
}
argv, err = argvFor("command", map[string]any{"command": `node add "the box" --adopted`})
if err != nil || len(argv) != 4 || argv[2] != "the box" {
argv, err = argvFor("command", map[string]any{"command": `plan "the box" --json`})
if err != nil || len(argv) != 3 || argv[1] != "the box" {
t.Fatalf("double quotes group: %q %v", argv, err)
}
if _, err := argvFor("command", map[string]any{"command": " "}); err == nil {
@@ -355,3 +359,59 @@ func TestTheControllerAnnouncesTheVerbsItServes(t *testing.T) {
}
}
}
// The generic verb only reads (novox/hq ADR 0266): an allow list of reading forms, and every line that
// writes, issues, sets a key or reveals a secret refused — the chain a review found ran through it: set the
// operator's key to one the caller holds, rotate secrets sealed to it, open them.
func TestTheCommandVerbOnlyReads(t *testing.T) {
for _, line := range []string{
"operator key set --replace k", "operator issue", "secret accept a b", "secret rotate a b c",
"secret recover a", "secret export a", "token issue --new x", "identity show", "broker users",
"api key", "licence show", "push anchor --why w", "assign novox m", "settings set m {}",
"settings clear m", "module add f", "module check /etc", "module forget m", "module issue m --node a",
"seat rename a b", "plans close p --why w", "plans go p", "plans retry p", "plans stop p",
"plans --json go p", "plans -n 3 close p", "plans --what-if r retry p", "doctor run", "conditions silence c --why w",
"retire approve x", "cleanup delete x", "delivery check", "delivery go x", "bus upgrade",
"mirrors --record x", "mirrors --confirm", "hand-act record x --why y --cause z", "serve", "migrate",
"prepare", "declare x", "overlay x", "facts", "merge-gate", "check-here", "build x", "rebuild x",
"cancel x", "kill x", "clear x", "replay x", "pause", "resume", "pin a b", "unpin a", "take x",
"converge", "adopt x", "rollout x", "upgrade x", "collect", "board", "builder", "ask x", "frobnicate",
} {
argv, err := argvFor("command", map[string]any{"command": line})
var policy *heldAtTheTerminal
if err == nil || !errors.As(err, &policy) {
t.Errorf("%q ran as %v (%v); the generic verb only reads", line, argv, err)
}
}
for _, line := range []string{
"status --json", "version", "seats --json", "healers", "hand-acts --days 3", "durations", "collection",
"images", "artifacts --collected", "data --machine a", "builds --log b", "queue", "plan ace --diff",
"plans", "plans plan-1", "doctor", "doctor probes", "doctor signals", "conditions", "conditions list",
"conditions show c", "conditions history", "node list", "node show ace", "module list",
"settings show m", "settings preferences", "retire list", "cleanup list", "delivery plan --repository r",
"delivery walks", "bus", "mirrors --json",
} {
if _, err := argvFor("command", map[string]any{"command": line}); err != nil {
t.Errorf("%q, a read, was refused: %v", line, err)
}
}
}
// What hands a caller a key, a credential or a secret is refused whichever verb composed it.
func TestNoVerbSetsTheOperatorsKeyOrRevealsASecret(t *testing.T) {
for _, argv := range [][]string{
{"operator", "key", "set"}, {"operator", "issue"}, {"identity"}, {"token", "issue"}, {"broker", "users"},
{"api"}, {"licence"}, {"secret", "export", "x"}, {"secret", "recover", "x"}, {"secret", "accept", "x"}, {"secret"},
} {
if err := terminalOnly(argv); err == nil {
t.Errorf("%v passed", argv)
}
}
if err := terminalOnly([]string{"secret", "rotate", "n", "m", "s"}); err != nil {
t.Errorf("rotating seals to the machine that uses the secret, and stays a verb's: %v", err)
}
// A policy refusal is not a verb this binary is behind on: every verb is still served.
if _, behind, err := seatToolHandlers(); err != nil || len(behind) != 0 {
t.Fatalf("behind %v: %v", behind, err)
}
}
+74 -2
View File
@@ -39,6 +39,8 @@ func secretCommand(ctx context.Context, args []string) error {
}
switch args[0] {
case "accept":
case "ask":
return secretAsk(ctx, args[1:])
case "rotate":
return secretRotate(ctx, args[1:])
case "recover":
@@ -55,6 +57,9 @@ func secretCommand(ctx context.Context, args []string) error {
provider := set.String("provider", "",
"the node providing <name>: the value becomes the PAIR credential between <module> on <node> "+
"and that provider, sealed to both — the vault's operator-delivered secret (ADR 0092)")
desk := set.String("at-desk", "",
"ask the operator for the value in a prompt that does not show it, on this machine's desk; the "+
"answer comes back sealed to this call alone (novox/hq ADR 0259 §10)")
local := set.String("local", "",
"with --provider: the name the credential goes by inside <module>, where its manifest keeps "+
"several for <name> (ADR 0094)")
@@ -65,6 +70,18 @@ func secretCommand(ctx context.Context, args []string) error {
return errors.New(secretUsage)
}
node, module, name := rest[0], rest[1], rest[2]
// A value comes from the terminal or the desk, never through a verb (the review of 2026-10-09, M4): a
// verb's caller may be an agent, and a value it chose would become what a module acts with.
if verb, through := throughAVerb(); through && *desk == "" {
return fmt.Errorf("a secret's value is given at the controller's terminal or at the desk (`give`), never "+
"through a verb (this line came through %q): nothing was read or sealed", verb)
}
if *desk != "" {
if *from != "" || *provider != "" {
return errors.New("--at-desk gives a module's own secret, and takes neither --from nor --provider")
}
return askAtDesk(ctx, node, module, name, *desk)
}
value, err := valueFor(node, module, name, *from)
if err != nil {
@@ -93,10 +110,26 @@ func secretCommand(ctx context.Context, args []string) error {
fmt.Printf(" run `push %s` and `push %s` to send it\n", *provider, node)
return nil
}
untilStart, err := open.inventory.AcceptGivenSecret(ctx, node, module, name, value)
// A trusted party's secret is announced before it is kept (the confirmation review of 2026-10-09, N1-give):
// on every channel, the one it replaces among them, which still runs on its old value until the next push.
// Not announced, it is not kept: a channel whose token changed unheard of answers for somebody else.
trusted, err := open.inventory.RunsAsItsOwnAccount(ctx, module)
if err != nil {
return err
}
untilStart, unannounced, err := keepGiven(trusted,
func() error { return announceSecretGiven(ctx, node, module, name, "at the controller's terminal") },
func() (bool, error) { return open.inventory.AcceptGivenSecret(ctx, node, module, name, value) })
if err != nil {
if trusted && unannounced != nil {
return fmt.Errorf("%s runs as an account of its own, and the change of its %s could not be announced on "+
"your channels first, so nothing was kept: %w", module, name, err)
}
return err
}
if unannounced != nil {
fmt.Printf(" this change could NOT be announced on the operator's channels: %v\n", unannounced)
}
// Not printed back, and there is nowhere it could be printed from: it is sealed to that
// machine and the mesh cannot read it again.
fmt.Printf("%s on %s now holds %q, sealed to that machine.\n", module, node, name)
@@ -117,8 +150,25 @@ func secretCommand(ctx context.Context, args []string) error {
return nil
}
// secretAsk is `secret ask <node> <module> <name> [--at <machine>]` (novox/hq ADR 0277): the operator is asked
// for a module's own secret in a prompt at the desk, which only they answer. The one `secret` line a verb may
// run beside rotate (givenAtTheDesk): it carries no value and answers none.
func secretAsk(ctx context.Context, args []string) error {
rest, flags := split(args)
set := flag.NewFlagSet("secret ask", flag.ContinueOnError)
at := set.String("at", "", "the machine the operator sits at, where the prompt opens; the module's machine when absent")
if err := set.Parse(flags); err != nil {
return err
}
if len(rest) != 3 {
return errors.New("secret ask <node> <module> <name> [--at <machine>]")
}
return askAtDesk(ctx, rest[0], rest[1], rest[2], *at)
}
const secretUsage = "secret rotate <node> <module> <name> [--why <text> [--cause <word>]]\n" +
"secret accept <node> <module> <name> [--from <file>] [--provider <node> [--local <name>]]\n" +
"secret ask <node> <module> <name> [--at <machine>]\n" +
"secret accept <node> <module> <name> [--from <file> | --at-desk <machine>] [--provider <node> [--local <name>]]\n" +
"secret recover <node> <module> <name> --key <operator-key> [--out <file>] [--from-export <file>] [--provider <node>]\n" +
"secret export [--out <file>]"
@@ -369,6 +419,8 @@ func valueFor(node, module, name, from string) (string, error) {
fmt.Fprintf(os.Stderr,
"reading %s's %q for %s from standard input; it is not echoed anywhere\n",
module, name, node)
// At a terminal, what is typed is not shown either: echo off while it is read.
defer hideTyping(os.Stdin)()
line, err := bufio.NewReader(os.Stdin).ReadString('\n')
if err != nil && line == "" {
return "", fmt.Errorf("nothing was given on standard input: %w", err)
@@ -452,3 +504,23 @@ func whoAsked() string {
}
return "the mesh"
}
// keepGiven keeps a value given at the controller's terminal, and announces it on the operator's channels
// (the confirmation review of 2026-10-09, N1-give). **A trusted party's — a module running as an account of its
// own: the router, a verified channel — is announced before it is kept, and not kept when the announcement
// fails**: a channel whose token changed unheard of answers for somebody else. Any other module's is kept first
// and announced after, and a failed announcement is said (unannounced) without undoing it.
func keepGiven(trusted bool, announce func() error, keep func() (bool, error)) (untilStart bool, unannounced, err error) {
if trusted {
if err := announce(); err != nil {
return false, err, err
}
untilStart, err = keep()
return untilStart, nil, err
}
untilStart, err = keep()
if err != nil {
return false, nil, err
}
return untilStart, announce(), nil
}

Some files were not shown because too many files have changed in this diff Show More