Author SHA1 Message Date
jochen 64bc138692 Retire the networking bundle and what the control plane stops shipping (hq ADR 0226)
networking required mesh-wireguard and nothing else; machines are assigned the network directly.
module forget refuses a provided module, so a retired one is removed at start once no machine has it.
Guard route-proxy's public account directory against a reissue.
2026-10-06 02:21:18 +02:00
mesh-admin e096b4595a Merge pull request 'Keep the account of the sent declaration over an older one (hq issue 267)' (#74) from fix/stale-report-overwrites into main 2026-10-05 23:47:10 +00:00
jochen 09c0c6b367 Keep the account of the sent declaration over an older one (hq issue 267)
The last report stored per node decides whether a release plan moves on,
and it was whichever arrived last. A report about a declaration the mesh
has moved past now records what it says about the machine but leaves the
account of the apply alone, so arrival order cannot undo the newer.
2026-10-06 01:45:35 +02:00
mesh-admin d1fc25f682 Merge pull request 'Catch up on merges the bus announced and never handed over (hq issue 266)' (#73) from fix/missed-merges-are-caught-up into main 2026-10-05 23:33:18 +00:00
mesh-admin eda457f415 Merge pull request 'Answer every seat call within ten seconds and keep what came of it (hq issue 265)' (#72) from fix/a-verb-answers-before-its-caller-gives-up into main 2026-10-05 23:33:11 +00:00
jochen 59f4d486b1 Catch up on merges the bus announced and never handed over (hq issue 266)
The controller acted only on what its events consumer handed it, so a merge
the bus skipped left modules behind with nothing said. The stream is now read
back every five minutes on a single-filter consumer, and any merge that would
still move a module after ten minutes is said and acted on.
2026-10-06 01:29:21 +02:00
jochen 801552c0eb Answer every seat call within ten seconds and keep what came of it (hq issue 265)
A push outlasted the console's 30s wait and, when it sent the bus its
changed user list, the broker's reload forgot the reply it may send:
the push happened and its caller was told it did not answer. Calls now
answer in full or as running with an id, a push answers before it
sends, refused answers are recorded on their call, and 'calls' reads
them back.
2026-10-06 01:14:58 +02:00
mesh-admin ede9bce6ef Merge pull request 'Refuse a verb argument the seat would pass over; a push without a machine says it is the whole mesh (hq issue 244)' (#71) from fix/verb-schemas into main 2026-10-05 22:43:07 +00:00
jochen 0f0028785c Refuse a verb argument the seat would pass over, and say a push is of the whole mesh
A push naming one machine reached the verb without it and pushed every
machine behind (hq issue 244). The controller now refuses any argument a
verb does not declare, any it composed its command line without, and a
switch that is not true or false; a push that names no machine says first
that it is the whole mesh. Tests walk every served verb: no argument is
ever ignored, and every flag of a verb's command, read from the source, is
in its schema or accounted for. plan gains files, push behind, builds and
plans limit.
2026-10-06 00:37:14 +02:00
mesh-admin 6fdcfad8d3 Merge pull request 'Report a provider that keeps failing a consumer in status (hq ADR 0224)' (#70) from feat/a-provider-failing-a-consumer-is-reported into main 2026-10-05 22:20:45 +00:00
jochen 8d9d33ae85 Report a provider that keeps failing a consumer in status (hq ADR 0224)
The identity provider failed every consumer for a day and status called the
mesh well (hq issue 179). The controller now follows every provider's
provisioner.failing/recovered, keeps the newest failing word per provider,
machine and consumer (migration 0065), and status, its JSON and node show
name it until it recovers. Every module that receives contributions is
granted the two events, so no manifest can forget them.
2026-10-06 00:13:23 +02:00
mesh-admin cc25baa563 Merge pull request 'Rename node-hosts-file to node-hostname, and refuse one seat claimed under two names (hq ADR 0223 part 3)' (#69) from hostname-module into main 2026-10-05 22:11:41 +00:00
mesh-admin 68a2ebdcc3 Merge pull request 'Retire node-resolver-config and the seat need only it used (hq ADR 0223 part 2, step 2 of 2)' (#68) from retire-resolv-conf into main 2026-10-05 22:08:03 +00:00
jochen 69b99eec68 Number the hostname seat migration 0064: it merges after the resolver-config one 2026-10-06 00:07:57 +02:00
jochen 09bd0eec4f Number the resolver-config migration 0063: it merges first 2026-10-06 00:07:54 +02:00
mesh-admin 222a38e050 Merge pull request 'Test resolv.conf as the uplink's, and refuse a second writer of a fact's path (hq ADR 0223 part 2, step 1 of 2)' (#67) from resolv-conf-to-uplink into main 2026-10-05 21:57:03 +00:00
jochen ee99a24f77 Rename node-hosts-file to node-hostname, and refuse one seat claimed under two names (hq ADR 0223)
The seat now covers /etc/hostname too. The migration keeps the old name as
an alias so hosts, still assigned while machines move, holds the same seat.
Claims were compared by spelling, so the old and new module would both have
held it on one machine; they are now compared by the seat they resolve to.
2026-10-05 23:43:15 +02:00
jochen f68521da28 Retire node-resolver-config and the seat need it alone used (hq ADR 0223)
The uplink's holder writes /etc/resolv.conf, so the seat that wrote it and
ADR 0220's dependency of it on the uplink have nothing left to say. The
migration deletes the store's row; nothing holds it once resolv-conf is
unassigned everywhere.
2026-10-05 23:40:39 +02:00
jochen 296064c799 Test the resolver file as the uplink's, and refuse a second writer of a fact's path (hq ADR 0223)
The catalogue moves /etc/resolv.conf from resolv-conf to the three uplink
modules. A rendered fact was not compared with other modules' paths, so two
modules could each write the resolver file on one machine, the last winning
every apply; a fact's path now counts as its module's.
2026-10-05 23:39:15 +02:00
mesh-admin df9231c734 Merge pull request 'A mesh seat may be replicated: the resolver held on two machines (hq ADR 0223)' (#65) from feat/the-mesh-has-two-resolvers into main 2026-10-05 20:48:23 +00:00
jochen 843b709b59 The registry-trust test reads the runtime's module, which now writes the trust (ADR 0222) 2026-10-05 22:47:43 +02:00
jochen f506fb34ec Let the mesh's resolver seat have several holders on record
musl takes the first reply from any listed nameserver, so a public fallback
beside the mesh's resolver answered NXDOMAIN for mesh names in every Alpine
container (hq ADR 0223). The fix is two mesh resolvers and no public one, which
needs mesh-dns-resolver held on two machines: a seat can now be replicated,
each holder recorded by 'seat <name> --add', checkClaims accepts every holder
on record and still refuses a second holder of any other mesh seat, a holder
answers its own requirement, and a roster fact gives each replicated seat's
holders, this machine first, so resolv-conf can list them. Migration 0062 keys
a holding by seat and assignment.
2026-10-05 22:42:53 +02:00
mesh-admin c34b937dd3 Merge pull request 'The private network writes nothing into the runtime's file; generated resources are collision-checked (hq ADR 0222, issue 190 — 3 of 3)' (#63) from fix/190-the-overlay-writes-no-runtime-file into main 2026-10-05 20:42:32 +00:00
mesh-admin d84c9699b3 Merge pull request 'Tell a module where a mesh seat's holder is reached: ${seat:<seat>:reach} (hq ADR 0222, issue 190 — 1 of 3)' (#62) from fix/190-seat-reach into main 2026-10-05 20:31:32 +00:00
mesh-admin 002d5e578c Merge pull request 'A named push sends no build a policy or a plan holds back (hq issue 259, ADR 0221)' (#64) from fix/259-a-named-push-sends-no-held-build into main 2026-10-05 20:26:50 +00:00
jochen 2421b82ad2 Keep a named push from sending builds a policy or a plan holds back
A named push flushed every other machine whose declaration differed from
what it was last sent (hq ADR 0083). Under an upgrade policy of `record`,
or a plan still waiting on its first machine (ADR 0218), every machine
running the module differs, so `push <one>` sent the held build to all of
them (hq issue 259).

Each send now records which build of each module it carried
(node.sent_builds, migration 0061). The cascade, and the bus holder added
to a named push, skip a machine any of whose modules would move to a
build its policy records or an open plan has not sent it, and say which
module, which build, why, and that `push <node>` sends it. A machine
whose last send was not recorded is held until it is named. The named
machine itself, a whole-mesh push and `push --behind` are unchanged.
2026-10-05 22:22:03 +02:00
jochen 0ebd48a6a8 The private network writes nothing into the runtime's file (hq issue 190)
daemon.json and docker.service belong to the docker module, which holds node-container-runtime
and now states the registry itself through ${seat:mesh-artifact-store:reach} (hq ADR 0222). The
overlay stops generating registry-trust and registry-trust-reload. A generated resource is now
held to the collision check every module is, so a second writer cannot come back through
computed code; resolution never saw what a generator declares.
2026-10-05 22:19:43 +02:00
jochen 67e291c02a Tell a module where a mesh seat's holder is reached (hq ADR 0222)
The container runtime's module must state the mesh's registry to the runtime it owns, so the
controller can stop writing that into the runtime's file (hq issue 190). ${seat:<seat>:reach}
answers host:port without a binding: nothing required, granted or minted, and the address is
one the mesh already composes into every reference it built. Only mesh-artifact-store is
answered; another seat is refused by name. Unanswered in a file written into as JSON, the empty
member is dropped, so the runtime is never told to trust "".
2026-10-05 22:16:23 +02:00
mesh-admin 8a400d165e Merge pull request 'Delete node-dns-resolver; resolver config needs the uplink (hq ADR 0220)' (#61) from feat/resolver-config-needs-the-uplink into main 2026-10-05 20:11:18 +00:00
jochen 53d3cd7ce9 Delete node-dns-resolver and make resolver config need the uplink
Nothing has claimed node-dns-resolver since the mesh moved to one resolver
(hq ADR 0194); seeding never removes a row, so a migration deletes it.

resolv.conf stays the mesh's only while the network manager is told to keep
off it, which the node-uplink holder does (ADR 0117). A seat's Needs makes
that a dependency checked at assignment by the ADR 0207 mechanism (hq ADR
0220). The two-claimants test keeps its intent with a synthetic module now
that resolved-split-dns leaves the catalogue.
2026-10-05 21:57:04 +02:00
mesh-admin 6648e4a5c8 Merge pull request 'The controller writes no /etc/hosts (hq ADR 0199)' (#60) from fix/the-controller-writes-no-hosts-file into main 2026-10-05 19:23:41 +00:00
jochen 7fa2568ce7 The controller writes no /etc/hosts (hq ADR 0199)
/etc/hosts is the file of the node-hosts-file seat's holder; the controller writes into no file
another seat's holder owns, and asks that holder if it ever needs a line there. The private
network's module stops asking for the node-names fact; every machine already asks the mesh's
one resolver for these names, and the host gives the region back at the next push.
2026-10-05 21:23:25 +02:00
mesh-admin 4b382ceffd Merge pull request 'A mesh seat's holder elsewhere answers before this machine's own provider (hq issue 258)' (#59) from fix/a-mesh-seat-answers-before-the-machines-own into main 2026-10-05 19:14:24 +00:00
jochen ce86d09d22 A mesh seat's holder elsewhere answers before this machine's own provider (issue 258)
A mesh-wide provision a machine could answer itself was bound to the local provider, with the
seat's holder and any pin consulted only for a provider on another machine. With every machine
still running its own resolver, each bound its resolver configuration to itself while the mesh's
one resolver was held and pinned elsewhere.
2026-10-05 21:14:01 +02:00
jochen 853be00ebe The runtime's file is the runtime module's: the resolver test expects docker to write live-restore (issue 190, ADR 0196)
The catalogue moves daemon.json's live-restore and the reload from resolv-conf to the docker
module, so no module writes another software's configuration. The test composes docker beside
the resolver modules and refuses resolv-conf writing the runtime's file.
2026-10-05 21:14:01 +02:00
mesh-admin e0ce2236dd Merge pull request 'The build queue is controlled through the controller and the build seat (hq ADR 0219)' (#57) from feat/the-build-queue-is-controlled into main 2026-10-05 18:22:01 +00:00
jochen 9873b3bf13 Keep a replay from moving the mesh backwards, and tighten the queue's edges (review of hq ADR 0219)
A registered replay asked now outranked newer asks of its module, and a plan
took any later outcome as its answer. A replay is now refused while the module
is asked anywhere, a plan module asked under an id is answered by that id
alone, and a rebuild of a commit asks what the module follows. An ask handed
back after a restart no longer reads as dead; a cancel that meets a start is
withdrawn; pause holds for an ask fetched as it lands; kill removes containers
before and after the build ends and says whether its outcome went out; a
holder may say only its own machine is paused.
2026-10-05 19:45:45 +02:00
jochen 541603c15c Announce the build agent's verbs, let a waiting build hear its cancel, and retry a stopped rollout (hq ADR 0219)
The holder's verbs were served and found by nothing; it now answers discovery
with its machine's four, as a runtime announces a seat's verb, so the console
finds <node>/node-build-agent.kill. A cancel publishes no outcome, so a build
waited for also looks at the cancelled set. A plan that stopped at its first
machine is retried by sending that machine the module again, unless a newer
plan holds it.
2026-10-05 19:26:40 +02:00
jochen 106507b1d3 Show and change the build queue through the controller, and have plans follow it (hq ADR 0219)
Nothing showed what waited for a build machine, and an ask could not be
dropped without leaving the plan that made it waiting for ever. New verbs:
queue, cancel, clear, rebuild, replay, kill, pause, resume, and plans retry.
Every ask a person drops is recorded failed through the same take-in as a
failed build; a plan keeps the id it asked each module under and matches
its outcome by it. replay is a dry run unless registered, and registering
an older commit than one registered since needs --older (hq issue 207).
A plan waiting on a seat paused on every holder says so and is not late;
a failed plan can be retried, and a rebuild joins the plan holding the
module instead of running beside it.
2026-10-05 19:17:56 +02:00
jochen e610f2d92c Let a build agent be paused, have a build killed, and end an ask cancelled as it took it (hq ADR 0219)
A queued ask could only be waited out and a running build only ended by
stopping the machine, which redelivered it elsewhere. The holder now serves
current, kill, pause and resume on its own machine's subjects; a kill ends
the build's process group and labelled containers and settles the ask as
failed, killed by hand; pause is kept in the workspace across a restart and
said on the bus. The controller writes cancelled ids to a cancelled set the
holder reads on taking an ask, closing the race a delete alone leaves.
2026-10-05 19:17:42 +02:00
mesh-admin f80b6cdbd1 Merge pull request 'Say a plan's first send in the machine's own time' (#56) from fix/a-plan-says-local-time into main 2026-10-05 16:35:25 +00:00
jochen 5dcf33db45 Say a plan's first send in the machine's own time, as every other line does
It is kept in UTC and was printed so: 16:32 beside log lines saying 18:32.
2026-10-05 18:35:13 +02:00
mesh-admin 6abce7e956 Merge pull request 'Judge each machine's report from its own send, so a first machine opens the gate (hq issue 256)' (#55) from fix/the-gate-reads-each-machine-from-its-own-send into main 2026-10-05 16:31:55 +00:00
jochen 0d2b304f08 Judge each machine's report from its own send, so a first machine opens the gate
With one machine first a module is sent twice, and the tier gate asked every
machine for a report after the second send: the first machine's report, made
between the two, read as stale and the plan waited for ever (hq issue 256).
Also round a plan's wait to the second, not the minute, so it is not 0s.
2026-10-05 18:29:59 +02:00
mesh-admin bdfbd0254f Merge pull request 'Delivery in order: grants before code, one machine first, a newer plan takes over (ADR 0218; hq issues 249, 252, 254)' (#54) from fix/delivery-in-order into main 2026-10-05 16:21:05 +00:00
jochen 16c5e78fa8 Stop a rollout whose first machine is silent, and choose one that is heard from (hq issue 249, ADR 0218)
A first machine that does not report within the bound now stops the
module's rollout, naming it. The first machine is the first by name heard
from lately; reports are judged by what the store says was last sent. A
plan that ends says what it built and never sent, and a failed send's
error is kept in the plan's note.
2026-10-05 18:17:52 +02:00
jochen ed90771382 Send the bus's machine before the grants, and only when its user list moved (hq issue 249)
Grants first could hold back the very declaration that lets the controller
issue them. The holder now goes first, then buckets and memberships, then
the rest; a membership that fails holds back only its own machine, and an
announcement whose send stopped at its grants is asked again. Whether the
holder must go first is read from a digest of the user list it was last
sent, not its whole declaration. Migration renumbered to 0058.
2026-10-05 18:17:52 +02:00
jochen 672d1f4ca1 Leave an announced move to the plan rolling the module out (hq issue 249)
The catalogue's upgrade announcement sent every machine one after another
without waiting for any to apply, beside the plan that now sends one
machine first. A module an open plan has not finished sending is the
plan's to roll out.
2026-10-05 18:17:52 +02:00
jochen 208901c6cc Let a newer plan supersede the older open plans of its repository (hq issue 254, ADR 0218)
A merge planned without looking at open plans, so two plans worked the
same modules and a stuck plan stayed open for ever. The newer plan folds
in what older plans of the same repository and branch had not built or
sent, and closes them as superseded. A person can close a stuck plan by
id with `plans close <id>`.
2026-10-05 18:17:52 +02:00
jochen 4ac5cfe3a7 Roll a plan's module out to one machine first (hq issue 249, ADR 0218)
A plan sent every machine running a module at once, ignoring the module's
upgrade policy. Unless the policy says together, the first machine by name
is sent, recorded in the plan, and the rest follow only once its report
after the send says it applied; a failed first machine stops the plan.
2026-10-05 18:17:52 +02:00
jochen 22660dc274 Issue grants before code, and the bus's machine first (hq issue 249)
A module's new state reached every machine before the permissions to use
it, which came only with a later push. Memberships and buckets now go
before declarations, the machine holding mesh-broker goes first when its
user list must change, and a grant that fails sends nothing and is an
error so the rollout is retried.
2026-10-05 18:17:52 +02:00
jochen d7f359c498 Read a new module's directory as its own, not as shared code (hq issue 252)
A merge adding a module the mesh has not registered rebuilt every module
built from the repository. A path under a directory known to hold modules
belongs to that module when its module.json is among the changed files.
2026-10-05 18:17:52 +02:00
mesh-admin ed25fd68e2 Merge pull request 'Hold every kept archive by a manifest so the store's collector keeps it (hq issue 253)' (#53) from fix/every-kept-archive-is-held into main 2026-10-05 16:15:15 +00:00
jochen 01c5ab2aab Hold every kept archive by a manifest so the store's collector keeps it (hq issue 253)
The store's garbage-collect marks only from manifests, and archives were
published as bare blobs, so the first real collection would delete every
archive the mesh keeps. PublishArchive now puts a deterministic OCI holder
manifest (empty config, one layer) beside each archive; the sweep holds every
kept archive before it lets anything go, which backfills existing bare blobs,
and lets go of an archive holder-first. A forgotten module no longer keeps its
five recent builds (ADR 0189). `collection [--json]` reports kept archives
held/unheld and what may be let go, so the dry run can be lifted on evidence.
2026-10-05 18:13:23 +02:00
mesh-admin bb3cd6437b Merge pull request 'Two tests that main broke: bus users after issue 195, and the event consumer made from now (issue 248)' (#52) from fix/the-bus-user-test-follows-issue-195 into main 2026-10-05 16:10:01 +00:00
jochen 0b07e68cb8 Publish the followed events once the controller's consumer exists
Made from now since issue 248, the consumer does not replay what was
published before it; the test raced the controller's start and published
first.
2026-10-05 18:04:40 +02:00
jochen 625d02862c Test the bus users as issue 195 made them: an account-reading module is one, another is not
The postgres-backed test still expected a module that declares no broker
secret to be a bus user, and failed on main since #270; it skips without a
database, so the change's own run did not see it.
2026-10-05 18:01:28 +02:00
mesh-admin e8478e208b Merge pull request 'Make the controller's event consumer from now, and give a stuck one a reset (hq issue 248)' (#51) from fix/a-consumer-on-a-history-stream-starts-from-now into main 2026-10-05 15:27:23 +00:00
jochen 5761737687 Name the issue the event consumer fix answers: 248 2026-10-05 17:15:49 +02:00
jochen 89ec48b9a9 Make the controller's event consumer from now, and give a stuck one a reset
A consumer made with the server's default replays everything a stream that
keeps history holds: the controller's EVENTS consumer, re-made that way,
replayed a week of merges and builds one at a time and held every new one
behind them. FromNow makes it start at the end; broker consumer-reset
re-makes a stuck one from now, refusing a work queue. hq issue 244.
2026-10-05 17:15:25 +02:00
jschoubben 869fb6d6bf Merge pull request 'The node-backup seat (hq ADR 0214, to-be 43)' (#49) from feat/node-backup into main 2026-10-05 10:12:34 +00:00
jschoubben d25b69178b The node-backup seat: a module contributes its backup, the mesh fills its directories
ADR 0214 / to-be 43: a node seat whose holder keeps nightly restore points of what every module on
the machine declares. A contribution of kind backup may name its module's own directories, filled
per module when placed. The catalogue check refuses a store provider that contributes no backup;
parsing does not, so the providers already running stay readable.
2026-10-05 11:42:40 +02:00
mesh-admin a7bb1e0b1c Merge pull request 'node-message-bus: the machine's D-Bus is a node seat (hq ADR 0215)' (#48) from feat/0215-node-message-bus into main 2026-10-05 09:34:14 +00:00
jochen 5eaed84271 node-message-bus: the machine's D-Bus is a node seat (hq ADR 0215) 2026-10-05 11:34:03 +02:00
jschoubben 326b1aec14 Merge pull request 'A dry-run build is taken in by nothing (hq issue 240)' (#47) from fix/a-dry-run-build-is-not-taken-in into main 2026-10-05 09:30:34 +00:00
jschoubben 134d039ff8 Take no dry run in: mark it on the request, echo it on the outcome, set it aside
A dry run of an unreviewed branch was heard by the daemon like any build, registered, and its
definition reached a machine (novox/hq issue 240). The mark now travels with the build and the
daemon records, registers and plans nothing for it.
2026-10-05 09:50:20 +02:00
jschoubben d90c6ab93a Merge pull request 'The mesh's one resolver: its seat, a provider's address, zones, and a node's hosts file (hq ADR 0194, 0196, 0199)' (#251) from feat/mesh-dns-resolver into main 2026-10-04 15:44:31 +00:00
mesh-admin 6b7d2ec49b Merge pull request 'Compose a bus user only for a module that can read an account (hq issue 195)' (#270) from fix/195-only-modules-that-read-an-account-are-bus-users into main 2026-10-04 15:34:06 +00:00
jochen 4ed1057df3 Compose a bus user only for a module that can read an account
Every assigned module was composed as a bus user, though only one declaring
a broker secret can ever be issued an account; the rest were named on every
status, plan and push as credentials never minted (137 now), burying the
real gaps. Their durable consumers are now derived from what the runtime
carries, so nothing they hear changes. The composed file is unchanged:
those users had no password and were already left out. Fixes hq issue 195.
2026-10-04 17:32:50 +02:00
jschoubben 1f4c67a01b The mesh's one resolver: its seat, a provider's address, zones, and a node's hosts file (hq ADR 0194, 0196, 0199)
- mesh-dns-resolver: a mesh seat delivering wildcard-resolution, so every node's resolver
  configuration resolves to its one holder; node-dns-resolver kept until nothing claims it.
- ${bound:<provision>:address}: the providing machine's private address, for the one consumer
  that cannot use a name — a machine's resolver configuration.
- zone: a module declares the zone it answers and the listen that answers it; the controller
  settles it per node, refuses duplicates and shadowing, and hands the resolver .Zones to forward.
- node-hosts-file: a node seat whose holder owns /etc/hosts, with entries/add/remove.
The resolver tests follow the catalogue: no runtime dns (containers copy the machine's resolvers),
live-restore held by resolv-conf, resolv.conf naming the resolver by address then a public one.
2026-10-04 17:32:08 +02:00
mesh-admin 5474ea2d41 Merge pull request 'A seat says what it receives, and its holder places every module's contribution (hq ADR 0212)' (#269) from feat/0212-contributions-to-a-seat into main 2026-10-04 14:48:13 +00:00
jochen b7912172af A seat says what it receives, and its holder places every module's contribution (hq ADR 0212)
Each contribution grain was a manifest field and a renderer of its own; a module now contributes
to any seat with a kind that seat receives, the holder places it with
${contribution:<seat>:<kind>}, and the contribution depends on the seat. node-hotkeys is the
first new seat to receive (triggers); the display session receives window-manager config.
2026-10-04 16:48:01 +02:00
mesh-admin b36babcd7d Merge pull request 'node-power: the power seat, and code for its moments (hq ADR 0211)' (#268) from feat/0211-node-power into main 2026-10-04 13:59:28 +00:00
jochen 49cf0aa562 node-power: the power seat, and code for its moments placed by its holder (hq ADR 0211)
A module that needs code after waking wrote into the service manager's sleep units; it now
contributes shell code for a named moment, which derives a dependency on node-power.
2026-10-04 15:59:17 +02:00
mesh-admin 5a4f73f761 Merge pull request 'A contribution depends on the seat that receives it; a collision is refused at assign (hq ADR 0210, issue 235)' (#267) from feat/0210-a-contribution-depends-on-its-seat into main 2026-10-04 13:45:46 +00:00
jochen 6af891e358 A contribution depends on the seat that receives it, and a collision is refused at assign (hq ADR 0210, issue 235)
The environment and shell contributions were written nowhere on a node without their holder;
they now derive a dependency on node-environment, node-login-shell or node-display-server, met
and refused as ADR 0207's are. Two modules declaring one package, path or unit made the node
unresolvable after the assignment was recorded; that is refused first now, because no later
assignment can complete it.
2026-10-04 15:45:35 +02:00
mesh-admin 568f55fd2e Merge pull request 'Refuse an unmet seat dependency the catalogue could meet (hq ADR 0207 §4)' (#266) from feat/0207-refuse-unmet-seat-dependencies into main 2026-10-04 11:07:52 +00:00
jochen 35314175f2 Refuse an unmet seat dependency the catalogue could meet (hq ADR 0207 §4)
status reported no unmet dependency on any node once systemd, pacman and docker
were assigned to all four (to-be 42), which is the condition ADR 0207 set for the
switch. A dependency no catalogue module could meet stays a report before and
after the switch, as assign already said it: there is no remedy to name.
2026-10-04 13:07:45 +02:00
mesh-admin ec2e6255a9 Merge pull request 'The unmet-dependency report names only the nodes an act touched (hq ADR 0207)' (#265) from fix/unheld-report-names-only-the-nodes-acted-on into main 2026-10-04 10:52:34 +00:00
jochen f3f34a170e Hold ssh-client to its new shape: an include region first, the mesh's hosts in config.d (mesh-catalog #266) 2026-10-04 12:52:03 +02:00
jochen e2622fd031 An act says the unmet seat dependencies of the node it acted on, not the mesh's (hq ADR 0207)
assign and unassign say only what they changed on their node; push <node>
lists that node's, push to many counts each and points at status. The
once-per-change log is the serving controller's alone: a one-shot command
starts with no memory, so it logged every node on every call.
2026-10-04 12:50:25 +02:00
mesh-admin 3ee32970ef Merge pull request 'Seat dependencies (hq ADR 0207), the graphical session's seats and display provisions (ADR 0208), groups from several modules' (#264) from feat/0207-a-module-depends-on-the-seats-that-apply-its-resources into main 2026-10-04 10:43:11 +00:00
jochen 11b654499b Several modules may add groups to one account; its shell and home stay one module's
The host only ever adds groups, so the container runtime's module can put the
operator in its group while the shell's module sets the same account's shell.
2026-10-04 12:42:00 +02:00
jochen d69e19103c The graphical session's seats, a display's machine reach, and the session's slots (hq ADR 0208)
Seed the eleven node seats with the verbs they start with. A provision may
have the machine's reach: a requirement for it resolves only to a provider
in the node's own set, is never pulled in, and is refused naming who could.
A shell contribution's for gains xinitrc and xresources, placed only by the
holder of node-display-server.
2026-10-04 12:38:53 +02:00
jochen 10f948e970 A module depends on the node seats that apply its resources (hq ADR 0207)
Seed node-package-manager and node-container-runtime. Derive each module's
dependencies from its declared service, package and container resources;
judge them over the node's whole set, exempting the foundation. Refuse at
assign (several modules may go on as one act) and at unassign of the last
holder; report at composition in status, behind one switch.
2026-10-04 12:34:11 +02:00
mesh-admin f421d4588c Merge pull request 'A grant secret belongs to whoever provisions (hq 225); the sweep skips what it will not address (hq 226); a container publishes only what it declares (hq 227)' (#263) from fix/a-grant-secret-is-read-by-the-account-that-provisions into main 2026-10-04 10:27:30 +00:00
jschoubben 74b0dab34c A container publishes only a port its module declares (hq issue 227)
The short form is a question the mesh answers: "80" means publish what the
software calls 80, and the mesh fills in the machine's half from the port it
assigned. It can only assign one for a port the module declared, so a number
appearing nowhere in listens gets no assignment and reaches the machine as
written — which is how the photo module asked for port 80 on the node whose
reverse proxy holds it.

Four modules publish 80 quite safely, because they declare 80. The difference
is the declaration, not the number. A catalogue-wide test now says so; it
names all three offenders against the catalogue as it was.
2026-10-04 12:25:27 +02:00
jschoubben 41b20b2782 A grant secret belongs to whoever provisions, and the sweep skips what it will not address
Issue 225. The mesh seals one credential per consumer beside the provider's
contributions file, and wrote it root-owned. That was right while a module's
own code ran in a container as root; ADR 0198 moved that code under the node's
runtime, as the node's account, and the secret stayed root's. On the control
machine two consumers went unprovisioned for three hours and the only sign
was a line reading 'secret not readable yet', 4330 times.

The same sentence is already written for a module's own secrets a few hundred
lines above — 'a root-owned 0600 file is one that process cannot read'. This
is that rule reaching the other kind of secret the mesh writes for a module.

Issue 226. The sweep met a reference recorded with the store's old address,
read 'I will not address this' as 'the store refuses everything', and
collected none of the 1681 it had found. Two changes: references from build
records are read through Recorded, where the provenance is known — not in
LetGo, which cannot tell one registry host from another and must stay strict
— and a reference the sweep will not address is now ErrNotOurs, skipped,
never a reason to stop. Only the store refusing ends a sweep.

make check: the two failures both fail on main as well — the resolver test
(hq 202/203) and the service-manager test, which reads this machine's own
shell environment.
2026-10-04 12:21:49 +02:00
mesh-admin 912e9f4e85 Merge pull request 'Assert every declared state's bucket on each push (hq ADR 0201)' (#262) from fix/buckets-on-push into main 2026-10-04 09:21:25 +00:00
jochen babd7b2f47 Assert every declared state's bucket on each push, before the memberships that name it (novox/hq ADR 0201)
The raise at start was the only place buckets were asserted, so a module
registered and assigned since had none until the control plane restarted —
found on the first module to declare state.
2026-10-04 11:13:34 +02:00
mesh-admin 892dfd1d08 Merge pull request 'Module state is hq ADR 0201 after all' (#261) from fix/module-state-is-0201 into main 2026-10-04 09:03:15 +00:00
jochen cfac579392 Module state is hq ADR 0201 after all: the derived-value record moved to 0202 on hq main 2026-10-04 11:02:42 +02:00
mesh-admin fe0d295490 Merge pull request 'Module state is hq ADR 0202 (0201 landed first for a provider's derivations)' (#258) from fix/adr-0202-module-state into main 2026-10-04 09:01:22 +00:00
mesh-admin d8a0238e02 Merge pull request 'The account's environment and the shell's contributions (hq ADR 0203, ADR 0204, to-be 41 WP2)' (#260) from feat/the-shell-and-its-environment into main 2026-10-04 08:49:35 +00:00
jochen dcf710a8d5 Merge remote-tracking branch 'origin/main' into feat/the-shell-and-its-environment 2026-10-04 10:31:03 +02:00
mesh-admin a2003ab616 Merge pull request 'while-stopped names the container as the machine knows it (hq ADR 0189)' (#259) from fix/while-stopped-names-the-composed-id into main 2026-10-04 02:18:44 +00:00
jochen f19a2254ac Compose the account's environment and the shell's code from every module (hq ADR 0203, 0204)
A module contributes environment variables, PATH entries and shell code in named slots;
the holder of the matching seat places them with ${environment:posix|systemd} and
${shell:<shell>:<slot>}. Rendered in module order with a naming line per contribution,
PATH entries added only when missing, machine facts resolved first. A variable two
modules set, or a placeholder outside its seat's holder, is refused at parse (the
catalogue check) and at composition. Filled after every other placeholder pass, so no
scanner ever reads a shell's own ${...}.
2026-10-04 04:03:50 +02:00
jochen 7d46e48b26 The account's environment and the login shell are the mesh's seats (hq ADR 0203, 0204)
node-environment says which module writes the account's environment; node-login-shell
replaces the module-declared login-shell, so a second shell claims it rather than
declaring a rival, and execute is the mesh's contract. login-shell is refused as a
module's seat name. Seeded into a live store by the existing additive seeding.
2026-10-04 04:03:50 +02:00
jochen 78915f9f7a Module state is hq ADR 0202: 0201 landed first for a provider's derivations 2026-10-04 03:44:50 +02:00
160 changed files with 16792 additions and 1028 deletions
+386
View File
@@ -0,0 +1,386 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// What a holder of the build seat answers for, on its own machine (novox/hq ADR 0219).
//
// **The queue is the controller's; the build running here is this machine's.** The controller can
// see and change what waits in the seat's queue, but an ask a holder already took is a process tree
// on this machine and containers in this machine's runtime, and only this machine can end them. So
// the holder serves four verbs on the seat's subjects for this machine: what it is building, kill
// it, pause, resume.
//
// **One build at a time** (ADR 0190), which is also what builder.Said assumes — a package global
// set per build — so "the build running here" is one or none, and kill names it by id so a call
// that arrives as one build ends and the next begins cannot end the wrong one.
// holder is this machine's state as a holder of the build seat.
type holder struct {
on, seat string
// workspace is where the paused flag is kept, so a holder restarted while paused stays paused
// rather than silently taking work again.
workspace string
// say publishes this machine's state: whether it takes work (link.HolderState).
say func(link.HolderState) error
// remove runs what a kill needs outside the build: the containers left behind.
remove builder.Runner
mu sync.Mutex
paused bool
running *running
}
// running is the build this machine is doing.
type running struct {
request link.BuildRequest
step string
started time.Time
cancel context.CancelFunc
killed bool
// returned is the build's own work having ended, before a kill or not; outcome is what was then
// announced, and announced whether it went out.
returned bool
outcome string
announced bool
done chan struct{}
}
// pausedFile is where the flag lives in the workspace.
func pausedFile(workspace string) string { return filepath.Join(workspace, ".mesh-builder-paused") }
// newHolder reads the paused flag the workspace keeps.
func newHolder(on, seat, workspace string, say func(link.HolderState) error) *holder {
h := &holder{on: on, seat: seat, workspace: workspace, say: say, remove: plainRun}
if _, err := os.Stat(pausedFile(workspace)); err == nil {
h.paused = true
}
return h
}
// Paused is asked by the taking loop before every fetch.
func (h *holder) Paused() bool {
h.mu.Lock()
defer h.mu.Unlock()
return h.paused
}
// setPaused records the flag in the workspace first and then in memory, so what this holder says
// it is and what it would be after a restart never differ.
func (h *holder) setPaused(paused bool) error {
path := pausedFile(h.workspace)
if paused {
if err := os.MkdirAll(h.workspace, 0o755); err != nil {
return err
}
if err := os.WriteFile(path, []byte(time.Now().UTC().Format(time.RFC3339)+"\n"), 0o644); err != nil {
return fmt.Errorf("cannot keep the paused flag in %s: %w", path, err)
}
} else if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove the paused flag %s: %w", path, err)
}
h.mu.Lock()
h.paused = paused
h.mu.Unlock()
h.announce()
return nil
}
// announce says whether this machine takes work. Never fatal: the flag is kept either way, and the
// controller reading an older state is a plan read as late rather than a build lost.
func (h *holder) announce() {
if h.say == nil {
return
}
if err := h.say(link.HolderState{On: h.on, Paused: h.Paused(), At: time.Now().UTC().Format(time.RFC3339Nano)}); err != nil {
fmt.Fprintf(os.Stderr, "cannot say whether this machine takes builds: %v\n", err)
}
}
// stateWhilePaused says this machine's state at start, and again every while it stays paused, so
// a pause outlives the events stream's retention.
func (h *holder) stateWhilePaused(ctx context.Context, every time.Duration) {
h.announce()
tick := time.NewTicker(every)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
if h.Paused() {
h.announce()
}
}
}
}
// begin records the build this machine took, with the cancel that ends it.
func (h *holder) begin(request link.BuildRequest, cancel context.CancelFunc) *running {
r := &running{request: request, started: time.Now(), cancel: cancel, done: make(chan struct{})}
h.mu.Lock()
h.running = r
h.mu.Unlock()
return r
}
// end clears it, and lets a kill waiting on it know it is over.
func (h *holder) end(r *running) {
h.mu.Lock()
if h.running == r {
h.running = nil
}
h.mu.Unlock()
close(r.done)
}
// stepped records the step a build is at, for `current`.
func (h *holder) stepped(r *running, step string) {
h.mu.Lock()
r.step = step
h.mu.Unlock()
}
// currentBuild is what `current` answers.
type currentBuild struct {
On string `json:"on"`
Paused bool `json:"paused"`
Running *struct {
ID string `json:"id"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
Step string `json:"step,omitempty"`
Started string `json:"started"`
Elapsed string `json:"elapsed"`
} `json:"running,omitempty"`
Said string `json:"said"`
}
func (h *holder) current() currentBuild {
h.mu.Lock()
defer h.mu.Unlock()
out := currentBuild{On: h.on, Paused: h.paused}
taking := "taking builds"
if h.paused {
taking = "paused, taking no new build"
}
if h.running == nil {
out.Said = fmt.Sprintf("%s is building nothing; %s", h.on, taking)
return out
}
r := h.running
elapsed := time.Since(r.started).Round(time.Second)
out.Running = &struct {
ID string `json:"id"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
Step string `json:"step,omitempty"`
Started string `json:"started"`
Elapsed string `json:"elapsed"`
}{r.request.ID, r.request.Repository, r.request.Path, r.request.Ref, r.step,
r.started.UTC().Format(time.RFC3339), elapsed.String()}
out.Said = fmt.Sprintf("%s is building %s (%s) at %s for %s; %s",
h.on, r.request.Repository, r.request.ID, orNothing(r.step), elapsed, taking)
return out
}
// The bounds a kill keeps: each pass removing containers, and the wait for the build to end between
// them. The worst case — 15s, 20s, 15s — is inside what the controller waits for the answer
// (killAnswer in the controller's queue.go, 75s).
var (
killRemoves = 15 * time.Second
killWaits = 20 * time.Second
)
// kill ends the build with this id, if it is the one running here and still working: its context
// cancelled — which kills each command's process group — and the containers it started removed by
// their label, once at once and again after the build has ended, so one created while it was being
// killed is not left. The build's own goroutine announces it failed, killed by hand, and settles the
// ask so it is not redelivered; the answer says whether that happened.
func (h *holder) kill(id string) (string, error) {
h.mu.Lock()
r := h.running
if r == nil || r.request.ID != id {
doing := "nothing"
if r != nil {
doing = r.request.ID
}
h.mu.Unlock()
return "", fmt.Errorf("%s is not building %s; it is building %s. `queue` says where an ask is", h.on, id, doing)
}
if r.returned {
h.mu.Unlock()
return "", fmt.Errorf("%s on %s has already ended on its own and is saying how; `builds` shows it", id, h.on)
}
r.killed = true
cancel, done := r.cancel, r.done
h.mu.Unlock()
cancel()
removed, removeErr := h.removeContainers(id)
ended := false
select {
case <-done:
ended = true
case <-time.After(killWaits):
}
again, againErr := h.removeContainers(id)
removed += again
if removeErr == nil {
removeErr = againErr
}
containers := fmt.Sprintf("%d container(s) it started removed", removed)
if removeErr != nil {
containers = "its containers could not all be listed or removed: " + removeErr.Error()
}
if !ended {
return fmt.Sprintf("killed %s on %s: %s; the build has not finished ending yet — `builds` says when "+
"its outcome is in", id, h.on, containers), nil
}
h.mu.Lock()
outcome, announced := r.outcome, r.announced
h.mu.Unlock()
if !announced {
return fmt.Sprintf("killed %s (%s) on %s: its commands ended and %s, and its outcome could not be "+
"announced — the ask is not settled and will be handed out again", id, r.request.Repository, h.on,
containers), nil
}
return fmt.Sprintf("killed %s (%s) on %s: its commands ended, %s, and its outcome announced as failed, %s — "+
"settled, so it is not handed to another machine", id, r.request.Repository, h.on, containers, outcome), nil
}
// removeContainers is one pass of removing what the build left, bounded.
func (h *holder) removeContainers(id string) (int, error) {
cleanup, stop := context.WithTimeout(context.Background(), killRemoves)
defer stop()
return builder.RemoveContainersOf(cleanup, h.remove, id)
}
// returned records that the build's work ended, and says whether a kill came first — only then is
// the build killed; an error it ended with on its own is its own outcome.
func (h *holder) returned(r *running) bool {
h.mu.Lock()
defer h.mu.Unlock()
r.returned = true
return r.killed
}
// said records the outcome announced, and whether it went out, for a kill to answer with.
func (h *holder) said(r *running, outcome string, announced bool) {
h.mu.Lock()
r.outcome, r.announced = outcome, announced
h.mu.Unlock()
}
// handlers are the seat's verbs, as this machine answers them.
func (h *holder) handlers() map[string]link.ToolHandler {
return map[string]link.ToolHandler{
"current": func(context.Context, json.RawMessage) (any, error) { return h.current(), nil },
"kill": func(_ context.Context, raw json.RawMessage) (any, error) {
var args struct {
ID string `json:"id"`
}
if err := json.Unmarshal(raw, &args); err != nil || strings.TrimSpace(args.ID) == "" {
return nil, errors.New("kill needs the build's id")
}
said, err := h.kill(strings.TrimSpace(args.ID))
if err != nil {
return nil, err
}
return map[string]any{"said": said}, nil
},
"pause": func(context.Context, json.RawMessage) (any, error) {
if err := h.setPaused(true); err != nil {
return nil, err
}
said := h.on + " is paused: it takes no new build until resumed"
if c := h.current(); c.Running != nil {
said += "; " + c.Running.ID + " runs on and finishes"
}
return map[string]any{"said": said, "paused": true}, nil
},
"resume": func(context.Context, json.RawMessage) (any, error) {
if err := h.setPaused(false); err != nil {
return nil, err
}
return map[string]any{"said": h.on + " takes builds again", "paused": false}, nil
},
}
}
func orNothing(s string) string {
if s == "" {
return "its start"
}
return s
}
// plainRun runs a command outside any build: no line reaches a build's log, because the build whose
// log it would be is the one being ended.
func plainRun(ctx context.Context, dir, name string, args ...string) (string, error) {
cmd := exec.CommandContext(ctx, name, args...)
cmd.Dir = dir
out, err := cmd.CombinedOutput()
if err != nil {
return string(out), fmt.Errorf("%s %s: %w: %s", name, strings.Join(args, " "), err, strings.TrimSpace(string(out)))
}
return string(out), nil
}
// announcement is what this holder answers discovery with (novox/hq ADR 0195, ADR 0197): this
// machine's verbs of the seat, in the shape every tool runtime announces a seat's verb — kind seat,
// the module answering, the seat, scope node, the machine, its description and argument schema — so
// the console finds `<node>/node-build-agent.kill` by searching, as it finds any seat's verb.
//
// One service per machine, named for the seat and identified by the machine, so the answer is this
// machine's four verbs and nothing more. The verbs are the compiled seat row's: a build machine has no
// store, and what it serves is what this binary was built to serve.
func announcement(seat, module, node string) micro.Info {
s, _ := catalogue.SeatNamed(seat)
var endpoints []micro.EndpointInfo
for _, v := range s.Serves {
schema, _ := json.Marshal(v.Input)
endpoints = append(endpoints, micro.EndpointInfo{
Name: seat + "__" + v.Name,
Subject: link.NodeSeatToolSubject(seat, v.Name, node),
// The queue group the verbs are served in, as every runtime announces its own.
QueueGroup: "seat." + seat,
Metadata: map[string]string{
"kind": "seat", "module": module, "tool": v.Name, "seat": seat, "scope": "node",
"node": node, "interchangeable": "false", "description": v.Description, "schema": string(schema),
},
})
}
return micro.Info{
ServiceIdentity: micro.ServiceIdentity{Name: seat, ID: node, Version: "0.1.0",
Metadata: map[string]string{"seat": seat, "scope": "node", "node": node, "module": module}},
Description: "what the build running on " + node + " is, and ending, pausing and resuming it (novox/hq ADR 0219)",
Endpoints: endpoints,
}
}
// moduleOf is the module a credential was issued for: its user is `<node>.<module>`.
func moduleOf(user string) string {
if _, module, ok := strings.Cut(user, "."); ok && module != "" {
return module
}
return "build-agent"
}
+148
View File
@@ -0,0 +1,148 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// A holder paused stays paused across its restart: the flag is kept in its workspace (novox/hq ADR
// 0219), and what it says about itself follows.
func TestAPausedHolderStaysPausedAcrossARestart(t *testing.T) {
workspace := t.TempDir()
var said []link.HolderState
h := newHolder("ace", link.TheBuildMachine, workspace, func(s link.HolderState) error {
said = append(said, s)
return nil
})
if h.Paused() {
t.Fatal("a new holder starts paused")
}
if _, err := h.handlers()["pause"](context.Background(), json.RawMessage(`{}`)); err != nil {
t.Fatal(err)
}
if !h.Paused() || len(said) != 1 || !said[0].Paused || said[0].On != "ace" {
t.Fatalf("paused: %v, said %+v", h.Paused(), said)
}
again := newHolder("ace", link.TheBuildMachine, workspace, nil)
if !again.Paused() {
t.Fatal("restarted, the holder forgot it was paused")
}
if _, err := again.handlers()["resume"](context.Background(), json.RawMessage(`{}`)); err != nil {
t.Fatal(err)
}
if newHolder("ace", link.TheBuildMachine, workspace, nil).Paused() {
t.Fatal("resumed, the holder came back paused")
}
}
// kill ends the build with that id — its context, which ends its commands — removes what it left by
// label, and refuses an id it is not building.
func TestKillEndsTheBuildRunningHereAndNoOther(t *testing.T) {
h := newHolder("ace", link.TheBuildMachine, t.TempDir(), nil)
var removed []string
h.remove = func(_ context.Context, _ string, name string, args ...string) (string, error) {
removed = append(removed, name+" "+strings.Join(args, " "))
if args[0] == "ps" {
return "c1\n", nil
}
return "", nil
}
kill := h.handlers()["kill"]
if _, err := kill(context.Background(), json.RawMessage(`{"id":"build-1"}`)); err == nil {
t.Fatal("killed a build while none ran")
}
building, cancel := context.WithCancel(context.Background())
r := h.begin(link.BuildRequest{ID: "build-1", Repository: "novox/a"}, cancel)
h.stepped(r, "image")
if c := h.current(); c.Running == nil || c.Running.ID != "build-1" || c.Running.Step != "image" {
t.Fatalf("current says %+v", c)
}
if _, err := kill(context.Background(), json.RawMessage(`{"id":"build-2"}`)); err == nil ||
!strings.Contains(err.Error(), "build-1") {
t.Fatalf("killed another id: %v", err)
}
// The build's own goroutine: it ends when its context does, as a build's commands do, and
// announces what came of it.
go func() {
<-building.Done()
if h.returned(r) {
h.said(r, link.KilledByHand, true)
}
h.end(r)
}()
answer, err := kill(context.Background(), json.RawMessage(`{"id":"build-1"}`))
if err != nil {
t.Fatal(err)
}
if building.Err() == nil {
t.Fatal("the build's context was not cancelled")
}
if !r.killed {
t.Error("the build is not marked killed, so it would be announced as an ordinary failure")
}
said := answer.(map[string]any)["said"].(string)
if !strings.Contains(said, "2 container(s)") || !strings.Contains(said, "announced as failed") || !strings.Contains(said, link.KilledByHand) {
t.Errorf("kill said %q", said)
}
// Removed at the kill and again once the build had ended: a container made in between is caught.
if len(removed) != 4 || !strings.Contains(removed[0], "label=mesh.build=build-1") || !strings.Contains(removed[2], "label=mesh.build=build-1") {
t.Errorf("removed %v", removed)
}
if c := h.current(); c.Running != nil {
t.Errorf("after the kill current says %+v", c)
}
}
// A holder announces this machine's verbs of the seat as the console reads a seat's verb, and no more.
func TestAHolderAnnouncesItsMachinesVerbsForTheConsole(t *testing.T) {
info := announcement(link.TheBuildMachine, moduleOf("ace.build-agent"), "ace")
if info.Name != "node-build-agent" || info.ID != "ace" || len(info.Endpoints) != 4 {
t.Fatalf("announced %s/%s with %d endpoints", info.Name, info.ID, len(info.Endpoints))
}
kill := info.Endpoints[1]
md := kill.Metadata
if kill.Subject != "mesh.seat.node-build-agent.tool.kill.ace" || kill.QueueGroup != "seat.node-build-agent" || md["kind"] != "seat" || md["seat"] != "node-build-agent" ||
md["scope"] != "node" || md["node"] != "ace" || md["tool"] != "kill" || md["module"] != "build-agent" ||
!strings.Contains(md["description"], "killed by hand") || !strings.Contains(md["schema"], `"id"`) {
t.Fatalf("kill is announced as %+v", kill)
}
body, err := json.Marshal(info)
if err != nil || len(body) > 8*1024 {
t.Fatalf("the answer is %d bytes (%v)", len(body), err)
}
}
// A build that ended on its own as the kill arrived says what it did: the kill is refused, and its
// own error is its outcome. One whose outcome could not be announced is not said to be settled.
func TestAKillArrivingAfterTheBuildEndedIsRefusedAndAnUnannouncedKillSaysSo(t *testing.T) {
h := newHolder("ace", link.TheBuildMachine, t.TempDir(), nil)
h.remove = func(context.Context, string, string, ...string) (string, error) { return "", nil }
_, cancel := context.WithCancel(context.Background())
r := h.begin(link.BuildRequest{ID: "build-1"}, cancel)
if killed := h.returned(r); killed {
t.Fatal("a build nobody killed reads as killed")
}
if _, err := h.kill("build-1"); err == nil || !strings.Contains(err.Error(), "ended on its own") {
t.Fatalf("killed a build that had ended: %v", err)
}
h.end(r)
building, cancel := context.WithCancel(context.Background())
r = h.begin(link.BuildRequest{ID: "build-2"}, cancel)
go func() {
<-building.Done()
if h.returned(r) {
h.said(r, link.KilledByHand, false)
}
h.end(r)
}()
said, err := h.kill("build-2")
if err != nil || strings.Contains(said, "announced as failed") || !strings.Contains(said, "could not be announced") {
t.Fatalf("kill said %q (%v)", said, err)
}
}
+92 -12
View File
@@ -19,11 +19,13 @@ import (
"context"
"encoding/json"
"fmt"
"log"
"net/url"
"os"
"os/signal"
"strings"
"syscall"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builder"
@@ -107,48 +109,96 @@ func run() error {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
machine, err := takeWorkFrom(credential, on)
js, seat, err := dialFor(credential)
if err != nil {
return err
}
defer js.Close()
// **This machine's holder: paused or not, and the build it is running** (novox/hq ADR 0219). The
// paused flag is read from the workspace before anything is taken, so a holder restarted while
// paused takes nothing.
h := newHolder(on, seat, workspace, func(state link.HolderState) error {
body, err := json.Marshal(state)
if err != nil {
return err
}
_, err = js.Context().Publish(link.BuildPausedOf(seat, on), body)
return err
})
if h.Paused() {
fmt.Fprintf(os.Stderr, "paused (kept in %s): taking no build until resumed\n", pausedFile(workspace))
}
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused})
defer machine.Close()
// The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired
// one serves none, and a subscription its holder has no grant for would be refused for ever.
if seat == link.TheBuildMachine {
stopServing, err := link.OverNATS{Conn: js.Conn()}.ServeNodeSeatTools(seat, on, h.handlers(),
log.New(os.Stderr, "", 0))
if err != nil {
return err
}
defer stopServing()
// And says so, for the console to find (novox/hq ADR 0197).
stopAnnouncing, err := link.OverNATS{Conn: js.Conn()}.Announce(
announcement(seat, moduleOf(credential.User), on), log.New(os.Stderr, "", 0))
if err != nil {
return err
}
defer stopAnnouncing()
go h.stateWhilePaused(ctx, time.Hour)
}
fmt.Fprintf(os.Stderr, "building for the mesh, publishing to %s\n", registry)
publisher := builder.Registry{Address: registry, Run: builder.Command}
return machine.Take(ctx, func(ctx context.Context, work link.Build) {
answer(ctx, publisher, on, workspace, work)
answer(ctx, publisher, on, workspace, work, h)
})
}
// takeWorkFrom opens this machine's link to whichever bus the mesh is on.
// dialFor opens this machine's link to whichever bus the mesh is on, and says which build seat it
// holds.
//
// **One place chooses**, as everywhere else the bus change went (novox/hq ADR 0116 step 5): a build
// machine told about both would take work from one and answer on the other, and every log line would
// say it was fine.
func takeWorkFrom(credential Credential, on string) (link.BuildMachine, error) {
func dialFor(credential Credential) (*broker.JetStream, string, error) {
// **The credential names the bus, and there is one** (novox/hq ADR 0131, design 28 task 5.5).
// A credential for the mesh's bus carries user, password and fingerprint beside the address,
// and that is enough to dial it, pinned.
if !credential.onTheNewBus() {
return nil, fmt.Errorf("the credential at hand names %q, which is not the mesh's bus", credential.URL)
return nil, "", fmt.Errorf("the credential at hand names %q, which is not the mesh's bus", credential.URL)
}
js, err := broker.DialPinned(credential.natsURL(), credential.Fingerprint)
if err != nil {
return nil, err
return nil, "", err
}
// **The seat this machine serves is the one its credential claims** (novox/hq ADR 0190, the
// handover): the mesh issues a build machine's credential naming the seat its module claims,
// and one binary serves the old role as `builder` and the new as `build-agent` from that alone.
seat := link.BuildSeatClaimed(credential.seatsClaimed())
fmt.Fprintf(os.Stderr, "taking build work as a holder of %s\n", seat)
return link.MachineOverNATSOn(js, on, seat), nil
return js, seat, nil
}
// answer does one build and says what happened, whichever way it went.
func answer(ctx context.Context, publisher builder.Publisher, on, workspace string, work link.Build) {
func answer(ctx context.Context, publisher builder.Publisher, on, workspace string, work link.Build, h *holder) {
request := work.Request()
// **Its own context, so it can be killed alone** (novox/hq ADR 0219): cancelled by `kill`, it ends
// this build's commands and nothing else; the machine's own context ending — a SIGTERM — still
// reaches it through the parent, and that keeps today's meaning below.
building, cancel := context.WithCancel(ctx)
defer cancel()
var mine *running
if h != nil {
mine = h.begin(request, cancel)
defer h.end(mine)
}
// **First thing, and to stdout.** A build request that arrives and produces no visible line until
// it either finishes or fails is indistinguishable from one that never arrived — which cost a long
// diagnosis against a running mesh, chasing "the handler never fired" when the truth was only that
@@ -162,6 +212,9 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
say := func(step, message string) {
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
work.Say(step, message)
if mine != nil && step != "run" && step != "output" {
h.stepped(mine, step)
}
}
builder.Said = say
defer func() { builder.Said = nil }()
@@ -171,7 +224,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
result := link.BuildResult{
ID: request.ID, Repository: request.Repository, Path: request.Path,
Ref: request.Ref, On: on, Source: request.Source,
Ref: request.Ref, On: on, Source: request.Source, DryRun: request.DryRun,
}
what := "building " + request.Repository
if request.Path != "" {
@@ -188,11 +241,24 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// The package-registry credential is a build input, so it is resolved before the clone: a
// build that could not have resolved its dependencies is refused in front of the reason, not
// after a clone that then fails at npm ci.
built, err = builder.Build(ctx, builder.Command, publisher,
// Every container it starts is labelled with its id, so a kill finds what outlived the
// docker client (ADR 0219).
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
if err != nil {
// Only a build the kill ended: the kill came before its work did. One that finished — built, or
// failed on its own — in the moment the kill arrived says what it did, and the kill is refused.
killed := false
if mine != nil {
killed = h.returned(mine) && err != nil
}
if killed {
// **Killed by hand is the outcome, whatever the build was doing** (novox/hq ADR 0219): the
// error it ended with is the kill's consequence, not a fault of the source.
result.Failed = link.KilledByHand
say("failed", link.KilledByHand)
} else if err != nil {
// A failure is a result. A build that fails and says nothing is indistinguishable from a
// builder that is not running, and those want completely different responses.
result.Failed = err.Error()
@@ -217,7 +283,21 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
}
}
if err := work.Announce(ctx, result); err != nil {
// A killed build is announced on a context of its own: the build's was the one cancelled, and the
// outcome must go out and the ask be settled — acknowledged, never redelivered to another machine
// to be built again. A machine being stopped is the other case and keeps its meaning: the
// parent's context is gone, nothing is announced or settled, and the ask is redelivered.
announcing := ctx
if killed {
fresh, stop := context.WithTimeout(context.Background(), 30*time.Second)
defer stop()
announcing = fresh
}
announceErr := work.Announce(announcing, result)
if mine != nil {
h.said(mine, result.Failed, announceErr == nil)
}
if err := announceErr; err != nil {
// Said, not fatal: the build happened. A build reported as failed because announcing it
// failed is a lie about work that was done — and the request stays unsettled below only if
// nothing was said at all, so another machine can try.
+147 -22
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/broker"
"slices"
"sort"
"strings"
@@ -39,7 +40,10 @@ import (
//
// It costs a resolution per machine. Assignment is a person typing a command, and being told which
// machines this just blocked is worth more than the milliseconds.
func assign(ctx context.Context, open *stores, node, module string) (string, error) {
func assign(ctx context.Context, open *stores, node string, modules ...string) (string, error) {
if len(modules) == 0 {
return "", fmt.Errorf("assign %s names no module", node)
}
// Held while it is recorded, so it cannot land between a converge's preview and its flip and
// be taken without ever having been previewed (novox/hq ADR 0100).
ctx, release, err := holdNodes(ctx, open, []string{node})
@@ -47,6 +51,16 @@ func assign(ctx context.Context, open *stores, node, module string) (string, err
return "", err
}
defer release()
// **The one assignment refused for what the node lacks** (novox/hq ADR 0207). Everything else
// an assignment leaves unresolved is kept, because assignment is not an ordering; a module whose
// resources are applied through a seat nothing on the node holds is refused, because that order
// — the service manager, the package manager and the runtime before anything that installs,
// runs or contains — is the mesh's to keep. Several modules in one act are judged together, so
// holders that depend on each other go on in one command.
shelf, before, err := seatDependenciesOnAssign(ctx, open, node, modules)
if err != nil {
return "", err
}
// **Before the new assignment can unsettle a seat somebody holds only by being alone**
// (novox/hq 04-ISSUES/170): what the mesh derived so far is written down, and then the
// assignment resolves against a record rather than against a coincidence.
@@ -54,65 +68,176 @@ func assign(ctx context.Context, open *stores, node, module string) (string, err
if err != nil {
return "", err
}
fresh, err := open.inventory.Assign(ctx, node, module)
if err != nil {
return "", err
var lines []string
var added []string
for _, module := range modules {
fresh, err := open.inventory.Assign(ctx, node, module)
if err != nil {
return strings.Join(lines, "\n"), err
}
if !fresh {
// Nothing changed, and saying "is assigned" would read as an action. One node runs one
// of each — the module's name is the assignment's identity (novox/hq ADR 0115).
lines = append(lines, fmt.Sprintf(
"%s already runs %s — one node runs one of each (ADR 0115); nothing changed", node, module))
continue
}
added = append(added, module)
lines = append(lines, fmt.Sprintf("%s is assigned %s", node, module))
}
if !fresh {
// Nothing changed, and saying "is assigned" would read as an action. One node runs one
// of each — the module's name is the assignment's identity (novox/hq ADR 0115).
return fmt.Sprintf("%s already runs %s — one node runs one of each (ADR 0115); nothing changed",
node, module), nil
if len(added) == 0 {
return strings.Join(lines, "\n"), nil
}
said := fmt.Sprintf("%s is assigned %s", node, module)
answer := strings.Join(lines, "\n")
for _, line := range settled {
said += "\n " + line
answer += "\n " + line
}
// What this act changed about this node's unmet seat dependencies, and nothing else (novox/hq
// ADR 0207): a dependency of a module just assigned, or one this assignment met. The rest of the
// node's list, and every other node's, is `status`'s.
for _, line := range unheldChange(shelf, node, before, append(append([]string(nil), before...), added...)) {
answer += "\n " + line
}
// Its bus credential, in the same act (novox/hq issue 203): an assignment pushed before its
// credential exists delivers a process that cannot authenticate and crash-loops until somebody
// runs a second verb and a second push. Issued here when the module speaks on the bus and has
// no credential yet; kept when it has one, so re-assigning rotates nothing.
if line := issueOnAssign(ctx, open, node, module); line != "" {
said += "\n " + line
for _, module := range added {
if line := issueOnAssign(ctx, open, node, module); line != "" {
answer += "\n " + line
}
}
plan, _, err := planFor(ctx, open, node)
if err != nil {
// Kept, and still refused. Both halves are the answer, and the rest of the mesh is still
// worth reporting: this machine's refusal is rarely the only consequence.
return said + blockedElsewhere(ctx, open, node), err
return answer + blockedElsewhere(ctx, open, node), err
}
// Kept, and cannot be hosted here. Said at once rather than discovered at push: a module whose
// capability the machine lacks is on the wrong machine, and the assignment records what a person
// meant while this line says it will not run until it moves. The rest of the node still pushes.
isAdded := map[string]bool{}
for _, m := range added {
isAdded[m] = true
}
for _, u := range plan.Unhostable {
if u.Module != module {
if !isAdded[u.Module] {
continue
}
for _, c := range u.Missing {
said += "\n but " + catalogue.WrongMachine(u.Module, c, node)
answer += "\n but " + catalogue.WrongMachine(u.Module, c, node)
}
}
return said + fmt.Sprintf("\n run `push %s` to send it", node) +
return answer + fmt.Sprintf("\n run `push %s` to send it", node) +
blockedElsewhere(ctx, open, node), nil
}
// unassign takes a module off a node. What it leaves behind is the host's business: a directory
// seatDependenciesOnAssign is the refusal ADR 0207 makes at assignment, or nothing, with the
// catalogue and the node's assignments it was judged against. Modules already assigned are not new
// and are not judged again.
func seatDependenciesOnAssign(ctx context.Context, open *stores, node string, modules []string) (
map[string]catalogue.Manifest, []string, error) {
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return nil, nil, err
}
assigned, err := open.inventory.Assigned(ctx, node)
if err != nil {
return nil, nil, err
}
already := map[string]bool{}
for _, a := range assigned {
already[a] = true
}
var adding []string
for _, m := range modules {
if !already[m] {
adding = append(adding, m)
}
}
// The lines AssignRefusal says beside an assignment it lets through are said by unheldChange
// with everything else this act changed, so they are not said twice.
if _, err := catalogue.AssignRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err
}
// Two modules declaring one package, path or unit is refused before anything is recorded
// (novox/hq ADR 0210, 04-ISSUES/235): kept, the node would not resolve until one came off again.
if err := catalogue.CollisionRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err
}
return shelf, assigned, nil
}
// unassign takes modules off a node. What they leave behind is the host's business: a directory
// holding anything the mesh did not put there is kept (novox/hq ADR 0030).
//
// It reports the rest of the mesh for the same reason assign does, and more sharply: taking a
// module off one machine is the ordinary way to stop providing something to another, and nothing
// about the command's own output would ever have said so.
func unassign(ctx context.Context, open *stores, node, module string) (string, error) {
//
// **Refused when it takes away the last holder of a seat a module left on the node depends on**
// (novox/hq ADR 0207) — the other side of refusing that module's assignment without one. Several
// modules in one act are judged together, so a holder and its dependents come off in one command.
func unassign(ctx context.Context, open *stores, node string, modules ...string) (string, error) {
if len(modules) == 0 {
return "", fmt.Errorf("unassign %s names no module", node)
}
ctx, release, err := holdNodes(ctx, open, []string{node})
if err != nil {
return "", err
}
defer release()
if err := open.inventory.Unassign(ctx, node, module); err != nil {
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
return fmt.Sprintf("%s no longer runs %s — run `push %s` to make it so",
node, module, node) + blockedElsewhere(ctx, open, node), nil
assigned, err := open.inventory.Assigned(ctx, node)
if err != nil {
return "", err
}
// Every one checked before any is taken off, so a refusal leaves the node as it was.
runs := map[string]bool{}
for _, a := range assigned {
runs[a] = true
}
for _, module := range modules {
if !runs[module] {
return "", fmt.Errorf("%s is not assigned to %s", module, node)
}
}
if err := catalogue.UnassignRefusal(shelf, node, assigned, modules); err != nil {
return "", err
}
for _, module := range modules {
if err := open.inventory.Unassign(ctx, node, module); err != nil {
return "", err
}
}
answer := fmt.Sprintf("%s no longer runs %s — run `push %s` to make it so",
node, strings.Join(modules, ", "), node)
var left []string
for _, a := range assigned {
if !slices.Contains(modules, a) {
left = append(left, a)
}
}
for _, line := range unheldChange(shelf, node, assigned, left) {
answer += "\n " + line
}
return answer + blockedElsewhere(ctx, open, node), nil
}
// splitModules is a surface's one `module` field as the modules it names: several, comma-separated,
// are one act (novox/hq ADR 0207), so the holders that depend on each other go on together from the
// command API and the controller seat's verbs as they do from the command line.
func splitModules(field string) []string {
var out []string
for _, m := range strings.Split(field, ",") {
if m = strings.TrimSpace(m); m != "" {
out = append(out, m)
}
}
return out
}
// blockedElsewhere is every OTHER machine that cannot be worked out as things now stand.
+8
View File
@@ -111,6 +111,14 @@ func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testin
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
// The runtime's trust is the runtime's module's to write (novox/hq ADR 0222): a stand-in for it
// asks where this machine reaches the store, as the docker module does.
register(t, open, catalogue.Manifest{Module: "runtime", Version: "1",
Resources: []map[string]any{{"id": "daemon", "type": "file", "path": "/etc/docker/daemon.json",
"into": "json", "content": `{"insecure-registries": ["${seat:mesh-artifact-store:reach}"]}` + "\n"}}})
if _, err := assign(ctx, open, "laptop", "runtime"); err != nil {
t.Fatal(err)
}
on := map[string]bool{"anchor": true, "laptop": true}
node, port, found, err := artifactStoreOnNetwork(ctx, open.inventory, on)
+1 -1
View File
@@ -85,7 +85,7 @@ func reportsReaching(t *testing.T, open *stores, reachable []link.Reach, held ..
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
if err := open.inventory.RecordSent(ctx, record.ID, digestOf(body), nil); err != nil {
t.Fatal(err)
}
if _, err := (link.Enrolment{Inventory: open.inventory}).Heard(ctx, link.Report{
+2 -2
View File
@@ -95,10 +95,10 @@ func commands(who Authenticator) http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("POST /assign", acting(who, true, func(ctx context.Context, open *stores, in request) (string, error) {
return assign(ctx, open, in.Node, in.Module)
return assign(ctx, open, in.Node, splitModules(in.Module)...)
}))
mux.HandleFunc("POST /unassign", acting(who, true, func(ctx context.Context, open *stores, in request) (string, error) {
return unassign(ctx, open, in.Node, in.Module)
return unassign(ctx, open, in.Node, splitModules(in.Module)...)
}))
// Adoption (novox/hq ADR 0100): the same acts as `take`, `converge` and `adopt`.
mux.HandleFunc("POST /take", acting(who, true, func(ctx context.Context, open *stores, in request) (string, error) {
+39 -10
View File
@@ -392,22 +392,34 @@ func buildBehind(ctx context.Context, wait time.Duration) error {
//
// Separated from the command so `--behind` can walk a list without a second path to the same act.
func buildOne(ctx context.Context, source buildSource, path, ref string, wait time.Duration) error {
_, err := buildOneAsked(ctx, source, path, ref, wait, false)
return err
}
// buildOneAsked is buildOne answering the id it asked with — what a plan keeps to match the outcome
// by (novox/hq ADR 0219) — and, for an ask not waited for, optionally a dry run: built and looked
// at, never taken in (issue 240), which is what `replay` asks unless told to register.
func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wait time.Duration,
dryRun bool) (string, error) {
if dryRun && wait != 0 {
return "", errors.New("a dry run waited for is `build --dry-run`")
}
// Before anything is asked of a builder: a source on a seat nobody holds is refused here, with
// the reason, rather than sent to a machine to fail at `git clone`.
repository, err := cloneFrom(ctx, source)
if err != nil {
return err
return "", err
}
ident, err := openIdentity(ctx)
if err != nil {
return err
return "", err
}
defer ident.Close()
server, err := connectLink(ctx, nil, nil, nil)
if err != nil {
return err
return "", err
}
defer server.Close()
@@ -420,6 +432,7 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
Ref: ref,
Held: heldBy(ctx),
Seats: seatBases(ctx),
DryRun: dryRun,
}
fmt.Printf("asked for %s", source)
if source.Seat != "" {
@@ -438,7 +451,7 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
if err != nil {
return err
return "", err
}
defer ask.Close()
fmt.Printf(" of %s\n", seat)
@@ -449,26 +462,31 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
// is still here. A tool call cannot hold a connection for the minutes a build takes; it
// follows the build by its id instead.
if err := ask.Ask(ctx, request); err != nil {
return err
return "", err
}
if dryRun {
fmt.Printf("asked as a dry run, not waited for: `builds --log %s` follows it as it runs; "+
"its outcome is not taken in\n", request.ID)
return request.ID, nil
}
fmt.Printf("asked, not waited for: `builds --log %s` follows it as it runs, and `builds` "+
"shows what came of it; the module is registered when the outcome comes\n", request.ID)
return nil
return request.ID, nil
}
result, err := ask.Submit(ctx, request, wait)
if err != nil {
return err
return request.ID, err
}
open, err := openStores(ctx)
if err != nil {
return err
return request.ID, err
}
defer open.Close()
manifest, kept, err := takeIn(ctx, open.inventory, result)
if err != nil {
return err
return request.ID, err
}
// Said as recorded: what each artifact is, not where this builder happened to push it.
for _, made := range kept.Made {
@@ -478,7 +496,7 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
manifest.Module, manifest.Version, result.On, short(result.Commit))
saysWhenThePolicyActs(ctx, open.inventory, manifest.Module)
fmt.Printf(" run `assign <node> %s` to put it somewhere\n", manifest.Module)
return nil
return request.ID, nil
}
// saysWhenThePolicyActs tells whoever built a module that its upgrade policy will send the
@@ -592,6 +610,7 @@ func buildAndShow(ctx context.Context, source buildSource, path, ref string, wai
ID: link.NewBuildID(time.Now()),
Repository: repository, Path: path, Ref: ref,
Held: heldBy(ctx), Seats: seatBases(ctx),
DryRun: true,
}, wait)
if err != nil {
return err
@@ -627,6 +646,8 @@ type answers struct {
reported []inventory.Reported
// plans is what the last merges produced and where each stands (novox/hq ADR 0162).
plans []inventory.Plan
// paused is whether the build seat takes work, which a plan waiting on it says (ADR 0219).
paused pauseView
// refused is why a machine cannot be worked out at all, by name. A different thing from every
// other answer here: those are about a machine that was told something, and this is about one
// that cannot be told anything — it never reaches waiting, because nothing was computed for it
@@ -650,6 +671,14 @@ type answers struct {
// public name on the machine went dark. The holds were correct; they were recorded only in the
// machine's own state file, and the one visible symptom was a count that did not add up.
untaken map[string]map[string]int
// unheld is every module on a machine whose resources are applied through a seat nothing on
// that machine holds (novox/hq ADR 0207), with the modules that could hold it. Reported, not
// refused, until the switch — and while there is any, the mesh is not all well: the order the
// machines' modules are built in is the mesh's to keep, and this is where it says it is not kept.
unheld []catalogue.Unheld
// failing is every consumer a provider says it keeps failing (novox/hq ADR 0224): a provider's
// journal was the only place that said so for a day (04-ISSUES/179).
failing []inventory.ProviderStanding
}
// heldBy is every artifact this mesh has built, for a build that may need one as its base.
+81
View File
@@ -0,0 +1,81 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// consoleWaits is how long the console waits for an answer (mesh-tools node-tools/internal/bus
// RequestTimeout) — the shortest wait of a caller the mesh ships.
const consoleWaits = 30 * time.Second
// **A call's one answer is never later than its caller or the bus allow** (novox/hq issue 265): a
// holder answers within AnswerWithin, which must be inside both the console's wait and the window the
// bus gives an answer. Before, the console waited 30s, the bus 60s, and a push ran as long as it ran.
func TestAVerbAnswersInsideEveryWaitOnIt(t *testing.T) {
if link.AnswerWithin >= consoleWaits/2 {
t.Errorf("a call answers within %s: not well inside the console's %s", link.AnswerWithin, consoleWaits)
}
if link.AnswerWithin >= broker.ResponseTTL {
t.Errorf("a call answers within %s, after the bus stops permitting an answer at %s", link.AnswerWithin, broker.ResponseTTL)
}
}
// A push — named or through command — answers before it runs: it sends the machine holding the bus
// first, and the broker reloading its user list forgets the answer it was about to permit.
func TestAPushAnswersBeforeItSends(t *testing.T) {
for _, c := range []struct {
verb string
args map[string]any
want bool
}{
{"push", map[string]any{"node": "anchor"}, true},
{"push", map[string]any{}, true},
{"command", map[string]any{"command": "push anchor"}, true},
{"command", map[string]any{"command": "push --behind"}, true},
{"command", map[string]any{"command": "builds"}, false},
{"status", map[string]any{}, false},
{"assign", map[string]any{"node": "anchor", "module": "m"}, false},
} {
argv, err := argvFor(c.verb, c.args)
if err != nil {
t.Fatalf("%s %v: %v", c.verb, c.args, err)
}
if got := answersFirst(argv); got != c.want {
t.Errorf("%s %v answers first: %v, want %v", c.verb, c.args, got, c.want)
}
}
}
// `calls` is served, takes a call's id and nothing else, and says plainly when it holds no such call.
func TestCallsIsServedAndSaysWhatItKeeps(t *testing.T) {
handlers, behind, err := seatToolHandlers()
if err != nil || len(behind) != 0 {
t.Fatalf("%v %v", behind, err)
}
calls, ok := handlers["calls"]
if !ok {
t.Fatal("calls is not served")
}
if _, err := calls(context.Background(), json.RawMessage(`{"node":"anchor"}`)); err == nil ||
!strings.Contains(err.Error(), `"node"`) {
t.Errorf("calls took an argument it does not declare: %v", err)
}
if _, err := calls(context.Background(), json.RawMessage(`{"call":"call-0-0"}`)); err == nil ||
!strings.Contains(err.Error(), "not across a restart") {
t.Errorf("an unknown call was not said plainly: %v", err)
}
got, err := calls(context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if _, listed := got.(map[string]any)["calls"]; !listed {
t.Errorf("calls answered %v", got)
}
}
+73 -6
View File
@@ -31,7 +31,12 @@ func collect(ctx context.Context, inv *inventory.Inventory) {
fmt.Fprintf(os.Stderr, "could not work out what the artifact store may let go of: %v\n", err)
return
}
if len(references) == 0 {
kept, err := inv.KeptArchives(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not work out which archives the artifact store keeps: %v\n", err)
return
}
if len(references) == 0 && len(kept) == 0 {
return
}
shelf, err := inv.Catalogue(ctx)
@@ -59,8 +64,27 @@ func collect(ctx context.Context, inv *inventory.Inventory) {
defer stop()
store := artifacts.Store{Address: address}
// **Hold before letting go** (novox/hq issue 253). The store's collector keeps only what a
// manifest names, and archives were published as bare blobs, so every kept archive is first
// held by its manifest — which backfills the ones published before holders, a few at a time
// as builds come, and is two HEADs each once done. A kept archive that could not be held stops
// the sweep before it deletes anything: "everything kept is held" is the precondition the
// collector's safety rests on, and a store refusing a hold would refuse the deletes too.
wrote, missing, err := holdKept(within, store, kept)
if wrote > 0 {
fmt.Fprintf(os.Stderr, "the artifact store now holds %d more kept archive(s) by a manifest\n", wrote)
}
if missing > 0 {
fmt.Fprintf(os.Stderr, "%d archive(s) the mesh keeps are not in the artifact store at all; "+
"`collection` lists them\n", missing)
}
if err != nil {
fmt.Fprintf(os.Stderr, "not every kept archive could be held, so nothing was let go: %v\n", err)
return
}
var done []string
var left int
var left, skipped int
for i, reference := range references {
if i >= mostPerSweep || within.Err() != nil {
left = len(references) - i
@@ -73,10 +97,22 @@ func collect(ctx context.Context, inv *inventory.Inventory) {
done = append(done, reference)
continue
}
// **Stopped at the first refusal, not pushed through.** A store that refuses one refuses
// all of them — deletion disabled, the store down, the network gone — so going on would
// be a hundred identical failures and a hundred identical log lines in front of whoever
// was building something.
if errors.Is(err, artifacts.ErrNotOurs) {
// **A fact about this record, so this record is skipped** (novox/hq issue 226). Not
// marked collected — the mesh did not remove it and should not claim to — and not a
// reason to stop, because the store was never asked. One of these at the front of
// the oldest-first order ended every sweep until this.
skipped++
if skipped == 1 {
fmt.Fprintf(os.Stderr,
"the sweep will not address %s and went on: %v\n", reference, err)
}
continue
}
// **Stopped at the first refusal by the STORE, not pushed through.** A store that refuses
// one refuses all of them — deletion disabled, the store down, the network gone — so
// going on would be a hundred identical failures and a hundred identical log lines in
// front of whoever was building something.
fmt.Fprintf(os.Stderr, "the artifact store kept %s, so nothing more was asked of it: %v\n",
reference, err)
left = len(references) - i
@@ -97,6 +133,37 @@ func collect(ctx context.Context, inv *inventory.Inventory) {
if left > 0 {
fmt.Fprintf(os.Stderr, "%d more to collect; the next build asks again\n", left)
}
if skipped > 0 {
fmt.Fprintf(os.Stderr, "%d artifact(s) the sweep will not address were skipped\n", skipped)
}
}
// holdKept holds every kept archive by its manifest, stopping at the first refusal by the store.
// Answers how many holders it wrote and how many kept archives the store does not have.
//
// A missing archive is counted rather than fatal: there is nothing to hold, and that is a fact
// for an operator to read (`collection`), not a reason to stop collecting what is not kept. A
// reference the store cannot be asked about is skipped as the deletion loop skips one
// (novox/hq issue 226).
func holdKept(ctx context.Context, store artifacts.Store, kept []string) (wrote, missing int, err error) {
for _, reference := range kept {
if err := ctx.Err(); err != nil {
return wrote, missing, fmt.Errorf("ran out of time before %s: %w", reference, err)
}
did, err := store.Hold(ctx, reference)
switch {
case err == nil:
if did {
wrote++
}
case errors.Is(err, artifacts.Gone):
missing++
case errors.Is(err, artifacts.ErrNotOurs):
default:
return wrote, missing, fmt.Errorf("holding %s: %w", reference, err)
}
}
return wrote, missing, nil
}
// mostPerSweep is how many artifacts one sweep will ask about. Enough that a mesh building
+166
View File
@@ -0,0 +1,166 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"strings"
"github.com/novox/mesh-controller/internal/artifacts"
)
// What the artifact store keeps, whether each kept archive is held, and what the sweep may let go
// (novox/hq issue 253, ADR 0189).
//
// **The question to answer before the store's collector runs for real.** The collector deletes
// every blob no manifest names, and archives were published as bare blobs, so the nightly step
// runs `--dry-run` until every archive the mesh keeps is held by its manifest. The sweep holds
// them as builds come; this says how far that has got — "0 unheld" is the number that lets the
// dry run go.
//
// Reads and changes nothing: each kept archive is asked about with HEADs only. Reached over the
// console through the mesh-controller seat's `command` verb (`collection --json`), which needs no
// new verb in the seat's row.
type collectionReport struct {
// Store is the artifact store as this machine reached it; empty when it is not on the network.
Store string `json:"store"`
// KeptArchives is how many archives the mesh keeps, for either reason.
KeptArchives int `json:"kept_archives"`
// Held is how many of them the store holds by their manifest.
Held int `json:"held"`
// Unheld are the kept archives the collector would delete tonight if it ran for real.
Unheld []string `json:"unheld"`
// Missing are kept archives the store does not have at all.
Missing []string `json:"missing"`
// Unasked is how many could not be asked about, and why the asking stopped.
Unasked int `json:"unasked"`
Stopped string `json:"stopped,omitempty"`
// Eligible is what the sweep may let go of: made by the mesh, kept for no reason, not yet
// collected — split by kind.
Eligible int `json:"eligible"`
EligibleImages int `json:"eligible_images"`
EligibleArchives int `json:"eligible_archives"`
// SafeToCollect is whether every kept archive was asked about and every one is held.
SafeToCollect bool `json:"safe_to_collect"`
}
func collectionCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("collection", flag.ContinueOnError)
asJSON := set.Bool("json", false, "answer as JSON")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 0 {
return errors.New("collection [--json]")
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
kept, err := inv.KeptArchives(ctx)
if err != nil {
return err
}
eligible, err := inv.ToCollect(ctx)
if err != nil {
return err
}
report := collectionReport{KeptArchives: len(kept), Eligible: len(eligible), Unheld: []string{}, Missing: []string{}}
for _, reference := range eligible {
if strings.Contains(reference, "/blobs/") {
report.EligibleArchives++
} else {
report.EligibleImages++
}
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
return err
}
report.Store, err = artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return err
}
if report.Store == "" {
report.Unasked = len(kept)
report.Stopped = "this mesh has no artifact store on its network"
} else {
report.Unasked, report.Stopped = askHeld(ctx, artifacts.Store{Address: report.Store}, kept, &report)
}
report.SafeToCollect = report.Unasked == 0 && len(report.Unheld) == 0
if *asJSON {
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
return encoder.Encode(report)
}
printCollection(report)
return nil
}
// askHeld asks the store about each kept archive, stopping at the first answer that is not about
// the archive: a store that cannot be reached for one cannot be for the next, and a page of
// identical failures says less than one line.
func askHeld(ctx context.Context, store artifacts.Store, kept []string, report *collectionReport) (int, string) {
for i, reference := range kept {
held, err := store.Held(ctx, reference)
switch {
case err == nil && held:
report.Held++
case err == nil:
report.Unheld = append(report.Unheld, reference)
case errors.Is(err, artifacts.Gone):
report.Missing = append(report.Missing, reference)
case errors.Is(err, artifacts.ErrNotOurs):
// KeptArchives names only the mesh's own; counted as unasked if one ever is not.
report.Unasked++
default:
return report.Unasked + len(kept) - i, fmt.Sprintf("asking about %s: %v", reference, err)
}
}
return report.Unasked, ""
}
func printCollection(r collectionReport) {
store := r.Store
if store == "" {
store = "(not on the network)"
}
fmt.Printf("artifact store %s\n", store)
fmt.Printf("kept archives %d\n", r.KeptArchives)
fmt.Printf(" held %d\n", r.Held)
fmt.Printf(" unheld %d\n", len(r.Unheld))
fmt.Printf(" missing %d\n", len(r.Missing))
if r.Unasked > 0 {
fmt.Printf(" not asked %d (%s)\n", r.Unasked, r.Stopped)
}
fmt.Printf("eligible to let go %d (%d images, %d archives)\n", r.Eligible, r.EligibleImages, r.EligibleArchives)
if len(r.Unheld) > 0 {
fmt.Println("\nunheld — the store's collector would delete these; the next build's sweep holds them:")
for _, reference := range r.Unheld {
fmt.Printf(" %s\n", reference)
}
}
if len(r.Missing) > 0 {
fmt.Println("\nmissing — kept by the mesh, not in the store:")
for _, reference := range r.Missing {
fmt.Printf(" %s\n", reference)
}
}
fmt.Println()
if r.SafeToCollect {
fmt.Println("every kept archive is held: the store's collector may run for real")
} else {
fmt.Println("NOT every kept archive is known to be held: keep the store's collector on --dry-run")
}
}
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// recordingDelivery is a delivery that writes down what was done, in order, and fails where told.
type recordingDelivery struct {
did []string
grantErr error
declareErr error
}
func (r *recordingDelivery) grant(_ context.Context, sending []readyNode) error {
for _, s := range sending {
r.did = append(r.did, "grant "+s.node)
}
return r.grantErr
}
func (r *recordingDelivery) declare(_ context.Context, s readyNode, _ []byte) (string, error) {
if r.declareErr != nil {
return "", r.declareErr
}
r.did = append(r.did, "declare "+s.node)
return "digest-" + s.node, nil
}
func ready(names ...string) []readyNode {
var out []readyNode
for _, n := range names {
out = append(out, readyNode{node: n, declared: sendable{Resources: []map[string]any{{"id": "x"}}}})
}
return out
}
// novox/hq issue 249: the grants that come with a module's new declarations are issued before any
// machine is sent the code that uses them — except the machine holding the bus, whose declaration
// carries the controller's own right to issue them, and goes first.
func TestGrantsAreIssuedBeforeTheDeclarations(t *testing.T) {
d := &recordingDelivery{}
digests, err := deliver(t.Context(), d, "", ready("anchor", "laptop"))
if err != nil {
t.Fatal(err)
}
want := []string{"grant anchor", "grant laptop", "declare anchor", "declare laptop"}
if !reflect.DeepEqual(d.did, want) {
t.Fatalf("delivered in the order %v, wanted %v", d.did, want)
}
if digests["anchor"] != "digest-anchor" || digests["laptop"] != "digest-laptop" {
t.Fatalf("the digests sent were not answered: %v", digests)
}
held := &recordingDelivery{}
if _, err := deliver(t.Context(), held, "broker", ready("broker", "anchor")); err != nil {
t.Fatal(err)
}
want = []string{"declare broker", "grant broker", "grant anchor", "declare anchor"}
if !reflect.DeepEqual(held.did, want) {
t.Fatalf("with the bus's machine in the send: %v, wanted %v", held.did, want)
}
}
// A grant that cannot be issued holds back the machines it concerns and is an error the caller
// retries on — never "until the next push" — and the bus's own machine is sent regardless, so the
// grant that would let the controller issue memberships is never held behind them.
func TestAGrantThatFailsHoldsBackWhatItConcerns(t *testing.T) {
// A failure naming no machine (the buckets): everything but the bus's machine.
d := &recordingDelivery{grantErr: errors.New("the bus refused the bucket")}
_, err := deliver(t.Context(), d, "broker", ready("broker", "anchor", "laptop"))
if err == nil || !errors.Is(err, errGrants) || !strings.Contains(err.Error(), "the bus refused the bucket") ||
!strings.Contains(err.Error(), "anchor, laptop not sent") {
t.Fatalf("a failed grant was not said as the send's failure: %v", err)
}
if want := []string{"declare broker", "grant broker", "grant anchor", "grant laptop"}; !reflect.DeepEqual(d.did, want) {
t.Fatalf("delivered %v, wanted the bus's machine alone", d.did)
}
// A membership that failed for one machine: that machine alone.
one := &recordingDelivery{grantErr: &grantsRefused{nodes: map[string]error{"laptop": errors.New("no")}}}
_, err = deliver(t.Context(), one, "", ready("anchor", "laptop"))
if !errors.Is(err, errGrants) || !strings.Contains(err.Error(), "laptop not sent") {
t.Fatalf("one machine's refused membership was not said: %v", err)
}
if want := []string{"grant anchor", "grant laptop", "declare anchor"}; !reflect.DeepEqual(one.did, want) {
t.Fatalf("delivered %v, wanted anchor sent and laptop held back", one.did)
}
// The announced upgrade that hit it is asked again.
if !errors.Is(askAgainOnGrants(err), link.ErrTryAgain) {
t.Fatal("an announcement whose send stopped at its grants is not asked again")
}
if other := errors.New("laptop could not be resolved"); errors.Is(askAgainOnGrants(other), link.ErrTryAgain) {
t.Fatal("any failure is asked again, not only a grant's")
}
// Nothing to send is nothing granted either.
none := &recordingDelivery{grantErr: errors.New("never asked")}
if _, err := deliver(t.Context(), none, "", nil); err != nil || len(none.did) != 0 {
t.Fatalf("an empty send granted or failed: %v %v", none.did, err)
}
}
// Whether the bus's machine goes first is read from the user list alone, by its digest.
func TestTheBusMachineIsBehindByItsUserListAlone(t *testing.T) {
list := "users: [a, b]"
if userListBehind(list, digestOf([]byte(list))) {
t.Fatal("the list it was sent reads as behind")
}
if !userListBehind(list, digestOf([]byte("users: [a]"))) || !userListBehind(list, "") {
t.Fatal("a changed or never-sent list reads as current")
}
if userListBehind("", "") {
t.Fatal("a machine sent no list reads as behind")
}
}
// The machine holding the bus goes first: its declaration carries the user list the new grants are
// checked against. Among the machines it is moved to the front; not among them it is added only
// when it is behind.
func TestTheMachineHoldingTheBusIsSentFirst(t *testing.T) {
for _, c := range []struct {
what string
names []string
holder string
behind bool
want []string
}{
{"among them", []string{"ace", "g14", "novox"}, "novox", false, []string{"novox", "ace", "g14"}},
{"not among them, behind", []string{"ace", "g14"}, "novox", true, []string{"novox", "ace", "g14"}},
{"not among them, current", []string{"ace", "g14"}, "novox", false, []string{"ace", "g14"}},
{"nothing holds the bus", []string{"ace", "g14"}, "", true, []string{"ace", "g14"}},
{"only it", []string{"novox"}, "novox", false, []string{"novox"}},
} {
if got := brokerFirst(c.names, c.holder, c.behind); !reflect.DeepEqual(got, c.want) {
t.Errorf("%s: sent in the order %v, wanted %v", c.what, got, c.want)
}
}
}
+38
View File
@@ -0,0 +1,38 @@
package main
import (
"encoding/json"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// A dry run's outcome is looked at, never taken in (novox/hq issue 240). The daemon here holds no
// store at all, so anything that tried to record or register would fail rather than pass quietly.
func TestADryRunsOutcomeIsTakenInByNothing(t *testing.T) {
err := builds{}.Built(t.Context(), link.BuildResult{
ID: "build-1", Repository: "ssh://forge/app.git", Ref: "unreviewed", Module: "app", DryRun: true,
Manifest: json.RawMessage(`{"module":"app","version":"1"}`),
})
if err != nil {
t.Fatalf("a dry run's outcome was not simply set aside: %v", err)
}
}
// The mark survives the wire both ways: asked as a dry run, answered as one.
func TestTheDryRunMarkTravelsWithTheBuild(t *testing.T) {
raw, _ := json.Marshal(link.BuildRequest{ID: "build-1", Repository: "r", DryRun: true})
var asked link.BuildRequest
if err := json.Unmarshal(raw, &asked); err != nil || !asked.DryRun {
t.Fatalf("the request lost its dry-run mark: %s", raw)
}
raw, _ = json.Marshal(link.BuildResult{ID: "build-1", DryRun: true})
var answered link.BuildResult
if err := json.Unmarshal(raw, &answered); err != nil || !answered.DryRun {
t.Fatalf("the outcome lost its dry-run mark: %s", raw)
}
raw, _ = json.Marshal(link.BuildResult{ID: "build-2"})
if string(raw) != `{"id":"build-2","repository":"","on":""}` {
t.Fatalf("an ordinary outcome carries a dry-run mark: %s", raw)
}
}
+241
View File
@@ -0,0 +1,241 @@
package main
import (
"context"
"fmt"
"io"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A push sends no build a policy or a plan holds back, except to the machine it names (novox/hq
// issue 259, ADR 0221).
//
// A named push ends by sending every other machine whose declaration differs from what it was last
// sent (ADR 0083), so that a grant the push's work minted reaches the provider in the same act. A
// digest cannot say why a machine differs. A module whose upgrade policy records rather than rolls
// out makes every machine running it differ from the merge on, and so did a module whose plan was
// still waiting on its first machine (ADR 0218): `push <anchor>` sent all four machines the build
// that was meant to be walked through the mesh one machine at a time, and a fault in it was met
// everywhere at once.
//
// What tells the two apart is which build of each module the machine was last sent, kept with every
// send. A machine any of whose modules would move to a build its policy or an open plan holds back
// is not sent by a push that did not name it; the push says which, and why, and how to send it.
// heldBack is why a push that did not name a machine must not send it, empty when it may.
//
// `modules` is what the machine would be sent now; `sent` and `known` what it was last sent, as
// Inventory.SentBuilds answers. A module moves when the build it would carry is not the one the
// machine was last sent — including a module the machine was never sent at all. A move is held when
// the module's policy records rather than rolls out, or when an open plan has not yet sent this
// machine (planStillToSend). A machine whose last send was not recorded is held whole: what it carried
// is not known, so a held upgrade cannot be told from anything else.
func heldBack(node string, modules []string, sent map[string]string, known bool,
current map[string]inventory.CurrentBuild, plans []inventory.Plan) []string {
if !known {
return []string{"which builds it was last sent is not known — it was last sent before the " +
"mesh kept them, or sent a declaration by hand"}
}
var why []string
for _, m := range modules {
now := current[m]
was, carried := sent[m]
if carried && was == now.Commit {
continue
}
move := fmt.Sprintf("%s would move %sto %s", m, fromBuild(was, carried), buildName(now.Commit))
if !now.RollOut {
why = append(why, move+", which its upgrade policy records rather than rolls out")
continue
}
if id := planStillToSend(plans, m, node); id != "" {
why = append(why, move+", which "+id+" has not sent it yet (one machine first)")
}
}
sort.Strings(why)
return why
}
// planStillToSend is the open plan that has a module's new build still to send this machine, or empty:
// one holding the module that has neither finished sending it nor sent it here first, and has not
// failed it (novox/hq ADR 0218). The same reading rolledOutByAPlan makes for the whole module, made
// per machine.
func planStillToSend(plans []inventory.Plan, module, node string) string {
for _, p := range plans {
s, holds := p.Modules[module]
if !p.Open() || !holds {
continue
}
if s == nil {
return p.ID
}
if s.SentAt != nil || s.State == "failed" {
continue
}
first := false
for _, n := range s.First {
if n == node {
first = true
}
}
if !first {
return p.ID
}
}
return ""
}
func fromBuild(was string, carried bool) string {
if !carried {
return "(never sent it) "
}
return "from " + buildName(was) + " "
}
func buildName(commit string) string {
if commit == "" {
return "a build with no source"
}
return shortCommit(commit)
}
// heldMachines reads, for each machine named, why a push that did not name it must not send it
// (heldBack), and answers only the machines held. A machine whose set cannot be worked out is left
// to the send, which says why.
func heldMachines(ctx context.Context, open *stores, names []string) (map[string][]string, error) {
out := map[string][]string{}
if len(names) == 0 {
return out, nil
}
inv := open.inventory
current, err := inv.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, err
}
for _, node := range names {
plan, _, err := planFor(ctx, open, node)
if err != nil {
continue
}
modules := make([]string, 0, len(plan.Modules))
for _, m := range plan.Modules {
modules = append(modules, m.Module)
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil {
return nil, err
}
if why := heldBack(node, modules, sent, known, current, plans); len(why) > 0 {
out[node] = why
}
}
return out, nil
}
// sayHeld is what a push says about a machine it left behind on purpose: that it is behind, why it
// was not sent, that whatever else it is owed waits with it, and the command that sends it.
func sayHeld(w io.Writer, node string, why []string) {
fmt.Fprintf(w, "\n%s is behind and was not sent: %s. A push sends no build a policy or a plan "+
"holds back to a machine it did not name (novox/hq ADR 0221), so anything else it is owed — a "+
"grant from this push among it — waits with it. `push %s` sends it\n",
node, strings.Join(why, "; "), node)
}
// flushBehind is the end of a named push: every other machine now behind is sent too, by name, over
// as many rounds as the sends take to settle (novox/hq issue 057, ADR 0083) — except a machine whose
// modules would move to a build a policy or a plan holds back, which is named and left (ADR 0221).
//
// `handled` is every machine already sent or already said; it is not considered again. Answers the
// machines that could not be composed, as refusals.
func flushBehind(ctx context.Context, open *stores, nodes []inventory.Node, handled map[string]bool,
compose func(held context.Context, node string) (sendable, error), d delivery, holder string,
w io.Writer) ([]string, error) {
inv := open.inventory
var refusals []string
// Bounded by the node count: a node is marked handled the round it is considered and is never
// considered twice, so the loop cannot run more than len(nodes) rounds. The bound is a guard
// against a logic error, not a real limit — if it were ever hit, that is a bug rather than a
// cascade legitimately still converging, so it is said rather than passed over in silence.
rounds := 0
for {
would, err := wouldSend(ctx, open, nodes)
if err != nil {
return refusals, err
}
behind, err := inv.Waiting(ctx, would)
if err != nil {
return refusals, err
}
var also []string
for _, m := range behind {
if !handled[m.Node] {
also = append(also, m.Node)
}
}
if len(also) == 0 {
return refusals, nil
}
if rounds++; rounds > len(nodes) {
fmt.Fprintf(w, "\nstopped cascading after %d rounds with %s still behind — this "+
"should not happen; run `push --behind` to finish\n",
rounds-1, strings.Join(also, ", "))
return refusals, nil
}
sort.Strings(also)
held, err := heldMachines(ctx, open, also)
if err != nil {
return refusals, err
}
var sending []string
for _, name := range also {
// Every candidate this round is marked handled — the sent ones so they are not
// re-listed, the held ones because they stay held, and the refused ones so a machine
// that cannot be composed does not make the loop spin on it for ever.
handled[name] = true
if why, isHeld := held[name]; isHeld {
sayHeld(w, name, why)
continue
}
sending = append(sending, name)
}
if len(sending) == 0 {
continue
}
fmt.Fprintf(w, "\nthis push left %s behind — a provision granted from there, or a "+
"declaration since changed; sending it too\n", strings.Join(sending, ", "))
// Tolerantly, exactly as the named send: a machine that cannot be composed is collected as
// a refusal and reported at the end, and the others are still sent (novox/hq ADR 0066).
// Held for this round only, and after the last round's were given back, so two pushes
// cascading into each other's machines never each wait on the other.
refused, err := sendRound(ctx, open, sending, compose, d, holder)
refusals = append(refusals, refused...)
if err != nil {
return refusals, err
}
}
}
// composeForPush is how a push composes one machine: its set resolved, what it cannot host and what
// is left out of it said, and its declaration allocated.
func composeForPush(open *stores, gens map[string]catalogue.Generator) func(held context.Context, node string) (sendable, error) {
return func(held context.Context, node string) (sendable, error) {
plan, settings, err := planFor(held, open, node)
if err != nil {
return sendable{}, err
}
reportUnhostable(node, plan)
declared, err := declarationWith(held, open, node, plan, settings, gens, Allocating)
if err == nil {
reportLeftOut(node, declared)
}
return declared, err
}
}
+335
View File
@@ -0,0 +1,335 @@
package main
import (
"bytes"
"context"
"encoding/json"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// novox/hq issue 259, ADR 0221: a push that did not name a machine does not send it a build its
// upgrade policy records rather than rolls out, nor one an open plan has not sent it yet. Anything
// else that moved is still a consequence the push sends (ADR 0083).
func TestAHeldBuildHoldsAMachineANamedPushDidNotName(t *testing.T) {
current := map[string]inventory.CurrentBuild{
"resolver": {Commit: "c2c2c2c2c2"},
"agent": {Commit: "a2", RollOut: true},
"network": {},
}
modules := []string{"network", "resolver", "agent"}
sent := map[string]string{"network": "", "resolver": "c1c1c1c1c1", "agent": "a2"}
// A module whose policy records moved: held, naming it, both builds and why.
why := heldBack("laptop", modules, sent, true, current, nil)
if len(why) != 1 || !strings.Contains(why[0], "resolver would move from c1c1c1c1 to c2c2c2c2") ||
!strings.Contains(why[0], "upgrade policy records") {
t.Fatalf("a recorded upgrade did not hold the machine: %v", why)
}
// Nothing moved — what differs is a grant, a peer, a setting: not held (issue 057).
sent["resolver"] = "c2c2c2c2c2"
if why := heldBack("laptop", modules, sent, true, current, nil); len(why) != 0 {
t.Fatalf("a machine whose builds are all current was held: %v", why)
}
// A module whose policy rolls out moved, and no plan holds it: sent, as before.
sent["agent"] = "a1"
if why := heldBack("laptop", modules, sent, true, current, nil); len(why) != 0 {
t.Fatalf("a rolled-out upgrade no plan holds was held: %v", why)
}
// The last send's builds are not known: held whole.
if why := heldBack("laptop", modules, nil, false, current, nil); len(why) != 1 ||
!strings.Contains(why[0], "not known") {
t.Fatalf("a machine whose last send was not recorded was not held: %v", why)
}
// A module the machine was never sent, under a recording policy: held, and said so.
delete(sent, "resolver")
sent["agent"] = "a2"
if why := heldBack("laptop", modules, sent, true, current, nil); len(why) != 1 ||
!strings.Contains(why[0], "resolver would move (never sent it) to c2c2c2c2") {
t.Fatalf("a module never sent under a recording policy: %v", why)
}
}
// ADR 0218 meets ADR 0083: a plan waiting on its first machine has not sent the rest, and a push
// naming some other machine must not send them for it.
func TestAPlanWaitingOnItsFirstMachineHoldsTheRest(t *testing.T) {
at := time.Now()
current := map[string]inventory.CurrentBuild{"agent": {Commit: "a2", RollOut: true}}
sent := map[string]string{"agent": "a1"}
waiting := []inventory.Plan{{ID: "plan-7", State: inventory.PlanRolling, Modules: map[string]*inventory.PlanModule{
"agent": {State: "built", First: []string{"ace"}, FirstAt: &at}}}}
why := heldBack("g14", []string{"agent"}, sent, true, current, waiting)
if len(why) != 1 || !strings.Contains(why[0], "plan-7 has not sent it yet") {
t.Fatalf("a machine the plan has not reached was not held: %v", why)
}
// The first machine itself was sent by the plan: not held by it.
if why := heldBack("ace", []string{"agent"}, sent, true, current, waiting); len(why) != 0 {
t.Fatalf("the plan's first machine was held: %v", why)
}
// Built but not yet sent anywhere, or not yet built: the plan has it still to send.
for what, s := range map[string]*inventory.PlanModule{"built, unsent": {State: "built"}, "unasked": nil} {
plans := []inventory.Plan{{ID: "plan-8", State: inventory.PlanBuilding,
Modules: map[string]*inventory.PlanModule{"agent": s}}}
if why := heldBack("ace", []string{"agent"}, sent, true, current, plans); len(why) != 1 {
t.Errorf("%s: not held: %v", what, why)
}
}
// Sent everywhere, failed, or a plan no longer open: the plan holds nothing back.
for what, plans := range map[string][]inventory.Plan{
"sent everywhere": {{ID: "p", State: inventory.PlanRolling, Modules: map[string]*inventory.PlanModule{
"agent": {State: "built", First: []string{"ace"}, FirstAt: &at, SentAt: &at}}}},
"failed": {{ID: "p", State: inventory.PlanRolling, Modules: map[string]*inventory.PlanModule{
"agent": {State: "failed"}}}},
"closed": {{ID: "p", State: inventory.PlanDone, Modules: map[string]*inventory.PlanModule{
"agent": {State: "built"}}}},
} {
if why := heldBack("g14", []string{"agent"}, sent, true, current, plans); len(why) != 0 {
t.Errorf("%s: held: %v", what, why)
}
}
}
// A send records the build of each module it carried; a module left out of it keeps the build it
// was last sent, since the machine keeps that one.
func TestASendCarriesTheCurrentBuildsAndALeftOutModuleKeepsItsOwn(t *testing.T) {
current := map[string]inventory.CurrentBuild{"a": {Commit: "a2"}, "b": {Commit: "b2"}, "c": {}}
got := carriedBuilds([]string{"a", "b", "c"}, map[string]string{"b": "a setting does not compose"},
current, map[string]string{"a": "a1", "b": "b1"})
if want := map[string]string{"a": "a2", "b": "b1", "c": ""}; !reflect.DeepEqual(got, want) {
t.Fatalf("carried %v, wanted %v", got, want)
}
// Not known before: the left-out module is not recorded at all, so it reads as never sent.
got = carriedBuilds([]string{"a", "b"}, map[string]string{"b": "x"}, current, nil)
if want := map[string]string{"a": "a2"}; !reflect.DeepEqual(got, want) {
t.Fatalf("carried %v, wanted %v", got, want)
}
}
// recordedDelivery sends nothing and records each send as the mesh does, so the next comparison
// reads the machine as current — and writes down which machines it declared.
type recordedDelivery struct {
inv *inventory.Inventory
declared []string
}
func (r *recordedDelivery) grant(context.Context, []readyNode) error { return nil }
func (r *recordedDelivery) declare(ctx context.Context, s readyNode, body []byte) (string, error) {
r.declared = append(r.declared, s.node)
return recordSent(ctx, r.inv, s.node, body, s.declared.Builds)
}
// aResolver is a module built from a repository, at a commit, with something on the machine that
// says which build it is.
func aResolver(t *testing.T, open *stores, commit string, asked time.Time) {
t.Helper()
m := catalogue.Manifest{Module: "resolver", Version: "1", Resources: []map[string]any{
{"id": "zones", "type": "file", "path": "/etc/resolver/zones", "content": "built from " + commit},
}}
if err := open.inventory.RegisterModule(t.Context(), m, inventory.Source{
Repository: "novox/mesh-catalog", Path: "modules/resolver", BuiltFrom: commit, Asked: asked}); err != nil {
t.Fatal(err)
}
}
// A third machine on the private network: once it is sent, every other machine's peers change with
// it, which is a consequence a push must still send — no build moved.
func aThirdMachine(t *testing.T, open *stores) {
t.Helper()
ctx := t.Context()
record, err := open.inventory.AddNode(ctx, "spare")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.SetPlace(ctx, "spare", "spare.example:51820", "here", false, "10.77.0.3"); err != nil {
t.Fatal(err)
}
reported, err := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}, {"name": "wireguard", "present": true},
{"name": "systemd", "present": true}}})
if err != nil {
t.Fatal(err)
}
var profile map[string]any
if err := json.Unmarshal(reported, &profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordProfile(ctx, record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordOverlayKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "spare", overlay.Name); err != nil {
t.Fatal(err)
}
}
// The issue as it happened, against the real stores: a change merged with the policy `record`, a push
// naming the anchor, and the laptop — running the same module — left with what it had, by name.
func TestANamedPushLeavesAMachineAPolicyHoldsBack(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := time.Now().Add(-time.Hour)
aResolver(t, open, "c1c1c1c1c1", asked)
for _, node := range []string{"anchor", "laptop"} {
if _, err := inv.Assign(ctx, node, "resolver"); err != nil {
t.Fatal(err)
}
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
compose := composeForPush(open, gens)
d := &recordedDelivery{inv: inv}
if _, err := sendRound(ctx, open, []string{"anchor", "laptop"}, compose, d, ""); err != nil {
t.Fatal(err)
}
if builds, known, err := inv.SentBuilds(ctx, "laptop"); err != nil || !known || builds["resolver"] != "c1c1c1c1c1" {
t.Fatalf("the send did not record the build it carried: %v %v %v", builds, known, err)
}
digestOfLaptop := func() string {
sent, err := inv.Outstanding(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
return sent
}
before := digestOfLaptop()
// The change merges; the policy is the default, record. `push anchor` sends the anchor...
aResolver(t, open, "c2c2c2c2c2", asked.Add(time.Minute))
d.declared = nil
if _, err := sendRound(ctx, open, []string{"anchor"}, compose, d, ""); err != nil {
t.Fatal(err)
}
// ...and its cascade leaves the laptop, saying so.
var said bytes.Buffer
d.declared = nil
refused, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"anchor": true}, compose, d, "", &said)
if err != nil || len(refused) != 0 {
t.Fatalf("the cascade failed: %v %v", refused, err)
}
if len(d.declared) != 0 {
t.Fatalf("the cascade sent %v a build its policy records", d.declared)
}
if digestOfLaptop() != before {
t.Fatal("the laptop's last send moved: it was sent the held build")
}
for _, want := range []string{"laptop is behind and was not sent", "resolver would move from c1c1c1c1 to c2c2c2c2",
"upgrade policy records", "`push laptop` sends it"} {
if !strings.Contains(said.String(), want) {
t.Errorf("the push did not say %q:\n%s", want, said.String())
}
}
// Held and owed something else at once — a peer joined: still not sent, and both said: why it
// is held, and that what else it is owed waits with it.
aThirdMachine(t, open)
d.declared = nil
if _, err := sendRound(ctx, open, []string{"spare"}, compose, d, ""); err != nil {
t.Fatal(err)
}
d.declared = nil
said.Reset()
if _, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"anchor": true, "spare": true},
compose, d, "", &said); err != nil {
t.Fatal(err)
}
if len(d.declared) != 0 || digestOfLaptop() != before {
t.Fatalf("a held machine owed a consequence was sent: %v", d.declared)
}
if !strings.Contains(said.String(), "resolver would move") || !strings.Contains(said.String(), "anything else it is owed") {
t.Fatalf("the push did not say both:\n%s", said.String())
}
// A policy that rolls out: the laptop is a consequence like any other, and sent.
if err := inv.SetUpgradeOf(ctx, "resolver", inventory.Upgrade{RollOut: true}); err != nil {
t.Fatal(err)
}
said.Reset()
if _, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"anchor": true, "spare": true},
compose, d, "", &said); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(d.declared, []string{"laptop"}) || digestOfLaptop() == before {
t.Fatalf("a rolled-out upgrade's machine was not sent: %v\n%s", d.declared, said.String())
}
if builds, _, _ := inv.SentBuilds(ctx, "laptop"); builds["resolver"] != "c2c2c2c2c2" {
t.Fatalf("the new send did not record the new build: %v", builds)
}
}
// Issue 057's case is unchanged: a machine whose builds are all current and whose declaration moved
// for another reason is sent by a push that names someone else.
func TestANamedPushStillSendsAConsequenceNothingHolds(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
aResolver(t, open, "c1c1c1c1c1", time.Now().Add(-time.Hour))
if _, err := inv.Assign(ctx, "laptop", "resolver"); err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
compose := composeForPush(open, gens)
d := &recordedDelivery{inv: inv}
if _, err := sendRound(ctx, open, []string{"anchor", "laptop"}, compose, d, ""); err != nil {
t.Fatal(err)
}
// `push spare`, the machine just placed: the others' peers change with it.
aThirdMachine(t, open)
if _, err := sendRound(ctx, open, []string{"spare"}, compose, d, ""); err != nil {
t.Fatal(err)
}
d.declared = nil
var said bytes.Buffer
if _, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"spare": true}, compose, d, "", &said); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(d.declared, []string{"anchor", "laptop"}) {
t.Fatalf("a consequence nothing holds was not sent: %v\n%s", d.declared, said.String())
}
if strings.Contains(said.String(), "was not sent") {
t.Fatalf("a machine nothing holds was said to be held:\n%s", said.String())
}
// A machine whose last send was not recorded — a declaration sent by hand — is held until named.
record, err := inv.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, record.ID, "sent-by-hand", nil); err != nil {
t.Fatal(err)
}
d.declared = nil
said.Reset()
if _, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"spare": true}, compose, d, "", &said); err != nil {
t.Fatal(err)
}
// The anchor, the hub, may still be settling from the machine placed above; the laptop is the
// question.
if slices.Contains(d.declared, "laptop") || !strings.Contains(said.String(), "laptop is behind and was not sent") {
t.Fatalf("a machine whose last send is not known was sent: %v\n%s", d.declared, said.String())
}
}
+4 -4
View File
@@ -18,16 +18,16 @@ func TestASendRoundGivesItsHoldBackOnEveryWayOut(t *testing.T) {
plain := func(context.Context, string) (sendable, error) {
return sendable{Resources: []map[string]any{{"id": "x"}}}, nil
}
failing := func(readyNode, []byte) error { return errors.New("the broker went away") }
fine := func(readyNode, []byte) error { return nil }
failing := &recordingDelivery{declareErr: errors.New("the broker went away")}
fine := &recordingDelivery{}
for name, round := range map[string]func() error{
"a body that cannot be marshalled": func() error {
_, err := sendRound(ctx, open, []string{"anchor"}, unmarshallable, fine)
_, err := sendRound(ctx, open, []string{"anchor"}, unmarshallable, fine, "")
return err
},
"a send that fails": func() error {
_, err := sendRound(ctx, open, []string{"anchor"}, plain, failing)
_, err := sendRound(ctx, open, []string{"anchor"}, plain, failing, "")
return err
},
} {
+71
View File
@@ -6,6 +6,8 @@ import (
"sort"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// recordDerivedHolders writes down who holds each mesh-scoped seat that nobody was ever recorded
@@ -90,3 +92,72 @@ func recordDerivedHolders(ctx context.Context, open *stores) ([]string, error) {
}
return said, nil
}
// replicatedHolders is, for each replicated mesh seat, every machine on the private network holding
// it — by internal name, at its private address (novox/hq ADR 0223). What a machine's resolver file
// lists for `mesh-dns-resolver`; the rendering puts the machine itself first when it is one.
//
// **The holders on record, and only the sole claimant when there are none** — the same answer the
// resolver gives about who holds (ADR 0131, issue 170). An assignment standing beside the holders,
// eligible and silent, is not listed: it becomes a holder by `seat <name> --add`, an act, never by
// being assigned. Two claimants with nothing on record are refused at resolution, so neither is
// listed here. A holder off the private network is left out: a resolver named at an address nothing
// answers is a lookup that waits out its timeout on every name.
func replicatedHolders(ctx context.Context, inv *inventory.Inventory,
shelf map[string]catalogue.Manifest) (map[string]map[string]string, error) {
var replicated []catalogue.Seat
for _, s := range catalogue.Seats() {
if s.Replicated && s.Scope == catalogue.ScopeMesh {
replicated = append(replicated, s)
}
}
if len(replicated) == 0 {
return nil, nil
}
recorded, err := inv.Holdings(ctx)
if err != nil {
return nil, err
}
places, err := onTheNetwork(ctx, inv, shelf)
if err != nil {
return nil, err
}
address := map[string]string{}
for _, p := range places {
address[p.Name] = p.Address
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return nil, err
}
out := map[string]map[string]string{}
for _, seat := range replicated {
var nodes []string
for _, h := range recorded {
if hs, ok := catalogue.SeatNamed(h.Claim); ok && hs.Name == seat.Name && h.Scope == seat.Scope {
nodes = append(nodes, h.Node)
}
}
if len(nodes) == 0 {
var derived []string
for _, e := range entries {
for _, c := range e.Manifest.Claims {
if cs, ok := catalogue.SeatNamed(c.Name); ok && cs.Name == seat.Name && c.At() == seat.Scope {
derived = append(derived, e.On...)
}
}
}
if len(derived) == 1 {
nodes = derived
}
}
at := map[string]string{}
for _, n := range nodes {
if address[n] != "" {
at[overlay.InternalName(n)] = address[n]
}
}
out[seat.Name] = at
}
return out, nil
}
+1 -1
View File
@@ -77,7 +77,7 @@ func TestASendIsRecordedEvenWhenTheSenderIsBeingCancelled(t *testing.T) {
}
cancel() // the sender is going away: its context is cancelled between the send and the record
body := []byte(`{"declaration":1,"resources":[]}`)
digest, err := recordSent(ctx, inv, "anchor", body)
digest, err := recordSent(ctx, inv, "anchor", body, nil)
if err != nil {
// NodeByName on the cancelled context may itself refuse; the record must still be possible
// through the detached context, so look the node up again on a live one.
+40 -6
View File
@@ -73,6 +73,23 @@ func run() error {
return askCommand(ctx, args[1:])
case "builds":
return buildsCommand(ctx, args[1:])
// The build queue, controlled by hand (novox/hq ADR 0219).
case "queue":
return queueCommand(ctx, args[1:])
case "cancel":
return cancelCommand(ctx, args[1:])
case "clear":
return clearCommand(ctx, args[1:])
case "rebuild":
return rebuildCommand(ctx, args[1:])
case "replay":
return replayCommand(ctx, args[1:])
case "kill":
return killCommand(ctx, args[1:])
case "pause", "resume":
return pauseCommand(ctx, args[0], args[1:])
case "collection":
return collectionCommand(ctx, args[1:])
case "plans":
return plansCommand(ctx, args[1:])
case "pin":
@@ -177,10 +194,11 @@ func usage() {
seats [--json] every seat this mesh defines, what it delivers, and who holds it
seat rename <from> <to> rename a seat; its former name still resolves (ADR 0122)
seat <name> --to <node>/<module> hand a seat to that assignment as one act; never empty in between (ADR 0131)
seat <name> --add <node>/<module> add a holder beside the others, for a replicated seat (ADR 0223)
board [--listen ADDR] the same three questions, as a page that holds nothing
api --issuer URL [--listen A] assign and unassign over http, for a surface that is not here
assign <node> <module> put a module on a node
unassign <node> <module> take it off
assign <node> <module>... put modules on a node, judged together (ADR 0207)
unassign <node> <module>... take them off
take <node> <module> preview a module's cutover on an adopted node: what runs beside
what it declares; --yes <digest> cuts it over as previewed
converge <node> [--yes <digest>] [--filter nftables] preview, then make, an adopted node converged
@@ -200,6 +218,15 @@ func usage() {
build --behind build every module the mesh holds older than its source
build --on <module> rebuild every module that stands on this module's artifacts, bases first
builds [<module>] what has been built lately, and what came of it
queue [--json] every ask in the build queue: waiting, in flight (where, how long), dead
cancel <id> drop a waiting or dead ask; recorded failed, cancelled by hand
clear [--dead] cancel every waiting ask (and the dead ones); never one in flight
rebuild <module|build-id> ask the module's source again, or that build's, under a new id
replay <build-id> [--register [--older]] that build's commit again; a dry run unless --register
kill <id> end a build where it runs; recorded failed, killed by hand
pause [<node>] / resume [<node>] the build seat's holder there, or every holder, takes nothing new / again
plans retry <id> ask a failed plan's failed builds again, and carry the plan on
collection [--json] kept archives held/unheld by a manifest, and what the sweep may let go
builder issue <name> a broker account for a build machine, scoped to build work,
delivered as the builder module's broker secret (module add it first)
licence add|list|use|key model access, under the name a person calls it
@@ -253,6 +280,13 @@ func parseAround(set *flag.FlagSet, args []string) ([]string, error) {
// registered, the same as the waiting command does. Said either way, so the daemon's log tells what
// became of a build nobody was watching.
func (b builds) Built(ctx context.Context, result link.BuildResult) error {
// **A dry run is looked at, never taken in** (novox/hq issue 240). On 2026-10-04 a dry run of an
// unmerged branch was heard here like any build, registered, and its definition reached a machine
// before anyone had reviewed it.
if result.DryRun {
fmt.Printf("%s: a dry run of %s on %s, not taken in\n", result.ID, result.Repository, result.Ref)
return nil
}
manifest, _, err := takeIn(ctx, b.inv, result)
// When it was asked, so a plan takes as its outcome only a build asked for it or after it
// (novox/hq 04-ISSUES/219). Zero when the id does not say.
@@ -261,7 +295,7 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
case err != nil && result.Failed != "":
fmt.Printf("%s: %v\n", result.ID, err)
if result.Module != "" {
planBuilt(ctx, b.open, result.Module, result.Commit, result.Failed, asked)
planBuilt(ctx, b.open, result.Module, result.Commit, result.Failed, asked, result.ID)
} else {
planFailedBuild(ctx, b.open, result)
}
@@ -270,18 +304,18 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
// Not a failure: the module is already at what a later request built. A plan that asked
// before that later request is answered by it; one that asked after it ignores this.
fmt.Printf("%s: %v\n", result.ID, err)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked, result.ID)
return nil
case err != nil:
fmt.Printf("%s: heard and recorded, and not registered: %v\n", result.ID, err)
if manifest.Module != "" {
planBuilt(ctx, b.open, manifest.Module, result.Commit, err.Error(), asked)
planBuilt(ctx, b.open, manifest.Module, result.Commit, err.Error(), asked, result.ID)
}
return nil
}
fmt.Printf("%s: %s %s registered, built on %s from %s\n",
result.ID, manifest.Module, manifest.Version, result.On, short(result.Commit))
saysWhenThePolicyActs(ctx, b.inv, manifest.Module)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked, result.ID)
return nil
}
+131
View File
@@ -0,0 +1,131 @@
package main
import (
"context"
"fmt"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A merge the bus announced and the controller never acted on is caught up** (novox/hq issue 266).
//
// The forge's poll announces every merge on the events stream, and the controller acts on what its
// consumer there hands it. On 2026-10-06 one merge was on the stream and never handed over: the bus
// server moved the consumer past it — a fault of consumers with several filters in the server the
// mesh ran — and the controller, which only acts on what it is handed, said nothing. The modules
// built from that repository stayed behind and were built by hand.
//
// So the stream is read back on a timer, on a consumer of its own filtered on merges alone, and every
// announcement older than mergeGrace is judged as SourceMoved would judge it. **No record of what
// was handled is kept, because none is needed**: acting on a merge marks every module it moved as
// looked at since, so an announcement already acted on reads as history and moves nothing. One that
// would still move something was never acted on — it is said, and acted on now.
const (
// mergeGrace is how long an announcement is left to the controller's own consumer before it is
// judged missed. That consumer hands over one event at a time, and a merge waits behind a build
// outcome that is being acted on; acting on a merge itself asks builds and does not wait for them.
mergeGrace = 10 * time.Minute
// mergeLookBack is how far back a pass reads. A merge missed longer ago than this was missed by a
// controller that was not running this, and is the operator's to look at, not a surprise rebuild.
mergeLookBack = 24 * time.Hour
// mergeCatchUpEvery is how often the stream is read back.
mergeCatchUpEvery = 5 * time.Minute
)
// merges is what reads back the forge's announcements; the link server, or a test's list.
type merges interface {
AnnouncedMerges(ctx context.Context, since time.Time) ([]link.AnnouncedMerge, error)
}
// catchingUpOnMerges reads back the forge's announcements on a timer, until the context ends.
func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) {
f := following{open}
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return nil, nil, err
}
read, err := open.inventory.ReadRepositories(ctx)
return entries, read, err
}
failing := ""
tick := time.NewTicker(mergeCatchUpEvery)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
err := catchUpOnMerges(ctx, time.Now(), announced, catalogued, f.SourceMoved, func(format string, args ...any) {
fmt.Printf(format+"\n", args...)
})
// A pass that cannot read says so once, not every five minutes, and says when it reads again.
why := ""
if err != nil {
why = err.Error()
}
if why != failing {
if why != "" {
fmt.Printf("merges the bus may not have handed over cannot be looked for: %s\n", why)
} else {
fmt.Println("merges the bus may not have handed over are looked for again")
}
failing = why
}
}
}
// catchUpOnMerges is one pass: every announcement older than mergeGrace that acting on would still
// move something is said and acted on, oldest first.
//
// Judged twice: once against the catalogue as the pass found it, and again just before acting,
// because acting on an earlier missed merge of the same repository may have moved what a later one
// would have.
func catchUpOnMerges(ctx context.Context, now time.Time, announced merges,
catalogued func(context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error),
act func(context.Context, link.SourceMoved) error, say func(string, ...any)) error {
all, err := announced.AnnouncedMerges(ctx, now.Add(-mergeLookBack))
if err != nil {
return err
}
entries, read, err := catalogued(ctx)
if err != nil {
return err
}
for _, a := range all {
if now.Sub(a.At) < mergeGrace {
continue
}
if len(wouldMove(a.SourceMoved, entries, read)) == 0 {
continue
}
if entries, read, err = catalogued(ctx); err != nil {
return err
}
moves := wouldMove(a.SourceMoved, entries, read)
if len(moves) == 0 {
continue
}
var names []string
for _, e := range moves {
names = append(names, e.Manifest.Module)
}
say("%s/%s merged into %s (%.8s), announced %s ago, and the controller never acted on it: the bus "+
"did not hand the announcement over (novox/hq issue 266). %s %s behind it; acting on it now",
a.Owner, a.Repo, a.Base, a.Commit, now.Sub(a.At).Round(time.Minute), readableList(names),
isAre(len(names)))
if err := act(ctx, a.SourceMoved); err != nil {
say("%s/%s moved to %.8s and the mesh could not act on it: %v; the next pass tries again",
a.Owner, a.Repo, a.Commit, err)
continue
}
if entries, read, err = catalogued(ctx); err != nil {
return err
}
}
return nil
}
+180
View File
@@ -0,0 +1,180 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// announcedList is the events stream's merges as a test gives them.
type announcedList []link.AnnouncedMerge
func (a announcedList) AnnouncedMerges(_ context.Context, since time.Time) ([]link.AnnouncedMerge, error) {
var out []link.AnnouncedMerge
for _, m := range a {
if !m.At.Before(since) {
out = append(out, m)
}
}
return out, nil
}
// aCatalogue is what the inventory holds, changed the way acting on a merge changes it: every module
// the merge moved is marked as looked at (inventory.SourceMoved writes source_seen = now()).
type aCatalogue struct {
entries []inventory.Entry
acted []string
fail error
}
func (c *aCatalogue) read(context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
return append([]inventory.Entry(nil), c.entries...), nil, nil
}
func (c *aCatalogue) act(now func() time.Time) func(context.Context, link.SourceMoved) error {
return func(_ context.Context, m link.SourceMoved) error {
if c.fail != nil {
return c.fail
}
c.acted = append(c.acted, m.Repo+"@"+m.Commit[:8])
for _, moved := range wouldMove(m, c.entries, nil) {
for i := range c.entries {
if c.entries[i].Manifest.Module == moved.Manifest.Module {
c.entries[i].Source.Head = m.Commit
c.entries[i].Source.Seen = now()
}
}
}
return nil
}
}
func at(s string) time.Time {
t, err := time.Parse(time.RFC3339, s)
if err != nil {
panic(err)
}
return t
}
func announced(repo, commit, mergedAt, onTheBus string, paths ...string) link.AnnouncedMerge {
return link.AnnouncedMerge{
SourceMoved: link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Commit: commit,
MergedAt: mergedAt, Paths: paths,
CloneURL: "http://forge.internal:20000/novox/" + repo + ".git"},
At: at(onTheBus),
}
}
func built(module, repo, path, commit, seen string) inventory.Entry {
e := fromRepo(module, "http://forge.internal:20000/novox/"+repo+".git", path)
e.Source.BuiltFrom, e.Source.Head, e.Source.Seen = commit, commit, at(seen)
return e
}
// **novox/hq issue 266, as it happened.** The forge announced a merge of the tools repository on the
// events stream; the bus never handed it to the controller, which acted on the merges around it and
// not on this one, and said nothing. Read back from the stream, it is the one merge that would still
// move something — so it is said and acted on, once, and only after the controller's own consumer
// has had its time with it.
func TestAMergeTheBusNeverHandedOverIsActedOnLate(t *testing.T) {
cat := &aCatalogue{entries: []inventory.Entry{
built("mesh-tools", "mesh-tools", "", "8b789578aaaaaaaa", "2026-10-04T15:24:32Z"),
built("node-tools", "mesh-tools", "node-tools", "8b789578aaaaaaaa", "2026-10-04T15:24:32Z"),
// Acted on when it was announced: looked at after it was merged.
built("gitea", "mesh-catalog", "modules/gitea", "5c2157b8bbbbbbbb", "2026-10-05T22:39:21Z"),
}}
stream := announcedList{
// Nothing the mesh holds is built from the records repository.
announced("hq", "88f7f79fcccccccc", "2026-10-05T22:43:00Z", "2026-10-05T22:43:04Z", "04-ISSUES/x.md"),
// Acted on: its module was looked at since.
announced("mesh-catalog", "78328d4adddddddd", "2026-10-05T22:39:00Z", "2026-10-05T22:39:21Z", "modules/gitea/x.ts"),
// Never handed over.
announced("mesh-tools", "9730bd89c3e48d0e", "2026-10-05T22:46:47Z", "2026-10-05T22:47:06Z",
"node-tools/internal/console/console.go"),
}
var said []string
say := func(format string, args ...any) { said = append(said, fmt.Sprintf(format, args...)) }
clock := at("2026-10-05T22:50:00Z")
now := func() time.Time { return clock }
pass := func() {
t.Helper()
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(now), say); err != nil {
t.Fatal(err)
}
}
pass()
if len(cat.acted) != 0 {
t.Fatalf("a merge three minutes old was taken from the controller's own consumer: %v", cat.acted)
}
clock = at("2026-10-05T22:58:00Z")
pass()
if strings.Join(cat.acted, ",") != "mesh-tools@9730bd89" {
t.Fatalf("acted on %v, wanted the one merge never handed over", cat.acted)
}
if len(said) != 1 || !strings.Contains(said[0], "novox/mesh-tools merged into main (9730bd89)") ||
!strings.Contains(said[0], "mesh-tools and node-tools are behind it") {
t.Fatalf("the missed merge was not said as one: %q", said)
}
clock = at("2026-10-05T23:03:00Z")
pass()
if len(cat.acted) != 1 || len(said) != 1 {
t.Fatalf("a merge acted on was acted on again: %v %q", cat.acted, said)
}
}
// A merge that changed none of the held modules' files moves nothing, so it is never "missed"; one
// that could not be acted on is said and tried again on the next pass.
func TestAMissedMergeThatCouldNotBeActedOnIsTriedAgain(t *testing.T) {
cat := &aCatalogue{
entries: []inventory.Entry{built("gitea", "mesh-catalog", "modules/gitea", "5c2157b8bbbbbbbb", "2026-10-05T20:00:00Z")},
fail: errors.New("the store is restarting"),
}
stream := announcedList{
announced("mesh-catalog", "aaaaaaaa11111111", "2026-10-05T21:00:00Z", "2026-10-05T21:00:10Z",
"modules/plex/module.json", "modules/plex/x.ts"),
announced("mesh-catalog", "bbbbbbbb22222222", "2026-10-05T21:10:00Z", "2026-10-05T21:10:10Z", "modules/gitea/x.ts"),
}
var said []string
say := func(format string, args ...any) { said = append(said, fmt.Sprintf(format, args...)) }
clock := at("2026-10-05T22:00:00Z")
now := func() time.Time { return clock }
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(now), say); err != nil {
t.Fatal(err)
}
if len(said) != 2 || !strings.Contains(said[1], "could not act on it") {
t.Fatalf("a failed catch-up was not said: %q", said)
}
cat.fail = nil
clock = at("2026-10-05T22:05:00Z")
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(now), say); err != nil {
t.Fatal(err)
}
if strings.Join(cat.acted, ",") != "mesh-catalog@bbbbbbbb" {
t.Fatalf("acted on %v, wanted only the merge that changed a held module", cat.acted)
}
}
// A merge older than the look-back is left to the operator: a controller that did not run this
// missed it, and acting on it days later would be a surprise rebuild.
func TestAMergeOlderThanTheLookBackIsLeftAlone(t *testing.T) {
cat := &aCatalogue{entries: []inventory.Entry{built("gitea", "mesh-catalog", "modules/gitea", "5c2157b8bbbbbbbb", "2026-10-01T00:00:00Z")}}
stream := announcedList{announced("mesh-catalog", "cccccccc33333333", "2026-10-03T00:00:00Z", "2026-10-03T00:00:05Z", "modules/gitea/x.ts")}
clock := at("2026-10-05T22:00:00Z")
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(func() time.Time { return clock }),
func(string, ...any) {}); err != nil {
t.Fatal(err)
}
if len(cat.acted) != 0 {
t.Fatalf("a merge of three days ago was acted on: %v", cat.acted)
}
}
+14 -7
View File
@@ -40,7 +40,12 @@ func providedModules() []catalogue.Manifest {
// and neither could be swapped for anything, which is the test of whether a thing is a
// module at all (novox/hq ADR 0040). They existed because computed output needed somewhere
// to live, and now a module says where it wants it — `facts` in its own manifest.
overlay.Manifest(), overlay.DomainManifest(),
//
// **And the bundle that required it is gone** (novox/hq ADR 0226): `networking` named this
// module's requirement and nothing else, so every machine carried two modules for one
// network. A machine is assigned the private network itself; what a release stops shipping
// is retired at the next start (stores.go, Inventory.RetireUnshipped).
overlay.Manifest(),
} {
var m catalogue.Manifest
b, _ := json.Marshal(raw)
@@ -334,8 +339,10 @@ func moduleCommand(ctx context.Context, args []string) error {
}
func assignCommand(ctx context.Context, verb string, args []string) error {
if len(args) != 2 {
return fmt.Errorf("%s <node> <module>", verb)
// Several modules in one act (novox/hq ADR 0207): holders that depend on each other — the
// service manager and the package manager — can only go on, or come off, together.
if len(args) < 2 {
return fmt.Errorf("%s <node> <module> [<module>…]", verb)
}
open, err := openStores(ctx)
if err != nil {
@@ -349,7 +356,7 @@ func assignCommand(ctx context.Context, verb string, args []string) error {
if verb == "unassign" {
act = unassign
}
said, err := act(ctx, open, args[0], args[1])
said, err := act(ctx, open, args[0], args[1:]...)
if said != "" {
fmt.Println(said)
}
@@ -492,8 +499,8 @@ const theBrokerSeat = "mesh-broker"
// says. Only when nothing holds the seat yet (genesis raised the broker as plumbing and no module
// has adopted it) does the hub stand in, which is where the foundation is by convention.
//
// "On the overlay" is what `whereEveryoneIs` answers — a machine that RESOLVED the networking
// module — not "has an address", which is true of every placed machine and says nothing about
// "On the overlay" is what `whereEveryoneIs` answers — a machine that RESOLVED the private network
// — not "has an address", which is true of every placed machine and says nothing about
// whether anything can reach it (novox/hq issue 059). A node not on the overlay — at genesis,
// before any `overlay place`, which is when the builder's account is issued — keeps the genesis
// address, so nothing about bring-up changes. This is issue 055, corrected by 059.
@@ -659,7 +666,7 @@ func issueWith(ctx context.Context, inv *inventory.Inventory, m catalogue.Manife
// durable subscription nobody reads.
if consumer, needed := broker.ConsumerFor(broker.Principal{
Kind: broker.KindModule, Node: node, Module: m.Module,
Emits: m.Emits, Consumes: m.Consumes, Serves: m.Tools,
Emits: m.EmitsAll(), Consumes: m.Consumes, Serves: m.Tools,
}); needed {
if busAddress == "" {
fmt.Printf(" %s consumes; its consumer is created when the bus is reachable (`push`, then "+
+2 -18
View File
@@ -275,25 +275,9 @@ func network(ctx context.Context, inv *inventory.Inventory, on map[string]bool,
if err != nil {
return nil, err
}
// No registry trust is composed here any more: the container runtime's module states it, told
// where the store is reached by ${seat:mesh-artifact-store:reach} (novox/hq ADR 0222, issue 190).
g, err := overlay.From(nodes, cidr, "")
if g != nil {
// The artifact store, as this network reaches it. Found rather than configured: the
// provider is whichever module offers it, on whichever machine holds that module — and if
// nothing does yet (genesis raises the registry before the catalogue knows it), there is
// no trust to write and nothing is written (novox/hq ADR 0082).
//
// Refused rather than composed without it when the question could not be answered: a
// declaration missing the trust because a lookup failed is a machine that cannot pull,
// delivered by a push that reported success — and nothing recomposes it until the next
// push (the shape of novox/hq issues 042/048, reappearing as a race).
at, port, found, storeErr := artifactStoreOnNetwork(ctx, inv, on)
if storeErr != nil {
return nil, fmt.Errorf("finding the artifact store this network reaches: %w", storeErr)
}
if found {
g.TrustRegistry(overlay.InternalName(at) + ":" + port)
}
}
if err != nil && len(refused) > 0 {
// The network is missing something, and some machines could not be resolved at all. Those
// are almost always the same fact: a node that does not resolve contributes nothing, so
+8 -8
View File
@@ -254,11 +254,10 @@ func theResolver(t *testing.T) catalogue.Manifest {
return m
}
// The resolver is handed every machine on the private network as a wildcard, the same set and the
// same source as the hosts file, and is handed it again when a machine leaves — through the
// module's own manifest asking for the fact, with no module of the mesh's own in between (hal
// dnsmasq-app conversion, novox/hq 08-connectivity). The runtime on that machine is pointed at the
// machine's own address, where the resolver answers for its containers.
// The resolver is handed every machine on the private network as a wildcard, and is handed it again
// when a machine leaves — through the module's own manifest asking for the fact, with no module of the
// mesh's own in between (hal dnsmasq-app conversion, novox/hq 08-connectivity). It is the mesh's one
// resolver (ADR 0194), and the container runtime is given no resolver of its own (ADR 0196).
func TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
@@ -293,11 +292,12 @@ func TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves(t *t
t.Errorf("the resolver's machines lack %q:\n%s", want, first)
}
}
// The container runtime is given no resolver of its own (novox/hq ADR 0196): it copies its
// machine's, which name the mesh's resolver first. A `dns` key would be a second account of where a
// container asks, read only when the runtime starts.
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "dnsmasq.runtime-dns" {
if !strings.Contains(r["content"].(string), `"10.77.0.1"`) || r["into"] != "json" {
t.Errorf("the runtime is not pointed at this machine's own address, written into its file: %v", r)
}
t.Errorf("the resolver still writes the runtime's own dns: %v", r)
}
}
+46 -1
View File
@@ -386,8 +386,12 @@ func brokerCommand(ctx context.Context, args []string) error {
if len(args) > 0 && args[0] == "accounts" {
return busAccounts(ctx, args[1:])
}
if len(args) > 0 && args[0] == "consumer-reset" {
return consumerReset(args[1:])
}
if len(args) == 0 || args[0] != "show" {
return errors.New("broker show | broker certificate [--check] --into <directory> | broker accounts --into <file>")
return errors.New("broker show | broker certificate [--check] --into <directory> | broker accounts --into <file> | " +
"broker consumer-reset <stream> <consumer>")
}
known, err := broker.FromEnvironment()
if errors.Is(err, broker.ErrNotConfigured) {
@@ -407,6 +411,34 @@ func brokerCommand(ctx context.Context, args []string) error {
return nil
}
// consumerReset re-makes one consumer on a stream that keeps history to start from now (novox/hq issue
// 248): the way out of a consumer replaying a week of announcements, said rather than done by hand. A
// person's act — what was pending is dropped — so it is a command, and nothing calls it on its own.
func consumerReset(args []string) error {
if len(args) != 2 {
return errors.New("broker consumer-reset <stream> <consumer>, e.g. broker consumer-reset EVENTS controller")
}
address, err := broker.BusAddress()
if err != nil {
return err
}
js, err := broker.Dial(address)
if err != nil {
return fmt.Errorf("cannot reach the bus: %w", err)
}
defer js.Close()
before, after, err := js.ResetConsumer(args[0], args[1])
if err != nil {
return err
}
fmt.Printf("consumer %s on %s re-made to deliver from now\n", args[1], args[0])
fmt.Printf(" before: delivers %s, delivered to %d, acknowledged to %d, %d pending, %d unacknowledged\n",
before.DeliverPolicy, before.Delivered, before.AckFloor, before.Pending, before.AckPending)
fmt.Printf(" after: delivers %s, %d pending; what was pending is dropped. A holder bound to it may need its "+
"process restarted to bind again\n", after.DeliverPolicy, after.Pending)
return nil
}
// heardFrom says when a node was last heard from, in a form somebody can act on.
//
// "never" and "an hour ago" are different answers and are kept different. A node that has never
@@ -473,6 +505,19 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
fmt.Printf(" public domain %s\n", domain)
}
// A provider here failing a consumer, or a consumer here failed (novox/hq ADR 0224). Before the
// capabilities, because it is something not working now and they are a description.
failing, err := failingProviders(ctx, inv)
if err != nil {
return err
}
if here := failingOn(failing, name); len(here) > 0 {
fmt.Printf("\n %d consumer(s) a provider keeps failing, here or for a module here:\n", len(here))
for _, line := range failingLines(here, time.Now()) {
fmt.Printf(" %s\n", line)
}
}
held, err := inv.Profile(ctx, name)
if err != nil {
return err
+8
View File
@@ -147,6 +147,14 @@ func TestAMergeRebuildsTheModulesItChanged(t *testing.T) {
{"a module the mesh does not hold", merge([]string{"modules/plex/index.ts"}, false), ""},
{"nothing said about the files", merge(nil, false), "gitea,keycloak"},
{"more files than were listed", merge([]string{"modules/gitea/index.ts"}, true), "gitea,keycloak"},
// novox/hq issue 252: a module the mesh has never registered is still a module, when the merge
// shows it is one — and a directory that may be shared code is still shared.
{"a new module beside a held one", merge([]string{"modules/gitea/x", "modules/newmod/module.json"}, false), "gitea"},
{"a new module's other files", merge([]string{"modules/newmod/index.ts", "modules/newmod/module.json"}, false), ""},
{"a module removed", merge([]string{"modules/gone/module.json"}, false), ""},
{"a directory with no manifest", merge([]string{"modules/lib/x.go"}, false), "gitea,keycloak"},
{"a file directly among the modules", merge([]string{"modules/README.md"}, false), "gitea,keycloak"},
{"the root's files still", merge([]string{"tsconfig.json"}, false), "gitea,keycloak"},
} {
if got := named(whatTheMergeTouched(candidates, known, c.m)); got != c.want {
t.Errorf("%s: rebuilt %q, wanted %q", c.what, got, c.want)
+243 -17
View File
@@ -8,8 +8,10 @@ import (
"flag"
"fmt"
"os"
"slices"
"sort"
"strings"
"sync"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
@@ -127,6 +129,7 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
// a mesh-wide gatherer may pass over — see notResolvable.
return catalogue.Resolution{}, nil, notResolvable{err}
}
logUnheld(nodeName, resolved.Unheld)
// The credential for each thing this node takes from elsewhere. Made once and kept, so the
// password a provider is told to create is the one its consumer was given — and sealed to
@@ -394,9 +397,49 @@ func declarationWith(ctx context.Context, open *stores, node string,
if err != nil {
return sendable{}, err
}
return sendable{Resources: composed.Resources, Adoption: adoption,
Received: composed.Received, Mesh: with.Mesh,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut}, nil
out := sendable{Resources: composed.Resources, Adoption: adoption,
Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut}
// And which build of each module it carries, for the send to record (novox/hq issue 259, ADR
// 0221). Read only on the send path: a question about what would be sent records nothing.
if choosing == Allocating {
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return sendable{}, err
}
before, known, err := open.inventory.SentBuilds(ctx, node)
if err != nil {
return sendable{}, err
}
if !known {
before = nil
}
names := make([]string, 0, len(plan.Modules))
for _, m := range plan.Modules {
names = append(names, m.Module)
}
out.Builds = carriedBuilds(names, composed.LeftOut, current, before)
}
return out, nil
}
// carriedBuilds is the build of each module a declaration carries, as a send records it (novox/hq
// issue 259): the module's current build for each module in it, and for a module left out of it
// (ADR 0163, rule 6) the build it was last sent, since the machine keeps that one — or nothing, when
// that is not known. Never nil, so a send through here always records what it knows.
func carriedBuilds(modules []string, leftOut map[string]string, current map[string]inventory.CurrentBuild,
before map[string]string) map[string]string {
out := map[string]string{}
for _, m := range modules {
if _, left := leftOut[m]; left {
if was, kept := before[m]; kept {
out[m] = was
}
continue
}
out[m] = current[m].Commit
}
return out
}
// sortedKeysOf is a map's keys, sorted — so what a declaration says it left out does not move
@@ -657,6 +700,20 @@ func renderingFor(ctx context.Context, open *stores, node string,
machines[name] = at
}
// And every zone a module in the mesh answers itself (novox/hq ADR 0199), for the mesh's resolver
// to forward.
zones, err := zonesInTheMesh(ctx, open)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// And who holds each replicated seat, where (novox/hq ADR 0223): every machine's resolver file
// lists every holder of the mesh's resolver.
replicas, err := replicatedHolders(ctx, inv, shelf)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// **The bus is never public** (novox/hq ADR 0169). It was a foundation port — widened from the
// broker's own `from: mesh` to from-anywhere on the broker's host, so a machine could enrol
// before it had an address on the private network. A machine joins through the tunnel now, and
@@ -722,18 +779,90 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// Where this machine reaches each mesh seat's holder, for ${seat:<seat>:reach} (novox/hq ADR
// 0222): the artifact store as this network reaches it, which the container runtime is told to
// trust (ADR 0082). The same address composed into every reference the mesh built.
var reach map[string]string
if artifactStore != "" {
reach = map[string]string{"mesh-artifact-store": artifactStore}
}
return catalogue.Rendering{
BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
Machines: machines,
Suffix: overlay.Suffix(), MeshRange: meshRange, TunnelInterface: overlay.Interface, Accounts: accounts, Foundation: foundation,
Machines: machines, Zones: zones, Holders: replicas,
Suffix: overlay.Suffix(), MeshRange: meshRange, TunnelInterface: overlay.Interface, Accounts: accounts, Foundation: foundation,
Kept: kept, Adopted: record.Adopted, OutwardLinks: outwardLinks,
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, Built: built,
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, SeatReach: reach, Built: built,
BusUsers: busUsers,
}, record, nil
}
// zonesInTheMesh is every zone a module in the mesh declares, where the mesh placed it (novox/hq ADR
// 0199): the zone settled from that node's settings, the node's private address, the port the
// answering listen is published on there.
//
// Read across every machine's resolution, as the roster once read routed names: a node whose set does
// not compose declares nothing and is passed over, so one broken machine does not cost the rest their
// zones; a store that cannot be read is raised, naming the machine, because returning the zones
// without it would withdraw them from the resolver as if the operator had (novox/hq 04-ISSUES/152).
// What the mesh refuses about the zones together — one declared twice, one shadowing the mesh's
// suffix or a node's public domain — is refused here, by name.
func zonesInTheMesh(ctx context.Context, open *stores) ([]catalogue.ZoneAt, error) {
inv := open.inventory
places, err := inv.Overlays(ctx)
if err != nil {
return nil, fmt.Errorf("where the machines are cannot be read: %w", err)
}
address := map[string]string{}
for _, p := range places {
if strings.TrimSpace(p.Address) != "" {
address[p.Name] = p.Address
}
}
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, fmt.Errorf("which machines the mesh has cannot be read: %w", err)
}
var zones []catalogue.ZoneAt
var public []string
for _, n := range nodes {
plan, _, err := planFor(ctx, open, n.Name)
switch {
case unresolvable(err):
continue
case err != nil:
return nil, fmt.Errorf("the zones %s answers cannot be read: %w", n.Name, err)
}
if plan.PublicDomain != "" {
public = append(public, plan.PublicDomain)
}
for _, m := range plan.Modules {
if m.Zone == nil {
continue
}
at := address[n.Name]
if at == "" {
// Not on the private network yet: nothing could reach its answerer.
continue
}
published, layers, err := portsGivenOn(ctx, inv, n.Name, m)
if err != nil {
return nil, fmt.Errorf("the zone %s declares on %s cannot be read: %w", m.Module, n.Name, err)
}
z, err := catalogue.ZoneOn(m, layers, published, n.Name, at)
if err != nil {
return nil, err
}
zones = append(zones, *z)
}
}
if problems := catalogue.ZonesProblems(zones, overlay.Suffix(), public); len(problems) > 0 {
return nil, fmt.Errorf("the mesh's zones cannot be forwarded:\n - %s", strings.Join(problems, "\n - "))
}
return zones, nil
}
// certificateFor is what the mesh certifies about one machine's internal name.
//
// It reaches across two contexts and reads neither one's store from the other: `inventory` knows
@@ -1266,6 +1395,20 @@ func composeBusUsers(ctx context.Context, inv *inventory.Inventory,
//
// Asked of what this push resolves to rather than of the seat's holder mesh-wide: the file is a
// resource of that module, so the question is whether it is here.
list, missing, err := busUserList(ctx, inv, onThisNode)
if len(missing) > 0 {
fmt.Printf("the bus's user list leaves out %d user(s) the mesh has minted no credential "+
"for: %s. Each is a user that cannot connect until one is issued\n",
len(missing), strings.Join(missing, ", "))
}
return list, err
}
// busUserList is composeBusUsers without saying anything: the list, and the users left out of it
// for want of a credential. Asked on every send to decide whether the machine holding the bus must
// go first (novox/hq issue 249), where saying the same missing users each time would bury them.
func busUserList(ctx context.Context, inv *inventory.Inventory,
onThisNode []catalogue.Manifest) (string, []string, error) {
holdsTheBus := false
for _, m := range onThisNode {
if m.BusUsers != "" && m.ClaimsSeat("mesh-broker") {
@@ -1273,37 +1416,33 @@ func composeBusUsers(ctx context.Context, inv *inventory.Inventory,
}
}
if !holdsTheBus {
return "", nil
return "", nil, nil
}
records, err := inv.BusRecords(ctx)
if err != nil {
return "", err
return "", nil, err
}
users, err := broker.Users(records)
if err != nil {
return "", err
return "", nil, err
}
kept, err := inv.BusUsers(ctx)
if err != nil {
return "", err
return "", nil, err
}
hashes := make(map[string]string, len(kept))
for name, u := range kept {
hashes[name] = u.PasswordHash
}
filled, missing := broker.WithPasswords(users, hashes)
if len(missing) > 0 {
fmt.Printf("the bus's user list leaves out %d user(s) the mesh has minted no credential "+
"for: %s. Each is a user that cannot connect until one is issued\n",
len(missing), strings.Join(missing, ", "))
}
if len(filled) == 0 {
return "", fmt.Errorf(
return "", missing, fmt.Errorf(
"this machine runs the bus and not one user has a credential, so the composed list " +
"would refuse every connection in the mesh")
}
return broker.ComposeAccounts(filled)
list, err := broker.ComposeAccounts(filled)
return list, missing, err
}
// providerModuleOf is which module answers a need on the providing node: the one in this node's
@@ -1325,3 +1464,90 @@ func providerModuleOf(resolved catalogue.Resolution, open *stores, ctx context.C
}
return ""
}
// unheldLogged is what was last logged about each node's unmet seat dependencies, so the log says
// each change once (novox/hq ADR 0207), on stderr so `status --json` stays a document.
//
// **The serving controller's log only.** planFor runs for every node on every push, assignment and
// status — `blockedElsewhere` alone resolves the whole mesh — and a one-shot command starts with an
// empty memory, so every node's report was "a change" and a push printed the whole mesh's list,
// burying the line about the node it acted on. A command says what concerns its own act instead
// (unheldChange, reportUnheldPushed); the full list is `status`'s.
var (
unheldLogged = map[string]string{}
unheldLoggedMu sync.Mutex
logUnheldChanges bool
)
// logUnheld logs a node's unmet seat dependencies when they differ from what was last logged for
// it, including when they become none — in the serving controller, and nowhere else.
func logUnheld(node string, unheld []catalogue.Unheld) {
if !logUnheldChanges {
return
}
lines := make([]string, 0, len(unheld))
for _, u := range unheld {
lines = append(lines, u.String())
}
now := strings.Join(lines, "\n")
unheldLoggedMu.Lock()
before, seen := unheldLogged[node]
unheldLogged[node] = now
unheldLoggedMu.Unlock()
if (seen && before == now) || (!seen && now == "") {
return
}
if now == "" {
fmt.Fprintf(os.Stderr, "%s: every seat its modules depend on is held (novox/hq ADR 0207)\n", node)
return
}
fmt.Fprintf(os.Stderr, "%s: %d unmet seat dependenc(ies), reported and not refused (novox/hq ADR 0207):\n %s\n",
node, len(lines), strings.Join(lines, "\n "))
}
// unheldChange is what an act on one node changed about its unmet seat dependencies, judged over
// its assignments before and after (novox/hq ADR 0207): each dependency now unmet that was not —
// which includes every one of a module just assigned — and each now met that was not. Nothing about
// any other node, and nothing that was already true before the act.
func unheldChange(shelf map[string]catalogue.Manifest, node string, before, after []string) []string {
judge := func(names []string) map[string]catalogue.Unheld {
var set []catalogue.Manifest
for _, n := range names {
if m, known := shelf[n]; known {
set = append(set, m)
}
}
out := map[string]catalogue.Unheld{}
for _, u := range catalogue.UnheldDependencies(shelf, node, set, nil) {
out[u.Module+" "+u.Seat] = u
}
return out
}
was, now := judge(before), judge(after)
var lines []string
for _, k := range sortedNames(now) {
if _, already := was[k]; !already {
lines = append(lines, "but "+now[k].String())
}
}
for _, k := range sortedNames(was) {
if _, still := now[k]; still {
continue
}
u := was[k]
if !slices.Contains(after, u.Module) {
continue // went with its module, which says nothing about the seat
}
lines = append(lines, fmt.Sprintf("and %s on %s now has %s held", u.Module, node, u.Seat))
}
return lines
}
func sortedNames[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
+401
View File
@@ -0,0 +1,401 @@
package main
import (
"context"
"fmt"
"os"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A plan follows what is done to the build queue (novox/hq ADR 0219).
//
// Two things a person does to builds by hand would otherwise leave a plan saying something untrue:
//
// - **Pausing the build seat.** A plan whose builds wait in the queue of a seat whose every holder
// is paused is not late — nothing will take its asks until somebody resumes — and saying LATE
// sends a reader looking for a fault that is a decision. It says what it waits on instead.
// - **A build that failed, been cancelled or killed, and is asked again.** `plans retry` re-asks a
// failed plan's failed modules and the plan goes on from that tier as if they had built the
// first time; `rebuild` of a module a plan holds unbuilt joins that plan rather than running
// beside it — beside it, the plan would either ask it again or stay failed on an outcome the
// rebuild has already replaced.
// pauseView is whether the build seat takes work: the holders that said they are paused, and
// whether that is every holder.
type pauseView struct {
Nodes []string
All bool
}
// pausedWaiting is what a plan waits on when the build seat is paused under it, or false: a plan
// building, whose current tier has an ask outstanding, while every holder of the seat is paused.
func pausedWaiting(p inventory.Plan, pause pauseView, now time.Time) (string, bool) {
if p.State != inventory.PlanBuilding || !pause.All || len(pause.Nodes) == 0 || p.Tier >= len(p.Tiers) {
return "", false
}
var asked *time.Time
for _, m := range p.Tiers[p.Tier] {
if s := p.Modules[m]; s != nil && s.State == "asked" && s.AskedAt != nil {
if asked == nil || s.AskedAt.Before(*asked) {
asked = s.AskedAt
}
}
}
if asked == nil {
return "", false
}
waited := now.Sub(*asked)
said := fmt.Sprintf("waiting: the build seat is paused on %s (asked %s ago)",
strings.Join(pause.Nodes, ", "), waited.Round(time.Second))
// Not late — a pause is a decision — but a pause forgotten is a plan that never moves, so one held
// longer than a day is named.
if waited > pausedTooLong {
said += " — PAUSED OVER A DAY: `resume` takes builds again"
}
return said, true
}
// pausedTooLong is how long a plan may wait on a paused build seat before it says the pause is long.
const pausedTooLong = 24 * time.Hour
// awaitsABuild is whether any open plan has an ask outstanding in its current tier — the only case
// where the seat being paused changes what a plan says.
func awaitsABuild(plans []inventory.Plan) bool {
for _, p := range plans {
if p.State != inventory.PlanBuilding || p.Tier >= len(p.Tiers) {
continue
}
for _, m := range p.Tiers[p.Tier] {
if s := p.Modules[m]; s != nil && s.State == "asked" {
return true
}
}
}
return false
}
// buildSeatPause reads whether the build seat's holders take work, from what each last said on the
// bus (link.HolderState) — read only when a plan waits on a build, so a mesh with nothing building
// does not dial the bus to say so. Anything unreadable is said and read as not paused: a plan then
// reads as late, which is what it said before this existed.
func buildSeatPause(ctx context.Context, inv *inventory.Inventory, plans []inventory.Plan) pauseView {
if !awaitsABuild(plans) {
return pauseView{}
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return pauseView{}
}
seat := buildSeatAmong(entries)
holders := holdersAmong(entries, seat)
if len(holders) == 0 {
return pauseView{}
}
address, err := broker.BusAddress()
if err != nil {
return pauseView{}
}
js, err := broker.Dial(address)
if err != nil {
fmt.Fprintf(os.Stderr, "could not reach the bus to read whether the build seat is paused: %v\n", err)
return pauseView{}
}
defer js.Close()
said, err := link.PausedSaid(js, seat, holders)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read whether the build seat is paused: %v\n", err)
return pauseView{}
}
return pauseOf(holders, said)
}
// pauseOf is the view from what each holder said.
func pauseOf(holders []string, said map[string]link.HolderState) pauseView {
var v pauseView
for _, n := range holders {
if said[n].Paused {
v.Nodes = append(v.Nodes, n)
}
}
sort.Strings(v.Nodes)
v.All = len(holders) > 0 && len(v.Nodes) == len(holders)
return v
}
// failedIn is the modules of a plan's current tier that failed to build, sorted.
func failedIn(p inventory.Plan) []string {
if p.Tier >= len(p.Tiers) {
return nil
}
var out []string
for _, m := range p.Tiers[p.Tier] {
if s := p.Modules[m]; s != nil && s.State == "failed" {
out = append(out, m)
}
}
sort.Strings(out)
return out
}
// newerOpenPlan is an open plan of the same repository and branch made after this one: the plan
// that holds what this one held now (issue 254, ADR 0218).
func newerOpenPlan(p inventory.Plan, plans []inventory.Plan) (inventory.Plan, bool) {
for _, q := range plans {
if q.ID == p.ID || !q.Open() || !strings.EqualFold(q.Repository, p.Repository) ||
(p.Branch != "" && q.Branch != "" && p.Branch != q.Branch) || !q.Created.After(p.Created) {
continue
}
return q, true
}
return inventory.Plan{}, false
}
// retryRefusal is why a plan cannot be retried, or nothing.
func retryRefusal(p inventory.Plan, plans []inventory.Plan) error {
switch {
case p.State == inventory.PlanDone:
return fmt.Errorf("%s is done; there is nothing to retry", p.ID)
case p.State == inventory.PlanSuperseded:
return fmt.Errorf("%s was %s — what it had not built is in that plan", p.ID, p.Note)
case p.Open():
return fmt.Errorf("%s is still %s; nothing in it failed to retry — `rebuild <module>` asks one module again", p.ID, p.State)
}
if len(failedIn(p)) > 0 {
if q, found := newerOpenPlan(p, plans); found {
return fmt.Errorf("%s supersedes it: a newer merge of %s (%s at %s) is open, and retrying %s would build "+
"what that one replaced", q.ID, q.Repository, q.ID, short(q.Commit), p.ID)
}
return nil
}
stopped := stoppedRollouts(p)
if len(stopped) == 0 {
return fmt.Errorf("nothing in tier %d of %s failed to build or stopped rolling out — it stopped at: %s",
p.Tier, p.ID, p.Note)
}
// **A rollout is retried unless the module has moved on**: a newer plan holding it sends — or
// sent — a newer build, and sending this one again would put the older build back on its machines.
for _, m := range stopped {
if q, found := newerPlanFor(m, p, plans); found {
return fmt.Errorf("%s has a newer plan, %s (%s, %s at %s): sending %s's build of it again would put "+
"the older build back", m, q.ID, q.State, q.Repository, short(q.Commit), p.ID)
}
}
return nil
}
// stoppedRollouts is the modules of a plan's current tier whose rollout stopped at its first machine
// (issue 249, ADR 0218): built, sent to the first machine, never to the rest, and why it stopped kept.
func stoppedRollouts(p inventory.Plan) []string {
if p.Tier >= len(p.Tiers) {
return nil
}
var out []string
for _, m := range p.Tiers[p.Tier] {
if s := p.Modules[m]; s != nil && s.State == "built" && s.FirstAt != nil && s.SentAt == nil &&
len(s.First) > 0 && s.Why != "" {
out = append(out, m)
}
}
sort.Strings(out)
return out
}
// newerPlanFor is a plan made after this one that holds the module, superseded ones aside.
func newerPlanFor(module string, p inventory.Plan, plans []inventory.Plan) (inventory.Plan, bool) {
for _, q := range plans {
if q.ID == p.ID || q.State == inventory.PlanSuperseded || !q.Created.After(p.Created) {
continue
}
for _, tier := range q.Tiers {
for _, m := range tier {
if m == module {
return q, true
}
}
}
}
return inventory.Plan{}, false
}
// sendRollout sends machines what the mesh would send them now, answering the ones it sent. A
// variable so a test of a retried rollout needs no machine.
var sendRollout = sendToEach
// resumed sets a failed plan building again once nothing in its tier is failed.
func resumed(p *inventory.Plan, why string) {
if p.State == inventory.PlanFailed && len(failedIn(*p)) == 0 {
p.State = inventory.PlanBuilding
p.Note = why
}
}
// retryPlan asks a failed plan's failed modules again, under new ids, and sets it building again at
// that tier: what follows is the plan going on as if they had built the first time.
func retryPlan(ctx context.Context, open *stores, id string) (string, error) {
inv := open.inventory
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return "", err
}
defer release()
p, err := inv.PlanByID(ctx, id)
if err != nil {
return "", err
}
plans, err := inv.OpenPlans(ctx)
if err != nil {
return "", err
}
recent, err := inv.RecentPlans(ctx, 50)
if err != nil {
return "", err
}
plans = append(plans, recent...)
if err := retryRefusal(p, plans); err != nil {
return "", err
}
if len(failedIn(p)) == 0 {
return retryRollouts(ctx, open, &p)
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return "", err
}
byName := map[string]inventory.Entry{}
for _, e := range entries {
byName[e.Manifest.Module] = e
}
failed := failedIn(p)
var asked []string
for _, m := range failed {
askModule(ctx, &p, m, byName)
if s := p.Modules[m]; s.State == "asked" {
asked = append(asked, m+" as "+s.Build)
}
}
resumed(&p, fmt.Sprintf("tier %d retried by hand: %s asked again", p.Tier, strings.Join(failed, ", ")))
if err := inv.SavePlan(ctx, p); err != nil {
return "", err
}
if p.State != inventory.PlanBuilding {
return "", fmt.Errorf("%s could not be resumed: %s", p.ID, p.Note)
}
return fmt.Sprintf("%s retried at tier %d of %d: asked %s; the plan goes on from there as any plan does",
p.ID, p.Tier, len(p.Tiers), strings.Join(asked, ", ")), nil
}
// joinAPlan asks a module again for the plan that holds it unbuilt or failed, if one does: open plans
// first, then the most recent failed one that nothing newer supersedes. Says whether it joined one.
func joinAPlan(ctx context.Context, open *stores, module string) (bool, string, error) {
inv := open.inventory
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return false, "", err
}
defer release()
openPlans, err := inv.OpenPlans(ctx)
if err != nil {
return false, "", err
}
recent, err := inv.RecentPlans(ctx, 20)
if err != nil {
return false, "", err
}
p, found := planHolding(module, openPlans, recent)
if !found {
return false, "", nil
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return false, "", err
}
byName := map[string]inventory.Entry{}
for _, e := range entries {
byName[e.Manifest.Module] = e
}
was := p.State
askModule(ctx, &p, module, byName)
s := p.Modules[module]
resumed(&p, fmt.Sprintf("tier %d: %s rebuilt by hand", p.Tier, module))
if err := inv.SavePlan(ctx, p); err != nil {
return false, "", err
}
if s.State != "asked" {
return true, "", fmt.Errorf("%s could not be asked for %s: %s", module, p.ID, s.Why)
}
said := fmt.Sprintf("rebuild asked as %s, joining %s at tier %d (%s at %s): the plan takes this build as %s's outcome",
s.Build, p.ID, p.Tier, p.Repository, short(p.Commit), module)
switch {
case was == inventory.PlanFailed && p.State == inventory.PlanBuilding:
said += "; the plan had failed and builds again from this tier"
case was == inventory.PlanFailed:
said += "; the plan stays failed while " + strings.Join(failedIn(p), ", ") + " failed too — `plans retry " + p.ID + "` asks them"
}
return true, said, nil
}
// planHolding is the plan a rebuild of a module joins: an open plan whose current tier holds it not
// yet built, else the newest failed plan whose current tier does, and that no open plan of its
// repository supersedes.
func planHolding(module string, openPlans, recent []inventory.Plan) (inventory.Plan, bool) {
holds := func(p inventory.Plan) bool {
if p.Tier >= len(p.Tiers) {
return false
}
for _, m := range p.Tiers[p.Tier] {
if m == module {
s := p.Modules[m]
return s == nil || s.State != "built"
}
}
return false
}
for _, p := range openPlans {
if holds(p) {
return p, true
}
}
sorted := append([]inventory.Plan(nil), recent...)
sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].Created.After(sorted[j].Created) })
for _, p := range sorted {
if p.State != inventory.PlanFailed || !holds(p) {
continue
}
if _, superseded := newerOpenPlan(p, openPlans); superseded {
continue
}
return p, true
}
return inventory.Plan{}, false
}
// retryRollouts sends each module whose rollout stopped to the machines it was first sent to, again,
// records that send as the first anew, and sets the plan rolling: from there it goes on as the plan
// would have — the rest sent once those report they applied it, the next tier after (ADR 0218).
func retryRollouts(ctx context.Context, open *stores, p *inventory.Plan) (string, error) {
var said []string
for _, m := range stoppedRollouts(*p) {
s := p.Modules[m]
sent, err := sendRollout(ctx, open, s.First)
if err != nil {
return "", fmt.Errorf("%s could not be sent to %s again, so %s stays failed: %w",
m, strings.Join(s.First, ", "), p.ID, err)
}
now := time.Now().UTC()
s.First, s.FirstAt, s.Why = sent, &now, ""
said = append(said, m+" to "+strings.Join(sent, ", "))
}
p.State = inventory.PlanRolling
p.Note = fmt.Sprintf("tier %d retried by hand; sent %s first again", p.Tier, strings.Join(said, "; "))
if err := open.inventory.SavePlan(ctx, *p); err != nil {
return "", err
}
return fmt.Sprintf("%s retried at tier %d of %d: sent %s first again; the rest follow once it reports it "+
"applied, as the plan would have", p.ID, p.Tier, len(p.Tiers), strings.Join(said, "; ")), nil
}
+415 -156
View File
@@ -8,6 +8,7 @@ import (
"errors"
"flag"
"fmt"
"io"
"log"
"os"
"sort"
@@ -62,6 +63,9 @@ func connectLink(ctx context.Context, inv *inventory.Inventory, enroller link.En
}
func serve(ctx context.Context) error {
// The one process whose log is read over time, so the one that says each change to a node's
// unmet seat dependencies once (novox/hq ADR 0207).
logUnheldChanges = true
open, err := openStores(ctx)
if err != nil {
return err
@@ -123,11 +127,19 @@ func serve(ctx context.Context) error {
if err := server.Follows(following{open}); err != nil {
return err
}
// And the merges the bus announced and never handed over, read back on a timer and acted on late
// rather than never (novox/hq issue 266).
go catchingUpOnMerges(ctx, open, server)
// And a catalogue that has just started, asking for what it missed. The same type answers
// both: what a build meant and what the builds were are two questions about one record.
if err := server.Answers(following{open}); err != nil {
return err
}
// And what providers say about consumers they keep failing, kept for `status` (novox/hq ADR
// 0224): a provider's journal must not be the only place that says so.
if err := server.Watches(standings{inv}); err != nil {
return err
}
// And the mesh's own verbs, as the seat this control plane holds (novox/hq ADR 0154). Served
// from the store's row, so what the seat declares is what is answered.
@@ -146,6 +158,8 @@ func serve(ctx context.Context) error {
if !isNATS {
return errors.New("the mesh's verbs are served over the bus, and this control plane is not on it")
}
// A call that outlasts its caller's patience is followed by `calls` (novox/hq issue 265).
link.Calls.Follow = catalogue.ControllerSeatName + ".calls"
stopServing, err := bus.ServeSeatTools(catalogue.ControllerSeatName, handlers, log.New(os.Stdout, "", log.LstdFlags))
if err != nil {
return err
@@ -213,8 +227,9 @@ func declare(ctx context.Context, args []string) error {
}
// Written down like every other send (novox/hq issue 204): a declaration a person sent by hand
// is still what the machine was last told, and status must not read it as current for the one
// the mesh would compose.
if _, err := recordSent(ctx, inv, node, raw); err != nil {
// the mesh would compose. Which builds it carried is recorded as not known (novox/hq issue 259):
// the mesh did not compose it, so a push that does not name this machine treats it as held.
if _, err := recordSent(ctx, inv, node, raw, nil); err != nil {
return err
}
fmt.Printf("sent %s a signed declaration (%d bytes)\n", node, len(raw))
@@ -312,7 +327,7 @@ func pushCommand(ctx context.Context, args []string) error {
if len(needsOne) == 0 {
// Said rather than doing nothing quietly. "Nothing needed one" and "this did not run"
// must never look the same.
fmt.Println("every machine is doing what it was told")
fmt.Println("no machine named: a push of the whole mesh, and every machine is doing what it was told — nothing sent")
return nil
}
}
@@ -353,6 +368,46 @@ func pushCommand(ctx context.Context, args []string) error {
}
asked = append(asked, n.Name)
}
// **A push that named no machine says so, first.** Through the console a machine the caller
// meant to name could be lost on the way (novox/hq issue 244): the call arrived empty, ran as
// `push --behind`, and every machine behind was pushed by someone who thought they had pushed one.
// Its whole-mesh reach is the first line of the answer, with the machines it is about to send.
if len(args) == 0 {
which := "every machine"
if *behind {
which = "every machine that is behind"
}
fmt.Printf("no machine named: this is a push of the WHOLE mesh — %s (%d): %s\n",
which, len(asked), strings.Join(asked, ", "))
}
// **The machine holding the bus first** (novox/hq issue 249): its declaration carries the bus's
// user list, and a module's new grants are refused by the bus until that list says them. Among
// the machines asked it goes first; not among them and behind, it is added — a named push whose
// module gained a state would otherwise send the code and leave the right to use it for the
// cascade below, after.
holder, holderBehind, err := brokerBehind(ctx, open, asked)
if err != nil {
return err
}
// **Not when its own modules are held back** (novox/hq issue 259, ADR 0221): added rather than
// named, it is sent its whole declaration, and a build its policy records or a plan has not sent
// it yet would go with the user list. Named and left, with what that costs.
saidHeld := map[string]bool{}
if holderBehind && len(args) == 1 {
held, err := heldMachines(ctx, open, []string{holder})
if err != nil {
return err
}
if why, isHeld := held[holder]; isHeld {
sayHeld(os.Stdout, holder, why)
fmt.Printf("%s holds the bus, and the user list it would carry has changed: until it is "+
"sent, the bus may refuse what this push's machines were newly granted\n", holder)
holderBehind = false
saidHeld[holder] = true
}
}
asked = brokerFirst(asked, holder, holderBehind)
// Held from composing to sending, so a converge on one of them cannot send between the two
// and be overtaken by what was composed before it (novox/hq ADR 0100).
@@ -360,6 +415,9 @@ func pushCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
// Each machine's unmet seat dependencies (novox/hq ADR 0207), said after the sends: in full
// for a machine named, as a count for each of many — the full list is `status`'s.
unheld := map[string][]catalogue.Unheld{}
sending, refusals := composeEach(asked, allotting(held, inv), func(node string) (sendable, error) {
plan, settings, err := planFor(held, open, node)
if err != nil {
@@ -369,6 +427,7 @@ func pushCommand(ctx context.Context, args []string) error {
// healthy modules beside it are still resolved and sent. Reported so it is not silently
// dropped — the remedy is to move it, and until then the rest of the node converges.
reportUnhostable(node, plan)
unheld[node] = plan.Unheld
// The private network is in here with everything else. It used to be composed separately
// and prepended, which meant every machine with an address was on it and no machine could
// be kept off. It is a module now, so it arrives the way a module does.
@@ -379,34 +438,21 @@ func pushCommand(ctx context.Context, args []string) error {
return declared, err
})
sentDigest := map[string]string{}
defer release()
for _, s := range sending {
// The number is inside the signed bytes, so a replayed older declaration cannot borrow a
// newer one's (novox/hq 04-ISSUES/107); it was taken when the composition began (issue 204).
body, err := s.declared.Body()
if err != nil {
return err
}
if err := link.Declare(ctx, server.Bus(), ident, s.node, body, 15*time.Second); err != nil {
return err
}
// After it is away, not before. A digest recorded for something that failed to send would
// make the machine look current for a declaration it never received.
digest, err := recordSent(ctx, inv, s.node, body)
if err != nil {
return err
}
sentDigest[s.node] = digest
fmt.Printf("sent %s %d resource(s)\n", s.node, len(s.declared.Resources))
}
release()
fmt.Printf("\n%d node(s) told\n", len(sending))
// And each machine's memberships, as every other send does (ADR 0160): a push is the one most
// operators run, and on 2026-10-01 it was the one path that issued none.
if err := issueMemberships(ctx, open, server, sending); err != nil {
// Each machine's memberships first, then the declarations (novox/hq issue 249, ADR 0160): a push
// is the one most operators run, and on 2026-10-01 it was the one path that issued none.
bus := overTheBus{open: open, server: server, signer: ident}
sentDigest, err := deliver(ctx, bus, holder, sending)
if err != nil {
return err
}
release()
told := make([]string, 0, len(sending))
for _, r := range sending {
told = append(told, r.node)
}
fmt.Printf("\n%d node(s) told: %s\n", len(sending), strings.Join(told, ", "))
reportUnheldPushed(os.Stdout, len(args) == 1, asked, unheld)
// **A named push leaves the mesh consistent, not just the machine it named** (novox/hq
// issue 057, ADR 0083). Assigning a cross-node consumer mints a provision, and the PROVIDER's
@@ -416,86 +462,21 @@ func pushCommand(ctx context.Context, args []string) error {
//
// Compared against what each machine was last SENT, not against a before/after of this push:
// the mint usually happened at `assign` or `module issue`, before this command ran, so the
// only durable signal is "what it should be" versus "what it last received". A machine behind
// for an unrelated reason is caught here too, which is not a cost — a named push that knew a
// machine was behind and left it so would be the very silence this removes. Bounded: a
// only durable signal is "what it should be" versus "what it last received". Bounded: a
// flushed send may itself mint, so this converges over a few rounds.
//
// **Except a machine a policy or a plan holds back** (novox/hq issue 259, ADR 0221): one whose
// modules would move to a build their upgrade policy records rather than rolls out, or that an
// open plan has not sent it yet. It is named, with why, and left for a push that names it.
if len(args) == 1 {
flushed := map[string]bool{args[0]: true}
// Bounded by the node count: a node is marked flushed the round it is handled and is
// never handled twice, so the loop cannot run more than len(nodes) rounds. The bound is
// a guard against a logic error, not a real limit — if it were ever hit, that is a bug
// rather than a cascade legitimately still converging, so it is said rather than passed
// over in silence, unlike the earlier fixed cap that could stop a real cascade short.
rounds := 0
for {
would, err := wouldSend(ctx, open, nodes)
if err != nil {
return err
}
behind, err := inv.Waiting(ctx, would)
if err != nil {
return err
}
var also []string
for _, m := range behind {
if !flushed[m.Node] {
also = append(also, m.Node)
}
}
if len(also) == 0 {
break
}
if rounds++; rounds > len(nodes) {
fmt.Printf("\nstopped cascading after %d rounds with %s still behind — this "+
"should not happen; run `push --behind` to finish\n",
rounds-1, strings.Join(also, ", "))
break
}
sort.Strings(also)
fmt.Printf("\nthis push left %s behind — a provision granted from there, or a "+
"declaration since changed; sending it too\n", strings.Join(also, ", "))
// Tolerantly, exactly as the named send above: a machine that cannot be composed is
// collected as a refusal and reported at the end, and the others are still sent
// (novox/hq ADR 0066). The earlier cut routed these through sendTo, which is
// all-or-nothing — so one swept machine's compose error failed the operator's named
// push and skipped its --wait, the very intolerance the main path exists to avoid.
// Held for this round only, and after the last round's were given back, so two pushes
// cascading into each other's machines never each wait on the other.
refused, err := sendRound(ctx, open, also,
func(held context.Context, node string) (sendable, error) {
plan, settings, err := planFor(held, open, node)
if err != nil {
return sendable{}, err
}
reportUnhostable(node, plan)
declared, err := declarationWith(held, open, node, plan, settings, gens, Allocating)
if err == nil {
reportLeftOut(node, declared)
}
return declared, err
},
func(s readyNode, body []byte) error {
if err := link.Declare(ctx, server.Bus(), ident, s.node, body,
15*time.Second); err != nil {
return err
}
if _, err := recordSent(ctx, inv, s.node, body); err != nil {
return err
}
fmt.Printf("sent %s %d resource(s)\n", s.node, len(s.declared.Resources))
return nil
})
refusals = append(refusals, refused...)
if err != nil {
return err
}
// Every candidate this round is marked handled — the sent ones so they are not
// re-listed, and the refused ones so a machine that cannot be composed does not make
// the loop spin on it for ever. Its refusal is already in the report.
for _, name := range also {
flushed[name] = true
}
handled := map[string]bool{args[0]: true}
for n := range saidHeld {
handled[n] = true
}
refused, err := flushBehind(ctx, open, nodes, handled, composeForPush(open, gens), bus, holder, os.Stdout)
refusals = append(refusals, refused...)
if err != nil {
return err
}
}
@@ -612,10 +593,10 @@ func composeEach(names []string, allot func(node string) (int64, error),
// sendRound holds the named nodes, composes each and sends each that composed, and gives the hold
// back on every way out — a body that cannot be marshalled and a send that fails included
// (novox/hq ADR 0100). A node that cannot be composed is a refusal, not an error: the others are
// still sent.
// still sent. Their memberships go before their declarations, as every send's do (issue 249).
func sendRound(ctx context.Context, open *stores, names []string,
compose func(held context.Context, node string) (sendable, error),
send func(s readyNode, body []byte) error) ([]string, error) {
d delivery, holder string) ([]string, error) {
held, release, err := holdNodes(ctx, open, names)
if err != nil {
return nil, err
@@ -624,18 +605,246 @@ func sendRound(ctx context.Context, open *stores, names []string,
sending, refused := composeEach(names, allotting(held, open.inventory), func(node string) (sendable, error) {
return compose(held, node)
})
for _, s := range sending {
body, err := s.declared.Body()
if err != nil {
return refused, err
}
if err := send(s, body); err != nil {
return refused, err
}
if _, err := deliver(held, d, holder, sending); err != nil {
return refused, err
}
return refused, nil
}
// delivery is the two acts of sending machines what they should be, apart, so the order between
// them is one function's and can be read and tested there (novox/hq issue 249).
type delivery interface {
// grant issues what the machines' modules may do — each module's state raised and its
// membership issued — for every machine about to be sent.
grant(ctx context.Context, sending []readyNode) error
// declare sends one machine its declaration and records it sent, answering the digest.
declare(ctx context.Context, s readyNode, body []byte) (string, error)
}
// errGrants marks a send that stopped because what the machines' modules may do could not be issued
// (novox/hq issue 249). Nothing about the machines is wrong; asked again, it is likely to work, so an
// announcement that hits it is held and asked again.
var errGrants = errors.New("what the machines' modules may do on the bus could not be issued, and code " +
"sent before its grants is refused there")
// grantsRefused is a grant that failed for some machines and not others: their memberships could not
// be issued, by machine, and only those machines are held back.
type grantsRefused struct{ nodes map[string]error }
func (g *grantsRefused) Error() string {
names := make([]string, 0, len(g.nodes))
for n := range g.nodes {
names = append(names, n)
}
sort.Strings(names)
return fmt.Sprintf("the memberships of %s could not be issued; the first: %v",
strings.Join(names, ", "), g.nodes[names[0]])
}
// deliver sends the machines their declarations: **the machine holding the bus, then the grants,
// then the rest** (novox/hq issue 249).
//
// A merge gave a module a new state; its bundle reached every machine within a minute, and the
// machines' permissions on the bus did not include the state until somebody pushed by hand: the code
// arrived before the right to use it. A module that read its new state on start failed its start; the
// one that was there retried for two minutes. The memberships were issued after the declarations —
// "because the runtime it is for arrives with it" — and a membership is retained last-per-subject on
// the bus (internal/link/bus.go), so issued first it waits for the runtime that arrives after it. A
// runtime still on the old code merely holds a grant it does not use yet.
//
// **The holder's declaration before the grants, though.** The bus's user list travels in it, and the
// controller's own right to publish memberships and raise buckets is in that list (the precedent of
// issue 183): grants first, and a grant the controller is not yet allowed to make would hold the very
// declaration that allows it — a lock only a hand on the broker could open. A runtime already running
// on that machine follows a membership issued after its declaration, as it always has.
//
// **A grant that cannot be issued holds back what it concerns, and says so as an error.** It used to
// be said and passed over — "the machines keep what they derive until the next push" — which reported
// a rollout done that had delivered code its machines could not run. A membership that failed holds
// back its own machine; a failure that names no machine (the buckets) holds back every machine but the
// holder, already sent. The error carries errGrants, so the caller's rollout is not marked sent and is
// tried again.
//
// The grants are issued, not waited on: a membership is a retained message the runtime reads when it
// comes, and the bus answers its publication; nothing here waits for a runtime to have read one.
func deliver(ctx context.Context, d delivery, holder string, sending []readyNode) (map[string]string, error) {
digests := map[string]string{}
if len(sending) == 0 {
return digests, nil
}
send := func(s readyNode) error {
// The number is inside the signed bytes, so a replayed older declaration cannot borrow a
// newer one's (novox/hq 04-ISSUES/107); it was taken when the composition began (issue 204).
body, err := s.declared.Body()
if err != nil {
return err
}
digest, err := d.declare(ctx, s, body)
if err != nil {
return err
}
digests[s.node] = digest
return nil
}
var rest []readyNode
for _, s := range sending {
if holder != "" && s.node == holder {
if err := send(s); err != nil {
return digests, err
}
continue
}
rest = append(rest, s)
}
held := map[string]error{}
if err := d.grant(ctx, sending); err != nil {
var some *grantsRefused
if !errors.As(err, &some) {
var names []string
for _, s := range rest {
names = append(names, s.node)
}
if len(names) == 0 {
return digests, fmt.Errorf("%w: %w", errGrants, err)
}
return digests, fmt.Errorf("%w; %s not sent: %w", errGrants, strings.Join(names, ", "), err)
}
held = some.nodes
}
var notSent []string
for _, s := range rest {
if _, refused := held[s.node]; refused {
notSent = append(notSent, s.node)
continue
}
if err := send(s); err != nil {
return digests, err
}
}
if len(notSent) > 0 {
return digests, fmt.Errorf("%w; %s not sent: %w", errGrants, strings.Join(notSent, ", "),
&grantsRefused{nodes: held})
}
if len(held) > 0 {
// Only the holder's own memberships failed, and it was sent before them.
return digests, fmt.Errorf("%w: %w", errGrants, &grantsRefused{nodes: held})
}
return digests, nil
}
// overTheBus is delivery as the mesh does it: memberships on the bus, declarations signed.
type overTheBus struct {
open *stores
server *link.Server
signer link.Signer
// indent is put before each "sent" line, for the callers whose output is nested.
indent string
}
func (b overTheBus) grant(ctx context.Context, sending []readyNode) error {
return issueMemberships(ctx, b.open, b.server, sending)
}
func (b overTheBus) declare(ctx context.Context, s readyNode, body []byte) (string, error) {
if err := link.Declare(ctx, b.server.Bus(), b.signer, s.node, body, 15*time.Second); err != nil {
return "", err
}
// After it is away, not before. A digest recorded for something that failed to send would make
// the machine look current for a declaration it never received.
digest, err := recordSent(ctx, b.open.inventory, s.node, body, s.declared.Builds)
if err != nil {
return "", err
}
if s.declared.BusUsers != "" {
// And the user list it carried, so the next send reads whether it must go first from the
// list alone (novox/hq issue 249). On the same outliving context as the send's record.
kept, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
err := b.open.inventory.RecordSentBusUsers(kept, s.node, digestOf([]byte(s.declared.BusUsers)))
cancel()
if err != nil {
return "", err
}
}
fmt.Printf("%ssent %s %d resource(s)\n", b.indent, s.node, len(s.declared.Resources))
return digest, nil
}
// brokerFirst is the machines to send in the order a grant needs (novox/hq issue 249): the machine
// holding the bus first — its declaration carries the bus's user list (composeBusUsers), and a
// module's new permissions are refused by the bus until that list says them. Among the machines it
// is moved to the front; not among them, it is added only when it is behind.
func brokerFirst(names []string, holder string, behind bool) []string {
if holder == "" {
return names
}
present := false
rest := make([]string, 0, len(names))
for _, n := range names {
if n == holder {
present = true
continue
}
rest = append(rest, n)
}
if !present && !behind {
return names
}
return append([]string{holder}, rest...)
}
// brokerBehind is the machine holding the bus — the one whose declaration carries the user list —
// and, when it is not among the machines named, whether the user list it would be sent now differs
// from the one it was last sent (novox/hq issue 249).
//
// **The user list alone, not the whole declaration.** Read from the whole declaration, any change
// pending on that machine — an upgrade its policy records rather than rolls out — went with every
// send anywhere, and a module running there always put it in its first wave. A digest of the list
// last sent is kept for this (ADR 0043: the list is composed on each push, never kept itself).
func brokerBehind(ctx context.Context, open *stores, names []string) (string, bool, error) {
inv := open.inventory
holders, err := seatHolders(ctx, inv)
if err != nil {
return "", false, err
}
h, held := holders[theBrokerSeat]
if !held || h.Node == "" {
return "", false, nil
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
return "", false, err
}
if m, known := shelf[h.Module]; !known || m.BusUsers == "" {
// A holder that is sent no user list carries no grant: nothing to send first.
return "", false, nil
}
for _, n := range names {
if n == h.Node {
return h.Node, false, nil
}
}
plan, _, err := planFor(ctx, open, h.Node)
if err != nil {
// It cannot be worked out: sending it would refuse the whole send, and `plan` says why.
return h.Node, false, nil
}
list, _, err := busUserList(ctx, inv, plan.Modules)
if err != nil {
return h.Node, false, nil
}
sent, err := inv.SentBusUsers(ctx, h.Node)
if err != nil {
return "", false, err
}
return h.Node, userListBehind(list, sent), nil
}
// userListBehind is whether the user list composed now is not the one last sent, by its digest. An
// empty list composed is never behind: there is nothing for it to carry.
func userListBehind(now, sentDigest string) bool {
return now != "" && digestOf([]byte(now)) != sentDigest
}
// couldNotBeResolved is what a push ends with when some machines could not be worked out.
//
// **After the rest have been sent, never instead of sending them.** It is still an error, because
@@ -658,23 +867,36 @@ func couldNotBeResolved(refusals []string, sent int) error {
// consumer and refused on the provider would leave one end holding a credential the other has
// never heard of — which is the state this whole mechanism exists to make impossible.
func sendTo(ctx context.Context, open *stores, names []string) error {
_, err := sendToEach(ctx, open, names)
return err
}
// sendToEach is sendTo, answering the machines it sent: those named, and before them the machine
// holding the bus when its user list must go first (novox/hq issue 249) — so a caller that waits for
// the machines it sent waits for that one too.
func sendToEach(ctx context.Context, open *stores, names []string) ([]string, error) {
inv := open.inventory
ident, err := openIdentity(ctx)
if err != nil {
return err
return nil, err
}
defer ident.Close()
gens, err := generators(ctx, open)
if err != nil {
return err
return nil, err
}
holder, behind, err := brokerBehind(ctx, open, names)
if err != nil {
return nil, err
}
names = brokerFirst(names, holder, behind)
// Held from composing to sending (novox/hq ADR 0100); a caller that holds them already —
// converge, which flips the node and then sends it — is not made to wait on itself.
ctx, release, err := holdNodes(ctx, open, names)
if err != nil {
return err
return nil, err
}
defer release()
@@ -703,33 +925,29 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
sending = append(sending, readyNode{name, declared})
}
if len(refusals) > 0 {
return fmt.Errorf("nothing was sent. %d machine(s) could not be resolved:\n\n%s",
return nil, fmt.Errorf("nothing was sent. %d machine(s) could not be resolved:\n\n%s",
len(refusals), strings.Join(refusals, "\n\n"))
}
server, err := connectLink(ctx, nil, nil, nil)
if err != nil {
return err
return nil, err
}
defer server.Close()
for _, s := range sending {
body, err := s.declared.Body()
if err != nil {
return err
}
if err := link.Declare(ctx, server.Bus(), ident, s.node, body, 15*time.Second); err != nil {
return err
}
if _, err := recordSent(ctx, inv, s.node, body); err != nil {
return err
}
fmt.Printf(" sent %s %d resource(s)\n", s.node, len(s.declared.Resources))
}
// And every assignment on those machines its membership (novox/hq ADR 0160): composed from the
// same records the bus's accounts are, so what a runtime serves and what its account may are one
// composition. Issued after the declaration, because the runtime it is for arrives with it.
return issueMemberships(ctx, open, server, sending)
// composition. **Issued before the declarations** (novox/hq issue 249): the runtime the
// membership is for arrives with the declaration, and a membership waits for it on the bus; the
// code arriving first was refused its own state until somebody pushed.
if _, err := deliver(ctx, overTheBus{open: open, server: server, signer: ident, indent: " "}, holder, sending); err != nil {
return nil, err
}
sent := make([]string, 0, len(sending))
for _, s := range sending {
sent = append(sent, s.node)
}
return sent, nil
}
// issueMemberships publishes the membership of every module on the machines just sent.
@@ -747,11 +965,26 @@ func issueMemberships(ctx context.Context, open *stores, server *link.Server, se
if !ok {
return nil
}
// The declarations are sent and recorded by now; a membership that cannot be issued is said
// and does not unsay them. Every runtime without one serves the shape it derives (ADR 0160), so
// the push stands, the first failure is named once, and the next push tries again.
issued, failed := 0, 0
var first error
// **Every declared state's bucket, before the memberships that name it** (novox/hq ADR 0201). The
// raise at start asserts them too, but a module registered and assigned since would otherwise have
// its bucket only after the control plane next restarts — found the first time a module declared
// state: its bundle asked for a bucket that did not exist. Idempotent and cheap.
//
// **A failure here is the send's failure** (novox/hq issue 249). It was said and the push stood,
// because the declarations were already away; they are sent after this now — all but the bus's
// own machine, sent before it (deliver) — and a module whose state does not exist is a module
// that fails its start, so they are not sent and the caller tries again rather than reporting the
// rollout done.
buckets, err := open.inventory.DeclaredBuckets(ctx)
if err != nil {
return fmt.Errorf("the modules' state could not be read, so no bucket was asserted: %w", err)
}
if _, err := broker.RaiseBuckets(broker.OnConn(bus.Conn), buckets); err != nil {
return fmt.Errorf("the modules' state could not be asserted on the bus: %w", err)
}
// Every membership is tried, and the first failure named once.
issued := 0
refused := map[string]error{}
for _, s := range sent {
node := s.node
for _, d := range records.Assigned[node] {
@@ -772,10 +1005,9 @@ func issueMemberships(ctx context.Context, open *stores, server *link.Server, se
return err
}
if err := bus.PublishMembership(ctx, node, d.Module, body); err != nil {
if first == nil {
first = err
if refused[node] == nil {
refused[node] = fmt.Errorf("%s: %w", d.Module, err)
}
failed++
continue
}
issued++
@@ -784,9 +1016,10 @@ func issueMemberships(ctx context.Context, open *stores, server *link.Server, se
if issued > 0 {
fmt.Printf(" issued %d membership(s)\n", issued)
}
if failed > 0 {
fmt.Printf(" %d membership(s) could not be issued; the first: %v — the machines keep what "+
"they derive until the next push\n", failed, first)
if len(refused) > 0 {
// Returned, never passed over (novox/hq issue 249): the declarations of the machines they are
// for are not sent, and the rollout that asked is tried again rather than waiting for a push.
return &grantsRefused{nodes: refused}
}
return nil
}
@@ -898,6 +1131,11 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
if err := broker.RaiseSeats(js, inventory.MeshSeats(), holders); err != nil {
return err
}
// And each work queue's cancelled set (novox/hq ADR 0219), so a holder taking an ask can ask
// whether it was cancelled the moment it took it.
if err := broker.RaiseCancelledSets(js, inventory.MeshSeats()); err != nil {
return err
}
// Every module's state (novox/hq ADR 0201), from the catalogue: a bucket exists from
// registration, so a module reading one may watch it before its owner runs anywhere. One that
// nothing declares any more is said and kept — what it holds is data.
@@ -926,13 +1164,9 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
return err
}
hearing := 0
for _, p := range users {
consumer, needed := broker.ConsumerFor(p)
if !needed {
continue
}
if err := js.EnsureConsumer(consumer); err != nil {
return fmt.Errorf("how %s on %s hears what it consumes: %w", p.Module, p.Node, err)
for _, c := range broker.ConsumersOf(users) {
if err := js.EnsureConsumer(c.Consumer); err != nil {
return fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err)
}
hearing++
}
@@ -999,7 +1233,11 @@ func allot(ctx context.Context, inv *inventory.Inventory, node string) (int64, e
// never wrote it down: status read "applied, current" over a machine that had just been sent
// something else. What was sent was sent; the record of it must not depend on the sender living
// another second. Bounded, so a store that is away does not hold a dying process open for ever.
func recordSent(ctx context.Context, inv *inventory.Inventory, node string, body []byte) (string, error) {
//
// And the build of each module it carried (novox/hq issue 259, ADR 0221), nil when that is not known:
// what tells a machine held back by a policy or a plan from one a push left behind.
func recordSent(ctx context.Context, inv *inventory.Inventory, node string, body []byte,
builds map[string]string) (string, error) {
kept, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
record, err := inv.NodeByName(kept, node)
@@ -1007,8 +1245,29 @@ func recordSent(ctx context.Context, inv *inventory.Inventory, node string, body
return "", err
}
digest := digestOf(body)
if err := inv.RecordSent(kept, record.ID, digest); err != nil {
if err := inv.RecordSent(kept, record.ID, digest, builds); err != nil {
return "", err
}
return digest, nil
}
// reportUnheldPushed says what a push's machines lack of the seats their modules depend on
// (novox/hq ADR 0207): every line for a machine the push named, since that is the machine somebody
// is looking at, and one line per machine otherwise — a list per machine across the mesh is the
// hundred lines that buried the one that mattered. Nothing for a machine that lacks nothing.
func reportUnheldPushed(w io.Writer, named bool, asked []string, unheld map[string][]catalogue.Unheld) {
for _, node := range asked {
lines := unheld[node]
if len(lines) == 0 {
continue
}
if named {
fmt.Fprintf(w, "\n%s has %d unmet seat dependenc(ies) (novox/hq ADR 0207):\n", node, len(lines))
for _, u := range lines {
fmt.Fprintf(w, " %s\n", u)
}
continue
}
fmt.Fprintf(w, "%s: %d unmet seat dependenc(ies) — see `status`\n", node, len(lines))
}
}
+709
View File
@@ -0,0 +1,709 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The build queue, controlled by hand (novox/hq ADR 0219).
//
// Seen whole — what waits, what runs where and for how long, what was handed out as often as it
// may be and never settled — and changed through the controller: an ask cancelled, the queue
// cleared, a module rebuilt, a build replayed. What one machine is running is that machine's
// holder's to end or pause, and the controller asks it (`kill`, `pause`, `resume`).
//
// **Everything that drops an ask leaves a failed outcome for it**, taken in exactly as a build that
// failed is (takeIn, then the plan): a plan waiting on an ask a person removed fails, saying so,
// rather than waiting for ever on an answer nobody will give.
// holderAsks is how long a holder's verb is waited for; killAnswer how long its kill is — longer
// than the holder's worst case (its two passes removing containers and its wait between, 50s).
const (
holderAsks = 15 * time.Second
killAnswer = 75 * time.Second
)
// dialTheBus opens the controller's own connection, for a command that reads or changes the queue.
func dialTheBus() (*broker.JetStream, error) {
address, err := broker.BusAddress()
if err != nil {
return nil, err
}
js, err := broker.Dial(address)
if err != nil {
return nil, fmt.Errorf("cannot reach the bus: %w", err)
}
return js, nil
}
// queueCommand prints every ask in the build seat's work queue.
func queueCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("queue", flag.ContinueOnError)
asJSON := set.Bool("json", false, "the queue as JSON")
if _, err := parseAround(set, args); err != nil {
return err
}
js, err := dialTheBus()
if err != nil {
return err
}
defer js.Close()
seat := buildSeatHeld(ctx)
q, err := link.ReadQueue(ctx, js, seat)
if err != nil {
return err
}
if *asJSON {
body, err := json.MarshalIndent(q, "", " ")
if err != nil {
return err
}
fmt.Println(string(body))
return nil
}
fmt.Print(queueText(q, time.Now()))
return nil
}
// queueText is the queue as a person reads it: a summary line, then each kind in queue order.
// Never what an ask carries as `held` — that is every artifact the mesh has built.
func queueText(q link.Queue, now time.Time) string {
var b strings.Builder
waiting, running, dead := q.Of(link.AskWaiting), q.Of(link.AskInFlight), q.Of(link.AskDead)
fmt.Fprintf(&b, "%s: %d waiting, %d in flight, %d dead\n", q.Seat, len(waiting), len(running), len(dead))
ago := func(t time.Time) string {
if t.IsZero() {
return "asked at an unknown time"
}
return "asked " + now.Sub(t).Round(time.Second).String() + " ago"
}
what := func(a link.QueuedAsk) string {
s := a.Repository
if a.Path != "" {
s += " at " + a.Path
}
if a.Ref != "" {
s += " on " + a.Ref
}
return s
}
if len(running) > 0 {
fmt.Fprintln(&b, "\nin flight:")
for _, a := range running {
where := "taken, not yet said where"
switch {
case a.On != "":
where = fmt.Sprintf("on %s for %s", a.On, now.Sub(a.Started).Round(time.Second))
case a.Was != "":
where = fmt.Sprintf("handed back by %s, to be handed out again", a.Was)
}
fmt.Fprintf(&b, " %-26s %s — %s, %s (seq %d)\n", a.ID, what(a), where, ago(a.AskedAt), a.Seq)
}
}
if len(waiting) > 0 {
fmt.Fprintln(&b, "\nwaiting:")
for _, a := range waiting {
fmt.Fprintf(&b, " %-26s %s — %s (seq %d)\n", a.ID, what(a), ago(a.AskedAt), a.Seq)
}
}
if len(dead) > 0 {
fmt.Fprintf(&b, "\ndead — handed out as often as the worker allows (%d) and never settled, still held:\n", q.MaxDeliver)
for _, a := range dead {
fmt.Fprintf(&b, " %-26s %s — %s (seq %d)\n", a.ID, what(a), ago(a.AskedAt), a.Seq)
}
}
switch {
case len(q.Asks) == 0:
fmt.Fprintln(&b, "nothing is asked of it")
default:
fmt.Fprintln(&b, "\n`cancel <id>` drops a waiting or dead ask, `clear` every waiting one, `kill <id>` ends one in flight")
}
return b.String()
}
// cancelCommand drops one waiting or dead ask.
func cancelCommand(ctx context.Context, args []string) error {
if len(args) != 1 {
return errors.New("cancel <build id>")
}
js, err := dialTheBus()
if err != nil {
return err
}
defer js.Close()
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
seat := buildSeatHeld(ctx)
q, err := link.ReadQueue(ctx, js, seat)
if err != nil {
return err
}
ask, found := q.Find(args[0])
if !found {
return fmt.Errorf("%s is not in the %s queue: it was built, cancelled, or never asked — `builds` says "+
"what came of it", args[0], seat)
}
said, err := cancelAsk(ctx, js, open, seat, ask)
if err != nil {
return err
}
fmt.Println(said)
return nil
}
// cancelAsk drops one ask from the queue and records it failed, cancelled by hand.
//
// **In this order, and each step for a reason** (novox/hq ADR 0219):
// 1. an ask a machine says it is building is refused: deleting its message ends nothing a machine
// is running — that is `kill`, on the machine running it;
// 2. its id goes into the seat's cancelled set, so a holder that fetches it from now on ends it;
// 3. for a waiting ask, the worker is read again: one handed out since the queue was read was
// taken in the moment of cancelling, and the cancel is withdrawn and refused — the holder either
// read the mark first and ends it as cancelled, or is building it;
// 4. for an ask the worker counts handed out that no machine is building — taken and not yet said,
// handed back after a restart, or past its deliveries while nobody pulls — the holder's own look
// at the set is waited out, and a start heard since withdraws and refuses the cancel: a holder
// that took it before the mark is building it, and only `kill` ends that;
// 5. the message is deleted by its sequence;
// 6. the failed outcome is taken in as any failed build's is, and the plan that asked fails.
func cancelAsk(ctx context.Context, js *broker.JetStream, open *stores, seat string, ask link.QueuedAsk) (string, error) {
if ask.State == link.AskInFlight && ask.On != "" {
return "", fmt.Errorf("%s is in flight on %s: cancelling drops an ask nobody is building. `kill %s` "+
"ends the build where it runs", ask.ID, ask.On, ask.ID)
}
if err := link.MarkCancelled(ctx, js, seat, ask.ID); err != nil {
return "", err
}
withdraw := func() {
if err := link.UnmarkCancelled(ctx, js, seat, ask.ID); err != nil {
fmt.Fprintf(os.Stderr, "could not withdraw the cancel of %s: %v — a holder taking it ends it as cancelled\n", ask.ID, err)
}
}
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: seat, Accepts: []string{"build"}})
switch ask.State {
case link.AskWaiting:
if taken, err := takenSince(js, worker, ask.Seq); err != nil {
withdraw()
return "", err
} else if taken {
withdraw()
return "", fmt.Errorf("%s was taken by a holder as it was cancelled, so the cancel is withdrawn. If the "+
"holder read it first it ends the ask as %s and its outcome says so; otherwise it is building — "+
"`queue` says which, and `kill %s` ends it", ask.ID, link.CancelledByHand, ask.ID)
}
case link.AskInFlight:
if started, err := startedSince(ctx, js, seat, ask); err != nil {
withdraw()
return "", err
} else if started != "" {
withdraw()
return "", fmt.Errorf("%s has started on %s since it was read, so the cancel is withdrawn: `kill %s` "+
"ends it there", ask.ID, started, ask.ID)
}
}
if err := js.Context().DeleteMsg(worker.Stream, ask.Seq); err != nil && !errors.Is(err, nats.ErrMsgNotFound) &&
!errors.Is(err, jetstream.ErrMsgNotFound) && !strings.Contains(err.Error(), "no message found") {
return "", fmt.Errorf("%s is marked cancelled and could not be deleted from the queue (seq %d): %w — a "+
"holder taking it ends it as cancelled", ask.ID, ask.Seq, err)
}
recordCancelled(ctx, open, ask)
return fmt.Sprintf("cancelled %s (%s, %s): deleted from the %s queue and recorded failed, %s — a plan "+
"that asked for it fails with that", ask.ID, ask.Repository, ask.State, seat, link.CancelledByHand), nil
}
// holderLooks is how long a cancel waits for a holder that took the ask before the mark to say it
// started: longer than a holder's look at the cancelled set (3s) and its start that follows.
var holderLooks = 5 * time.Second
// startedSince waits out a holder's look at the cancelled set and says the machine that started the
// ask since it was read, or nothing. A start it ended as cancelled comes with its outcome and is not
// a start of a build.
func startedSince(ctx context.Context, js *broker.JetStream, seat string, ask link.QueuedAsk) (string, error) {
select {
case <-ctx.Done():
return "", ctx.Err()
case <-time.After(holderLooks):
}
since := time.Now().Add(-7 * 24 * time.Hour)
if !ask.AskedAt.IsZero() {
since = ask.AskedAt.Add(-time.Minute)
}
heard, err := link.ReadBuildEvents(ctx, js, seat, since)
if err != nil {
return "", err
}
s, ok := heard.Started[ask.ID]
if !ok || heard.Outcomes[ask.ID] {
return "", nil
}
at, _ := time.Parse(time.RFC3339Nano, s.At)
if ask.Started.IsZero() || at.After(ask.Started) {
return s.On, nil
}
return "", nil
}
// takenSince is whether the worker has handed out the ask at this sequence.
func takenSince(js *broker.JetStream, worker broker.Consumer, seq uint64) (bool, error) {
info, err := js.Context().ConsumerInfo(worker.Stream, worker.Name)
if errors.Is(err, nats.ErrConsumerNotFound) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("cannot read the worker of the queue again: %w", err)
}
return info.Delivered.Stream >= seq, nil
}
// recordCancelled takes the failed outcome in the way the daemon takes in any build's: recorded,
// then the plan that asked for it — by the id it asked with, or by repository and path.
func recordCancelled(ctx context.Context, open *stores, ask link.QueuedAsk) {
r := ask.Request
result := link.BuildResult{ID: ask.ID, Repository: ask.Repository, Path: ask.Path, Ref: ask.Ref,
Source: r.Source, DryRun: r.DryRun, Failed: link.CancelledByHand}
_ = builds{open.inventory, open}.Built(ctx, result)
}
// clearCommand cancels every waiting ask, and with --dead every dead one too.
func clearCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("clear", flag.ContinueOnError)
dead := set.Bool("dead", false, "the dead asks too")
if _, err := parseAround(set, args); err != nil {
return err
}
js, err := dialTheBus()
if err != nil {
return err
}
defer js.Close()
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
seat := buildSeatHeld(ctx)
said, err := clearQueue(ctx, js, open, seat, *dead)
fmt.Print(said)
return err
}
// clearQueue cancels what clear names, each as `cancel` would, and never anything in flight.
func clearQueue(ctx context.Context, js *broker.JetStream, open *stores, seat string, dead bool) (string, error) {
q, err := link.ReadQueue(ctx, js, seat)
if err != nil {
return "", err
}
var b strings.Builder
cancelled, refused := 0, 0
for _, a := range q.Asks {
if a.State == link.AskInFlight || (a.State == link.AskDead && !dead) {
continue
}
said, err := cancelAsk(ctx, js, open, seat, a)
if err != nil {
refused++
fmt.Fprintf(&b, " %s: %v\n", a.ID, err)
continue
}
cancelled++
fmt.Fprintf(&b, " %s\n", said)
}
what := "waiting"
if dead {
what = "waiting and dead"
}
fmt.Fprintf(&b, "%d %s ask(s) cancelled", cancelled, what)
if refused > 0 {
fmt.Fprintf(&b, ", %d could not be", refused)
}
fmt.Fprintln(&b)
if n := len(q.Of(link.AskInFlight)); n > 0 {
fmt.Fprintf(&b, "%d in flight, left running: `kill <id>` ends one where it runs\n", n)
}
if n := len(q.Of(link.AskDead)); n > 0 && !dead {
fmt.Fprintf(&b, "%d dead, left: `clear --dead` cancels them too\n", n)
}
if refused > 0 {
return b.String(), fmt.Errorf("%d ask(s) could not be cancelled", refused)
}
return b.String(), nil
}
// rebuildCommand asks the module's current source again — or, given a build's id, that build's
// repository, path and ref — under a new id.
func rebuildCommand(ctx context.Context, args []string) error {
if len(args) != 1 {
return errors.New("rebuild <module | build id>")
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
entries, err := inv.Catalogued(ctx)
if err != nil {
return err
}
var source buildSource
var path, ref, module string
if b, found, err := inv.BuildByID(ctx, args[0]); err != nil {
return err
} else if found {
source, module = sourceOfBuild(b, entries)
path, ref = b.Path, b.Ref
// **A build at a commit is rebuilt at what its module follows now** (novox/hq ADR 0219, issue
// 219): asked now, a rebuild of an old commit would be the newest ask of the module and roll
// that commit out over everything since. Building that commit again is `replay`, which says
// what it would do and refuses to register it while anything newer is asked or registered.
if e, known := entryNamed(entries, module); known && ref != "" && followedBranch(ref) == "" {
ref = followedBranch(e.Source.Ref)
follows := ref
if follows == "" {
follows = "the repository's default branch"
}
fmt.Printf("%s was built at commit %s; a rebuild asks what %s follows now, %s — `replay %s` "+
"builds that commit\n", b.ID, short(b.Ref), module, follows, b.ID)
}
} else if e, known := entryNamed(entries, args[0]); known {
// As a plan asks it: the branch it follows, never a commit a build once named (issue 215).
source = buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
path, ref, module = e.Source.Path, followedBranch(e.Source.Ref), e.Manifest.Module
} else {
return fmt.Errorf("%s is neither a module the catalogue holds nor a build the mesh recorded", args[0])
}
// **A module a plan holds unbuilt, or failed, joins that plan** rather than running beside it: the
// plan would ask it again, or stay failed on an outcome a rebuild has replaced.
if module != "" {
joined, said, err := joinAPlan(ctx, open, module)
if err != nil {
return err
}
if joined {
fmt.Println(said)
return nil
}
}
id, err := askABuild(ctx, source, path, ref)
if err != nil {
return err
}
fmt.Printf("rebuild asked as %s\n", id)
return nil
}
// entryNamed is the catalogue's entry for a module.
func entryNamed(entries []inventory.Entry, name string) (inventory.Entry, bool) {
for _, e := range entries {
if e.Manifest.Module == name {
return e, true
}
}
return inventory.Entry{}, false
}
// sourceOfBuild is where a recorded build's repository is asked from: the catalogued module's own
// source when the build is of one — on its seat, so the outcome registers it as the mesh records it
// (ADR 0111) — else the repository as it was cloned.
func sourceOfBuild(b inventory.Build, entries []inventory.Entry) (buildSource, string) {
for _, e := range entries {
if (b.Module != "" && e.Manifest.Module == b.Module) ||
(b.Module == "" && repositoryMatches(e.Source.Repository, b.Repository) && e.Source.Path == b.Path) {
return buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}, e.Manifest.Module
}
}
return buildSource{Repository: b.Repository}, b.Module
}
// replayCommand asks a recorded build's repository and path again at the commit it built.
func replayCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("replay", flag.ContinueOnError)
register := set.Bool("register", false, "register what it builds, as any build is")
older := set.Bool("older", false, "with --register: even though a newer build of the module is registered")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 1 {
return errors.New("replay <build id> [--register [--older]]")
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
b, found, err := inv.BuildByID(ctx, positionals[0])
if err != nil {
return err
}
if !found {
return fmt.Errorf("no build %s is recorded", positionals[0])
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return err
}
source, module := sourceOfBuild(b, entries)
var history []inventory.Build
if module != "" {
if history, err = inv.Builds(ctx, module, 100); err != nil {
return err
}
}
// What is asked of the module and not yet answered, when the replay would be registered.
var outstanding []string
if *register {
js, err := dialTheBus()
if err != nil {
return err
}
q, err := link.ReadQueue(ctx, js, buildSeatHeld(ctx))
js.Close()
if err != nil {
return err
}
plans, err := inv.OpenPlans(ctx)
if err != nil {
return err
}
outstanding = outstandingFor(module, b.Repository, b.Path, q, plans)
}
if err := replayRefusal(b, module, history, *register, *older, outstanding); err != nil {
return err
}
id, err := buildOneAsked(ctx, source, b.Path, b.Commit, 0, !*register)
if err != nil {
return err
}
fmt.Println(replaySaid(b, module, id, *register))
return nil
}
// replayRefusal is why a replay is not asked, or nothing (novox/hq ADR 0219, issue 207).
//
// A replay re-asks a recorded build at the commit it built, under a new id — and an id is when it
// was asked, which is what orders builds of one module (issue 219). So a registered replay of an
// older commit is newer than everything since, and would roll that older commit out as the module's
// current version: what issue 207 recorded happening by accident. It is therefore a dry run unless
// --register says otherwise, and --register is refused when a newer build of a different commit is
// registered, unless --older says that is the point.
//
// **And refused while anything newer is outstanding**, --older or not: an ask of the module in the
// queue, or an open plan holding it unbuilt. Asked now, the replay would be newer than those asks,
// and their builds — made from newer source — would be recorded and never registered (issue 219
// orders by ask), the plan waiting on them sending the older commit instead.
func replayRefusal(b inventory.Build, module string, history []inventory.Build, register, older bool,
outstanding []string) error {
if b.Commit == "" {
return fmt.Errorf("%s recorded no commit — it failed before it knew what it was building, so there is "+
"nothing to replay; `rebuild %s` asks its repository, path and ref again", b.ID, b.ID)
}
if older && !register {
return errors.New("--older only says what --register may do; a dry run registers nothing")
}
if register && len(outstanding) > 0 {
return fmt.Errorf("%s is asked and not yet answered — %s — and a registered replay asked now would "+
"replace what those build (novox/hq issue 219). Wait for them, or `replay %s` without --register to look",
orNone(module), strings.Join(outstanding, "; "), b.ID)
}
if !register || older {
return nil
}
for _, h := range history {
if h.ID == b.ID || !h.Worked() || h.Commit == b.Commit {
continue
}
if h.AskedOrAt().After(b.AskedOrAt()) {
return fmt.Errorf("a newer build of %s is registered — %s, from %s — and registering a replay of %s "+
"would roll that older commit out as %s's current version (novox/hq issue 207). `replay %s` "+
"without --register looks at it; --register --older registers it anyway",
module, h.ID, short(h.Commit), short(b.Commit), module, b.ID)
}
}
return nil
}
// replaySaid is what a replay prints once asked: what it builds, and what becomes of the outcome.
func replaySaid(b inventory.Build, module, id string, register bool) string {
what := b.Repository
if module != "" {
what = module
}
said := fmt.Sprintf("replaying %s (%s) at %s as %s", b.ID, what, short(b.Commit), id)
if !register {
return said + "\n a dry run: the outcome is looked at and not taken in — nothing is recorded or " +
"registered, and nothing is sent. `builds --log " + id + "` follows it; `--register` registers it"
}
return said + "\n registered when it is built, as the module's current version — newer than every build " +
"asked before now — and rolled out as its policy says. `builds --log " + id + "` follows it"
}
// killCommand finds the machine running a build and asks its holder to end it.
func killCommand(ctx context.Context, args []string) error {
if len(args) != 1 {
return errors.New("kill <build id>")
}
id := args[0]
js, err := dialTheBus()
if err != nil {
return err
}
defer js.Close()
seat := buildSeatHeld(ctx)
since := time.Now().Add(-7 * 24 * time.Hour)
if at, ok := link.BuildAskedAt(id); ok {
since = at.Add(-time.Minute)
}
heard, err := link.ReadBuildEvents(ctx, js, seat, since)
if err != nil {
return err
}
started, ok := heard.Started[id]
if !ok {
return fmt.Errorf("no machine has said it started %s: if it waits in the queue, `cancel %s` drops it", id, id)
}
if heard.Outcomes[id] {
return fmt.Errorf("%s has already ended on %s — `builds` says how", id, started.On)
}
answer, err := link.AskSeatTool(ctx, js.Conn(), seat, "kill", started.On, map[string]string{"id": id}, killAnswer)
if err != nil {
return err
}
return printHolderAnswer(started.On, answer)
}
// pauseCommand asks one machine's holder, or every holder's, to pause or resume.
func pauseCommand(ctx context.Context, verb string, args []string) error {
if len(args) > 1 {
return fmt.Errorf("%s [node]", verb)
}
js, err := dialTheBus()
if err != nil {
return err
}
defer js.Close()
seat := buildSeatHeld(ctx)
nodes := args
if len(nodes) == 0 {
if nodes, err = buildSeatHolders(ctx, seat); err != nil {
return err
}
if len(nodes) == 0 {
return fmt.Errorf("nothing holds %s, so there is nothing to %s", seat, verb)
}
}
failed := 0
for _, node := range nodes {
answer, err := link.AskSeatTool(ctx, js.Conn(), seat, verb, node, map[string]string{}, holderAsks)
if err != nil {
fmt.Printf("%s: %v\n", node, err)
failed++
continue
}
if err := printHolderAnswer(node, answer); err != nil {
failed++
}
}
if failed > 0 {
return fmt.Errorf("%d of %d machine(s) did not %s", failed, len(nodes), verb)
}
return nil
}
// printHolderAnswer prints what a holder said, or its refusal as the command's failure.
func printHolderAnswer(node string, answer link.Answer) error {
if answer.Error != "" {
fmt.Printf("%s: %s\n", node, answer.Error)
return errors.New(answer.Error)
}
var said struct {
Said string `json:"said"`
}
if json.Unmarshal(answer.Result, &said) == nil && said.Said != "" {
fmt.Printf("%s: %s\n", node, said.Said)
return nil
}
fmt.Printf("%s: %s\n", node, string(answer.Result))
return nil
}
// buildSeatHolders is every machine an assigned module holding the build seat runs on.
func buildSeatHolders(ctx context.Context, seat string) ([]string, error) {
open, err := openStores(ctx)
if err != nil {
return nil, err
}
defer open.Close()
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return nil, err
}
return holdersAmong(entries, seat), nil
}
// holdersAmong is the machines of every catalogued module claiming the seat, sorted, once each.
func holdersAmong(entries []inventory.Entry, seat string) []string {
seen := map[string]bool{}
var out []string
for _, e := range entries {
if !e.Manifest.ClaimsSeat(seat) {
continue
}
for _, n := range e.On {
if !seen[n] {
seen[n] = true
out = append(out, n)
}
}
}
sort.Strings(out)
return out
}
// outstandingFor is everything asked of a module and not yet answered: its asks in the build queue,
// by repository and path, and every open plan holding it not yet built.
func outstandingFor(module, repository, path string, q link.Queue, plans []inventory.Plan) []string {
var out []string
for _, a := range q.Asks {
if repositoryMatches(a.Repository, repository) && a.Path == path {
out = append(out, fmt.Sprintf("%s %s in the queue", a.ID, a.State))
}
}
if module == "" {
return out
}
for _, p := range plans {
if !p.Open() {
continue
}
for _, tier := range p.Tiers {
for _, m := range tier {
if m == module {
if st := p.Modules[m]; st == nil || st.State != "built" {
out = append(out, fmt.Sprintf("%s holds it not yet built", p.ID))
}
}
}
}
}
return out
}
+772
View File
@@ -0,0 +1,772 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"reflect"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The build queue, controlled by hand (novox/hq ADR 0219), and the plans that follow it.
//
// docker run -d --rm --name bq-nats -p 14294:4222 nats:2.10-alpine -js
// make postgres PG_PORT=55566 PG_CONTAINER=bq-pg
// MESH_TEST_NATS=nats://127.0.0.1:14294 \
// MESH_TEST_POSTGRES='postgres://postgres:check@127.0.0.1:55566/postgres?sslmode=disable' \
// go test ./cmd/mesh-controller/ -run 'Queue|Cancel|Clear|Retry|Rebuild|Replay|Paused'
// asksRecorded makes every ask a plan makes return the next id in a row, and says which were asked.
func asksRecorded(t *testing.T) *[]string {
t.Helper()
var asked []string
was := askABuild
askABuild = func(_ context.Context, source buildSource, path, ref string) (string, error) {
id := link.NewBuildID(time.Now().Add(time.Duration(len(asked)) * time.Millisecond))
asked = append(asked, source.Repository+"#"+id)
return id, nil
}
t.Cleanup(func() { askABuild = was })
return &asked
}
// twoTiers registers two modules, b standing on a, each with a source a plan asks.
func twoTiers(t *testing.T, open *stores) {
t.Helper()
for _, name := range []string{"a", "b"} {
if err := open.inventory.RegisterModule(t.Context(), catalogue.Manifest{Module: name, Version: "1"},
inventory.Source{Repository: "novox/" + name, Seat: "git", Ref: "main", BuiltFrom: "c0ffee", Head: "c0ffee"}); err != nil {
t.Fatal(err)
}
}
}
// A failed plan is retried: its failed module asked again under a new id, the plan building at that
// tier, and when that build comes in the plan goes on and asks its next tier.
func TestAFailedPlanIsRetriedAndGoesOnThroughItsLaterTiers(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asked := asksRecorded(t)
twoTiers(t, open)
before := time.Now().UTC().Add(-time.Hour)
failed := inventory.Plan{ID: "plan-retry", Repository: "novox/a", Branch: "main", Commit: "c0ffee",
Created: before, State: inventory.PlanFailed, Tier: 0, Tiers: [][]string{{"a"}, {"b"}},
Note: "a failed to build in tier 0",
Modules: map[string]*inventory.PlanModule{"a": {State: "failed", AskedAt: &before, Build: "build-1",
Why: link.KilledByHand}}}
if err := open.inventory.SavePlan(ctx, failed); err != nil {
t.Fatal(err)
}
said, err := retryPlan(ctx, open, failed.ID)
if err != nil {
t.Fatal(err)
}
p, err := open.inventory.PlanByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
a := p.Modules["a"]
if p.State != inventory.PlanBuilding || a.State != "asked" || a.Build == "" || a.Build == "build-1" || a.Why != "" {
t.Fatalf("after retry the plan is %s and a is %+v", p.State, a)
}
if !strings.Contains(said, a.Build) {
t.Errorf("retry does not say the new id: %q", said)
}
if len(*asked) != 1 {
t.Fatalf("asked %v", *asked)
}
// The killed build's own late outcome is not this ask's; the new one's is, and tier 1 follows.
planBuilt(ctx, open, "a", "c0ffee", link.KilledByHand, before, "build-1")
if p, _ = open.inventory.PlanByID(ctx, failed.ID); p.State != inventory.PlanBuilding {
t.Fatalf("the old ask's outcome failed the retried plan: %s %q", p.State, p.Note)
}
asking, _ := link.BuildAskedAt(a.Build)
planBuilt(ctx, open, "a", "c0ffee", "", asking, a.Build)
p, err = open.inventory.PlanByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
if p.Tier != 1 || p.Modules["b"] == nil || p.Modules["b"].State != "asked" || p.Modules["b"].Build == "" {
t.Fatalf("the retried plan did not go on to tier 1: tier %d, %s, b %+v", p.Tier, p.State, p.Modules["b"])
}
if len(*asked) != 2 {
t.Fatalf("asked %v", *asked)
}
}
// What retry refuses, and says why.
func TestRetryRefusesWhatItCannotResume(t *testing.T) {
at := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
plan := func(id, state string, created time.Time) inventory.Plan {
return inventory.Plan{ID: id, Repository: "novox/mesh-catalog", Branch: "main", Commit: id + "c0ffee",
Created: created, State: state, Tiers: [][]string{{"a"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "failed"}}}
}
failed := plan("plan-1", inventory.PlanFailed, at)
for _, c := range []struct {
p inventory.Plan
others []inventory.Plan
says string
}{
{plan("plan-d", inventory.PlanDone, at), nil, "is done"},
{plan("plan-s", inventory.PlanSuperseded, at), nil, "was"},
{plan("plan-o", inventory.PlanBuilding, at), nil, "still building"},
{failed, []inventory.Plan{plan("plan-2", inventory.PlanRolling, at.Add(time.Hour))}, "plan-2 supersedes it"},
} {
err := retryRefusal(c.p, c.others)
if err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: %v, wanted it to say %q", c.p.ID, err, c.says)
}
}
stopped := failed
stopped.Modules = map[string]*inventory.PlanModule{"a": {State: "built"}}
stopped.Note = "a stopped at its first machine"
if err := retryRefusal(stopped, nil); err == nil || !strings.Contains(err.Error(), "nothing in tier 0") {
t.Errorf("a plan with nothing failed to build was retried: %v", err)
}
// Another branch's newer plan, an older one, and a failed one do not supersede it.
other := plan("plan-3", inventory.PlanBuilding, at.Add(time.Hour))
other.Branch = "release"
if err := retryRefusal(failed, []inventory.Plan{other, plan("plan-0", inventory.PlanBuilding, at.Add(-time.Hour)),
plan("plan-4", inventory.PlanFailed, at.Add(time.Hour))}); err != nil {
t.Errorf("refused for a plan that does not supersede it: %v", err)
}
}
// `rebuild` of a module a failed plan holds joins that plan; one held by nothing runs alone.
func TestARebuildJoinsThePlanHoldingTheModule(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asked := asksRecorded(t)
twoTiers(t, open)
before := time.Now().UTC().Add(-time.Hour)
failed := inventory.Plan{ID: "plan-join", Repository: "novox/a", Branch: "main", Commit: "c0ffee",
Created: before, State: inventory.PlanFailed, Tiers: [][]string{{"a"}, {"b"}},
Note: "a failed to build in tier 0",
Modules: map[string]*inventory.PlanModule{"a": {State: "failed", AskedAt: &before, Build: "build-1", Why: link.CancelledByHand}}}
if err := open.inventory.SavePlan(ctx, failed); err != nil {
t.Fatal(err)
}
if err := rebuildCommand(ctx, []string{"a"}); err != nil {
t.Fatal(err)
}
p, err := open.inventory.PlanByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
if p.State != inventory.PlanBuilding || p.Modules["a"].State != "asked" || p.Modules["a"].Build == "build-1" {
t.Fatalf("the rebuild did not join the failed plan: %s %+v", p.State, p.Modules["a"])
}
if len(*asked) != 1 {
t.Fatalf("asked %v", *asked)
}
// b is in a tier not yet reached: nothing holds it in its current tier, so it is asked alone.
if err := rebuildCommand(ctx, []string{"b"}); err != nil {
t.Fatal(err)
}
if p, _ = open.inventory.PlanByID(ctx, failed.ID); p.Modules["b"] != nil {
t.Fatalf("a module the plan has not reached joined it: %+v", p.Modules["b"])
}
if len(*asked) != 2 {
t.Fatalf("asked %v", *asked)
}
}
// A failed plan an open plan of its repository supersedes is not joined: the open one holds the module.
func TestARebuildJoinsTheOpenPlanBeforeASupersededFailedOne(t *testing.T) {
at := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
failed := inventory.Plan{ID: "plan-old", Repository: "novox/a", Branch: "main", Created: at, State: inventory.PlanFailed,
Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{"a": {State: "failed"}}}
newer := inventory.Plan{ID: "plan-new", Repository: "novox/a", Branch: "main", Created: at.Add(time.Hour),
State: inventory.PlanBuilding, Tiers: [][]string{{"x"}, {"a"}}, Modules: map[string]*inventory.PlanModule{}}
if _, found := planHolding("a", []inventory.Plan{newer}, []inventory.Plan{newer, failed}); found {
t.Fatal("joined a failed plan a newer open one supersedes")
}
if p, found := planHolding("a", nil, []inventory.Plan{failed}); !found || p.ID != "plan-old" {
t.Fatalf("did not join the failed plan holding it: %v %s", found, p.ID)
}
built := failed
built.Modules = map[string]*inventory.PlanModule{"a": {State: "built"}}
if _, found := planHolding("a", nil, []inventory.Plan{built}); found {
t.Fatal("joined a plan that has the module built")
}
}
// replay is a dry run unless registered, and registering an older commit than one registered since
// is refused unless --older says it is meant (novox/hq issue 207).
func TestReplayRefusesToRollAnOlderCommitOutUnlessToldTo(t *testing.T) {
asked := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
old := inventory.Build{ID: "build-old", Module: "a", Commit: "0ldc0mm1t", Asked: asked}
newer := inventory.Build{ID: "build-new", Module: "a", Commit: "n3wc0mm1t", Asked: asked.Add(time.Hour)}
history := []inventory.Build{newer, old}
if err := replayRefusal(old, "a", history, false, false, nil); err != nil {
t.Errorf("a dry run was refused: %v", err)
}
err := replayRefusal(old, "a", history, true, false, nil)
if err == nil || !strings.Contains(err.Error(), "build-new") || !strings.Contains(err.Error(), "--older") {
t.Errorf("registering an older commit than the one registered was not refused: %v", err)
}
if err := replayRefusal(old, "a", history, true, true, nil); err != nil {
t.Errorf("--register --older was refused: %v", err)
}
if err := replayRefusal(newer, "a", history, true, false, nil); err != nil {
t.Errorf("registering the newest build again was refused: %v", err)
}
// A newer build of the same commit, or one that failed, is not a newer version to roll back from.
same := inventory.Build{ID: "build-same", Module: "a", Commit: old.Commit, Asked: asked.Add(2 * time.Hour)}
broken := inventory.Build{ID: "build-broken", Module: "a", Failed: "no", Asked: asked.Add(3 * time.Hour)}
if err := replayRefusal(old, "a", []inventory.Build{broken, same, old}, true, false, nil); err != nil {
t.Errorf("refused for a newer build of the same commit or a failed one: %v", err)
}
if err := replayRefusal(inventory.Build{ID: "build-x", Failed: "clone"}, "", nil, false, false, nil); err == nil {
t.Error("a build that recorded no commit was replayed")
}
if err := replayRefusal(old, "a", history, false, true, nil); err == nil {
t.Error("--older without --register was taken")
}
// **Nothing newer outstanding**, --older or not: an ask of the module in the queue, or an open plan
// holding it unbuilt, would be replaced by a replay asked now (issue 219).
q := link.Queue{Asks: []link.QueuedAsk{
{ID: "build-queued", Repository: "https://forge.example/novox/a.git", State: link.AskWaiting},
{ID: "build-elsewhere", Repository: "https://forge.example/novox/b.git", State: link.AskWaiting},
{ID: "build-other-path", Repository: "https://forge.example/novox/a.git", Path: "sub", State: link.AskWaiting},
}}
plans := []inventory.Plan{
{ID: "plan-holds", State: inventory.PlanBuilding, Tiers: [][]string{{"x"}, {"a"}}, Modules: map[string]*inventory.PlanModule{}},
{ID: "plan-built", State: inventory.PlanRolling, Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{"a": {State: "built"}}},
{ID: "plan-failed", State: inventory.PlanFailed, Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{}},
}
outstanding := outstandingFor("a", "novox/a", "", q, plans)
if len(outstanding) != 2 || !strings.Contains(outstanding[0], "build-queued") || !strings.Contains(outstanding[1], "plan-holds") {
t.Fatalf("outstanding: %v", outstanding)
}
if err := replayRefusal(old, "a", history, true, true, outstanding); err == nil || !strings.Contains(err.Error(), "build-queued") {
t.Errorf("registered over an outstanding ask: %v", err)
}
if err := replayRefusal(old, "a", history, false, false, outstanding); err != nil {
t.Errorf("a dry run was refused for what is outstanding: %v", err)
}
if said := replaySaid(old, "a", "build-1", false); !strings.Contains(said, "dry run") || !strings.Contains(said, "--register") {
t.Errorf("a dry replay does not say what it is: %q", said)
}
if said := replaySaid(old, "a", "build-1", true); !strings.Contains(said, "registered") || !strings.Contains(said, "rolled out") {
t.Errorf("a registered replay does not say what it does: %q", said)
}
}
// A plan waiting on builds of a seat whose every holder is paused says so and is not late; a seat
// paused on some holders only is not a reason the plan is waiting.
func TestAPlanWaitingOnAPausedSeatSaysSoAndIsNotLate(t *testing.T) {
now := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
asked := now.Add(-12 * time.Minute)
p := inventory.Plan{ID: "plan-p", Repository: "novox/a", Commit: "c0ffee", State: inventory.PlanBuilding,
Updated: now.Add(-2 * time.Hour), Tiers: [][]string{{"a"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &asked}}}
all := pauseOf([]string{"g14", "ace"}, map[string]link.HolderState{"ace": {Paused: true}, "g14": {Paused: true}})
line := planLineWith(p, now, all)
if !strings.Contains(line, "waiting: the build seat is paused on ace, g14 (asked 12m0s ago)") || strings.Contains(line, "LATE") {
t.Errorf("a plan on a paused seat reads %q", line)
}
if st := planStatuses([]inventory.Plan{p}, now, all)[0]; st.Late || !strings.Contains(st.Waiting, "paused on ace, g14") {
t.Errorf("status --json says %+v", st)
}
if _, late := openPlans([]inventory.Plan{p}, all); late != 0 {
t.Errorf("a plan waiting on a paused seat counted late")
}
longAgo := now.Add(-25 * time.Hour)
forgotten := p
forgotten.Modules = map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &longAgo}}
if line := planLineWith(forgotten, now, all); !strings.Contains(line, "PAUSED OVER A DAY") || strings.Contains(line, "LATE") {
t.Errorf("a plan paused over a day reads %q", line)
}
some := pauseOf([]string{"g14", "ace"}, map[string]link.HolderState{"ace": {Paused: true}})
if some.All || !reflect.DeepEqual(some.Nodes, []string{"ace"}) {
t.Fatalf("%+v", some)
}
if line := planLineWith(p, now, some); !strings.Contains(line, "LATE") {
t.Errorf("a seat paused on one holder of two made the plan not late: %q", line)
}
if st := planStatuses([]inventory.Plan{p}, now, some)[0]; !st.Late {
t.Errorf("status --json: %+v", st)
}
// A plan rolling out, or with nothing asked, is not waiting on the seat.
rolling := p
rolling.State = inventory.PlanRolling
if _, paused := pausedWaiting(rolling, all, now); paused {
t.Error("a rolling plan reads as waiting on the build seat")
}
}
// The queue's verbs are the controller seat's, each to the command it names.
func TestTheQueueVerbsRunTheirCommands(t *testing.T) {
for _, c := range []struct {
verb string
args map[string]any
want []string
}{
{"queue", nil, []string{"queue"}},
{"cancel", map[string]any{"id": "build-1"}, []string{"cancel", "build-1"}},
{"clear", nil, []string{"clear"}},
{"clear", map[string]any{"dead": "true"}, []string{"clear", "--dead"}},
{"rebuild", map[string]any{"what": "gitea"}, []string{"rebuild", "gitea"}},
{"replay", map[string]any{"id": "build-1"}, []string{"replay", "build-1"}},
{"replay", map[string]any{"id": "build-1", "register": "true", "older": "true"}, []string{"replay", "build-1", "--register", "--older"}},
{"kill", map[string]any{"id": "build-1"}, []string{"kill", "build-1"}},
{"pause", nil, []string{"pause"}},
{"resume", map[string]any{"node": "ace"}, []string{"resume", "ace"}},
{"plans", map[string]any{"retry": "plan-1"}, []string{"plans", "retry", "plan-1"}},
} {
got, err := argvFor(c.verb, c.args)
if err != nil || !reflect.DeepEqual(got, c.want) {
t.Errorf("%s %v: %v %v, want %v", c.verb, c.args, got, err, c.want)
}
}
if _, err := argvFor("cancel", nil); err == nil {
t.Error("cancel without an id was taken")
}
declared := map[string]bool{}
for _, v := range catalogue.ControllerVerbs {
declared[v.Name] = true
}
for _, v := range []string{"queue", "cancel", "clear", "rebuild", "replay", "kill", "pause", "resume"} {
if !declared[v] {
t.Errorf("%s is not a verb of the controller seat", v)
}
}
}
// --- against a real bus -----------------------------------------------------------------------
// aBuildQueue is the build seat's queue, worker and cancelled set on a real server, the controller
// pointed at it, and a function that asks the seat one build.
func aBuildQueue(t *testing.T) (*broker.JetStream, func(id, repository string) link.BuildRequest) {
t.Helper()
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
t.Setenv(broker.NATSVar, url)
js, err := broker.Dial(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
seat := broker.DeclaredSeat{Name: link.TheBuildMachine, Accepts: []string{"build"},
Emits: []string{"started", "built", "log.*", "paused.*"}}
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
_ = js.Context().DeleteStream("SEAT_NODE_BUILD_AGENT")
if err := broker.RaiseSeats(js, []broker.DeclaredSeat{seat}, map[string]broker.Holder{
link.TheBuildMachine: {Node: "anchor", Module: "build-agent"}}); err != nil {
t.Fatal(err)
}
if err := broker.RaiseCancelledSets(js, []broker.DeclaredSeat{seat}); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_ = js.Context().DeleteStream("SEAT_NODE_BUILD_AGENT")
_ = js.Context().DeleteKeyValue(broker.CancelledSetName(link.TheBuildMachine))
_ = js.Context().PurgeStream(broker.EventsStream)
})
ask := func(id, repository string) link.BuildRequest {
r := link.BuildRequest{ID: id, Repository: repository, Held: map[string]string{"x/y": "secret-ish"}}
body, _ := json.Marshal(r)
if _, err := js.Context().Publish(link.BuildWork(), body); err != nil {
t.Fatal(err)
}
return r
}
return js, ask
}
// deadOne takes the oldest ask from the worker and hands it back as often as the worker allows.
func deadOne(t *testing.T, js *broker.JetStream) {
t.Helper()
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: link.TheBuildMachine, Accepts: []string{"build"}})
sub, err := js.Context().PullSubscribe(worker.Filters[0], worker.Name, nats.Bind(worker.Stream, worker.Name), nats.ManualAck())
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
for i := 0; i < worker.MaxDeliver; i++ {
msgs, err := sub.Fetch(1, nats.MaxWait(3*time.Second))
if err != nil {
t.Fatalf("delivery %d: %v", i+1, err)
}
_ = msgs[0].Nak()
}
// One more pull, as a holder always has one waiting: the server finds the ask past its deliveries
// then, and stops counting it pending.
if msgs, _ := sub.Fetch(1, nats.MaxWait(time.Second)); len(msgs) > 0 {
t.Fatalf("an ask past its deliveries was delivered again")
}
}
// cancel drops a waiting ask from the bus and records it failed, cancelled by hand — and the plan
// that asked for it, matched by the id, fails with it; an ask the plan's records name no module for
// is still found.
func TestCancelDeletesTheAskAndFailsThePlanThatAskedIt(t *testing.T) {
js, ask := aBuildQueue(t)
open := aMesh(t)
ctx := t.Context()
twoTiers(t, open)
id := link.NewBuildID(time.Now())
ask(id, "https://forge.example/novox/a.git")
asked, _ := link.BuildAskedAt(id)
plan := inventory.Plan{ID: "plan-cancel", Repository: "novox/a", Commit: "c0ffee", Created: asked,
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &asked, Build: id}}}
if err := open.inventory.SavePlan(ctx, plan); err != nil {
t.Fatal(err)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
if len(q.Asks) != 1 || q.Asks[0].State != link.AskWaiting || q.Asks[0].ID != id {
t.Fatalf("the queue reads %+v", q)
}
if text := queueText(q, time.Now()); !strings.Contains(text, "1 waiting, 0 in flight, 0 dead") ||
strings.Contains(text, "secret-ish") {
t.Errorf("the queue says:\n%s", text)
}
if err := cancelCommand(ctx, []string{id}); err != nil {
t.Fatal(err)
}
if q, _ = link.ReadQueue(ctx, js, link.TheBuildMachine); len(q.Asks) != 0 {
t.Fatalf("the ask is still queued: %+v", q.Asks)
}
if cancelled, err := link.IsCancelled(js.Conn(), link.TheBuildMachine, id); err != nil || !cancelled {
t.Errorf("the cancelled set does not hold it: %v %v", cancelled, err)
}
b, found, err := open.inventory.BuildByID(ctx, id)
if err != nil || !found || b.Failed != link.CancelledByHand {
t.Fatalf("the cancel is recorded as %+v (%v %v)", b, found, err)
}
p, err := open.inventory.PlanByID(ctx, plan.ID)
if err != nil {
t.Fatal(err)
}
if p.State != inventory.PlanFailed || p.Modules["a"].State != "failed" || p.Modules["a"].Why != link.CancelledByHand {
t.Fatalf("the plan that asked is %s, a %+v", p.State, p.Modules["a"])
}
if err := cancelCommand(ctx, []string{id}); err == nil {
t.Error("an ask cancelled already was cancelled again")
}
}
// clear cancels every waiting ask and leaves the dead ones unless told, and never one in flight.
func TestClearCancelsTheWaitingAndTheDeadOnlyWhenTold(t *testing.T) {
js, ask := aBuildQueue(t)
open := aMesh(t)
ctx := t.Context()
start := time.Now()
dead := link.NewBuildID(start)
ask(dead, "https://forge.example/novox/dead.git")
deadOne(t, js)
var waiting []string
for i := 1; i <= 2; i++ {
id := link.NewBuildID(start.Add(time.Duration(i) * time.Millisecond))
waiting = append(waiting, id)
ask(id, fmt.Sprintf("https://forge.example/novox/w%d.git", i))
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
if len(q.Of(link.AskDead)) != 1 || len(q.Of(link.AskWaiting)) != 2 || q.MaxDeliver != 5 {
t.Fatalf("the queue reads %+v", q)
}
said, err := clearQueue(ctx, js, open, link.TheBuildMachine, false)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(said, "2 waiting ask(s) cancelled") || !strings.Contains(said, "1 dead, left") {
t.Errorf("clear said:\n%s", said)
}
q, _ = link.ReadQueue(ctx, js, link.TheBuildMachine)
if len(q.Asks) != 1 || q.Asks[0].ID != dead || q.Asks[0].State != link.AskDead {
t.Fatalf("after clear the queue is %+v", q.Asks)
}
if _, err := clearQueue(ctx, js, open, link.TheBuildMachine, true); err != nil {
t.Fatal(err)
}
if q, _ = link.ReadQueue(ctx, js, link.TheBuildMachine); len(q.Asks) != 0 {
t.Fatalf("clear --dead left %+v", q.Asks)
}
for _, id := range append(waiting, dead) {
if b, found, _ := open.inventory.BuildByID(ctx, id); !found || b.Failed != link.CancelledByHand {
t.Errorf("%s is recorded as %+v", id, b)
}
}
}
// An ask in flight is not cancelled: kill ends it where it runs.
func TestCancelRefusesAnAskInFlight(t *testing.T) {
js, ask := aBuildQueue(t)
open := aMesh(t)
ctx := t.Context()
id := link.NewBuildID(time.Now())
ask(id, "https://forge.example/novox/a.git")
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: link.TheBuildMachine, Accepts: []string{"build"}})
sub, err := js.Context().PullSubscribe(worker.Filters[0], worker.Name, nats.Bind(worker.Stream, worker.Name), nats.ManualAck())
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
if _, err := sub.Fetch(1, nats.MaxWait(3*time.Second)); err != nil {
t.Fatal(err)
}
started, _ := json.Marshal(link.BuildStart{ID: id, On: "ace", At: time.Now().UTC().Format(time.RFC3339Nano)})
if _, err := js.Context().Publish(link.BuildStarted(), started); err != nil {
t.Fatal(err)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
a, _ := q.Find(id)
if a.State != link.AskInFlight || a.On != "ace" {
t.Fatalf("the taken ask reads %+v", a)
}
_, err = cancelAsk(ctx, js, open, link.TheBuildMachine, a)
if err == nil || !strings.Contains(err.Error(), "kill "+id) {
t.Fatalf("an ask in flight was cancelled: %v", err)
}
if _, found, _ := open.inventory.BuildByID(ctx, id); found {
t.Error("a refused cancel recorded an outcome")
}
}
// kill finds the machine from the build's start and asks that machine's holder; pause asks the
// machine named. Each prints what the holder answered, and a refusal is the command's failure.
func TestKillAndPauseAskTheHolderOnTheMachine(t *testing.T) {
js, _ := aBuildQueue(t)
ctx := t.Context()
asked := map[string]string{}
handlers := map[string]link.ToolHandler{
"kill": func(_ context.Context, raw json.RawMessage) (any, error) {
var args struct{ ID string }
_ = json.Unmarshal(raw, &args)
asked["kill"] = args.ID
if args.ID != "build-running" {
return nil, fmt.Errorf("ace is not building %s", args.ID)
}
return map[string]any{"said": "killed " + args.ID}, nil
},
"pause": func(context.Context, json.RawMessage) (any, error) {
asked["pause"] = "ace"
return map[string]any{"said": "ace is paused"}, nil
},
}
stop, err := link.OverNATS{Conn: js.Conn()}.ServeNodeSeatTools(link.TheBuildMachine, "ace", handlers, nil)
if err != nil {
t.Fatal(err)
}
defer stop()
for _, id := range []string{"build-running", "build-other"} {
started, _ := json.Marshal(link.BuildStart{ID: id, On: "ace", At: time.Now().UTC().Format(time.RFC3339Nano)})
if _, err := js.Context().Publish(link.BuildStarted(), started); err != nil {
t.Fatal(err)
}
}
if err := killCommand(ctx, []string{"build-running"}); err != nil {
t.Fatal(err)
}
if asked["kill"] != "build-running" {
t.Fatalf("the holder was asked %v", asked)
}
if err := killCommand(ctx, []string{"build-other"}); err == nil {
t.Error("the holder's refusal was not the command's")
}
if err := killCommand(ctx, []string{"build-never"}); err == nil || !strings.Contains(err.Error(), "cancel build-never") {
t.Errorf("a build nobody started: %v", err)
}
if err := pauseCommand(ctx, "pause", []string{"ace"}); err != nil || asked["pause"] != "ace" {
t.Fatalf("pause: %v %v", err, asked)
}
if err := pauseCommand(ctx, "resume", []string{"g14"}); err == nil {
t.Error("a machine nothing answers on was resumed")
}
}
// A plan that stopped at its first machine is retried: the module sent to that machine again, the
// send recorded as the first anew, and the plan goes on — unless a newer plan holds the module.
func TestAPlanStoppedAtItsFirstMachineIsRetried(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asked := asksRecorded(t)
twoTiers(t, open)
var sentTo [][]string
was := sendRollout
sendRollout = func(_ context.Context, _ *stores, names []string) ([]string, error) {
sentTo = append(sentTo, names)
return names, nil
}
t.Cleanup(func() { sendRollout = was })
long := time.Now().UTC().Add(-2 * time.Hour)
stopped := inventory.Plan{ID: "plan-rollout", Repository: "novox/a", Branch: "main", Commit: "c0ffee",
Created: long, State: inventory.PlanFailed, Tiers: [][]string{{"a"}, {"b"}},
Note: "a stopped at its first machine in tier 0: laptop refused what it was sent",
Modules: map[string]*inventory.PlanModule{"a": {State: "built", BuiltAt: &long, Commit: "c0ffee",
First: []string{"laptop"}, FirstAt: &long, Why: "laptop refused what it was sent"}}}
if err := open.inventory.SavePlan(ctx, stopped); err != nil {
t.Fatal(err)
}
// A newer plan holding a refuses it: sending the older build would put it back.
newer := inventory.Plan{ID: "plan-newer", Repository: "novox/other", Commit: "d00d", Created: long.Add(time.Hour),
State: inventory.PlanDone, Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{}}
if err := open.inventory.SavePlan(ctx, newer); err != nil {
t.Fatal(err)
}
if _, err := retryPlan(ctx, open, stopped.ID); err == nil || !strings.Contains(err.Error(), "plan-newer") {
t.Fatalf("retried under a newer plan: %v", err)
}
newer.State = inventory.PlanSuperseded
if err := open.inventory.SavePlan(ctx, newer); err != nil {
t.Fatal(err)
}
said, err := retryPlan(ctx, open, stopped.ID)
if err != nil {
t.Fatal(err)
}
if len(sentTo) != 1 || !reflect.DeepEqual(sentTo[0], []string{"laptop"}) || !strings.Contains(said, "laptop") {
t.Fatalf("sent %v; said %q", sentTo, said)
}
p, err := open.inventory.PlanByID(ctx, stopped.ID)
if err != nil {
t.Fatal(err)
}
a := p.Modules["a"]
if p.State != inventory.PlanRolling || a.FirstAt == nil || !a.FirstAt.After(long) || a.Why != "" {
t.Fatalf("after retry the plan is %s, a %+v", p.State, a)
}
// And it goes on: a records (its policy sends nothing more), so the next tier is asked.
advancePlans(ctx, open)
if p, _ = open.inventory.PlanByID(ctx, stopped.ID); p.Tier != 1 || p.Modules["b"] == nil || p.Modules["b"].State != "asked" {
t.Fatalf("the retried plan did not go on: tier %d %s %+v", p.Tier, p.State, p.Modules["b"])
}
if len(*asked) != 1 {
t.Fatalf("asked %v", *asked)
}
}
// A plan module asked under an id is settled by that id's outcome alone: a replay or a rebuild beside
// the plan, asked later, never answers it (novox/hq ADR 0219).
func TestAPlanIsAnsweredOnlyByTheBuildItAskedFor(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asked := time.Now().UTC().Add(-time.Minute)
plan := inventory.Plan{ID: "plan-own", Repository: "novox/a", Commit: "c0ffee", Created: asked,
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &asked, Build: "build-own"}}}
if err := open.inventory.SavePlan(ctx, plan); err != nil {
t.Fatal(err)
}
planBuilt(ctx, open, "a", "0ldc0mm1t", "", time.Now().UTC(), "build-replay")
p, _ := open.inventory.PlanByID(ctx, plan.ID)
if p.Modules["a"].State != "asked" {
t.Fatalf("a replay asked after the plan settled it: %+v", p.Modules["a"])
}
// From the records too: a later build of the module recorded is not the plan's.
recorded := map[string][]inventory.Build{"a": {{ID: "build-replay", Commit: "0ldc0mm1t", Asked: time.Now(), At: time.Now()}}}
if settleFromRecords(&p, p.Tiers[0], recorded, nil) {
t.Fatalf("the records settled it with another build: %+v", p.Modules["a"])
}
planBuilt(ctx, open, "a", "c0ffee", "", asked, "build-own")
if p, _ = open.inventory.PlanByID(ctx, plan.ID); p.Modules["a"].State != "built" || p.Modules["a"].Commit != "c0ffee" {
t.Fatalf("its own build did not settle it: %+v", p.Modules["a"])
}
}
// rebuild of a build made at a commit asks what the module follows now, never the commit.
func TestARebuildOfACommitAsksWhatTheModuleFollows(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asked := asksRecorded(t)
twoTiers(t, open)
var refs []string
was := askABuild
askABuild = func(c context.Context, source buildSource, path, ref string) (string, error) {
refs = append(refs, ref)
return was(c, source, path, ref)
}
if err := open.inventory.RecordBuild(ctx, inventory.Build{ID: "build-at-commit", Repository: "novox/a",
Ref: "0123456789abcdef0123456789abcdef01234567", Module: "a", Commit: "0123456789abcdef0123456789abcdef01234567"}); err != nil {
t.Fatal(err)
}
if err := rebuildCommand(ctx, []string{"build-at-commit"}); err != nil {
t.Fatal(err)
}
if len(refs) != 1 || refs[0] != "main" || len(*asked) != 1 {
t.Fatalf("asked %v at %v", *asked, refs)
}
}
// An ask the worker counts out that no machine said it started is cancelled only if no start comes
// while a holder looks: one that does withdraws the cancel, and kill is what ends it.
func TestCancelOfAnAskNobodySaidIsWithdrawnWhenItStarts(t *testing.T) {
js, ask := aBuildQueue(t)
open := aMesh(t)
ctx := t.Context()
was := holderLooks
holderLooks = 300 * time.Millisecond
t.Cleanup(func() { holderLooks = was })
id := link.NewBuildID(time.Now())
ask(id, "https://forge.example/novox/a.git")
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: link.TheBuildMachine, Accepts: []string{"build"}})
sub, err := js.Context().PullSubscribe(worker.Filters[0], worker.Name, nats.Bind(worker.Stream, worker.Name), nats.ManualAck())
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
if _, err := sub.Fetch(1, nats.MaxWait(3*time.Second)); err != nil {
t.Fatal(err)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
a, _ := q.Find(id)
if a.State != link.AskInFlight || a.On != "" {
t.Fatalf("the taken ask reads %+v", a)
}
// The holder that took it says it started, while the cancel waits.
go func() {
time.Sleep(100 * time.Millisecond)
started, _ := json.Marshal(link.BuildStart{ID: id, On: "ace", At: time.Now().UTC().Format(time.RFC3339Nano)})
_, _ = js.Context().Publish(link.BuildStarted(), started)
}()
if _, err := cancelAsk(ctx, js, open, link.TheBuildMachine, a); err == nil || !strings.Contains(err.Error(), "started on ace") {
t.Fatalf("a build that started was cancelled: %v", err)
}
if cancelled, _ := link.IsCancelled(js.Conn(), link.TheBuildMachine, id); cancelled {
t.Error("the withdrawn cancel is still marked")
}
if _, found, _ := open.inventory.BuildByID(ctx, id); found {
t.Error("a withdrawn cancel recorded an outcome")
}
}
+13 -1
View File
@@ -3,6 +3,7 @@ package main
import (
"encoding/json"
"fmt"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"sort"
"time"
@@ -73,6 +74,15 @@ type meshStatus struct {
// alone. A document without this called a machine well while a predecessor's chain refused
// what the mesh declared open.
Filtered []machineFiltered `json:"filtered,omitempty"`
// Unheld is every module on a machine whose resources are applied through a seat nothing on
// that machine holds, with the modules that could hold it (novox/hq ADR 0207). Absent when every
// dependency is met. Reported, not refused, until the switch.
Unheld []catalogue.Unheld `json:"unheld,omitempty"`
// Failing is every consumer a provider says it keeps failing, with the class of error, since
// when, and when it was last said (novox/hq ADR 0224). Absent when no provider says so. A
// document without this called the mesh well while the identity provider refused every consumer
// for a day (04-ISSUES/179).
Failing []inventory.ProviderStanding `json:"failing,omitempty"`
}
// machineFiltered is one rule set on a converged machine that the mesh did not write and that
@@ -171,7 +181,7 @@ func statusAsJSON(asked answers) ([]byte, error) {
out := meshStatus{Machines: len(nodes), Wrong: []machineDoing{},
Quiet: []machineQuiet{}, Behind: []moduleBehind{}, Waiting: []machineWaiting{},
Reported: []machineReported{}, Unresolved: []machineUnresolved{},
Network: asked.network, Adopted: adoptedNodes(nodes), Plans: planStatuses(asked.plans, time.Now())}
Network: asked.network, Adopted: adoptedNodes(nodes), Plans: planStatuses(asked.plans, time.Now(), asked.paused)}
// In a stated order, so two readings of an unchanged mesh are the same document.
untakenNodes := make([]string, 0, len(asked.untaken))
for name := range asked.untaken {
@@ -204,6 +214,8 @@ func statusAsJSON(asked answers) ([]byte, error) {
out.Filtered = append(out.Filtered, machineFiltered{Node: name, Where: x.Where, Owner: x.Owner, Refuses: x.Refuses})
}
}
out.Unheld = asked.unheld
out.Failing = asked.failing
for name := range asked.refused {
out.Unresolved = append(out.Unresolved, machineUnresolved{
Node: name, Problem: asked.refused[name]})
+433 -63
View File
@@ -184,6 +184,7 @@ func planOfMerge(m link.SourceMoved, moved []string, edges []inventory.Edge) inv
return inventory.Plan{
ID: fmt.Sprintf("plan-%d", time.Now().UnixNano()),
Repository: m.Owner + "/" + m.Repo,
Branch: m.Base,
Commit: m.Commit,
Created: time.Now().UTC(),
State: inventory.PlanBuilding,
@@ -192,6 +193,57 @@ func planOfMerge(m link.SourceMoved, moved []string, edges []inventory.Edge) inv
}
}
// supersededBy is what a newer plan takes over from the open plans it supersedes (novox/hq issue
// 254, ADR 0218): the modules they had not finished, and those plans closed as superseded.
//
// **A merge looked at no plan but its own.** Two merges of one repository a few minutes apart were
// two open plans asking for the same modules, each sending machines what it built; and a plan that
// would never move again — waiting on a report that could not come, at 97b1b2b — stayed open for
// ever beside the newer ones, read as work in progress by everyone who looked. The newer merge is the
// newer intent for that repository and branch, so its plan takes over: every open plan of the same
// repository and branch **created before it** — by the time the plans were made, never by comparing
// commits, which have no order of their own — gives up the modules it had not built, and those are
// planned again in the newer plan beside what the newer merge moved.
//
// "Not built" is a module not yet asked, or asked and not answered; **and a module built and not
// yet sent to its machines**, where its policy rolls it out: closed, the older plan would never send
// it, and the catalogue announces no move for a rebuild (issue 189), so the newer plan builds and
// sends it. A build the older plan asked still finishes and registers as any build does — ordered by
// when it was asked (issue 219), so the newer plan's ask, made later, is the one that stands.
//
// A plan with no branch recorded is from before branches were kept, and is superseded by the next
// plan of its repository: what it had not built is folded in, so nothing is lost by it.
func supersededBy(newer inventory.Plan, open []inventory.Plan, rollsOut func(string) bool) ([]string, []inventory.Plan) {
folded := map[string]bool{}
var closed []inventory.Plan
for _, old := range open {
if old.ID == newer.ID || !old.Open() || !strings.EqualFold(old.Repository, newer.Repository) ||
(old.Branch != "" && old.Branch != newer.Branch) || !old.Created.Before(newer.Created) {
continue
}
var took []string
for name, s := range old.Modules {
if s == nil || s.State != "built" || (s.SentAt == nil && rollsOut(name)) {
folded[name] = true
took = append(took, name)
}
}
sort.Strings(took)
old.State = inventory.PlanSuperseded
old.Note = fmt.Sprintf("superseded at tier %d by %s (%s at %s)", old.Tier, newer.ID, newer.Repository, short(newer.Commit))
if len(took) > 0 {
old.Note += "; " + strings.Join(took, ", ") + " planned there again"
}
closed = append(closed, old)
}
out := make([]string, 0, len(folded))
for name := range folded {
out = append(out, name)
}
sort.Strings(out)
return out, closed
}
// gates is what the next tier needs running from this one: a module of the tier that a later
// tier is built by — the runtime dependency — and whose policy rolls it out, must be applied by
// the machines running it before the next tier is asked. A base an image stands on need only be
@@ -230,6 +282,22 @@ func gates(p inventory.Plan, edges []inventory.Edge, rollsOut func(string) bool)
}
// applied says whether every machine running the module has reported since the module was built.
// appliedEach is applied with a moment of its own for each machine: the reports that count are the ones
// after that machine was sent the build (novox/hq issue 256).
func appliedEach(module string, since func(node string) time.Time, running []string, reports []inventory.Reported) (bool, []string) {
at := map[string]*time.Time{}
for _, r := range reports {
at[r.Node] = r.At
}
var waiting []string
for _, n := range running {
if t := at[n]; t == nil || t.Before(since(n)) {
waiting = append(waiting, n)
}
}
return len(waiting) == 0, waiting
}
func applied(module string, builtAt time.Time, running []string, reports []inventory.Reported) (bool, []string) {
at := map[string]*time.Time{}
for _, r := range reports {
@@ -256,37 +324,52 @@ func askTier(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan) e
for _, e := range entries {
byName[e.Manifest.Module] = e
}
now := time.Now().UTC()
for _, name := range p.Tiers[p.Tier] {
state := p.Modules[name]
if state == nil {
state = &inventory.PlanModule{}
p.Modules[name] = state
}
e, known := byName[name]
if !known {
state.State = "failed"
state.Why = "no longer in the catalogue"
p.State = inventory.PlanFailed
p.Note = name + " is no longer in the catalogue"
continue
}
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
fmt.Printf(" tier %d: ", p.Tier)
// The branch it follows, never a commit a build once named (novox/hq 04-ISSUES/215).
if err := buildOne(ctx, source, e.Source.Path, followedBranch(e.Source.Ref), 0); err != nil {
state.State = "failed"
state.Why = err.Error()
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("%s could not be asked for: %v", name, err)
continue
}
state.State = "asked"
state.AskedAt = &now
askModule(ctx, p, name, byName)
}
return nil
}
// askABuild is how a plan asks for one build, not waited for, and learns the id it asked under. A
// variable so a test of what a plan does around an ask needs no build machine.
var askABuild = func(ctx context.Context, source buildSource, path, ref string) (string, error) {
return buildOneAsked(ctx, source, path, ref, 0, false)
}
// askModule asks the build machine for one module of a plan and marks it asked, with the id it was
// asked under (novox/hq ADR 0219) — or failed, with the plan, when it could not be asked.
func askModule(ctx context.Context, p *inventory.Plan, name string, byName map[string]inventory.Entry) {
now := time.Now().UTC()
state := p.Modules[name]
if state == nil {
state = &inventory.PlanModule{}
p.Modules[name] = state
}
e, known := byName[name]
if !known {
state.State = "failed"
state.Why = "no longer in the catalogue"
p.State = inventory.PlanFailed
p.Note = name + " is no longer in the catalogue"
return
}
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
fmt.Printf(" tier %d: ", p.Tier)
// The branch it follows, never a commit a build once named (novox/hq 04-ISSUES/215).
id, err := askABuild(ctx, source, e.Source.Path, followedBranch(e.Source.Ref))
if err != nil {
state.State = "failed"
state.Why = err.Error()
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("%s could not be asked for: %v", name, err)
return
}
state.State = "asked"
state.AskedAt = &now
state.Build = id
state.Why, state.Commit, state.BuiltAt = "", "", nil
}
// planBuilt marks a module built (or failed) in every open plan whose current tier holds it, and
// advances what that completes. Called from the daemon's take-in of every outcome.
//
@@ -295,7 +378,7 @@ func askTier(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan) e
// the later plan's answer — it stood on the bases from before the later plan's merge, and taking it
// would send machines, and the next tier, what the later merge replaced. asked is zero when the
// build's request time is not known, and such an outcome is taken as before.
func planBuilt(ctx context.Context, open *stores, module, commit, failed string, asked time.Time) {
func planBuilt(ctx context.Context, open *stores, module, commit, failed string, asked time.Time, id string) {
inv := open.inventory
// One controller works the plans at a time (novox/hq issue 213); an outcome waits its turn rather
// than write over what the holder is about to save. Not taken, it is still in the build records,
@@ -331,7 +414,17 @@ func planBuilt(ctx context.Context, open *stores, module, commit, failed string,
state = &inventory.PlanModule{}
p.Modules[module] = state
}
if askedBefore(asked, state.AskedAt) {
// **The plan's own ask is its outcome, by id** (novox/hq ADR 0219); another build of the module
// is, as before, when it was asked at or after the plan's ask (issue 219).
// **A module asked under an id is answered by that id's outcome and no other** (novox/hq ADR
// 0219): a replay, a rebuild beside the plan, or an older ask finishing late is somebody else's
// build, made from other source, and settling the plan with it would send that. A plan from
// before ids were kept is matched as it was: by when the build was asked (issue 219).
if state.Build != "" {
if state.Build != id {
continue
}
} else if askedBefore(asked, state.AskedAt) {
continue
}
if failed != "" {
@@ -339,6 +432,10 @@ func planBuilt(ctx context.Context, open *stores, module, commit, failed string,
state.Why = failed
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("%s failed to build in tier %d", module, p.Tier)
sayUnsent(p, func(m string) bool {
u, err := inv.UpgradeOf(ctx, m)
return err == nil && u.RollOut
})
} else {
state.State = "built"
state.BuiltAt = &now
@@ -400,8 +497,17 @@ func advanceHeld(ctx context.Context, open *stores) {
moved, err := advanceOnce(ctx, open, p, edges, rollsOut)
if err != nil {
fmt.Printf("%s: %v\n", p.ID, err)
// Kept in the plan, so `plans` says why it has not moved rather than the log alone;
// the state is left as it was and the step is tried again on the next tick.
p.Note = "tier " + fmt.Sprint(p.Tier) + ": " + err.Error() + " — tried again"
if err := inv.SavePlan(ctx, *p); err != nil {
fmt.Printf("%s: cannot keep the plan: %v\n", p.ID, err)
}
break
}
if p.State == inventory.PlanFailed {
sayUnsent(p, rollsOut)
}
if err := inv.SavePlan(ctx, *p); err != nil {
fmt.Printf("%s: cannot keep the plan: %v\n", p.ID, err)
break
@@ -442,6 +548,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
// plan never hears it. The record is the fact; a build recorded after the ask is that tier's
// outcome, whoever was listening.
recorded := map[string][]inventory.Build{}
byID := map[string]inventory.Build{}
for _, m := range tier {
if s := p.Modules[m]; s != nil && s.State == "asked" {
builds, err := inv.Builds(ctx, m, 5)
@@ -449,9 +556,19 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
return false, err
}
recorded[m] = builds
// Its own ask's record, by id — found even when the outcome named no module (ADR 0219).
if s.Build != "" {
b, found, err := inv.BuildByID(ctx, s.Build)
if err != nil {
return false, err
}
if found {
byID[s.Build] = b
}
}
}
}
if settleFromRecords(p, tier, recorded) {
if settleFromRecords(p, tier, recorded, byID) {
return true, nil
}
// Asked: wait for every build.
@@ -470,6 +587,15 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
// a module that packages another repository's source, keeps its commit; the catalogue announces
// no move for it and its machines would keep the old image until somebody pushed (novox/hq
// issue 189). A module whose policy records is built and left, as its policy says.
//
// **One machine first, unless the module's policy says together** (novox/hq issue 249, ADR
// 0218). The plan sent every machine running the module at once, and the operator's policy —
// one at a time, stopping at the first that fails, which an announced upgrade honours — was not
// read here at all: a module whose new declarations broke it broke everywhere in the same
// minute. Now the first machine is sent, the plan records it and waits for that machine's report
// after the send to say it applied what it was sent; only then are the rest sent. A first machine
// that fails or refuses stops the module's rollout and the plan with it, the rest untouched.
var pending []string
for _, m := range tier {
state := p.Modules[m]
if state == nil || state.SentAt != nil || !rollsOut(m) {
@@ -479,17 +605,64 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
if err != nil {
return false, err
}
policy, err := inv.UpgradeOf(ctx, m)
if err != nil {
return false, err
}
var reports []inventory.Reported
if !policy.Together {
// Read for the choice of the first machine as well as for its report.
if reports, err = inv.LastReports(ctx); err != nil {
return false, err
}
}
now := time.Now().UTC()
state.SentAt = &now
if len(running) == 0 {
step := nextRollout(*state, running, policy.Together, reports, now, planWaitBound)
switch {
case step.failed != "":
state.Why = step.failed
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("%s stopped at its first machine in tier %d: %s; %s left as it was",
m, p.Tier, step.failed, orNone(strings.Join(step.rest, ", ")))
fmt.Printf("%s: %s\n", p.ID, p.Note)
return true, nil
case step.waiting != "":
pending = append(pending, fmt.Sprintf("%s on %s, sent first at %s", m, step.waiting, state.FirstAt.Local().Format("15:04")))
continue
case len(step.send) == 0:
// No machine runs it: nothing to send, and nothing to wait for.
state.SentAt = &now
continue
}
if err := sendTo(ctx, open, running); err != nil {
return false, fmt.Errorf("sending %s to %s after tier %d: %w", m, strings.Join(running, ", "), p.Tier, err)
// What sendToEach answers, not what was asked: the machine holding the bus is sent before
// the first when its user list must change (issue 249), and the plan waits for it too.
sent, err := sendToEach(ctx, open, step.send)
if err != nil {
// Not marked sent, so the next step tries again (issue 249): a grant that could not be
// issued is a send that did not happen.
return false, fmt.Errorf("sending %s to %s after tier %d: %w", m, strings.Join(step.send, ", "), p.Tier, err)
}
fmt.Printf("%s: tier %d built; sent %s to %s\n", p.ID, p.Tier, m, strings.Join(running, ", "))
if step.first {
state.First = sent
state.FirstAt = &now
p.State = inventory.PlanRolling
p.Note = fmt.Sprintf("tier %d built; sent %s to %s first", p.Tier, m, strings.Join(sent, ", "))
fmt.Printf("%s: tier %d built; sent %s to %s first, the rest once it reports it applied\n",
p.ID, p.Tier, m, strings.Join(sent, ", "))
return true, nil
}
state.SentAt = &now
fmt.Printf("%s: tier %d built; sent %s to %s\n", p.ID, p.Tier, m, strings.Join(sent, ", "))
return true, nil
}
if len(pending) > 0 {
note := "tier " + fmt.Sprint(p.Tier) + " built; waiting for " + strings.Join(pending, "; ") +
" to report it applied before the rest are sent"
changed := p.State != inventory.PlanRolling || p.Note != note
p.State = inventory.PlanRolling
p.Note = note
return changed, nil
}
// And wait for what the next tier needs running.
needed := gates(*p, edges, rollsOut)
if len(needed) > 0 {
@@ -516,10 +689,24 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
if state.BuiltAt != nil {
since = *state.BuiltAt
}
if state.SentAt != nil && state.SentAt.After(since) {
since = *state.SentAt
// **Each machine from its own send** (novox/hq issue 256). With one machine first (ADR 0218)
// a module is sent twice — the first machine, then the rest — and SentAt is the second.
// Asked of every machine, the first machine's report, made between the two sends, read as
// older than the build, and the gate waited for a report it already had, for ever.
sinceFor := func(node string) time.Time {
at := since
sent := state.SentAt
for _, n := range state.First {
if n == node {
sent = state.FirstAt
}
}
if sent != nil && sent.After(at) {
at = *sent
}
return at
}
if ok, on := applied(m, since, running, reports); !ok {
if ok, on := appliedEach(m, sinceFor, running, reports); !ok {
waiting = append(waiting, fmt.Sprintf("%s on %s", m, strings.Join(on, ", ")))
}
}
@@ -540,6 +727,110 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
return true, nil
}
// rolloutStep is what a plan does next with one built module's machines (novox/hq issue 249).
type rolloutStep struct {
// send is the machines to send now; first, whether they are the first machine's send.
send []string
first bool
// waiting names the first machines whose report the rest wait for.
waiting string
// failed says how a first machine did not take it; rest is what is then left alone.
failed string
rest []string
}
// nextRollout is the next step of one module's rollout in a plan (novox/hq issue 249, ADR 0218).
//
// Together, every machine running it at once, as the policy says. Otherwise one machine first — the
// first by name among those that have reported within the bound, so a laptop that is away is not
// the one the rest wait on; the first by name when none has; the same choice on every controller and
// every resume — and the rest once each machine the first send reached reports, about the
// declaration it was last sent, that it applied it. Compared by the store's own record of what was
// sent (Reported.Current), never by this controller's clock against the machine's.
//
// **A first machine that fails, refuses, or does not report within the bound stops the rollout
// there** (ADR 0218 §2), naming the machine; the rest are not sent. A wait with no end is not a
// rollout: it held the plan open for ever, read as work in progress (issue 254).
func nextRollout(s inventory.PlanModule, running []string, together bool, reports []inventory.Reported,
now time.Time, bound time.Duration) rolloutStep {
if len(running) == 0 {
return rolloutStep{}
}
if together {
return rolloutStep{send: running}
}
byNode := map[string]inventory.Reported{}
for _, r := range reports {
byNode[r.Node] = r
}
if s.FirstAt == nil {
sorted := append([]string{}, running...)
sort.Strings(sorted)
for _, n := range sorted {
if r, said := byNode[n]; said && r.At != nil && now.Sub(*r.At) <= bound {
return rolloutStep{send: []string{n}, first: true}
}
}
return rolloutStep{send: sorted[:1], first: true}
}
sentFirst := map[string]bool{}
for _, n := range s.First {
sentFirst[n] = true
}
var rest []string
for _, n := range running {
if !sentFirst[n] {
rest = append(rest, n)
}
}
var waiting, failed []string
for _, n := range s.First {
r, said := byNode[n]
// Only a report about what it was last sent says anything about this build.
if !said || r.At == nil || !r.Current {
waiting = append(waiting, n)
continue
}
switch r.Outcome {
case inventory.OutcomeApplied:
case inventory.OutcomeFailed, inventory.OutcomeRefused:
failed = append(failed, n+" "+r.Outcome+" what it was sent")
default:
waiting = append(waiting, n)
}
}
if len(failed) > 0 {
return rolloutStep{failed: strings.Join(failed, "; "), rest: rest}
}
if len(waiting) > 0 {
if now.Sub(*s.FirstAt) > bound {
return rolloutStep{failed: fmt.Sprintf("%s did not report it applied within %s",
strings.Join(waiting, ", "), bound), rest: rest}
}
return rolloutStep{waiting: strings.Join(waiting, ", ")}
}
return rolloutStep{send: rest}
}
// sayUnsent adds to an ended plan's note the modules it built and never sent (novox/hq issue 249).
// An announced move of a module a plan held was left to that plan; a plan that ends without
// sending it — failed elsewhere, or closed by hand — would leave its machines behind with nothing
// saying so. A module whose rollout stopped at its first machine is not among them: that stop was
// the point. Said once.
func sayUnsent(p *inventory.Plan, rollsOut func(string) bool) {
var unsent []string
for name, s := range p.Modules {
if s != nil && s.State == "built" && s.SentAt == nil && s.FirstAt == nil && rollsOut(name) {
unsent = append(unsent, name)
}
}
if len(unsent) == 0 || strings.Contains(p.Note, "built and never sent") {
return
}
sort.Strings(unsent)
p.Note += "; built and never sent: " + strings.Join(unsent, ", ") + " — `push --behind` sends them"
}
// planTicker advances open plans on a timer, for the steps outcomes alone cannot take.
func planTicker(ctx context.Context, open *stores) {
advancePlans(ctx, open)
@@ -556,15 +847,24 @@ func planTicker(ctx context.Context, open *stores) {
}
// planLine is one plan as `status` says it.
func planLine(p inventory.Plan, now time.Time) string {
func planLine(p inventory.Plan, now time.Time) string { return planLineWith(p, now, pauseView{}) }
// planLineWith is planLine knowing whether the build seat is paused (novox/hq ADR 0219): a plan
// waiting on builds nobody will take until a person resumes the seat says so, and is not late.
func planLineWith(p inventory.Plan, now time.Time, pause pauseView) string {
where := fmt.Sprintf("tier %d of %d", min(p.Tier+1, len(p.Tiers)), len(p.Tiers))
switch p.State {
case inventory.PlanDone:
return fmt.Sprintf("%s %s done, %d tier(s)", p.Repository, short(p.Commit), len(p.Tiers))
case inventory.PlanFailed:
return fmt.Sprintf("%s %s FAILED at %s: %s", p.Repository, short(p.Commit), where, p.Note)
case inventory.PlanSuperseded:
return fmt.Sprintf("%s %s %s", p.Repository, short(p.Commit), p.Note)
}
since := now.Sub(p.Updated).Round(time.Second)
if waiting, paused := pausedWaiting(p, pause, now); paused {
return fmt.Sprintf("%s %s %s, %s", p.Repository, short(p.Commit), where, waiting)
}
since := now.Sub(p.Updated).Round(time.Minute)
late := ""
if since > planWaitBound {
late = " — LATE"
@@ -578,20 +878,49 @@ func planLine(p inventory.Plan, now time.Time) string {
// planFailedBuild marks the module a failed build was for when the result names no module: by the
// repository and path the plan's modules were asked at.
//
// **By the id first** (novox/hq ADR 0219): a plan keeps the id it asked each module under, so an
// outcome that never learnt its module's name — cancelled, killed, failed at the clone — is matched
// to the module it was asked for exactly. Repository and path remain for a plan from before ids
// were kept.
func planFailedBuild(ctx context.Context, open *stores, result link.BuildResult) {
asked, _ := link.BuildAskedAt(result.ID)
if plans, err := open.inventory.OpenPlans(ctx); err == nil {
if module := moduleAskedAs(plans, result.ID); module != "" {
planBuilt(ctx, open, module, result.Commit, result.Failed, asked, result.ID)
return
}
}
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return
}
for _, e := range entries {
if repositoryMatches(e.Source.Repository, result.Repository) && e.Source.Path == result.Path {
asked, _ := link.BuildAskedAt(result.ID)
planBuilt(ctx, open, e.Manifest.Module, result.Commit, result.Failed, asked)
planBuilt(ctx, open, e.Manifest.Module, result.Commit, result.Failed, asked, result.ID)
return
}
}
}
// moduleAskedAs is the module an open plan's current tier asked for under this id, or nothing.
func moduleAskedAs(plans []inventory.Plan, id string) string {
if id == "" {
return ""
}
for _, p := range plans {
if p.Tier >= len(p.Tiers) {
continue
}
for _, m := range p.Tiers[p.Tier] {
if s := p.Modules[m]; s != nil && s.Build == id {
return m
}
}
}
return ""
}
func repositoryMatches(a, b string) bool {
trim := func(s string) string { return strings.ToLower(strings.TrimSuffix(s, ".git")) }
return trim(a) == trim(b) || strings.HasSuffix(trim(a), "/"+trim(b)) || strings.HasSuffix(trim(b), "/"+trim(a))
@@ -610,7 +939,7 @@ type planStatus struct {
Late bool `json:"late"`
}
func planStatuses(plans []inventory.Plan, now time.Time) []planStatus {
func planStatuses(plans []inventory.Plan, now time.Time, pause pauseView) []planStatus {
out := make([]planStatus, 0, len(plans))
for _, p := range plans {
ps := planStatus{ID: p.ID, Repository: p.Repository, Commit: p.Commit, State: p.State,
@@ -621,6 +950,10 @@ func planStatuses(plans []inventory.Plan, now time.Time) []planStatus {
ps.Waiting = "builds of tier " + fmt.Sprint(p.Tier)
}
ps.Late = now.Sub(p.Updated) > planWaitBound
// Paused is a person's decision, not lateness (novox/hq ADR 0219).
if waiting, paused := pausedWaiting(p, pause, now); paused {
ps.Waiting, ps.Late = waiting, false
}
}
out = append(out, ps)
}
@@ -628,12 +961,15 @@ func planStatuses(plans []inventory.Plan, now time.Time) []planStatus {
}
// openPlans is the open plans among the recent ones, and how many have waited past the bound.
func openPlans(plans []inventory.Plan) ([]inventory.Plan, int) {
func openPlans(plans []inventory.Plan, pause pauseView) ([]inventory.Plan, int) {
var open []inventory.Plan
late := 0
for _, p := range plans {
if p.Open() {
open = append(open, p)
if _, paused := pausedWaiting(p, pause, time.Now()); paused {
continue
}
if time.Since(p.Updated) > planWaitBound {
late++
}
@@ -666,7 +1002,7 @@ func plansCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
fmt.Printf("%s — %s\n", p.ID, planLine(p, now))
fmt.Printf("%s — %s\n", p.ID, planLineWith(p, now, buildSeatPause(ctx, inv, []inventory.Plan{p})))
for i, tier := range p.Tiers {
marker := " "
if i == p.Tier && p.Open() {
@@ -681,6 +1017,9 @@ func plansCommand(ctx context.Context, args []string) error {
if s.Commit != "" {
state += " from " + short(s.Commit)
}
if s.Build != "" && s.State != "built" {
state += " (" + s.Build + ")"
}
if s.Why != "" {
state += ": " + s.Why
}
@@ -693,7 +1032,28 @@ func plansCommand(ctx context.Context, args []string) error {
if *whatIf != "" {
return planWhatIf(ctx, inv, *whatIf, splitList(*paths), splitList(*modules))
}
if len(positionals) == 2 && positionals[0] == "stop" {
// `retry` (novox/hq ADR 0219): a failed plan's failed builds asked again, and the plan goes on.
if len(positionals) == 2 && positionals[0] == "retry" {
said, err := retryPlan(ctx, open, positionals[1])
if err != nil {
return err
}
fmt.Println(said)
return nil
}
// `stop`, or `close` (novox/hq issue 254): a person ending a plan that will not move again — one
// waiting on a report that cannot come — so it stops reading as work in progress. Marked failed
// with who ended it; what it asked still builds and registers.
if len(positionals) == 2 && (positionals[0] == "stop" || positionals[0] == "close") {
how := "stopped"
if positionals[0] == "close" {
how = "closed"
}
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return err
}
defer release()
p, err := inv.PlanByID(ctx, positionals[1])
if err != nil {
return err
@@ -702,25 +1062,16 @@ func plansCommand(ctx context.Context, args []string) error {
return fmt.Errorf("%s is already %s", p.ID, p.State)
}
p.State = inventory.PlanFailed
p.Note = "stopped by hand at tier " + fmt.Sprint(p.Tier)
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return err
}
defer release()
if p, err = inv.PlanByID(ctx, positionals[1]); err != nil {
return err
}
if !p.Open() {
return fmt.Errorf("%s is already %s", p.ID, p.State)
}
p.State = inventory.PlanFailed
p.Note = "stopped by hand at tier " + fmt.Sprint(p.Tier)
p.Note = how + " by hand at tier " + fmt.Sprint(p.Tier)
sayUnsent(&p, func(m string) bool {
u, err := inv.UpgradeOf(ctx, m)
return err == nil && u.RollOut
})
if err := inv.SavePlan(ctx, p); err != nil {
return err
}
fmt.Printf("%s stopped at tier %d of %d; what was asked still builds and registers, nothing further is asked\n",
p.ID, p.Tier, len(p.Tiers))
fmt.Printf("%s %s at tier %d of %d; what was asked still builds and registers, nothing further is asked\n",
p.ID, how, p.Tier, len(p.Tiers))
return nil
}
plans, err := inv.RecentPlans(ctx, *limit)
@@ -731,8 +1082,9 @@ func plansCommand(ctx context.Context, args []string) error {
fmt.Println("no merge has produced a plan yet")
return nil
}
pause := buildSeatPause(ctx, inv, plans)
for _, p := range plans {
fmt.Printf("%-28s %s\n", p.ID, planLine(p, now))
fmt.Printf("%-28s %s\n", p.ID, planLineWith(p, now, pause))
}
return nil
}
@@ -795,7 +1147,13 @@ func planWhatIf(ctx context.Context, inv *inventory.Inventory, repository string
how := "built; its policy records, so nothing is sent"
if u, err := inv.UpgradeOf(ctx, name); err == nil && u.RollOut {
running, _ := inv.Running(ctx, name)
reports, _ := inv.LastReports(ctx)
how = "built, then sent to " + orNone(strings.Join(running, ", "))
// One machine first unless the policy says together (novox/hq issue 249).
if first := nextRollout(inventory.PlanModule{}, running, u.Together, reports, time.Now(), planWaitBound); first.first && len(running) > 1 {
how = fmt.Sprintf("built, then sent to %s first and to the rest once it has applied it",
first.send[0])
}
rolls[name] = how
}
fmt.Printf(" %-22s %s\n", name, how)
@@ -828,7 +1186,12 @@ func splitList(s string) []string {
// settleFromRecords marks every module of the tier still `asked` built — or failed — from a build
// recorded after it was asked, and says whether it changed anything (novox/hq 04-ISSUES/214).
// Newest first, as Builds answers: the first record after the ask is the outcome of that ask.
func settleFromRecords(p *inventory.Plan, tier []string, recorded map[string][]inventory.Build) bool {
//
// The record of the plan's own ask, by its id, is that outcome before anything else (novox/hq ADR
// 0219): the plan asked under it, and a failure recorded without a module — cancelled, killed — is
// found by nothing else.
func settleFromRecords(p *inventory.Plan, tier []string, recorded map[string][]inventory.Build,
byID map[string]inventory.Build) bool {
changed := false
for _, m := range tier {
s := p.Modules[m]
@@ -837,6 +1200,10 @@ func settleFromRecords(p *inventory.Plan, tier []string, recorded map[string][]i
}
var outcome *inventory.Build
for i := range recorded[m] {
// Asked under an id, only that id's record answers (ADR 0219), and it is looked up below.
if s.Build != "" {
break
}
b := recorded[m][i]
if b.At.Before(*s.AskedAt) {
break
@@ -848,6 +1215,9 @@ func settleFromRecords(p *inventory.Plan, tier []string, recorded map[string][]i
}
outcome = &b
}
if own, found := byID[s.Build]; s.Build != "" && found {
outcome = &own
}
if outcome == nil {
continue
}
+35 -4
View File
@@ -145,7 +145,7 @@ func TestAPlanSettlesAnAskedBuildFromTheRecords(t *testing.T) {
// Only a build from before the ask: not this ask's outcome.
"builder": {{ID: "build-0", Commit: "06ea2168", At: asked.Add(-time.Hour)}},
}
if !settleFromRecords(&p, p.Tiers[0], records) {
if !settleFromRecords(&p, p.Tiers[0], records, nil) {
t.Fatal("nothing settled, though the controller's build is recorded after the ask")
}
if s := p.Modules["mesh-controller"]; s.State != "built" || s.Commit != "2ebbb799" || s.BuiltAt == nil {
@@ -162,13 +162,13 @@ func TestAPlanSettlesAnAskedBuildFromTheRecords(t *testing.T) {
late := map[string][]inventory.Build{"postgres": {
{ID: "build-old", Commit: "efff5415", Asked: asked.Add(-18 * time.Minute), At: asked.Add(12 * time.Minute)},
}}
if settleFromRecords(&r, r.Tiers[0], late) || r.Modules["postgres"].State != "asked" {
if settleFromRecords(&r, r.Tiers[0], late, nil) || r.Modules["postgres"].State != "asked" {
t.Errorf("an earlier ask's late outcome settled this ask: %+v", r.Modules["postgres"])
}
// Newest heard first: the earlier ask's late outcome, then this ask's own, heard before it.
late["postgres"] = append(late["postgres"], inventory.Build{ID: "build-mine", Commit: "4bcd5f73",
Asked: asked.Add(time.Second), At: asked.Add(5 * time.Minute)})
if !settleFromRecords(&r, r.Tiers[0], late) || r.Modules["postgres"].State != "built" ||
if !settleFromRecords(&r, r.Tiers[0], late, nil) || r.Modules["postgres"].State != "built" ||
r.Modules["postgres"].Commit != "4bcd5f73" {
t.Errorf("this ask's own outcome, heard before the earlier ask's, did not settle it: %+v", r.Modules["postgres"])
}
@@ -176,8 +176,39 @@ func TestAPlanSettlesAnAskedBuildFromTheRecords(t *testing.T) {
// A failure recorded after the ask fails the plan, as hearing it would have.
q := inventory.Plan{ID: "plan-2", Tiers: [][]string{{"x"}},
Modules: map[string]*inventory.PlanModule{"x": {State: "asked", AskedAt: &asked}}}
settleFromRecords(&q, q.Tiers[0], map[string][]inventory.Build{"x": {{ID: "b", Failed: "no", At: asked.Add(time.Minute)}}})
settleFromRecords(&q, q.Tiers[0], map[string][]inventory.Build{"x": {{ID: "b", Failed: "no", At: asked.Add(time.Minute)}}}, nil)
if q.State != inventory.PlanFailed || q.Modules["x"].State != "failed" {
t.Errorf("a recorded failure did not fail the plan: %+v %+v", q, q.Modules["x"])
}
}
// The first machine's report, made between the send to it and the send to the rest, opens the gate for it:
// each machine is judged from its own send, not from the last one (novox/hq issue 256).
func TestTheGateJudgesEachMachineFromItsOwnSend(t *testing.T) {
built := time.Date(2026, 10, 5, 18, 22, 0, 0, time.UTC)
firstSent := built.Add(31 * time.Second)
firstReported := built.Add(43 * time.Second)
restSent := built.Add(58 * time.Second)
restReported := built.Add(74 * time.Second)
reports := []inventory.Reported{
{Node: "ace", At: &firstReported},
{Node: "g14", At: &restReported},
}
since := func(node string) time.Time {
if node == "ace" {
return firstSent
}
return restSent
}
if ok, waiting := appliedEach("build-agent", since, []string{"ace", "g14"}, reports); !ok {
t.Fatalf("the gate still waits on %v, though each reported after its own send", waiting)
}
// The old reading, every machine from the last send, is what held the plan.
if ok, _ := applied("build-agent", restSent, []string{"ace", "g14"}, reports); ok {
t.Fatal("the single-moment reading should hold the first machine back")
}
early := built.Add(10 * time.Second)
if ok, waiting := appliedEach("build-agent", since, []string{"ace"}, []inventory.Reported{{Node: "ace", At: &early}}); ok || waiting[0] != "ace" {
t.Fatal("a report from before the machine was sent opened the gate")
}
}
+1 -1
View File
@@ -189,7 +189,7 @@ func readinessOf(ctx context.Context, inv *inventory.Inventory) (broker.Readines
// A third of the catalogue never does (novox/hq ADR 0120), and counting those as missing a credential
// would bury the ones that matter under a list nobody can act on.
func speaksOnTheBus(m catalogue.Manifest) bool {
return len(m.Emits) > 0 || len(m.Consumes) > 0 || len(m.Tools) > 0 ||
return len(m.EmitsAll()) > 0 || len(m.Consumes) > 0 || len(m.Tools) > 0 ||
len(m.DefinesSeats) > 0 || len(m.Uses) > 0 || len(m.Claims) > 0
}
+152
View File
@@ -0,0 +1,152 @@
package main
import (
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 249, ADR 0218: a plan rolls a module out to one machine first and the rest only
// once that machine has reported it applied; a module whose policy says together goes everywhere at
// once, as before.
func TestAPlanSendsOneMachineFirstAndTheRestAfterItsReport(t *testing.T) {
running := []string{"novox", "ace", "g14"}
sentAt := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
now := sentAt.Add(5 * time.Minute)
bound := 30 * time.Minute
after := sentAt.Add(time.Minute)
next := func(s inventory.PlanModule, running []string, together bool, reports []inventory.Reported) rolloutStep {
return nextRollout(s, running, together, reports, now, bound)
}
// Together: every machine at once.
if step := next(inventory.PlanModule{}, running, true, nil); !reflect.DeepEqual(step.send, running) || step.first {
t.Fatalf("a together policy did not send every machine at once: %+v", step)
}
// Otherwise the first by name, alone, when none has reported lately.
step := next(inventory.PlanModule{}, running, false, nil)
if !step.first || !reflect.DeepEqual(step.send, []string{"ace"}) {
t.Fatalf("the first send was %+v, wanted ace alone", step)
}
state := inventory.PlanModule{First: []string{"ace"}, FirstAt: &sentAt}
report := func(outcome string, current bool) []inventory.Reported {
return []inventory.Reported{{Node: "ace", At: &after, Outcome: outcome, Current: current},
{Node: "g14", At: &after, Outcome: inventory.OutcomeApplied, Current: true}}
}
// No report yet, or one about an older declaration than it was last sent: wait.
for what, reports := range map[string][]inventory.Reported{
"no report": nil,
"a report about older": report(inventory.OutcomeApplied, false),
} {
step := next(state, running, false, reports)
if len(step.send) != 0 || step.waiting != "ace" || step.failed != "" {
t.Errorf("%s: %+v, wanted to wait for ace", what, step)
}
}
// Applied what it was last sent: the rest, and only the rest.
step = next(state, running, false, report(inventory.OutcomeApplied, true))
if step.first || !reflect.DeepEqual(step.send, []string{"novox", "g14"}) {
t.Fatalf("after ace applied it the plan sent %+v, wanted novox and g14", step)
}
// Failed or refused: stop, the rest untouched.
for _, outcome := range []string{inventory.OutcomeFailed, inventory.OutcomeRefused} {
step := next(state, running, false, report(outcome, true))
if len(step.send) != 0 || !strings.Contains(step.failed, "ace "+outcome) ||
!reflect.DeepEqual(step.rest, []string{"novox", "g14"}) {
t.Errorf("a first machine that %s it: %+v", outcome, step)
}
}
// The machine holding the bus went with the first send: the rest wait for it too, and it is
// not sent again.
both := inventory.PlanModule{First: []string{"novox", "ace"}, FirstAt: &sentAt}
half := report(inventory.OutcomeApplied, true)
if step := next(both, running, false, half); step.waiting != "novox" {
t.Fatalf("the plan did not wait for the bus's machine sent first: %+v", step)
}
all := append(half, inventory.Reported{Node: "novox", At: &after, Outcome: inventory.OutcomeApplied, Current: true})
if step := next(both, running, false, all); !reflect.DeepEqual(step.send, []string{"g14"}) {
t.Fatalf("after both applied it the plan sent %+v, wanted g14 alone", step)
}
// One machine, or none: nothing is waited for that cannot come.
if step := next(inventory.PlanModule{}, nil, false, nil); len(step.send) != 0 || step.first {
t.Fatalf("a module nothing runs was sent: %+v", step)
}
if step := next(state, []string{"ace"}, false, report(inventory.OutcomeApplied, true)); len(step.send) != 0 || step.waiting != "" {
t.Fatalf("a module on one machine waited for more: %+v", step)
}
}
// ADR 0218 §2: a first machine that does not report within the bound stops the rollout there,
// naming the machine and the bound; the rest are left alone.
func TestAFirstMachineThatDoesNotReportStopsTheRollout(t *testing.T) {
sentAt := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
state := inventory.PlanModule{First: []string{"ace"}, FirstAt: &sentAt}
step := nextRollout(state, []string{"ace", "g14"}, false, nil, sentAt.Add(31*time.Minute), 30*time.Minute)
if !strings.Contains(step.failed, "ace did not report it applied within 30m") ||
!reflect.DeepEqual(step.rest, []string{"g14"}) || len(step.send) != 0 {
t.Fatalf("a silent first machine: %+v", step)
}
}
// The first machine is the first by name among those heard from lately: a laptop that is away is
// not the one the rest wait on. When none has been heard from, the first by name.
func TestTheFirstMachineIsOneThatHasReportedLately(t *testing.T) {
now := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
lately, long := now.Add(-time.Minute), now.Add(-3*time.Hour)
reports := []inventory.Reported{
{Node: "ace", At: &long}, {Node: "g14", At: &lately}, {Node: "novox", At: &lately},
}
step := nextRollout(inventory.PlanModule{}, []string{"novox", "ace", "g14"}, false, reports, now, 30*time.Minute)
if !step.first || !reflect.DeepEqual(step.send, []string{"g14"}) {
t.Fatalf("the first send was %+v, wanted g14, the first heard from lately", step)
}
}
// An announced move of a module an open plan is still rolling out is left to the plan: sending it
// here as well put the bundle on every machine at once (novox/hq issue 249).
func TestAnAnnouncedMoveIsLeftToThePlanRollingItOut(t *testing.T) {
sent := time.Now()
plans := []inventory.Plan{
{ID: "plan-done", State: inventory.PlanDone, Modules: map[string]*inventory.PlanModule{"agent": {}}},
{ID: "plan-1", State: inventory.PlanRolling, Modules: map[string]*inventory.PlanModule{
"agent": {State: "built", First: []string{"ace"}, FirstAt: &sent}, "gitea": {State: "built", SentAt: &sent}}},
}
if got := rolledOutByAPlan(plans, "agent"); got != "plan-1" {
t.Fatalf("a module the plan is rolling out was not left to it: %q", got)
}
for _, m := range []string{"gitea", "keycloak"} {
if got := rolledOutByAPlan(plans, m); got != "" {
t.Errorf("%s, which no plan will send, was left to %s", m, got)
}
}
}
// A plan that ends without sending what it built says so, with the remedy; a module whose rollout
// stopped at its first machine is not among them.
func TestAnEndedPlanSaysWhatItBuiltAndNeverSent(t *testing.T) {
at := time.Now()
p := inventory.Plan{State: inventory.PlanFailed, Note: "closed by hand at tier 1",
Modules: map[string]*inventory.PlanModule{
"agent": {State: "built"},
"stopped": {State: "built", First: []string{"ace"}, FirstAt: &at},
"sent": {State: "built", SentAt: &at},
"notes": {State: "built"},
"later": {},
}}
rollsOut := func(m string) bool { return m != "notes" }
sayUnsent(&p, rollsOut)
sayUnsent(&p, rollsOut)
if p.Note != "closed by hand at tier 1; built and never sent: agent — `push --behind` sends them" {
t.Fatalf("the note reads %q", p.Note)
}
}
@@ -0,0 +1,149 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Defends novox/hq ADR 0207 at the controller's acts: `assign` refuses a module whose resources a
// seat nothing on the node holds applies, `unassign` refuses taking the last holder from under its
// dependents, and `status` reports what composition does not yet refuse.
func serviceManagerHolder() catalogue.Manifest {
return catalogue.Manifest{Module: "systemd", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.ServiceManagerSeat, Scope: catalogue.ScopeNode,
Serves: []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal"}}},
Resources: []map[string]any{{"id": "systemd", "type": "package", "package": "systemd"}}}
}
func packageManagerHolder() catalogue.Manifest {
return catalogue.Manifest{Module: "pacman", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.PackageManagerSeat, Scope: catalogue.ScopeNode}},
Resources: []map[string]any{{"id": "refresh", "type": "service", "unit": "pacman-refresh.timer"}}}
}
func aDaemon() catalogue.Manifest {
return catalogue.Manifest{Module: "sshd", Version: "1",
Resources: []map[string]any{{"id": "sshd", "type": "service", "unit": "sshd.service"}}}
}
func TestAnAssignmentWithoutItsHolderIsRefusedAndNotKept(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, serviceManagerHolder())
register(t, open, packageManagerHolder())
register(t, open, aDaemon())
_, err := assign(ctx, open, "laptop", "sshd")
if err == nil {
t.Fatal("sshd went onto a machine nothing holds the service manager of")
}
for _, want := range []string{catalogue.ServiceManagerSeat, "systemd", "ADR 0207"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q:\n%v", want, err)
}
}
assigned, err := open.inventory.Assigned(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if contains(assigned, "sshd") {
t.Fatalf("a refused assignment was kept: %v", assigned)
}
// The holders depend on each other, so neither goes on alone — and both go on in one act.
if _, err := assign(ctx, open, "laptop", "systemd"); err == nil {
t.Fatal("systemd went on alone though its package needs a package manager")
}
if said, err := assign(ctx, open, "laptop", "systemd", "pacman"); err != nil {
t.Fatalf("the two holders assigned together were refused: %v\n%s", err, said)
}
if said, err := assign(ctx, open, "laptop", "sshd"); err != nil {
t.Fatalf("sshd beside its holder was refused: %v\n%s", err, said)
}
}
func TestTheControllerSeatsAssignTakesSeveralModulesAsOneAct(t *testing.T) {
argv, err := argvFor("assign", map[string]any{"node": "laptop", "module": "systemd, pacman"})
if err != nil {
t.Fatal(err)
}
if strings.Join(argv, " ") != "assign laptop systemd pacman" {
t.Errorf("the seat's assign became %v", argv)
}
}
func TestUnassigningTheLastHolderUnderItsDependentsIsRefused(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, serviceManagerHolder())
register(t, open, packageManagerHolder())
register(t, open, aDaemon())
if _, err := assign(ctx, open, "laptop", "systemd", "pacman", "sshd"); err != nil {
t.Fatal(err)
}
_, err := unassign(ctx, open, "laptop", "systemd")
if err == nil {
t.Fatal("the service manager came off a machine still running services")
}
for _, want := range []string{catalogue.ServiceManagerSeat, "sshd", "pacman"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not name %q:\n%v", want, err)
}
}
assigned, _ := open.inventory.Assigned(ctx, "laptop")
if !contains(assigned, "systemd") {
t.Fatalf("a refused unassignment took the module off anyway: %v", assigned)
}
if _, err := unassign(ctx, open, "laptop", "sshd"); err != nil {
t.Fatalf("a dependent could not come off: %v", err)
}
}
func TestStatusReportsAnUnheldDependencyWithoutRefusingTheMachine(t *testing.T) {
// The mesh as it ran before the switch (novox/hq ADR 0207 §4).
defer catalogue.EnforcingSeatDependencies(false)()
open := aMesh(t)
ctx := t.Context()
register(t, open, serviceManagerHolder())
register(t, open, aDaemon())
// Assigned straight into the store: a machine whose modules predate the rule, which is every
// machine on the day it ships.
if _, err := open.inventory.Assign(ctx, "laptop", "sshd"); err != nil {
t.Fatal(err)
}
asked, err := theThreeQuestions(ctx, open)
if err != nil {
t.Fatal(err)
}
if _, refused := asked.refused["laptop"]; refused {
t.Fatalf("an unmet dependency refused the machine before the switch: %s", asked.refused["laptop"])
}
if asked.well() {
t.Error("a mesh with an unheld dependency reads as all well")
}
got := printed(t, func() error { return printStatus(asked) })
for _, want := range []string{"unheld", "laptop", "sshd", catalogue.ServiceManagerSeat, "systemd"} {
if !strings.Contains(got, want) {
t.Errorf("status does not say %q:\n%s", want, got)
}
}
body, err := statusAsJSON(asked)
if err != nil {
t.Fatal(err)
}
var doc struct {
Unheld []catalogue.Unheld `json:"unheld"`
}
if err := json.Unmarshal(body, &doc); err != nil {
t.Fatal(err)
}
if len(doc.Unheld) != 1 || doc.Unheld[0].Module != "sshd" || doc.Unheld[0].Seat != catalogue.ServiceManagerSeat {
t.Errorf("the document's unheld is %+v", doc.Unheld)
}
}
+35 -9
View File
@@ -29,11 +29,13 @@ type seatHolder struct {
// seatRow is one seat and who holds it. Unheld is an answer — "this mesh has no X" — not a fault.
type seatRow struct {
Seat string `json:"seat"`
Scope string `json:"scope"`
Delivers string `json:"delivers,omitempty"`
Decision string `json:"decision"`
Holders []seatHolder `json:"holders"`
Seat string `json:"seat"`
Scope string `json:"scope"`
Delivers string `json:"delivers,omitempty"`
Decision string `json:"decision"`
// Replicated says the seat may be held on several machines at once (novox/hq ADR 0223).
Replicated bool `json:"replicated,omitempty"`
Holders []seatHolder `json:"holders"`
}
// seatsHeld is every seat the mesh defines with its holders, and every claim held that names no
@@ -47,7 +49,7 @@ func seatsHeld(seats []catalogue.Seat, held []catalogue.Held) ([]seatRow, []cata
rows := make([]seatRow, 0, len(seats))
for _, s := range seats {
row := seatRow{Seat: s.Name, Scope: s.Scope, Delivers: s.Delivers, Decision: s.Decision,
Holders: []seatHolder{}}
Replicated: s.Replicated, Holders: []seatHolder{}}
seen := map[seatHolder]bool{}
for _, h := range held {
// Resolve the held claim to a seat rather than comparing names, so a record naming a
@@ -104,9 +106,13 @@ func seatCommand(ctx context.Context, args []string) error {
return nil
}
if len(args) == 3 && args[1] == "--to" {
return handOver(ctx, args[0], args[2])
return handOver(ctx, args[0], args[2], false)
}
return fmt.Errorf("seat rename <from> <to> | seat <name> --to <node>/<module>")
if len(args) == 3 && args[1] == "--add" {
return handOver(ctx, args[0], args[2], true)
}
return fmt.Errorf("seat rename <from> <to> | seat <name> --to <node>/<module> | " +
"seat <name> --add <node>/<module>")
}
// handOver makes one assignment the holder of a seat, as one act, so the seat is never without a
@@ -119,7 +125,12 @@ func seatCommand(ctx context.Context, args []string) error {
// **not** checked is whether the module is running yet: that is what `push` confirms afterwards,
// and refusing to record a handover to a module the node has not started would make the handover
// impossible to do before the switch instead of as the switch.
func handOver(ctx context.Context, seatName, to string) error {
//
// **Or adds one holder beside the others, for a replicated seat** (novox/hq ADR 0223): `--add`
// records the named assignment as a further holder and leaves every holder on record as it is. A
// seat held once refuses it, naming `--to`; `--to` on a replicated seat replaces every holder with
// the one named, as it always did.
func handOver(ctx context.Context, seatName, to string, adding bool) error {
nodeName, module, ok := strings.Cut(to, "/")
if !ok || nodeName == "" || module == "" {
return fmt.Errorf("the new holder is named <node>/<module>, not %q", to)
@@ -135,6 +146,10 @@ func handOver(ctx context.Context, seatName, to string) error {
if !known {
return fmt.Errorf("%q is not a seat this mesh defines — `seats` lists them", seatName)
}
if adding && !seat.Replicated {
return fmt.Errorf("%s is held once per %s, so a second holder cannot be added beside the first — "+
"`seat %s --to %s` hands it over", seat.Name, seat.Scope, seat.Name, to)
}
assigned, err := inv.Assigned(ctx, nodeName)
if err != nil {
return err
@@ -157,6 +172,7 @@ func handOver(ctx context.Context, seatName, to string) error {
return fmt.Errorf("%s is assigned but not in the catalogue, which should not happen", module)
}
var was string
var held []string
holdings, err := inv.Holdings(ctx)
if err != nil {
return err
@@ -164,6 +180,7 @@ func handOver(ctx context.Context, seatName, to string) error {
for _, h := range holdings {
if hs, ok := catalogue.SeatNamed(h.Claim); ok && hs.Name == seat.Name {
was = h.Node
held = append(held, h.Node)
}
}
@@ -187,6 +204,15 @@ func handOver(ctx context.Context, seatName, to string) error {
} else if err := catalogue.CanHold(*m, seat); err != nil {
return fmt.Errorf("%s cannot hold %s: %w", module, seat.Name, err)
}
if adding {
if err := inv.AddSeatHolder(ctx, seat.Name, seat.Scope, nodeName, module); err != nil {
return err
}
fmt.Printf("%s is held by %s on %s, beside what was on record: %s\n", seat.Name, module, nodeName,
strings.Join(held, ", "))
fmt.Printf(" `push --behind` re-declares every machine that reads the seat's holders\n")
return nil
}
if err := inv.HoldSeat(ctx, seat.Name, seat.Scope, nodeName, module); err != nil {
return err
}
+323 -31
View File
@@ -36,19 +36,194 @@ type verbAnswer struct {
// argvFor is the command line a verb and its arguments become. Only the verbs the seat declares, and
// only the arguments each declares: a caller cannot reach a flag the schema did not name.
//
// **Nothing a caller sends is passed over** (novox/hq issue 244). An argument the verb does not
// declare is refused, naming it; a switch that is not "true" or "false" is refused; and an argument
// the verb declares but did not use for the command line it composed — given beside another that
// wins, or half of a shape — is refused too. On 2026-10-05 a push naming one machine reached the
// verb without the machine and ran as a push of every machine behind; a verb that answers "I did
// not take that" would have stopped it before anything was sent.
func argvFor(verb string, args map[string]any) ([]string, error) {
str := func(key string) string {
v, _ := args[key].(string)
return strings.TrimSpace(v)
a, err := readArguments(verb, args)
if err != nil {
return nil, err
}
need := func(keys ...string) error {
for _, k := range keys {
if str(k) == "" {
return fmt.Errorf("%s needs %q", verb, k)
argv, err := a.commandLine()
if len(a.misread) > 0 {
// The table and the command line disagree: the verb reads an argument no caller can see
// in its schema, so no caller could ever pass it.
return nil, fmt.Errorf("%s reads %s, which its schema does not declare — this build's verb "+
"table and its command lines disagree", verb, quoteAll(a.misread))
}
if err != nil {
return nil, err
}
if unused := a.unused(); len(unused) > 0 {
return nil, fmt.Errorf("%s did not use %s together with %s, and an argument a verb would pass over "+
"is refused: nothing was done", verb, quoteAll(unused), quoteAll(a.usedGiven()))
}
return argv, nil
}
// verbArguments are one call's arguments, checked against the verb's schema, and which of them the
// command line was composed from.
type verbArguments struct {
verb string
given map[string]string
used map[string]bool
declared map[string]bool
misread []string // arguments the command line read that the schema does not declare: a bug here
}
// controllerVerb is this binary's own definition of a verb: what it runs is what it declares, so the
// arguments are checked against the table compiled beside argvFor, not a row a newer or older build
// wrote.
func controllerVerb(name string) (catalogue.Verb, bool) {
for _, v := range catalogue.ControllerVerbs {
if v.Name == name {
return v, true
}
}
return catalogue.Verb{}, false
}
// declaredArguments are a schema's properties, and which of them are switches.
func declaredArguments(v catalogue.Verb) (names []string, switches map[string]bool) {
switches = map[string]bool{}
props, _ := v.Input["properties"].(map[string]any)
for name, p := range props {
names = append(names, name)
desc, _ := p.(map[string]any)
switch enum := desc["enum"].(type) {
case []string:
switches[name] = len(enum) == 2 && enum[0] == "true" && enum[1] == "false"
case []any:
switches[name] = len(enum) == 2 && enum[0] == "true" && enum[1] == "false"
}
}
sort.Strings(names)
return names, switches
}
// readArguments refuses what the verb does not take, before anything is composed.
func readArguments(verb string, args map[string]any) (*verbArguments, error) {
v, known := controllerVerb(verb)
if !known {
return nil, fmt.Errorf("%q is not a verb the %s seat serves", verb, catalogue.ControllerSeatName)
}
names, switches := declaredArguments(v)
declared := map[string]bool{}
for _, n := range names {
declared[n] = true
}
takes := "none"
if len(names) > 0 {
takes = quoteAll(names)
}
a := &verbArguments{verb: verb, given: map[string]string{}, used: map[string]bool{}, declared: declared}
keys := make([]string, 0, len(args))
for k := range args {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if !declared[k] {
return nil, fmt.Errorf("%s takes no argument %q — it takes %s; nothing was done", verb, k, takes)
}
var value string
switch x := args[k].(type) {
case nil:
continue
case string:
value = strings.TrimSpace(x)
case bool:
if !switches[k] {
return nil, fmt.Errorf("%s: %q is text, not true or false", verb, k)
}
value = fmt.Sprint(x)
default:
return nil, fmt.Errorf("%s: %q is text, and was given %T", verb, k, x)
}
if switches[k] {
switch value {
case "true":
case "false", "":
continue // said and off: the same as not given, and nothing passed over
default:
return nil, fmt.Errorf("%s: %q is \"true\" or \"false\", not %q", verb, k, value)
}
}
return nil
if value != "" {
a.given[k] = value
}
}
return a, nil
}
// str is one argument's value, marked as used.
func (a *verbArguments) str(key string) string {
if !a.declared[key] {
a.misread = append(a.misread, key)
}
a.used[key] = true
return a.given[key]
}
// on is a switch, marked as used.
func (a *verbArguments) on(key string) bool { return a.str(key) == "true" }
// need refuses a call missing a required argument, in the verb's own words.
func (a *verbArguments) need(keys ...string) error {
for _, k := range keys {
if a.str(k) == "" {
return fmt.Errorf("%s needs %q", a.verb, k)
}
}
return nil
}
// unused are the arguments given that the command line was not composed from.
func (a *verbArguments) unused() []string {
var out []string
for k := range a.given {
if !a.used[k] {
out = append(out, k)
}
}
sort.Strings(out)
return out
}
func (a *verbArguments) usedGiven() []string {
var out []string
for k := range a.given {
if a.used[k] {
out = append(out, k)
}
}
sort.Strings(out)
if len(out) == 0 {
return []string{"nothing"}
}
return out
}
func quoteAll(xs []string) string {
q := make([]string, len(xs))
for i, x := range xs {
if x == "nothing" {
q[i] = x
continue
}
q[i] = fmt.Sprintf("%q", x)
}
return strings.Join(q, ", ")
}
// commandLine composes the command. Every argument it reads is one it uses: a branch that reads an
// argument and then drops it would pass it over, which is what the check after it exists to refuse.
func (a *verbArguments) commandLine() ([]string, error) {
verb, str, on, need := a.verb, a.str, a.on, a.need
switch verb {
case "command":
// The generic verb: the command line as given, split as a shell would split it, with
@@ -65,6 +240,8 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
return nil, errors.New("command names no command")
}
return argv, nil
case "tools":
return nil, errors.New("tools is answered from the records, not by a command")
case "status":
return []string{"status", "--json"}, nil
case "nodes":
@@ -82,10 +259,14 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
if id := str("log"); id != "" {
return []string{"builds", "--log", id}, nil
}
if m := str("module"); m != "" {
return []string{"builds", m}, nil
argv := []string{"builds"}
if n := str("limit"); n != "" {
argv = append(argv, "-n", n)
}
return []string{"builds"}, nil
if m := str("module"); m != "" {
argv = append(argv, m)
}
return argv, nil
case "plans":
if r := str("repository"); r != "" {
argv := []string{"plans", "--what-if", r}
@@ -97,23 +278,69 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
}
return argv, nil
}
if id := str("stop"); id != "" {
return []string{"plans", "stop", id}, nil
for _, act := range []string{"stop", "close", "retry"} {
if id := str(act); id != "" {
return []string{"plans", act, id}, nil
}
}
if id := str("id"); id != "" {
return []string{"plans", id}, nil
}
return []string{"plans"}, nil
argv := []string{"plans"}
if n := str("limit"); n != "" {
argv = append(argv, "-n", n)
}
return argv, nil
// The build queue (novox/hq ADR 0219).
case "queue":
return []string{"queue"}, nil
case "cancel", "kill":
if err := need("id"); err != nil {
return nil, err
}
return []string{verb, str("id")}, nil
case "clear":
if on("dead") {
return []string{"clear", "--dead"}, nil
}
return []string{"clear"}, nil
case "rebuild":
if err := need("what"); err != nil {
return nil, err
}
return []string{"rebuild", str("what")}, nil
case "replay":
if err := need("id"); err != nil {
return nil, err
}
argv := []string{"replay", str("id")}
if on("register") {
argv = append(argv, "--register")
}
if on("older") {
argv = append(argv, "--older")
}
return argv, nil
case "pause", "resume":
if n := str("node"); n != "" {
return []string{verb, n}, nil
}
return []string{verb}, nil
case "plan":
if err := need("node"); err != nil {
return nil, err
}
if on("files") {
return []string{"plan", str("node"), "--files"}, nil
}
return []string{"plan", str("node"), "--json"}, nil
case "assign", "unassign":
if err := need("node", "module"); err != nil {
return nil, err
}
return []string{verb, str("node"), str("module")}, nil
// Several modules comma-separated, judged as one act (novox/hq ADR 0207): the holders of
// the seats that apply resources depend on each other and go on together.
return append([]string{verb, str("node")}, splitModules(str("module"))...), nil
case "pin":
if err := need("node", "provision", "from", "module"); err != nil {
return nil, err
@@ -129,8 +356,14 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
// what a person at a shell does too. A tool call that blocked for a push's whole apply would
// time out on every machine that takes a minute, and say nothing about the ones that did not.
if n := str("node"); n != "" {
// behind is not read here: given with a machine, it is refused as passed over — naming
// a machine and asking for every machine behind are two requests, and guessing one
// would push a machine nobody named, or not push one somebody did.
return []string{"push", n, "--wait", "0"}, nil
}
// No machine: the whole mesh, whether or not behind said so. The command's answer says it
// first, so a caller who meant one machine reads that it was not one.
on("behind")
return []string{"push", "--behind", "--wait", "0"}, nil
case "rotate":
if p := str("provision"); p != "" {
@@ -140,11 +373,17 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
}
return argv, nil
}
if str("node") != "" && str("module") != "" && str("secret") != "" {
_, node := a.given["node"]
_, module := a.given["module"]
_, secret := a.given["secret"]
if node || module || secret {
if err := need("node", "module", "secret"); err != nil {
return nil, fmt.Errorf("%w: a module's own secret is named by node, module and secret together", err)
}
return []string{"secret", "rotate", str("node"), str("module"), str("secret")}, nil
}
// Half of either shape: the command says its usage, which names both shapes, and that is
// the answer the caller needs.
// Neither shape: the command says its usage, which names both, and that is the answer the
// caller needs.
return []string{"rotate"}, nil
case "settings":
// `settings set|clear` at a shell (novox/hq issue 198). The values travel as an argument
@@ -152,15 +391,16 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
if err := need("module"); err != nil {
return nil, err
}
argv := []string{"settings", "set", str("module")}
switch {
case str("clear") == "true":
var argv []string
if on("clear") {
argv = []string{"settings", "clear", str("module")}
case str("values") != "":
argv = append(argv, str("values"))
} else {
argv = []string{"settings", "set", str("module")}
// Neither values nor clear: the command says its usage, which names both.
if v := str("values"); v != "" {
argv = append(argv, v)
}
}
// Neither values nor clear: the command says its usage, which names both, and that is the
// answer the caller needs — the same as `rotate` given half of either shape.
if n := str("node"); n != "" {
argv = append(argv, "--node", n)
}
@@ -192,11 +432,12 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
}
return argv, nil
}
return nil, fmt.Errorf("%q is not a verb the %s seat serves", verb, catalogue.ControllerSeatName)
return nil, fmt.Errorf("%q is a verb of the %s seat's table that this binary has no command line for",
verb, catalogue.ControllerSeatName)
}
// jsonVerbs are the verbs whose command speaks JSON, so the answer carries it as data as well.
var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true}
var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true, "collection": true}
// runVerb runs this binary with the given command line and gathers what it said.
func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
@@ -244,8 +485,22 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
handlers := map[string]link.ToolHandler{}
for _, v := range seat.Serves {
verb := v.Name
if verb == "tools" {
handlers[verb] = func(ctx context.Context, _ json.RawMessage) (any, error) {
if inProcess[verb] {
handlers[verb] = func(ctx context.Context, raw json.RawMessage) (any, error) {
args := map[string]any{}
if len(bytes.TrimSpace(raw)) > 0 {
if err := json.Unmarshal(raw, &args); err != nil {
return nil, fmt.Errorf("the arguments are not a JSON object: %w", err)
}
}
// Refused like any verb's: what a verb does not take is not ignored.
a, err := readArguments(verb, args)
if err != nil {
return nil, err
}
if verb == "calls" {
return callsAnswer(link.Calls, a.given["call"])
}
return seatTools(), nil
}
continue
@@ -284,12 +539,48 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
if err != nil {
return nil, err
}
if answersFirst(argv) {
// Before anything is sent: a push sends the bus's own machine first, and a broker
// reloading its user list forgets the answer it was about to permit (novox/hq issue 265).
link.Acknowledge(ctx)
}
return runVerb(ctx, argv)
}
}
return handlers, behind, nil
}
// inProcess are the verbs answered by this process rather than by a command it runs: `tools` from
// the records, `calls` from what this process served.
var inProcess = map[string]bool{"tools": true, "calls": true}
// answersFirst is a command line whose caller is answered before it runs: a push, by its verb or
// through `command`. A push sends the machine holding the bus first when its user list changed, the
// broker reloads, and a reload forgets every answer the bus was about to permit — so an answer
// waiting for the push to end was refused, every time the list had changed (novox/hq issue 265).
func answersFirst(argv []string) bool {
return len(argv) > 0 && argv[0] == "push"
}
// callsAnswer is what `calls` answers: the kept calls, newest first, without their answers — or
// one call whole.
func callsAnswer(log *link.CallLog, id string) (any, error) {
if id != "" {
c, ok := log.Get(id)
if !ok {
return nil, fmt.Errorf("no call %s is kept here: calls are kept by the controller that "+
"answered them, the last %d, and not across a restart — `calls` lists them", id, link.KeptCalls)
}
return c, nil
}
recent := log.Recent()
for i := range recent {
recent[i].Answer = nil
}
return map[string]any{"calls": recent, "kept": link.KeptCalls,
"note": "newest first; `calls` with a call's id gives its whole answer"}, nil
}
// seatTools is what `tools` answers: every seat with a protocol, and the tools each serves, from the
// mesh's own records — no holder in the path, so it is true while a holder restarts (design 33 §5).
func seatTools() map[string]any {
@@ -310,9 +601,10 @@ func seatTools() map[string]any {
}
// sampleArguments is one of every argument a verb's schema requires, so the check at start proves the
// verb runnable rather than that it happens to want the arguments the check guessed.
// verb runnable rather than that it happens to want the arguments the check guessed — and nothing
// more, since an argument a verb does not declare is refused.
func sampleArguments(v catalogue.Verb) map[string]any {
sample := map[string]any{"node": "x", "module": "x", "repository": "x"}
sample := map[string]any{}
switch required := v.Input["required"].(type) {
case []string:
for _, k := range required {
@@ -0,0 +1,365 @@
package main
import (
"encoding/json"
"go/ast"
"go/parser"
"go/token"
"path/filepath"
"reflect"
"sort"
"strconv"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The schemas the controller serves, verb by verb, as the console receives them: from the
// announcement, not the table — what is checked is what a caller is shown (novox/hq issue 244).
func servedSchemas(t *testing.T) map[string]catalogue.Verb {
t.Helper()
handlers, behind, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
if len(behind) != 0 {
t.Fatalf("this build cannot run %v of its own seat's verbs", behind)
}
served := map[string]catalogue.Verb{}
for _, e := range seatAnnouncement(handlers).Endpoints {
var input map[string]any
if err := json.Unmarshal([]byte(e.Metadata["schema"]), &input); err != nil {
t.Fatalf("%s announces a schema that is not JSON: %v", e.Name, err)
}
served[e.Metadata["tool"]] = catalogue.Verb{Name: e.Metadata["tool"], Input: input}
}
if len(served) != len(catalogue.ControllerVerbs) {
t.Fatalf("%d verbs served for %d in the table", len(served), len(catalogue.ControllerVerbs))
}
return served
}
// subsetsOf is every subset of the names, the empty one included.
func subsetsOf(names []string) [][]string {
var out [][]string
for mask := 0; mask < 1<<len(names); mask++ {
var s []string
for i, n := range names {
if mask&(1<<i) != 0 {
s = append(s, n)
}
}
out = append(out, s)
}
return out
}
func argumentsFor(subset []string, switches map[string]bool) map[string]any {
args := map[string]any{}
for _, n := range subset {
if switches[n] {
args[n] = "true"
} else {
args[n] = "x-" + n
}
}
return args
}
// saidOutright are the switches that say the default aloud, so the line is the same without them —
// on purpose, and only these.
var saidOutright = map[string]string{
"push": "behind", // the whole mesh is what no machine means; behind lets a caller say they meant it
}
// **No argument a caller gives is passed over** (novox/hq issue 244). For every verb served and
// every combination of the arguments its schema declares, the verb either refuses the call, or
// composes a command line that each given argument changed: taking any one away changes the line
// or makes it refused. An argument the line is the same without is one the verb ignored — the
// shape of the push that named a machine and pushed every machine behind.
func TestNoArgumentAVerbIsGivenIsPassedOver(t *testing.T) {
for name, v := range servedSchemas(t) {
if inProcess[name] {
continue
}
names, switches := declaredArguments(v)
for _, subset := range subsetsOf(names) {
args := argumentsFor(subset, switches)
argv, err := argvFor(name, args)
if err != nil {
if strings.Contains(err.Error(), "does not declare") {
t.Errorf("%s %v: %v", name, args, err)
}
continue
}
for _, dropped := range subset {
fewer := map[string]any{}
for k, val := range args {
if k != dropped {
fewer[k] = val
}
}
without, err := argvFor(name, fewer)
if err == nil && reflect.DeepEqual(argv, without) && saidOutright[name] != dropped {
t.Errorf("%s ignores %q given with %v: %v either way", name, dropped, subset, argv)
}
}
}
}
}
// **An argument a verb does not declare is refused, naming it — never dropped.** Every verb, with
// what it requires and one argument more; and `node` in particular, for every verb whose schema
// does not take a machine.
func TestAnArgumentAVerbDoesNotDeclareIsRefused(t *testing.T) {
for name, v := range servedSchemas(t) {
names, _ := declaredArguments(v)
strangers := []string{"no-such-argument"}
if !contains(names, "node") {
strangers = append(strangers, "node")
}
for _, stranger := range strangers {
args := sampleArguments(v)
args[stranger] = "x"
_, err := argvFor(name, args)
if inProcess[name] {
_, err = readArguments(name, args)
}
if err == nil || !strings.Contains(err.Error(), strconv.Quote(stranger)) {
t.Errorf("%s took %q, which it does not declare: %v", name, stranger, err)
}
}
}
if _, err := argvFor("clear", map[string]any{"dead": "yes"}); err == nil {
t.Error("a switch took a word that is neither true nor false, and would have read it as false")
}
if _, err := argvFor("node", map[string]any{"node": 7}); err == nil {
t.Error("a number was taken as a machine's name, or as no machine")
}
if argv, err := argvFor("clear", map[string]any{"dead": true}); err != nil || strings.Join(argv, " ") != "clear --dead" {
t.Errorf("a switch given as JSON true: %v %v", argv, err)
}
}
// The push that was the cause: a machine named is that machine; none named is the whole mesh, and
// naming one beside behind is refused rather than one of the two guessed.
func TestAPushIsOneMachineOrSaysItIsTheWholeMesh(t *testing.T) {
argv, err := argvFor("push", map[string]any{"node": "g1"})
if err != nil || strings.Join(argv, " ") != "push g1 --wait 0" {
t.Fatalf("a named push: %v %v", argv, err)
}
for _, args := range []map[string]any{{}, {"behind": "true"}} {
argv, err := argvFor("push", args)
if err != nil || strings.Join(argv, " ") != "push --behind --wait 0" {
t.Fatalf("a push of the whole mesh %v: %v %v", args, argv, err)
}
}
if _, err := argvFor("push", map[string]any{"node": "g1", "behind": "true"}); err == nil ||
!strings.Contains(err.Error(), `"behind"`) {
t.Fatalf("a named push with behind was taken: %v", err)
}
if _, err := argvFor("push", map[string]any{"machine": "g1"}); err == nil || !strings.Contains(err.Error(), `"machine"`) {
t.Fatalf("a push given the machine under another name ran as a push of every machine: %v", err)
}
}
// commandFlags is every flag set this package's commands parse, by the name the set is made with,
// and the flags defined on it — read from the source, so a flag added to a command is seen here
// without anyone remembering to.
func commandFlags(t *testing.T) map[string][]string {
t.Helper()
files, err := filepath.Glob("*.go")
if err != nil {
t.Fatal(err)
}
fset := token.NewFileSet()
out := map[string][]string{}
for _, f := range files {
if strings.HasSuffix(f, "_test.go") {
continue
}
file, err := parser.ParseFile(fset, f, nil, 0)
if err != nil {
t.Fatal(err)
}
for _, decl := range file.Decls {
fn, ok := decl.(*ast.FuncDecl)
if !ok || fn.Body == nil {
continue
}
sets := map[string]string{} // variable → the set's name
ast.Inspect(fn.Body, func(n ast.Node) bool {
if assign, ok := n.(*ast.AssignStmt); ok && len(assign.Lhs) == 1 && len(assign.Rhs) == 1 {
if call, ok := assign.Rhs[0].(*ast.CallExpr); ok && isSelector(call.Fun, "flag", "NewFlagSet") {
if id, ok := assign.Lhs[0].(*ast.Ident); ok {
if name, ok := stringLit(call.Args[0]); ok {
sets[id.Name] = name
out[name] = append(out[name], []string{}...)
}
}
}
}
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
recv, ok := sel.X.(*ast.Ident)
if !ok || sets[recv.Name] == "" {
return true
}
arg := 0
switch sel.Sel.Name {
case "Bool", "String", "Int", "Int64", "Uint", "Uint64", "Float64", "Duration", "Func", "BoolFunc", "TextVar":
case "BoolVar", "StringVar", "IntVar", "Int64Var", "UintVar", "Uint64Var", "Float64Var", "DurationVar", "Var":
arg = 1
default:
return true
}
if arg < len(call.Args) {
if flagName, ok := stringLit(call.Args[arg]); ok {
out[sets[recv.Name]] = append(out[sets[recv.Name]], flagName)
}
}
return true
})
}
}
return out
}
func isSelector(e ast.Expr, pkg, name string) bool {
sel, ok := e.(*ast.SelectorExpr)
if !ok {
return false
}
id, ok := sel.X.(*ast.Ident)
return ok && id.Name == pkg && sel.Sel.Name == name
}
func stringLit(e ast.Expr) (string, bool) {
lit, ok := e.(*ast.BasicLit)
if !ok || lit.Kind != token.STRING {
return "", false
}
s, err := strconv.Unquote(lit.Value)
return s, err == nil
}
// accountedFlags are the flags of a verb's command that are not an argument of the same name:
// carried by an argument named otherwise ("=argument"), or set by the verb itself, or withheld from
// the named verb on purpose — each with why. `command` reaches every flag of the binary regardless.
var accountedFlags = map[string]map[string]string{
"status": {"json": "set by the verb: the answer is data"},
"seats": {"json": "set by the verb: the answer is data"},
"queue": {"json": "not set: the verb answers the table a person reads"},
"plan": {"json": "set by the verb unless files is asked"},
"push": {"wait": "set by the verb to 0: a tool call cannot hold a connection for a whole apply"},
"build": {
"wait": "set by the verb to 0: the id follows the build (issue 176)",
"self": "set by the verb from the repository's form: a path on the forge, or a URL",
"dry-run": "withheld: a dry run answers only when the build ends, which a call cannot wait for; `command` reaches it",
"behind": "withheld: the named verb builds one named repository; `command` reaches the rest",
"on": "withheld: the named verb builds one named repository; `command` reaches the rest",
},
"builds": {"n": "=limit"},
"plans": {"n": "=limit", "what-if": "=repository"},
}
// **Every flag of the command a verb runs is in the verb's schema, or accounted for here.** Derived
// from the source, so a flag added to a command — or a verb added whose command takes flags —
// fails this until somebody decides, in writing, how a caller reaches it (novox/hq issue 244). And
// a flag accounted for that no longer exists fails too, so the table cannot rot into a list nobody
// reads.
func TestEveryFlagOfAVerbsCommandIsAnArgumentOrAccountedFor(t *testing.T) {
flags := commandFlags(t)
if len(flags["push"]) == 0 || len(flags["plan"]) == 0 {
t.Fatalf("reading the commands' flags found nothing for push or plan: %v", flags)
}
reached := map[string]map[string]bool{} // verb → flag sets its command lines reach
served := servedSchemas(t)
for name, v := range served {
if inProcess[name] || name == "command" {
continue
}
names, switches := declaredArguments(v)
for _, subset := range subsetsOf(names) {
argv, err := argvFor(name, argumentsFor(subset, switches))
if err != nil {
continue
}
// The flag set is named by the longest run of leading words that names one.
var words []string
for _, w := range argv {
if strings.HasPrefix(w, "-") {
break
}
words = append(words, w)
}
for n := len(words); n > 0; n-- {
if _, has := flags[strings.Join(words[:n], " ")]; has {
if reached[name] == nil {
reached[name] = map[string]bool{}
}
reached[name][strings.Join(words[:n], " ")] = true
break
}
}
}
}
used := map[string]map[string]bool{}
verbs := make([]string, 0, len(reached))
for verb := range reached {
verbs = append(verbs, verb)
}
sort.Strings(verbs)
for _, verb := range verbs {
declared, _ := declaredArguments(served[verb])
for set := range reached[verb] {
if used[set] == nil {
used[set] = map[string]bool{}
}
for _, flagName := range flags[set] {
why, accounted := accountedFlags[set][flagName]
switch {
case accounted && strings.HasPrefix(why, "="):
used[set][flagName] = true
if !contains(declared, strings.TrimPrefix(why, "=")) {
t.Errorf("%s: --%s of `%s` is said to be carried by %q, which the schema does not declare",
verb, flagName, set, strings.TrimPrefix(why, "="))
}
case accounted:
used[set][flagName] = true
case contains(declared, flagName):
default:
t.Errorf("%s runs `%s`, which takes --%s, and the verb's schema has no %q: declare it, "+
"or say in accountedFlags why a caller does not reach it", verb, set, flagName, flagName)
}
}
}
}
for set, fs := range accountedFlags {
for flagName := range fs {
if !used[set][flagName] {
t.Errorf("accountedFlags names --%s of `%s`, which no verb's command takes any more", flagName, set)
}
}
}
}
// Every verb's required arguments are properties of its schema: a schema that requires what it
// does not describe is the uncallable verb of issue 244 from the other side.
func TestEveryRequiredArgumentIsDescribed(t *testing.T) {
for name, v := range servedSchemas(t) {
names, _ := declaredArguments(v)
for k := range sampleArguments(v) {
if !contains(names, k) {
t.Errorf("%s requires %q and does not describe it", name, k)
}
}
}
}
+8 -26
View File
@@ -10,29 +10,6 @@ import (
"github.com/novox/mesh-controller/internal/catalogue"
)
// Every verb the mesh-controller seat declares is one this binary can run, with the arguments the
// schema names and no other (novox/hq ADR 0154, ADR 0035).
func TestEveryDeclaredVerbHasACommandLine(t *testing.T) {
for _, v := range catalogue.ControllerVerbs {
if v.Name == "tools" {
continue
}
args := map[string]any{}
props, _ := v.Input["properties"].(map[string]any)
for name := range props {
args[name] = "x"
}
argv, err := argvFor(v.Name, args)
if err != nil {
t.Errorf("%s: %v", v.Name, err)
continue
}
if argv[0] == "" {
t.Errorf("%s: empty command", v.Name)
}
}
}
// `builds` given a build's id reads that build's log from the bus rather than listing builds
// (novox/hq ADR 0157).
func TestBuildsWithAnIdReadsThatBuildsLog(t *testing.T) {
@@ -70,9 +47,14 @@ func TestRotateTakesAProvisionOrAnOwnSecret(t *testing.T) {
if strings.Join(argv, " ") != "secret rotate ace nodered api-token" {
t.Fatalf("an own secret: %v", argv)
}
argv, _ = argvFor("rotate", map[string]any{"node": "ace"})
if strings.Join(argv, " ") != "rotate" {
t.Fatalf("half an own secret falls to the command's usage: %v", argv)
if _, err := argvFor("rotate", map[string]any{"node": "ace"}); err == nil || !strings.Contains(err.Error(), `"module"`) {
t.Fatalf("half an own secret is refused, naming what it lacks: %v", err)
}
if argv, _ := argvFor("rotate", nil); strings.Join(argv, " ") != "rotate" {
t.Fatalf("neither shape falls to the command's usage: %v", argv)
}
if _, err := argvFor("rotate", map[string]any{"provision": "p", "node": "ace", "module": "m", "secret": "s"}); err == nil {
t.Fatal("both shapes at once were taken, and one of them passed over")
}
}
+8
View File
@@ -31,6 +31,10 @@ type sendable struct {
// the same composition as its received files, and every machine's private-network address.
Received map[string]map[string][]catalogue.Contribution
Mesh []string
// BusUsers is the bus's user list this declaration carries, empty for every machine but the one
// holding the bus; not sent apart from the file it is in. Its digest is recorded once sent, so
// whether that machine must go first is read from the list alone (novox/hq issue 249).
BusUsers string
// LeftOut is every module of the machine's set left out of this declaration because a stored
// setting cannot compose with its definition (novox/hq ADR 0163, rule 6), sorted. The host
// keeps that module's held things and touches none of its containers; a machine is told
@@ -39,6 +43,10 @@ type sendable struct {
LeftOut []string
// leftOutWhy is why each was, for push and plan to say; never on the wire.
leftOutWhy map[string]string
// Builds is the build of each module this declaration carries — module to the commit its build
// was made from — recorded with the send and never on the wire (novox/hq issue 259, ADR 0221).
// Composed only on the send path; nil records that it is not known.
Builds map[string]string
}
// adoptionEnvelope is what an adopted node is told about its mode. Taken is every module taken on
+120
View File
@@ -0,0 +1,120 @@
package main
import (
"context"
"fmt"
"strings"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A provider that keeps failing a consumer is a problem the controller reports (novox/hq ADR 0224).
//
// On 2026-10-05 the identity provider's provisioner failed every consumer from shortly after midnight
// until it was fixed by hand that night — 31,000 refused logins after its database was moved and its
// admin kept an older password — and `status` called the mesh well all day (novox/hq issue 179). A
// provider now announces a consumer it has failed for minutes; the controller keeps it until the
// provider says it recovered; and `status`, its JSON and `node show` name it, breaking "all well".
// standings keeps what providers say, in the inventory.
type standings struct{ inv *inventory.Inventory }
func (s standings) Stood(ctx context.Context, st link.Standing) (bool, error) {
return s.inv.KeepStanding(ctx, st.Failing, inventory.ProviderStanding{
Module: st.Module, ProviderNode: st.ProviderNode, Provision: st.Provider,
Consumer: st.Consumer, ConsumerNode: st.Node,
Class: st.Class, Error: st.Error, Since: st.Since, Attempts: st.Attempts,
})
}
// failingProviders is every consumer a provider still assigned where it ran says it keeps failing.
//
// **A provider no longer assigned is not asked about.** Its last word stays in the store, and is
// not a problem: nothing runs there to fail anybody. Assigned again, its first success for each
// consumer clears it.
func failingProviders(ctx context.Context, inv *inventory.Inventory) ([]inventory.ProviderStanding, error) {
all, err := inv.FailingProviders(ctx)
if err != nil {
return nil, fmt.Errorf("what providers say they keep failing cannot be read: %w", err)
}
assigned := map[string]map[string]bool{}
var out []inventory.ProviderStanding
for _, s := range all {
on, asked := assigned[s.ProviderNode]
if !asked {
modules, err := inv.Assigned(ctx, s.ProviderNode)
if err != nil {
// A provider on a machine the mesh no longer knows has nothing running to fail anybody.
modules = nil
}
on = map[string]bool{}
for _, m := range modules {
on[m] = true
}
assigned[s.ProviderNode] = on
}
if on[s.Module] {
out = append(out, s)
}
}
return out, nil
}
// failingLines is how status says them: one consumer per entry, the error under it, and a provider
// that stopped repeating itself said so.
func failingLines(list []inventory.ProviderStanding, now time.Time) []string {
var out []string
for _, s := range list {
where := s.Module
if s.ProviderNode != "" {
where += " on " + s.ProviderNode
}
whom := s.Consumer
if s.ConsumerNode != "" {
whom += " (" + s.ConsumerNode + ")"
}
out = append(out, fmt.Sprintf(" %-24s fails %s: %s, for %s (%d attempts since %s)",
where, whom, orUnclassed(s.Class), roughly(now.Sub(s.Since)), s.Attempts,
s.Since.Local().Format("2006-01-02 15:04")))
if e := strings.TrimSpace(s.Error); e != "" {
out = append(out, fmt.Sprintf(" %-24s %s", "", firstLine(e)))
}
if s.Quiet(now) {
out = append(out, fmt.Sprintf(" %-24s not said again for %s — the provider has stopped "+
"saying anything, so this is its last word", "", roughly(now.Sub(s.SaidAt))))
}
}
return out
}
func orUnclassed(class string) string {
if class == "" {
return "failing"
}
return class
}
// printFailing is the status section, said when there is anything to say.
func printFailing(list []inventory.ProviderStanding, now time.Time) {
if len(list) == 0 {
return
}
fmt.Printf("%d consumer(s) a provider keeps failing (ADR 0224):\n\n", len(list))
for _, line := range failingLines(list, now) {
fmt.Println(line)
}
fmt.Printf("\n the provider's journal has every attempt; it says recovered on its next success\n\n")
}
// failingOn is the standings that concern one machine: a provider running there, or a consumer.
func failingOn(list []inventory.ProviderStanding, node string) []inventory.ProviderStanding {
var out []inventory.ProviderStanding
for _, s := range list {
if s.ProviderNode == node || s.ConsumerNode == node {
out = append(out, s)
}
}
return out
}
+115
View File
@@ -0,0 +1,115 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A provider that keeps failing a consumer is a problem `status` names (novox/hq ADR 0224). On
// 2026-10-05 the identity provider refused every consumer for a day and status called the mesh well
// (04-ISSUES/179): this is that day, told to the controller the way the provider now tells it.
func TestAProviderFailingAConsumerBreaksAllWellUntilItRecovers(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "idp", Version: "1",
Receives: map[string]string{"oidc-client": "/var/lib/mesh/idp/mesh.json"}})
if _, err := assign(ctx, open, "anchor", "idp"); err != nil {
t.Fatal(err)
}
kept := standings{open.inventory}
since := time.Now().Add(-23 * time.Hour)
failing := link.Standing{Module: "idp", Failing: true, Provider: "oidc-client", ProviderNode: "anchor",
Consumer: "mesh_laptop_dashboard", Node: "laptop", Class: "credentials-rejected",
Error: `Keycloak token request failed: 401 {"error":"invalid_grant"}`, Since: since, Attempts: 31000}
if _, err := kept.Stood(ctx, failing); err != nil {
t.Fatal(err)
}
asked, err := theThreeQuestions(ctx, open)
if err != nil {
t.Fatal(err)
}
if asked.well() {
t.Fatal("a mesh whose identity provider fails a consumer reads as well")
}
said := printed(t, func() error { return printStatus(asked) })
for _, want := range []string{"1 consumer(s) a provider keeps failing", "idp on anchor",
"mesh_laptop_dashboard (laptop)", "credentials-rejected", "31000 attempts", "invalid_grant"} {
if !strings.Contains(said, want) {
t.Fatalf("status does not say %q:\n%s", want, said)
}
}
if strings.Contains(said, "all doing what they were told") {
t.Fatalf("status said all well beside a failing provider:\n%s", said)
}
body, err := statusAsJSON(asked)
if err != nil {
t.Fatal(err)
}
var doc struct {
Failing []inventory.ProviderStanding `json:"failing"`
}
if err := json.Unmarshal(body, &doc); err != nil || len(doc.Failing) != 1 || doc.Failing[0].Consumer != "mesh_laptop_dashboard" {
t.Fatalf("the document does not carry it: %v\n%s", err, body)
}
// Both machines' `node show` name it: where the provider runs, and where the consumer is.
for _, node := range []string{"anchor", "laptop"} {
shown := printed(t, func() error { return showNode(ctx, open.inventory, node) })
if !strings.Contains(shown, "a provider keeps failing") || !strings.Contains(shown, "mesh_laptop_dashboard") {
t.Fatalf("node show %s does not name it:\n%s", node, shown)
}
}
// Recovered: gone, and the mesh may be well again as far as this is concerned.
failing.Failing = false
if cleared, err := kept.Stood(ctx, failing); err != nil || !cleared {
t.Fatalf("%v %v", cleared, err)
}
asked, err = theThreeQuestions(ctx, open)
if err != nil {
t.Fatal(err)
}
if len(asked.failing) != 0 {
t.Fatalf("a recovered consumer is still named: %+v", asked.failing)
}
}
// A provider no longer assigned where it ran has nothing running to fail anybody: its last word is
// not a problem.
func TestAnUnassignedProvidersLastWordIsNotAProblem(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
if _, err := (standings{open.inventory}).Stood(ctx, link.Standing{Module: "gone", Failing: true,
ProviderNode: "anchor", Consumer: "x", Since: time.Now()}); err != nil {
t.Fatal(err)
}
asked, err := theThreeQuestions(ctx, open)
if err != nil {
t.Fatal(err)
}
if len(asked.failing) != 0 {
t.Fatalf("%+v", asked.failing)
}
}
func TestAProviderThatStoppedRepeatingItselfIsSaidToHaveGoneQuiet(t *testing.T) {
now := time.Now()
lines := strings.Join(failingLines([]inventory.ProviderStanding{{
Module: "idp", ProviderNode: "anchor", Consumer: "c", Class: "unreachable", Error: "connection refused\nmore",
Since: now.Add(-3 * time.Hour), SaidAt: now.Add(-2 * time.Hour), Attempts: 9,
}}, now), "\n")
for _, want := range []string{"unreachable, for 3h", "connection refused", "not said again for 2h"} {
if !strings.Contains(lines, want) {
t.Fatalf("%q not in:\n%s", want, lines)
}
}
if strings.Contains(lines, "more") {
t.Fatalf("more than the first line of an error:\n%s", lines)
}
}
+45 -3
View File
@@ -129,6 +129,10 @@ func printStatus(asked answers) error {
fmt.Println()
}
// A provider failing a consumer, beside machines failing what they were told: both are something
// not working now (novox/hq ADR 0224).
printFailing(asked.failing, time.Now())
if len(quiet) > 0 {
var said []string
for _, n := range quiet {
@@ -138,14 +142,14 @@ func printStatus(asked answers) error {
len(quiet), strings.Join(said, "\n "))
}
if open, late := openPlans(asked.plans); len(open) > 0 {
if open, late := openPlans(asked.plans, asked.paused); len(open) > 0 {
fmt.Printf("%d plan(s) open", len(open))
if late > 0 {
fmt.Printf(", %d waiting past %s", late, planWaitBound)
}
fmt.Println(":")
for _, p := range open {
fmt.Printf(" %s\n", planLine(p, time.Now()))
fmt.Printf(" %s\n", planLineWith(p, time.Now(), asked.paused))
}
fmt.Println()
}
@@ -282,6 +286,22 @@ func printStatus(asked answers) error {
fmt.Printf("\n `take <node> <module>` compares what runs against what it declares, and runs it\n\n")
}
if len(asked.unheld) > 0 {
// **Reported, and not refused yet** (novox/hq ADR 0207 §4). Each machine still resolves and
// is sent what it would be; this says which of its modules depend on a seat nothing there
// holds, until every machine has its holders and the switch makes it a refusal.
fmt.Printf("%d module dependenc(ies) on a seat nothing on the machine holds (unheld, ADR 0207):\n",
len(asked.unheld))
for _, u := range asked.unheld {
holders := "no module in the catalogue claims it yet"
if len(u.Holders) > 0 {
holders = "could be held by " + strings.Join(u.Holders, ", ")
}
fmt.Printf(" %-12s %-24s %-24s %s\n", u.Node, u.Module, u.Seat, holders)
}
fmt.Printf("\n `assign <node> <holder>` meets it; reported until every machine has its holders, then refused\n\n")
}
if adopted := adoptedNodes(nodes); len(adopted) > 0 {
// Said, because nothing forces the flip: a node left adopted is visible here rather than
// read as converged (novox/hq ADR 0100). Not a fault, so it does not break "all well".
@@ -386,10 +406,32 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
if err != nil {
return answers{}, err
}
// And which machines run a module whose resources a seat nothing there holds applies (novox/hq
// ADR 0207). Each machine resolved again rather than threaded through whoResolves, whose answer
// the private network is built from and should say nothing else; a machine that does not
// resolve is already in refused, and is passed over here.
for _, n := range out.nodes {
plan, _, err := planFor(ctx, open, n.Name)
if err != nil {
if unresolvable(err) {
continue
}
return answers{}, err
}
out.unheld = append(out.unheld, plan.Unheld...)
}
// And every consumer a provider says it keeps failing (novox/hq ADR 0224). Read from what the
// providers announced: nothing else in the mesh knows whether a provision is being made.
out.failing, err = failingProviders(ctx, inv)
if err != nil {
return answers{}, err
}
out.plans, err = inv.RecentPlans(ctx, 5)
if err != nil {
return answers{}, err
}
// Whether the build seat takes work, for a plan waiting on it (novox/hq ADR 0219).
out.paused = buildSeatPause(ctx, inv, out.plans)
// And which machines are not running what the mesh would send them. The same question as a
// module being behind its source, one level down: that one says the catalogue is out of date,
@@ -496,7 +538,7 @@ func untakenModules(ctx context.Context, inv *inventory.Inventory, nodes []inven
func (a answers) well() bool {
return len(a.wrong) == 0 && len(a.quiet) == 0 && len(a.behind) == 0 &&
len(a.waiting) == 0 && len(a.refused) == 0 && a.network == "" && len(a.untaken) == 0 &&
len(a.filtered) == 0
len(a.filtered) == 0 && len(a.unheld) == 0 && len(a.failing) == 0
}
// hostSplit is which machines report which host version, for every version more than one machine
+16
View File
@@ -73,6 +73,22 @@ func migrate(ctx context.Context) error {
}
fmt.Printf("provided %s\n", m.Module)
}
// And what an earlier release shipped and this one does not goes (novox/hq ADR 0226) — unless a
// machine still has it, which is said rather than overridden.
var shipped []string
for _, m := range provided {
shipped = append(shipped, m.Module)
}
retired, kept, err := inv.RetireUnshipped(ctx, shipped)
if err != nil {
return err
}
for _, name := range retired {
fmt.Printf("retired %s: the control plane no longer ships it\n", name)
}
for name, why := range kept {
fmt.Printf("kept %s, which the control plane no longer ships: %s\n", name, why)
}
// The seats the mesh ships with, into the table that now holds the set (novox/hq ADR 0122).
// Idempotent: fills an empty table on first boot, adds a seat a release ships, and leaves an
// operator's changes in the table as they are.
+95
View File
@@ -0,0 +1,95 @@
package main
import (
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 254, ADR 0218: a newer plan takes over what the older open plans of its repository
// and branch had not built, and closes them as superseded; another repository's plan, another
// branch's, and a plan made after it are left alone.
func TestANewerPlanSupersedesTheOlderOpenPlansOfItsRepository(t *testing.T) {
at := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
sent := at.Add(time.Minute)
plan := func(id, repository, branch string, created time.Time, modules map[string]*inventory.PlanModule) inventory.Plan {
return inventory.Plan{ID: id, Repository: repository, Branch: branch, Commit: id + "-commit",
Created: created, State: inventory.PlanRolling, Modules: modules}
}
older := plan("plan-1", "novox/mesh-catalog", "main", at, map[string]*inventory.PlanModule{
"gitea": {State: "built", SentAt: &sent}, // done with: stays done
"keycloak": {State: "asked"}, // asked, not answered: folded
"plex": {}, // not yet asked: folded
"agent": {State: "built"}, // built, rolls out, not sent: folded
"notes": {State: "built"}, // built, records: nothing to send
})
stuck := plan("plan-0", "Novox/Mesh-Catalog", "", at.Add(-time.Hour), map[string]*inventory.PlanModule{
"runtime": {State: "asked"},
})
other := plan("plan-2", "novox/mesh-controller", "main", at, map[string]*inventory.PlanModule{"mesh-controller": {}})
release := plan("plan-3", "novox/mesh-catalog", "release", at, map[string]*inventory.PlanModule{"lemurs": {}})
later := plan("plan-5", "novox/mesh-catalog", "main", at.Add(2*time.Hour), map[string]*inventory.PlanModule{"later": {}})
done := plan("plan-6", "novox/mesh-catalog", "main", at, map[string]*inventory.PlanModule{"finished": {}})
done.State = inventory.PlanDone
newer := plan("plan-4", "novox/mesh-catalog", "main", at.Add(time.Hour), nil)
newer.Commit = "97b1b2b0c0ffee"
rollsOut := func(m string) bool { return m != "notes" }
folded, closed := supersededBy(newer, []inventory.Plan{stuck, older, other, release, later, done, newer}, rollsOut)
if want := []string{"agent", "keycloak", "plex", "runtime"}; !reflect.DeepEqual(folded, want) {
t.Fatalf("folded %v, wanted %v", folded, want)
}
var ids []string
for _, p := range closed {
ids = append(ids, p.ID)
if p.State != inventory.PlanSuperseded || p.Open() {
t.Errorf("%s was left %s", p.ID, p.State)
}
if !strings.Contains(p.Note, "plan-4") || !strings.Contains(p.Note, "97b1b2b0") {
t.Errorf("%s does not name the plan that superseded it: %q", p.ID, p.Note)
}
}
if want := []string{"plan-0", "plan-1"}; !reflect.DeepEqual(ids, want) {
t.Fatalf("superseded %v, wanted %v — another repository, another branch, a later plan and a "+
"finished one are left alone", ids, want)
}
if other.State != inventory.PlanRolling {
t.Fatal("the plan handed in was changed in place")
}
if line := planLine(closed[1], time.Now()); !strings.Contains(line, "superseded") {
t.Fatalf("a superseded plan reads %q", line)
}
}
// novox/hq issue 254: a person closes a plan that will not move again, by its id.
func TestAPersonClosesAStuckPlan(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
stuck := inventory.Plan{ID: "plan-97b1b2b", Repository: "novox/mesh-catalog", Commit: "97b1b2b",
Created: time.Now().UTC(), State: inventory.PlanRolling, Tier: 1, Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "built"}, "b": {}}}
if err := open.inventory.SavePlan(ctx, stuck); err != nil {
t.Fatal(err)
}
if err := plansCommand(ctx, []string{"close", stuck.ID}); err != nil {
t.Fatal(err)
}
closed, err := open.inventory.PlanByID(ctx, stuck.ID)
if err != nil {
t.Fatal(err)
}
if closed.State != inventory.PlanFailed || !strings.Contains(closed.Note, "closed by hand") {
t.Fatalf("the plan was left %s: %q", closed.State, closed.Note)
}
if err := plansCommand(ctx, []string{"close", stuck.ID}); err == nil {
t.Fatal("a plan already closed was closed again")
}
if argv, err := argvFor("plans", map[string]any{"close": stuck.ID}); err != nil ||
!reflect.DeepEqual(argv, []string{"plans", "close", stuck.ID}) {
t.Fatalf("the seat's verb does not close a plan: %v %v", argv, err)
}
}
+108
View File
@@ -0,0 +1,108 @@
package main
import (
"bytes"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// An act says what it changed about the node it acted on, and nothing about the rest of the mesh
// (novox/hq ADR 0207): after the seat dependencies shipped, every `push <node>` and `assign` printed
// every node's unmet dependencies, a hundred lines around the one about the module just assigned.
func aContainer(name string) catalogue.Manifest {
return catalogue.Manifest{Module: name, Version: "1",
Resources: []map[string]any{{"id": name, "type": "container", "image": name}}}
}
func TestAnAssignmentSaysOnlyWhatItChangedOnItsOwnNode(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aContainer("web"))
register(t, open, aContainer("db"))
// anchor already lacks a runtime for db: true, and not this act's to say.
if _, err := open.inventory.Assign(ctx, "anchor", "db"); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "laptop", "db"); err != nil {
t.Fatal(err)
}
said, err := assign(ctx, open, "laptop", "web")
if err != nil {
t.Fatalf("%v\n%s", err, said)
}
if !strings.Contains(said, "web on laptop depends on "+catalogue.ContainerRuntimeSeat) {
t.Errorf("the assignment does not say what the module it assigned depends on:\n%s", said)
}
for _, not := range []string{"db on laptop", "db on anchor", "anchor:"} {
if strings.Contains(said, not) {
t.Errorf("the assignment says %q, which it did not change:\n%s", not, said)
}
}
}
func TestAnAssignmentThatMeetsADependencySaysSo(t *testing.T) {
shelf := map[string]catalogue.Manifest{
"web": aContainer("web"),
"docker": {Module: "docker", Claims: []catalogue.Claim{{Name: catalogue.ContainerRuntimeSeat}},
Resources: []map[string]any{{"id": "d", "type": "container", "image": "dind"}}},
}
lines := unheldChange(shelf, "laptop", []string{"web"}, []string{"web", "docker"})
if len(lines) != 1 || !strings.Contains(lines[0], "web on laptop now has "+catalogue.ContainerRuntimeSeat+" held") {
t.Errorf("meeting a dependency said %v", lines)
}
// Taking the dependent off says nothing: the dependency went with its module.
if lines := unheldChange(shelf, "laptop", []string{"web"}, nil); len(lines) != 0 {
t.Errorf("unassigning the dependent said %v", lines)
}
}
func TestAPushSaysANamedNodesDependenciesAndCountsTheRest(t *testing.T) {
unheld := map[string][]catalogue.Unheld{
"anchor": {{Node: "anchor", Module: "db", Seat: catalogue.ContainerRuntimeSeat}},
"laptop": {{Node: "laptop", Module: "web", Seat: catalogue.ContainerRuntimeSeat},
{Node: "laptop", Module: "sshd", Seat: catalogue.ServiceManagerSeat}},
}
var named bytes.Buffer
reportUnheldPushed(&named, true, []string{"laptop"}, unheld)
got := named.String()
if !strings.Contains(got, "laptop has 2 unmet") || !strings.Contains(got, "web on laptop") ||
!strings.Contains(got, "sshd on laptop") || strings.Contains(got, "anchor") {
t.Errorf("a named push said:\n%s", got)
}
var all bytes.Buffer
reportUnheldPushed(&all, false, []string{"anchor", "laptop", "quiet"}, unheld)
want := "anchor: 1 unmet seat dependenc(ies) — see `status`\nlaptop: 2 unmet seat dependenc(ies) — see `status`\n"
if all.String() != want {
t.Errorf("a push to every node said:\n%s\nwant\n%s", all.String(), want)
}
}
func TestOnlyTheServingControllerLogsEachChange(t *testing.T) {
read := func(f func()) string {
old := os.Stderr
r, w, _ := os.Pipe()
os.Stderr = w
f()
_ = w.Close()
os.Stderr = old
var b bytes.Buffer
_, _ = b.ReadFrom(r)
return b.String()
}
u := []catalogue.Unheld{{Node: "n1", Module: "web", Seat: catalogue.ContainerRuntimeSeat}}
if got := read(func() { logUnheld("n1", u) }); got != "" {
t.Errorf("a command logged:\n%s", got)
}
logUnheldChanges = true
defer func() { logUnheldChanges = false }()
if got := read(func() { logUnheld("n1", u) }); !strings.Contains(got, "web on n1") {
t.Errorf("the serving controller did not log a change:\n%s", got)
}
if got := read(func() { logUnheld("n1", u) }); got != "" {
t.Errorf("an unchanged report was logged again:\n%s", got)
}
}
+195 -38
View File
@@ -5,8 +5,10 @@ import (
"errors"
"flag"
"fmt"
"path"
"regexp"
"strings"
"sync"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
@@ -55,10 +57,26 @@ func (f following) Upgraded(ctx context.Context, u link.Upgraded) error {
return nil
}
// **A plan that holds the module rolls it out, and this does not** (novox/hq issue 249, ADR
// 0218). A merge's plan builds the module and sends it one machine first, the rest once that one
// has applied it; this announcement arrives as the build registers, and sending here too — one
// machine after another without waiting for any to apply — put the new bundle on every machine in
// the same minute, whatever the plan was waiting for. A move no plan answers (a build asked by
// hand) is still this handler's.
if plans, err := inv.OpenPlans(ctx); err != nil {
return notNow(err)
} else if id := rolledOutByAPlan(plans, u.Module); id != "" {
// Said with its remedy: a plan that ends without sending it — failed, or closed by hand — leaves
// these machines behind, which `status` lists and `push --behind` sends (novox/hq issue 249).
fmt.Printf("%s moved to %s; %s rolls it out to %s — if that plan ends without sending it, "+
"`status` lists them as behind and `push --behind` sends it\n",
u.Module, shortCommit(u.Commit), id, readableList(on))
return nil
}
if decision.Together {
fmt.Printf("%s moved to %s; sending %s together\n",
u.Module, shortCommit(u.Commit), readableList(on))
return sendTo(ctx, f.open, on)
return askAgainOnGrants(sendTo(ctx, f.open, on))
}
// One at a time, and stopping at the first that fails.
//
@@ -69,13 +87,34 @@ func (f following) Upgraded(ctx context.Context, u link.Upgraded) error {
u.Module, shortCommit(u.Commit), readableList(on))
for _, node := range on {
if err := sendTo(ctx, f.open, []string{node}); err != nil {
return fmt.Errorf("%s did not take %s, so the machines after it were left alone: %w",
node, u.Module, err)
return askAgainOnGrants(fmt.Errorf("%s did not take %s, so the machines after it were left alone: %w",
node, u.Module, err))
}
}
return nil
}
// askAgainOnGrants marks a send that stopped at its grants as one to ask again (novox/hq issue 249):
// an announcement handled by a send whose memberships could not be issued is held and redelivered,
// rather than taken as handled with the machines left on the old version.
func askAgainOnGrants(err error) error {
if err != nil && errors.Is(err, errGrants) && !errors.Is(err, link.ErrTryAgain) {
return fmt.Errorf("%w: %w", link.ErrTryAgain, err)
}
return err
}
// rolledOutByAPlan is the open plan that will send a module's machines its new build — one holding
// the module that has not finished sending it — or empty when none will (novox/hq issue 249).
func rolledOutByAPlan(plans []inventory.Plan, module string) string {
for _, p := range plans {
if s, holds := p.Modules[module]; p.Open() && holds && (s == nil || (s.SentAt == nil && s.State != "failed")) {
return p.ID
}
}
return ""
}
// readableList names machines the way a sentence does, because this is read by a person deciding
// whether an upgrade went where they expected.
func readableList(names []string) string {
@@ -228,6 +267,11 @@ func notNow(err error) error {
// (novox/hq 04-ISSUES/131). Nothing is pushed here: what a finished build does to the machines
// running the module is the upgrade's decision, taken when the catalogue announces it.
func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
// One merge acted on at a time, whoever hands it over: the bus, or the catch-up that reads back
// what the bus did not hand over (novox/hq issue 266). Each judges against what the other wrote.
actingOnMerges.Lock()
defer actingOnMerges.Unlock()
inv := f.open.inventory
entries, err := inv.Catalogued(ctx)
if err != nil {
@@ -238,34 +282,7 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
return notNow(err)
}
// Two kinds of module are affected by one merge, and they are affected differently.
//
// A module **built from** this repository and branch has moved: the mesh records the new commit
// as what its source now has, and only what the merge actually changed is rebuilt. A module that
// only **packages source from** it has not moved — its own source is somewhere else, at the
// commit it already records — so it is rebuilt and its record left alone. Writing this commit as
// its source would make it permanently behind a repository its manifest does not come from.
var from, packaging []inventory.Entry
already := 0
for _, e := range entries {
switch {
case sourceIs(e.Source, m):
if e.Source.BuiltFrom == m.Commit {
already++
continue
}
// **A merge older than the last look at the source is history, not a move.** The forge
// announces what it finds merged, and an old merge surfacing late would otherwise move
// the recorded head backwards and rebuild everything built from that repository, once
// per old merge (2026-09-28).
if isHistory(m.MergedAt, e.Source.Seen) {
continue
}
from = append(from, e)
case readsFrom(read[e.Manifest.Module], m):
packaging = append(packaging, e)
}
}
from, packaging, already := mergeCandidates(m, entries, read)
if len(from) == 0 && len(packaging) == 0 {
// "Already built from it" and "nothing reads it" are different facts, and reading the first
// as the second sends somebody looking for a broken trigger when the mesh is up to date.
@@ -323,6 +340,45 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
}
defer release()
plan := planOfMerge(m, movedNames, edges)
// **A newer plan supersedes the older open plans of this repository and branch** (novox/hq issue
// 254, ADR 0218): what they had not built is planned here again, and they are closed, so one plan
// works a repository's modules at a time and a stuck one ends at the next merge.
working, err := inv.OpenPlans(ctx)
if err != nil {
return notNow(err)
}
rollsOut := func(module string) bool {
u, err := inv.UpgradeOf(ctx, module)
return err == nil && u.RollOut
}
folded, superseded := supersededBy(plan, working, rollsOut)
if len(folded) > 0 {
held := map[string]bool{}
for _, e := range entries {
held[e.Manifest.Module] = true
}
names := map[string]bool{}
for _, name := range movedNames {
names[name] = true
}
var also []string
for _, name := range folded {
// One the catalogue no longer holds would fail the newer plan's ask; it is not this
// merge's to build.
if held[name] && !names[name] {
names[name] = true
movedNames = append(movedNames, name)
also = append(also, name)
}
}
if len(also) > 0 {
again := planOfMerge(m, movedNames, edges)
again.ID, again.Created = plan.ID, plan.Created
plan = again
fmt.Printf(" %s, left unbuilt by an older plan of %s, are planned here again\n",
strings.Join(also, ", "), plan.Repository)
}
}
if hasCycle(plan.Tiers, edges) {
fmt.Printf(" the last tier depends on itself: %s — built together, in no order\n",
strings.Join(plan.Tiers[len(plan.Tiers)-1], ", "))
@@ -330,6 +386,14 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
if err := inv.SavePlan(ctx, plan); err != nil {
return notNow(err)
}
// Closed after the newer plan is kept, never before: a controller replaced between the two leaves
// both open, which the next merge settles, rather than neither.
for _, old := range superseded {
if err := inv.SavePlan(ctx, old); err != nil {
return notNow(err)
}
fmt.Printf(" %s (%s at %s) is %s\n", old.ID, old.Repository, short(old.Commit), old.Note)
}
var tiers []string
for i, t := range plan.Tiers {
tiers = append(tiers, fmt.Sprintf("%d: %s", i, strings.Join(t, ", ")))
@@ -353,6 +417,57 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
return nil
}
// actingOnMerges keeps one merge acted on at a time (novox/hq issue 266).
var actingOnMerges sync.Mutex
// mergeCandidates is what one merge could move, judged against what the catalogue holds.
//
// Two kinds of module are affected by one merge, and they are affected differently.
//
// A module **built from** this repository and branch has moved: the mesh records the new commit as
// what its source now has, and only what the merge actually changed is rebuilt. A module that only
// **packages source from** it has not moved — its own source is somewhere else, at the commit it
// already records — so it is rebuilt and its record left alone. Writing this commit as its source
// would make it permanently behind a repository its manifest does not come from. `already` counts
// the modules built from this repository that are already built from this very commit.
func mergeCandidates(m link.SourceMoved, entries []inventory.Entry,
read map[string][]inventory.ReadRepository) (from, packaging []inventory.Entry, already int) {
for _, e := range entries {
switch {
case sourceIs(e.Source, m):
if e.Source.BuiltFrom == m.Commit {
already++
continue
}
// **A merge older than the last look at the source is history, not a move.** The forge
// announces what it finds merged, and an old merge surfacing late would otherwise move
// the recorded head backwards and rebuild everything built from that repository, once
// per old merge (2026-09-28).
if isHistory(m.MergedAt, e.Source.Seen) {
continue
}
from = append(from, e)
case readsFrom(read[e.Manifest.Module], m):
packaging = append(packaging, e)
}
}
return from, packaging, already
}
// wouldMove is the modules built from the merged repository that acting on this merge would mark as
// moved and rebuild — SourceMoved's judgement, made without acting (novox/hq issue 266). Empty for a
// merge already acted on: acting marks each of them as looked at, so the merge then reads as history.
//
// **Only the modules built from it, never the ones that merely package source from it.** Acting
// records nothing about those, so a merge acted on would go on reading as unacted for them, and be
// acted on again on every look. A merge that moves both is caught by the first kind, and acting on it
// rebuilds the second as well.
func wouldMove(m link.SourceMoved, entries []inventory.Entry,
read map[string][]inventory.ReadRepository) []inventory.Entry {
from, _, _ := mergeCandidates(m, entries, read)
return whatTheMergeTouched(from, entries, m)
}
// sourceIs is whether a recorded source is the repository and branch a merge announced. A source on
// the git seat is recorded as its path on the forge; one elsewhere as the URL it was cloned from.
// An empty recorded ref is the repository's default branch, which is what a merge into the base
@@ -423,25 +538,45 @@ func lastLookAt(entries []inventory.Entry, m link.SourceMoved) time.Time {
//
// A change inside *another* module's directory is that module's business and not this one's, even
// when the mesh does not hold that module: `known` is every module this repository is known to hold,
// whatever branch it was registered from. That is also the limit of this — a repository whose shared
// code sits inside a directory the mesh has never seen a module in reads as shared, and everything
// is rebuilt. Rebuilding too much is the safe direction: the fault this whole path exists for is a
// mesh that believes it is current and is not (novox/hq 04-ISSUES/131).
// whatever branch it was registered from.
//
// **And a module the mesh has never seen is still a module** (novox/hq issue 252). A merge adding a
// new module to the catalogue repository — `modules/newmod/module.json` and its files — read as a
// change to shared code, because `modules/newmod` was nobody's known directory, and every module
// built from the repository was rebuilt and rolled out for a module none of them is. So the
// directories that hold modules are known too: the parents of the known modules' directories
// (`modules`, never the root). A changed path `<parent>/<name>/…` belongs to the module at
// `<parent>/<name>` — held or not — and rebuilds nothing else, **provided it is shown to be a
// module**: its `module.json` is among the changed files (added, changed, or removed with it). A
// directory under the same parent whose manifest the merge did not touch may as well be a shared
// library (`modules/lib`), and that is still read as shared. Rebuilding too much remains the safe
// direction: the fault this whole path exists for is a mesh that believes it is current and is not
// (novox/hq 04-ISSUES/131). A file at the root, or directly in a parent, is shared as it always was.
func whatTheMergeTouched(candidates, known []inventory.Entry, m link.SourceMoved) []inventory.Entry {
// Nothing said about the files, or not all of them said: everything built from it is affected.
if len(m.Paths) == 0 || m.PathsTruncated {
return candidates
}
var dirs []string
parents := map[string]bool{}
for _, e := range known {
if e.Source.Path != "" && sameRepository(e.Source.Repository, m) {
dirs = append(dirs, e.Source.Path)
dir := strings.Trim(e.Source.Path, "/")
dirs = append(dirs, dir)
if parent := path.Dir(dir); parent != "." && parent != "/" {
parents[parent] = true
}
}
}
changed := map[string]bool{}
for _, p := range m.Paths {
if !insideAny(p, dirs) {
return candidates
changed[strings.TrimPrefix(p, "/")] = true
}
for _, p := range m.Paths {
if insideAny(p, dirs) || inAModuleOfItsOwn(p, parents, changed) {
continue
}
return candidates
}
var out []inventory.Entry
for _, e := range candidates {
@@ -452,6 +587,28 @@ func whatTheMergeTouched(candidates, known []inventory.Entry, m link.SourceMoved
return out
}
// inAModuleOfItsOwn is whether a changed file is inside a module directory the mesh does not know —
// `<parent>/<name>/…` under a directory known to hold modules, whose `module.json` the same merge
// changed (novox/hq issue 252). Such a file is that module's business and nobody else's.
func inAModuleOfItsOwn(p string, parents, changed map[string]bool) bool {
p = strings.TrimPrefix(p, "/")
for parent := range parents {
rest, under := strings.CutPrefix(p, parent+"/")
if !under {
continue
}
name, _, inADirectory := strings.Cut(rest, "/")
if !inADirectory || name == "" {
// A file directly in the parent — `modules/README.md` — is about all of them.
continue
}
if changed[parent+"/"+name+"/module.json"] {
return true
}
}
return false
}
// inside is whether a changed file is in a directory: that directory itself, or under it.
func inside(path, dir string) bool {
dir = strings.Trim(dir, "/")
+345
View File
@@ -0,0 +1,345 @@
package artifacts
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"strconv"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Every archive the store keeps is held by a manifest (novox/hq issue 253, ADR 0189).
//
// **The store's collector marks only from manifests.** The mesh's store is a stock registry, and
// its nightly `registry garbage-collect` walks every manifest in every repository, marks the blobs
// those manifests name, and deletes every blob it did not mark. An image is a manifest, so what
// the mesh keeps of an image survives. An archive was not: the builder put it in the store as a
// bare blob — upload, then `PUT ?digest=` — and nothing in the store names it. To the collector a
// bare blob is unreferenced, so the first real collection would have deleted every archive the
// mesh holds, kept or not, and every machine pinning a bundle would have found it gone. The
// collector runs `--dry-run` until this is true.
//
// **So each archive gets a holder**: the smallest OCI image manifest that names it — the empty
// config, one layer, nothing else — put in the archive's own repository, by digest, untagged. The
// collector marks it and so keeps the archive; the sweep lets go of an archive by deleting its
// holder first, which is what lets the bytes go at the next collection.
//
// **Nothing a machine reads changes.** The recorded reference stays
// `artifact-store://<module>/<artifact>/blobs/sha256:…`, and machines fetch the blob exactly as
// before. The holder is the store's bookkeeping, not a second way to reach anything.
//
// **Deterministic, so it never needs recording.** The holder is composed from the archive's digest
// and size alone, in a fixed field order with no timestamps or annotations, so the sweep can
// compute which manifest holds any archive from the reference it already has plus one HEAD for the
// size. No schema change, no second record that could disagree with the store.
const (
// mediaManifest is the type a holder is put and asked for as.
mediaManifest = "application/vnd.oci.image.manifest.v1+json"
// mediaEmpty is the OCI empty descriptor's type: a config that says nothing, for a manifest
// whose only purpose is to name its layer.
mediaEmpty = "application/vnd.oci.empty.v1+json"
// emptyDigest is the digest of `{}`, the empty config's content, fixed by the OCI spec.
emptyDigest = "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
// mediaArchive is the layer type an archive is held as. Every archive the builder publishes is
// `pack`'s gzipped tar, so this is the true type and not a placeholder — and it is a constant,
// not read from anywhere, because the holder must be recomputable from the reference alone.
mediaArchive = "application/vnd.oci.image.layer.v1.tar+gzip"
)
// emptyConfig is the content emptyDigest names.
var emptyConfig = []byte("{}")
// manifestAccept is what a manifest is asked for as. A registry answers a manifest HEAD only in a
// type the caller named, and answers 404 to a bare one for a manifest it holds perfectly well
// (measured 2026-09-28; internal/builder/registry.go says how that was found).
var manifestAccept = []string{
mediaManifest,
"application/vnd.docker.distribution.manifest.v2+json",
}
type descriptor struct {
MediaType string `json:"mediaType"`
Digest string `json:"digest"`
Size int64 `json:"size"`
}
type holderManifest struct {
SchemaVersion int `json:"schemaVersion"`
MediaType string `json:"mediaType"`
Config descriptor `json:"config"`
Layers []descriptor `json:"layers"`
}
// Holder is the manifest that holds an archive in the store, and its digest.
//
// A pure function of the archive's digest and size: the same two in give the same bytes out,
// always, because `encoding/json` writes a struct's fields in their declared order and there is
// nothing here that varies by when or where it was composed.
func Holder(digest string, size int64) (body []byte, holder string) {
body, err := json.Marshal(holderManifest{
SchemaVersion: 2,
MediaType: mediaManifest,
Config: descriptor{MediaType: mediaEmpty, Digest: emptyDigest, Size: int64(len(emptyConfig))},
Layers: []descriptor{{MediaType: mediaArchive, Digest: digest, Size: size}},
})
if err != nil {
// Marshalling a struct of strings and integers cannot fail.
panic(err)
}
sum := sha256.Sum256(body)
return body, "sha256:" + hex.EncodeToString(sum[:])
}
// Hold makes sure the store holds this archive by a manifest, and says whether it had to write one.
//
// Takes a reference as the mesh records it. An image is its own manifest and needs no holder, so
// it answers false and nothing is asked. Idempotent: a holder already there is left alone, which
// is what lets the sweep run it over every kept archive on every build and so backfill the bare
// blobs published before holders existed (novox/hq issue 253).
//
// Gone when the store does not hold the archive at all: there is nothing to hold, and that is a
// fact the caller reports rather than one this invents a remedy for.
func (s Store) Hold(ctx context.Context, reference string) (bool, error) {
repository, digest, archive, err := s.archive(reference)
if err != nil || !archive {
return false, err
}
size, err := s.blobSize(ctx, repository, digest)
if err != nil {
return false, err
}
return s.HoldBlob(ctx, repository, digest, size)
}
// Held is whether the store holds this archive by its manifest. Asks and changes nothing — the
// question an operator needs answered with "none unheld" before the collector is let loose.
//
// An image answers true: it is its own manifest. An archive the store does not have answers Gone.
func (s Store) Held(ctx context.Context, reference string) (bool, error) {
repository, digest, archive, err := s.archive(reference)
if err != nil {
return false, err
}
if !archive {
return true, nil
}
size, err := s.blobSize(ctx, repository, digest)
if err != nil {
return false, err
}
_, holder := Holder(digest, size)
return s.has(ctx, s.url(repository, "manifests", holder), manifestAccept...)
}
// HoldBlob puts the holder for a blob of this digest and size into its repository, unless it is
// there already. Answers whether it wrote one.
//
// The builder calls this with the size it has just uploaded; the sweep, through Hold, with the size
// the store reports. Both arrive at the same holder, which is the point of composing it.
func (s Store) HoldBlob(ctx context.Context, repository, digest string, size int64) (bool, error) {
if s.Address == "" {
return false, fmt.Errorf("this mesh has no artifact store on its network to hold %s/%s in", repository, digest)
}
body, holder := Holder(digest, size)
there, err := s.has(ctx, s.url(repository, "manifests", holder), manifestAccept...)
if err != nil {
return false, err
}
if there {
return false, nil
}
// The config must be in the repository before a manifest naming it is accepted: a registry
// refuses a manifest whose blobs it cannot find there, which is the property that makes a
// holder mean something.
if err := s.putBlob(ctx, repository, emptyDigest, emptyConfig); err != nil {
return false, err
}
// **By digest, never by tag.** A tag would be one more name to move and one more thing the
// collector's `--delete-untagged` would read as meaningful; the mesh names nothing by tag that
// it pins by digest, and an untagged manifest is kept by plain collection.
request, err := http.NewRequestWithContext(ctx, http.MethodPut,
s.url(repository, "manifests", holder), bytes.NewReader(body))
if err != nil {
return false, err
}
request.Header.Set("Content-Type", mediaManifest)
response, err := s.client().Do(request)
if err != nil {
return false, err
}
defer response.Body.Close()
if response.StatusCode != http.StatusCreated {
said, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
return false, fmt.Errorf("the artifact store refused to hold %s/%s: %s %s",
repository, digest, response.Status, strings.TrimSpace(string(said)))
}
return true, nil
}
// letGoOfHolder deletes the manifest holding an archive, before the archive's own link goes.
//
// **Holder first.** Deleting the blob link alone leaves a manifest still naming the blob, and the
// collector would keep its bytes for ever on the strength of it — the sweep would record the
// archive collected while the disk said otherwise. Deleting the holder first and failing before
// the link goes leaves an unheld archive that the next sweep still offers, which is safe.
//
// A store that no longer has the blob answers Gone: without its size the holder cannot be named,
// and without the blob there is nothing left for a holder to keep. A store that never had a holder
// for it — an archive published before holders, never backfilled — answers 404 to the delete, and
// that is the outcome wanted.
func (s Store) letGoOfHolder(ctx context.Context, repository, digest string) error {
size, err := s.blobSize(ctx, repository, digest)
if err != nil {
return err
}
_, holder := Holder(digest, size)
err = s.remove(ctx, s.url(repository, "manifests", holder), repository+"/manifests/"+holder)
if err == Gone {
return nil
}
return err
}
// archive reads a recorded reference into its repository and digest, and whether it is an archive
// at all. Refuses as ErrNotOurs anything the mesh did not put in its own store.
func (s Store) archive(reference string) (repository, digest string, archive bool, err error) {
path, kept := catalogue.InArtifactStore(reference)
if !kept {
return "", "", false, fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
if s.Address == "" {
return "", "", false, fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return "", "", false, err
}
return repository, digest, kind == "blobs", nil
}
// blobSize is how large the store says a blob is; Gone when it does not have it.
func (s Store) blobSize(ctx context.Context, repository, digest string) (int64, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodHead, s.url(repository, "blobs", digest), nil)
if err != nil {
return 0, err
}
response, err := s.client().Do(request)
if err != nil {
return 0, fmt.Errorf("cannot reach the artifact store at %s: %w", s.Address, err)
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
case http.StatusNotFound:
return 0, Gone
default:
return 0, fmt.Errorf("the artifact store answered %s for %s/blobs/%s", response.Status, repository, digest)
}
// Read from the header rather than ContentLength: a HEAD's ContentLength is what the response
// says it would have sent, which Go reports faithfully, but a proxy in between is free to drop
// it, and the header is what the registry itself wrote.
if length := response.Header.Get("Content-Length"); length != "" {
if n, err := strconv.ParseInt(length, 10, 64); err == nil && n >= 0 {
return n, nil
}
}
if response.ContentLength >= 0 {
return response.ContentLength, nil
}
return 0, fmt.Errorf("the artifact store holds %s/blobs/%s and will not say how large it is", repository, digest)
}
// putBlob uploads a small blob unless the repository already has it: ask where, then put it there
// naming the digest — the registry's own two steps, the same the builder takes for an archive.
func (s Store) putBlob(ctx context.Context, repository, digest string, body []byte) error {
if there, err := s.has(ctx, s.url(repository, "blobs", digest)); err != nil {
return err
} else if there {
return nil
}
start, err := http.NewRequestWithContext(ctx, http.MethodPost,
"http://"+s.Address+"/v2/"+repository+"/blobs/uploads/", nil)
if err != nil {
return err
}
begun, err := s.client().Do(start)
if err != nil {
return fmt.Errorf("cannot start an upload to %s: %w", repository, err)
}
begun.Body.Close()
if begun.StatusCode != http.StatusAccepted {
return fmt.Errorf("the artifact store answered %s when asked where to put a blob in %s", begun.Status, repository)
}
where := begun.Header.Get("Location")
if where == "" {
return fmt.Errorf("the artifact store accepted an upload to %s and said nowhere to put it", repository)
}
if strings.HasPrefix(where, "/") {
where = "http://" + s.Address + where
}
separator := "?"
if strings.Contains(where, "?") {
separator = "&"
}
put, err := http.NewRequestWithContext(ctx, http.MethodPut, where+separator+"digest="+digest, bytes.NewReader(body))
if err != nil {
return err
}
put.Header.Set("Content-Type", "application/octet-stream")
done, err := s.client().Do(put)
if err != nil {
return err
}
defer done.Body.Close()
if done.StatusCode != http.StatusCreated {
said, _ := io.ReadAll(io.LimitReader(done.Body, 4096))
return fmt.Errorf("the artifact store refused a blob in %s: %s %s", repository, done.Status, strings.TrimSpace(string(said)))
}
return nil
}
// has is whether the store answers 200 for a HEAD at that URL.
func (s Store) has(ctx context.Context, url string, accept ...string) (bool, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
if err != nil {
return false, err
}
for _, media := range accept {
request.Header.Add("Accept", media)
}
response, err := s.client().Do(request)
if err != nil {
return false, fmt.Errorf("cannot reach the artifact store at %s: %w", s.Address, err)
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
return true, nil
case http.StatusNotFound:
return false, nil
default:
return false, fmt.Errorf("the artifact store answered %s for %s", response.Status, url)
}
}
func (s Store) url(repository, kind, digest string) string {
return "http://" + s.Address + "/v2/" + repository + "/" + kind + "/" + digest
}
func (s Store) client() *http.Client {
if s.HTTP != nil {
return s.HTTP
}
return &http.Client{Timeout: 30 * time.Second}
}
+268
View File
@@ -0,0 +1,268 @@
package artifacts
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Every kept archive is held by a manifest (novox/hq issue 253, ADR 0189).
//
// Against an in-memory registry that keeps blobs and manifests per repository and refuses what a
// registry refuses — a blob whose digest does not match, a manifest whose digest does not match
// or whose blobs the repository does not have, a manifest asked for without an Accept naming its
// type. What is asserted is this side's decisions; the live test below asserts the registry's.
type memRegistry struct {
mu sync.Mutex
blobs map[string][]byte // repository + "@" + digest
manifests map[string][]byte // repository + "@" + digest
writes []string // every PUT and DELETE, as "METHOD path"
}
func digestOf(body []byte) string {
sum := sha256.Sum256(body)
return "sha256:" + hex.EncodeToString(sum[:])
}
func (m *memRegistry) serve(t *testing.T) Store {
t.Helper()
m.blobs = map[string][]byte{}
m.manifests = map[string][]byte{}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
m.mu.Lock()
defer m.mu.Unlock()
path := strings.TrimPrefix(r.URL.Path, "/v2/")
if r.Method == http.MethodPut || r.Method == http.MethodDelete {
m.writes = append(m.writes, r.Method+" "+r.URL.Path)
}
switch {
case r.Method == http.MethodPost && strings.HasSuffix(path, "/blobs/uploads/"):
repository := strings.TrimSuffix(path, "/blobs/uploads/")
w.Header().Set("Location", "/upload/"+repository+"?state=x")
w.WriteHeader(http.StatusAccepted)
case r.Method == http.MethodPut && strings.HasPrefix(r.URL.Path, "/upload/"):
repository := strings.TrimPrefix(r.URL.Path, "/upload/")
body, _ := io.ReadAll(r.Body)
digest := r.URL.Query().Get("digest")
if digest != digestOf(body) {
w.WriteHeader(http.StatusBadRequest)
return
}
m.blobs[repository+"@"+digest] = body
w.WriteHeader(http.StatusCreated)
case strings.Contains(path, "/blobs/"):
repository, digest, _ := strings.Cut(path, "/blobs/")
key := repository + "@" + digest
body, ok := m.blobs[key]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
switch r.Method {
case http.MethodHead:
w.Header().Set("Content-Length", strconv.Itoa(len(body)))
w.WriteHeader(http.StatusOK)
case http.MethodDelete:
delete(m.blobs, key)
w.WriteHeader(http.StatusAccepted)
default:
w.WriteHeader(http.StatusMethodNotAllowed)
}
case strings.Contains(path, "/manifests/"):
repository, digest, _ := strings.Cut(path, "/manifests/")
key := repository + "@" + digest
switch r.Method {
case http.MethodHead:
if _, ok := m.manifests[key]; !ok || !strings.Contains(r.Header.Get("Accept"), mediaManifest) {
w.WriteHeader(http.StatusNotFound)
return
}
w.WriteHeader(http.StatusOK)
case http.MethodPut:
body, _ := io.ReadAll(r.Body)
if digest != digestOf(body) {
w.WriteHeader(http.StatusBadRequest)
return
}
var named holderManifest
if err := json.Unmarshal(body, &named); err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
for _, d := range append([]descriptor{named.Config}, named.Layers...) {
if _, ok := m.blobs[repository+"@"+d.Digest]; !ok {
w.WriteHeader(http.StatusBadRequest)
fmt.Fprintf(w, "MANIFEST_BLOB_UNKNOWN %s", d.Digest)
return
}
}
m.manifests[key] = body
w.WriteHeader(http.StatusCreated)
case http.MethodDelete:
if _, ok := m.manifests[key]; !ok {
w.WriteHeader(http.StatusNotFound)
return
}
delete(m.manifests, key)
w.WriteHeader(http.StatusAccepted)
}
default:
w.WriteHeader(http.StatusNotFound)
}
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}
}
// bare puts an archive in the store the way the builder did before holders: a blob, nothing more.
func (m *memRegistry) bare(repository string, body []byte) string {
m.mu.Lock()
defer m.mu.Unlock()
digest := digestOf(body)
m.blobs[repository+"@"+digest] = body
return catalogue.ArtifactStoreScheme + repository + "/blobs/" + digest
}
func TestTheHolderIsComposedFromTheDigestAndSizeAlone(t *testing.T) {
// The sweep must arrive at the very manifest the builder wrote, with nothing recorded between
// them. Same inputs, same bytes — and a different size is a different holder, so a holder can
// never be mistaken for one of a different blob.
digest := "sha256:" + strings.Repeat("a", 64)
one, first := Holder(digest, 42)
two, second := Holder(digest, 42)
if !bytes.Equal(one, two) || first != second {
t.Fatalf("the same archive composed two holders:\n%s\n%s", one, two)
}
if _, other := Holder(digest, 43); other == first {
t.Fatal("a different size composed the same holder")
}
want := `{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json",` +
`"config":{"mediaType":"application/vnd.oci.empty.v1+json",` +
`"digest":"sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a","size":2},` +
`"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"` + digest + `","size":42}]}`
if string(one) != want {
t.Fatalf("the holder is\n%s\nwant\n%s", one, want)
}
if digestOf(emptyConfig) != emptyDigest {
t.Fatalf("the empty config's digest is %s, not %s", digestOf(emptyConfig), emptyDigest)
}
}
func TestHoldBackfillsABareArchiveAndIsIdempotent(t *testing.T) {
// The archives published before this have no holder. Hold, run over every kept archive on
// every sweep, writes one the first time and nothing after.
m := &memRegistry{}
store := m.serve(t)
ctx := context.Background()
body := []byte("a theme")
reference := m.bare("shell/config", body)
if held, err := store.Held(ctx, reference); err != nil || held {
t.Fatalf("a bare blob reads as held=%v (%v)", held, err)
}
wrote, err := store.Hold(ctx, reference)
if err != nil {
t.Fatal(err)
}
if !wrote {
t.Fatal("holding a bare archive wrote nothing")
}
_, holder := Holder(digestOf(body), int64(len(body)))
if _, ok := m.manifests["shell/config@"+holder]; !ok {
t.Fatalf("the store holds manifests %v; want %s", m.manifests, holder)
}
if held, err := store.Held(ctx, reference); err != nil || !held {
t.Fatalf("after holding, held=%v (%v)", held, err)
}
writes := len(m.writes)
wrote, err = store.Hold(ctx, reference)
if err != nil {
t.Fatal(err)
}
if wrote || len(m.writes) != writes {
t.Fatalf("holding again wrote %v", m.writes[writes:])
}
}
func TestAnImageNeedsNoHolderAndAMissingArchiveIsGone(t *testing.T) {
m := &memRegistry{}
store := m.serve(t)
ctx := context.Background()
// An image is its own manifest: nothing is asked.
wrote, err := store.Hold(ctx, catalogue.ArtifactStoreScheme+"web/app@sha256:"+strings.Repeat("b", 64))
if err != nil || wrote || len(m.writes) != 0 {
t.Fatalf("holding an image wrote=%v err=%v writes=%v", wrote, err, m.writes)
}
// An archive the store does not have is a fact to report, not something to invent a holder for.
_, err = store.Hold(ctx, catalogue.ArtifactStoreScheme+"web/config/blobs/sha256:"+strings.Repeat("c", 64))
if !errors.Is(err, Gone) {
t.Fatalf("holding a missing archive answered %v, want Gone", err)
}
// And a reference that is not the mesh's is refused as such.
if _, err := store.Hold(ctx, "docker.io/library/registry@sha256:abc"); !errors.Is(err, ErrNotOurs) {
t.Fatalf("holding a vendor's image answered %v, want ErrNotOurs", err)
}
}
func TestLettingGoOfAnArchiveDeletesItsHolderFirst(t *testing.T) {
// A holder left behind would keep the bytes through every collection while the record said
// collected; the link deleted first and the holder failing after would be that exactly.
m := &memRegistry{}
store := m.serve(t)
ctx := context.Background()
body := []byte("an old theme")
reference := m.bare("shell/config", body)
if _, err := store.Hold(ctx, reference); err != nil {
t.Fatal(err)
}
m.writes = nil
if err := store.LetGo(ctx, reference); err != nil {
t.Fatal(err)
}
_, holder := Holder(digestOf(body), int64(len(body)))
want := []string{
"DELETE /v2/shell/config/manifests/" + holder,
"DELETE /v2/shell/config/blobs/" + digestOf(body),
}
if strings.Join(m.writes, "\n") != strings.Join(want, "\n") {
t.Fatalf("the store was asked\n%s\nwant\n%s", strings.Join(m.writes, "\n"), strings.Join(want, "\n"))
}
if len(m.manifests) != 0 {
t.Fatalf("a holder survived: %v", m.manifests)
}
// Asked again, the archive is already gone, which is the outcome wanted.
if err := store.LetGo(ctx, reference); !errors.Is(err, Gone) {
t.Fatalf("letting go twice answered %v, want Gone", err)
}
}
func TestLettingGoOfAnUnheldArchiveStillDeletesIt(t *testing.T) {
// An archive published before holders and let go of before any sweep held it: the holder's
// delete answers 404, which is the outcome wanted, and the blob still goes.
m := &memRegistry{}
store := m.serve(t)
reference := m.bare("shell/config", []byte("never held"))
if err := store.LetGo(context.Background(), reference); err != nil {
t.Fatal(err)
}
if len(m.blobs) != 0 {
t.Fatalf("the blob survived: %v", m.blobs)
}
}
+130
View File
@@ -0,0 +1,130 @@
package artifacts
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The registry's own collector keeps a held archive and takes a bare one (novox/hq issue 253,
// ADR 0189).
//
// Everything else here is asserted against a fake, which can only say what this side asks. This is
// the one question a fake cannot answer — what `registry garbage-collect` actually does with what
// this side wrote — and it is the whole of whether the store's nightly step may stop being a dry
// run. Against the very image the mesh's store runs:
//
// docker run -d --rm --name mesh-controller-registry -p 15000:5000 \
// -e REGISTRY_STORAGE_DELETE_ENABLED=true registry:2.8.3
// MESH_TEST_REGISTRY=127.0.0.1:15000 MESH_TEST_REGISTRY_CONTAINER=mesh-controller-registry \
// go test -run Live ./internal/artifacts/
// docker stop mesh-controller-registry
//
// Skipped without both variables: it needs a registry it may write to and collect, and a container
// to run the collector in.
func TestLiveTheRegistrysCollectorKeepsWhatIsHeldAndTakesWhatIsNot(t *testing.T) {
address := os.Getenv("MESH_TEST_REGISTRY")
container := os.Getenv("MESH_TEST_REGISTRY_CONTAINER")
if address == "" || container == "" {
t.Skip("no MESH_TEST_REGISTRY / MESH_TEST_REGISTRY_CONTAINER; see this test's comment for the registry to raise")
}
ctx := context.Background()
store := Store{Address: address}
run := time.Now().UnixNano()
// Four archives in four repositories, each a different story. Distinct bytes per run, so a
// registry reused across runs cannot answer for an earlier one.
put := func(name string) (repository, digest string, body []byte) {
repository = fmt.Sprintf("live-%d/%s", run, name)
body = []byte(fmt.Sprintf("%s archive of run %d", name, run))
digest = digestOf(body)
if err := store.putBlob(ctx, repository, digest, body); err != nil {
t.Fatal(err)
}
return repository, digest, body
}
reference := func(repository, digest string) string {
return catalogue.ArtifactStoreScheme + repository + "/blobs/" + digest
}
// Published held — what PublishArchive now does.
heldRepo, heldDigest, heldBody := put("held")
if _, err := store.HoldBlob(ctx, heldRepo, heldDigest, int64(len(heldBody))); err != nil {
t.Fatalf("the registry refused a holder: %v", err)
}
// Published bare, as before, and never held: what the collector must take.
_, bareDigest, _ := put("bare")
// Published bare and then held by the sweep: the backfill.
backRepo, backDigest, backBody := put("backfilled")
if wrote, err := store.Hold(ctx, reference(backRepo, backDigest)); err != nil || !wrote {
t.Fatalf("backfilling wrote=%v: %v", wrote, err)
}
if held, err := store.Held(ctx, reference(backRepo, backDigest)); err != nil || !held {
t.Fatalf("after backfilling, held=%v: %v", held, err)
}
// Held, and then let go of by the sweep: holder first, then the link.
goneRepo, goneDigest, _ := put("let-go")
if _, err := store.Hold(ctx, reference(goneRepo, goneDigest)); err != nil {
t.Fatal(err)
}
if err := store.LetGo(ctx, reference(goneRepo, goneDigest)); err != nil {
t.Fatalf("letting go of a held archive: %v", err)
}
collected, err := exec.CommandContext(ctx, "docker", "exec", container,
"registry", "garbage-collect", "/etc/docker/registry/config.yml").CombinedOutput()
if err != nil {
t.Fatalf("the collector failed: %v\n%s", err, collected)
}
t.Logf("the collector said:\n%s", lastLines(string(collected), 12))
// What is asserted is the bytes on the store's disk, not what the running server answers: the
// server caches blob descriptors in memory and can answer for a blob the collector removed.
onDisk := func(digest string) bool {
hex := strings.TrimPrefix(digest, "sha256:")
path := "/var/lib/registry/docker/registry/v2/blobs/sha256/" + hex[:2] + "/" + hex + "/data"
return exec.CommandContext(ctx, "docker", "exec", container, "test", "-f", path).Run() == nil
}
if !onDisk(heldDigest) {
t.Error("the collector took an archive published held")
}
if !onDisk(backDigest) {
t.Error("the collector took an archive the sweep backfilled a holder for")
}
if onDisk(bareDigest) {
t.Error("the collector kept a bare archive — then the holders prove nothing, and this test is wrong")
}
if onDisk(goneDigest) {
t.Error("the collector kept an archive the sweep let go of: its holder outlived its link")
}
// And what survived is still fetched exactly as machines fetch it: the blob, by digest.
for repository, want := range map[string][]byte{heldRepo: heldBody, backRepo: backBody} {
digest := digestOf(want)
response, err := http.Get(catalogue.Routed(reference(repository, digest), address))
if err != nil {
t.Fatal(err)
}
got, _ := io.ReadAll(response.Body)
response.Body.Close()
if response.StatusCode != http.StatusOK || !bytes.Equal(got, want) {
t.Errorf("%s answered %s with %q after collection", repository, response.Status, got)
}
}
}
func lastLines(s string, n int) string {
lines := strings.Split(strings.TrimSpace(s), "\n")
if len(lines) > n {
lines = lines[len(lines)-n:]
}
return strings.Join(lines, "\n")
}
+38 -14
View File
@@ -1,17 +1,18 @@
// Package artifacts speaks to the mesh's artifact store over its own door.
//
// Only what the mesh needs that nothing else does: letting go of something it put there
// (novox/hq ADR 0189, issue 108). Pushing is the builder's, through the container runtime; reading
// (novox/hq ADR 0189, issue 108), and holding every archive it keeps by a manifest so the store's
// own collector does not take it (novox/hq issue 253). Pushing is the builder's, through the container runtime; reading
// is every machine's, through its runtime. This is the one operation that belongs to the thing
// holding the records, because it is the only one that is a decision rather than a transfer.
package artifacts
import (
"context"
"errors"
"fmt"
"net/http"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
@@ -26,7 +27,15 @@ type Store struct {
}
// Gone is the answer when the store does not hold it: the outcome wanted, already true.
var Gone = fmt.Errorf("the store does not hold it")
var Gone = errors.New("the store does not hold it")
// ErrNotOurs is a reference this sweep will not address: not the mesh's own, or naming nothing
// the store holds by digest.
//
// **A fact about the record, not about the store** (novox/hq issue 226). The two deserve opposite
// responses — skip one and go on, abandon the sweep for the other — and collapsing them into "an
// error" is how a cautious loop became one that did nothing while reporting the right number.
var ErrNotOurs = errors.New("not a reference into the mesh's artifact store")
// LetGo asks the store to drop one artifact the mesh recorded making.
//
@@ -34,16 +43,24 @@ var Gone = fmt.Errorf("the store does not hold it")
// an image, `…/blobs/sha256:…` for an archive — because that is the identity every record uses,
// and composes the address here at the moment of use.
//
// An archive is let go of in two deletes, its holder manifest and then the blob's link (novox/hq
// issue 253); an image in one.
//
// Returns Gone when the store answers that it does not have it. That is not a failure: the sweep
// wants the artifact absent, and it is. It is distinguished from success only so a caller can say
// which of the two happened.
func (s Store) LetGo(ctx context.Context, reference string) error {
// **Strict, and deliberately** (novox/hq issue 226). Only a reference the mesh keeps in its
// own vocabulary is addressed here. `Recorded` would read `docker.io/library/registry@sha256:…`
// as the mesh's too — it cannot tell one registry host from another — so normalising belongs
// where the provenance is known, which is the sweep reading its own build records, not here
// where the only job is to refuse anything that is not plainly ours.
path, kept := catalogue.InArtifactStore(reference)
if !kept {
// Nothing the mesh put in its own store. Refused rather than attempted: composing a
// delete for a reference of unknown shape is how a sweep reaches something that is not
// the mesh's.
return fmt.Errorf("%s is not a reference into the mesh's artifact store", reference)
// the mesh's. Distinguished from a store that refuses, so a sweep skips this and goes on.
return fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
if s.Address == "" {
return fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
@@ -52,17 +69,24 @@ func (s Store) LetGo(ctx context.Context, reference string) error {
if err != nil {
return err
}
url := "http://" + s.Address + "/v2/" + repository + "/" + kind + "/" + digest
if kind == "blobs" {
// **An archive's holder goes before the archive** (novox/hq issue 253): a manifest left
// naming the blob would keep its bytes through every collection while the record said
// collected. Gone here means the store has no such blob, so there is nothing to let go.
if err := s.letGoOfHolder(ctx, repository, digest); err != nil {
return err
}
}
return s.remove(ctx, s.url(repository, kind, digest), reference)
}
// remove asks the store to delete what is at url. Gone when it has no such thing.
func (s Store) remove(ctx context.Context, url, what string) error {
request, err := http.NewRequestWithContext(ctx, http.MethodDelete, url, nil)
if err != nil {
return err
}
client := s.HTTP
if client == nil {
client = &http.Client{Timeout: 30 * time.Second}
}
response, err := client.Do(request)
response, err := s.client().Do(request)
if err != nil {
return err
}
@@ -78,9 +102,9 @@ func (s Store) LetGo(ctx context.Context, reference string) error {
return fmt.Errorf(
"the artifact store refuses deletion: its server was started without it enabled "+
"(REGISTRY_STORAGE_DELETE_ENABLED), so nothing can be collected until the store "+
"module is applied again (novox/hq ADR 0189). Asking about %s", reference)
"module is applied again (novox/hq ADR 0189). Asking about %s", what)
default:
return fmt.Errorf("the artifact store answered %s for %s", response.Status, reference)
return fmt.Errorf("the artifact store answered %s for %s", response.Status, what)
}
}
@@ -95,5 +119,5 @@ func split(path string) (repository, kind, digest string, err error) {
if before, after, ok := strings.Cut(path, "/blobs/sha256:"); ok {
return before, "blobs", "sha256:" + after, nil
}
return "", "", "", fmt.Errorf("%q names nothing the store holds by digest", path)
return "", "", "", fmt.Errorf("%w: %q names nothing the store holds by digest", ErrNotOurs, path)
}
+42 -2
View File
@@ -20,6 +20,17 @@ func fakeStore(t *testing.T, answer int) (Store, *[]string) {
t.Helper()
var asked []string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/") {
// An archive's size, asked so its holder can be named (novox/hq issue 253). A store
// that does not have the thing does not have its blob either.
if answer == http.StatusNotFound {
w.WriteHeader(http.StatusNotFound)
return
}
w.Header().Set("Content-Length", "7")
w.WriteHeader(http.StatusOK)
return
}
if r.Method != http.MethodDelete {
t.Errorf("the store was asked %s %s; collecting is a delete", r.Method, r.URL.Path)
}
@@ -45,8 +56,14 @@ func TestAnImageAndAnArchiveAreAskedForAtTheirOwnEndpoints(t *testing.T) {
if err := store.LetGo(ctx, archive); err != nil {
t.Fatal(err)
}
want := []string{"/v2/web/app/manifests/sha256:abc123", "/v2/web/config/blobs/sha256:def456"}
if len(*asked) != 2 || (*asked)[0] != want[0] || (*asked)[1] != want[1] {
// The archive's holder goes first, then the archive (novox/hq issue 253).
_, holder := Holder("sha256:def456", 7)
want := []string{
"/v2/web/app/manifests/sha256:abc123",
"/v2/web/config/manifests/" + holder,
"/v2/web/config/blobs/sha256:def456",
}
if strings.Join(*asked, " ") != strings.Join(want, " ") {
t.Fatalf("the store was asked %v; want %v", *asked, want)
}
}
@@ -92,3 +109,26 @@ func TestAReferenceThatIsNotTheMeshsOwnIsNeverAsked(t *testing.T) {
t.Fatalf("the store was asked about %v", *asked)
}
}
// A reference this sweep will not address says so as ErrNotOurs, which is a fact about the
// record and not about the store (novox/hq issue 226).
//
// The sweep skips one and abandons itself for the other, so they cannot be the same error. The
// first live run met a reference recorded with the store's old address, read the refusal as "the
// store refuses everything", and collected none of the 1681 it had found.
func TestAReferenceThisSweepWillNotAddressIsToldApartFromAStoreRefusing(t *testing.T) {
store, asked := fakeStore(t, http.StatusAccepted)
for _, reference := range []string{
"docker.io/library/registry@sha256:abc123",
"127.0.0.1:5100/mesh-tools/build@sha256:abc123",
"1.4.2",
} {
err := store.LetGo(context.Background(), reference)
if !errors.Is(err, ErrNotOurs) {
t.Errorf("%s answered %v; a sweep must be able to skip it and go on", reference, err)
}
}
if len(*asked) != 0 {
t.Fatalf("the store was asked about %v", *asked)
}
}
+92
View File
@@ -0,0 +1,92 @@
package broker
import (
"context"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go/jetstream"
)
// A seat's cancelled set (novox/hq ADR 0219).
//
// **Cancelling an ask is deleting its message from the seat's work queue** — and that alone has a
// race no ordering on one side closes: a holder may fetch the ask in the moment between the
// controller reading the queue and deleting the message, and build what a person cancelled. So the
// controller first writes the ask's id into the seat's cancelled set, and a holder looks its ask up
// there after taking it and before building: listed, it terminates the ask and announces it failed
// rather than starting it.
//
// **A key-value bucket, because that is the shape the bus already has for "a small set the
// controller writes and many read cheaply"** (ADR 0201's state buckets): one direct read by key per
// ask taken — no consumer, no subscription a holder must keep, nothing that grows a holder's grants
// beyond one read subject. Kept beside the seat's own stream and worker and named like them, so the
// three objects of one work queue read as one family; asserted by the controller with the queue,
// and aged out with it — an id cancelled a week ago names an ask the queue no longer holds.
// CancelledSetName is the bucket holding a seat's cancelled asks.
func CancelledSetName(seat string) string { return "SEAT_" + upperSnake(seat) + "_cancelled" }
// hasCancelledSet is whether a seat's queue can be cancelled from: the work queues the controller
// asks, whose asks it alone shows and changes. A module's own seat's queue is that module's affair.
func hasCancelledSet(seat string) bool {
for _, s := range seatsTheControllerAsks {
if s == seat {
return true
}
}
return false
}
// IsCancelledSet says a bucket is a seat's cancelled set rather than a module's state — the mesh's
// own, and never one to report as state nothing declares.
func IsCancelledSet(bucket string) bool {
return strings.HasPrefix(bucket, "SEAT_") && strings.HasSuffix(bucket, "_cancelled")
}
// cancelledSetAge is how long a cancelled id is kept: as long as the seat's queue keeps an ask.
const cancelledSetAge = 7 * 24 * time.Hour
// A CancelledSetAsserter is what raising the cancelled sets needs of a connection.
type CancelledSetAsserter interface {
EnsureCancelledSet(seat string) error
}
// RaiseCancelledSets asserts the cancelled set of every work queue the controller asks.
func RaiseCancelledSets(a CancelledSetAsserter, seats []DeclaredSeat) error {
for _, s := range seats {
if len(s.Accepts) == 0 || !hasCancelledSet(s.Name) {
continue
}
if err := a.EnsureCancelledSet(s.Name); err != nil {
return fmt.Errorf("asserting the cancelled set of %s: %w", s.Name, err)
}
}
return nil
}
// EnsureCancelledSet creates a seat's cancelled set if absent and brings its options to match.
// Direct reads on, which is how a holder looks an id up with one request.
func (j *JetStream) EnsureCancelledSet(seat string) error {
js, err := jetstream.New(j.conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: CancelledSetName(seat),
Description: fmt.Sprintf("the asks of the %s seat cancelled by hand (novox/hq ADR 0219): written "+
"by the controller before it deletes an ask from the queue, read by a holder on taking one, "+
"so an ask fetched in that moment is ended rather than built", seat),
History: 1,
TTL: cancelledSetAge,
MaxValueSize: 4 * 1024,
MaxBytes: 4 * 1024 * 1024,
Storage: jetstream.FileStorage,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", CancelledSetName(seat), err)
}
return nil
}
+73
View File
@@ -0,0 +1,73 @@
package broker
import "testing"
// A build agent reads its seat's cancelled set by key and nothing more, answers its verbs on its own
// machine's subjects, and says whether it is paused under its own machine's name; the controller may
// ask any machine's holder its verbs (novox/hq ADR 0219).
func TestTheBuildQueueIsControlledWithTheGrantsItNeedsAndNoMore(t *testing.T) {
seat := Seat{Name: "node-build-agent", Scope: "node", Accepts: []string{"build"},
Emits: []string{"started", "built", "log.*", "paused.*"},
Serves: []string{"current", "kill", "pause", "resume"}}
holder, err := PermissionsFor(Principal{Kind: KindModule, Node: "ace", Module: "build-agent",
Holds: []Seat{seat}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
has(t, holder.Publish, "$JS.API.DIRECT.GET.KV_SEAT_NODE_BUILD_AGENT_cancelled.$KV.SEAT_NODE_BUILD_AGENT_cancelled.>")
hasNot(t, holder.Publish, "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>")
has(t, holder.Publish, "mesh.seat.node-build-agent.event.paused.ace")
hasNot(t, holder.Publish, "mesh.seat.node-build-agent.event.paused.*")
has(t, holder.Publish, "mesh.seat.node-build-agent.event.log.*")
has(t, holder.Subscribe, "mesh.seat.node-build-agent.tool.kill.ace")
hasNot(t, holder.Subscribe, "mesh.seat.node-build-agent.tool.kill.g14")
// A module's own seat's queue is its own affair: no cancelled set, no grant for one.
other, _ := PermissionsFor(Principal{Kind: KindModule, Node: "ace", Module: "telegram",
Holds: []Seat{{Name: "telegram-sender", Accepts: []string{"send"}}}, PasswordHash: "x"})
hasNot(t, other.Publish, "$JS.API.DIRECT.GET.KV_SEAT_TELEGRAM_SENDER_cancelled.$KV.SEAT_TELEGRAM_SENDER_cancelled.>")
controller, _ := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
has(t, controller.Publish, "mesh.seat.node-build-agent.tool.>")
if CancelledSetName("node-build-agent") != "SEAT_NODE_BUILD_AGENT_cancelled" ||
!IsCancelledSet("SEAT_NODE_BUILD_AGENT_cancelled") || IsCancelledSet("build-agent_cancelled") {
t.Error("the cancelled set is not named as its seat's family")
}
}
// Only the work queues the controller asks get a cancelled set.
func TestOnlyTheControllersQueuesHaveACancelledSet(t *testing.T) {
var asserted []string
a := asserterFunc(func(seat string) error { asserted = append(asserted, seat); return nil })
if err := RaiseCancelledSets(a, []DeclaredSeat{
{Name: "node-build-agent", Accepts: []string{"build"}},
{Name: "telegram-sender", Accepts: []string{"send"}},
{Name: "mesh-controller"},
}); err != nil {
t.Fatal(err)
}
if len(asserted) != 1 || asserted[0] != "node-build-agent" {
t.Fatalf("asserted %v", asserted)
}
}
type asserterFunc func(string) error
func (f asserterFunc) EnsureCancelledSet(seat string) error { return f(seat) }
// A seat's cancelled set is never reported as state nothing declares.
func TestACancelledSetIsNotUndeclaredState(t *testing.T) {
undeclared, err := RaiseBuckets(fakeBuckets{names: []string{"SEAT_NODE_BUILD_AGENT_cancelled", "gone_state"}}, nil)
if err != nil {
t.Fatal(err)
}
if len(undeclared) != 1 || undeclared[0] != "gone_state" {
t.Fatalf("undeclared %v", undeclared)
}
}
type fakeBuckets struct{ names []string }
func (f fakeBuckets) EnsureBucket(Bucket) error { return nil }
func (f fakeBuckets) BucketNames() ([]string, error) { return f.names, nil }
+91
View File
@@ -0,0 +1,91 @@
package broker
import (
"fmt"
"os"
"testing"
"time"
"github.com/nats-io/nats.go"
)
// A consumer that reacts to announcements, made on a stream that keeps a week of them, starts from now:
// made from the start it replays every merge and build of that week (novox/hq issue 248). And a reset
// re-makes a stuck one from now, its configuration otherwise kept, and refuses a work queue.
//
// docker run -d --rm --name t -p 14231:4222 nats:2.10-alpine -js
// MESH_TEST_NATS=nats://127.0.0.1:14231 go test ./internal/broker/ -run TestAConsumerMadeFromNow
func TestAConsumerMadeFromNowNeverReplaysTheStreamsHistory(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
js, err := Dial(url)
if err != nil {
t.Fatal(err)
}
defer js.Close()
const stream = "HISTORY_TEST"
_ = js.js.DeleteStream(stream)
if _, err := js.js.AddStream(&nats.StreamConfig{Name: stream, Subjects: []string{"history.>"}, Storage: nats.MemoryStorage}); err != nil {
t.Fatal(err)
}
defer func() { _ = js.js.DeleteStream(stream) }()
for i := 0; i < 5; i++ {
if _, err := js.js.Publish("history.merged", []byte(fmt.Sprint(i))); err != nil {
t.Fatal(err)
}
}
fresh := Consumer{Name: "fresh", Stream: stream, Filters: []string{"history.merged"}, Push: true,
AckWaitSeconds: 30, MaxDeliver: 5, MaxAckPending: 1, FromNow: true}
if err := js.EnsureConsumer(fresh); err != nil {
t.Fatal(err)
}
info, _ := js.js.ConsumerInfo(stream, "fresh")
if info.NumPending != 0 || info.Config.DeliverPolicy != nats.DeliverNewPolicy {
t.Fatalf("a consumer made from now holds %d of the stream's past (policy %v)", info.NumPending, info.Config.DeliverPolicy)
}
_, _ = js.js.Publish("history.merged", []byte("new"))
time.Sleep(100 * time.Millisecond)
if info, _ = js.js.ConsumerInfo(stream, "fresh"); info.NumPending != 1 {
t.Fatalf("a new announcement is not pending: %d", info.NumPending)
}
// Asserted again, it keeps where it is.
if err := js.EnsureConsumer(fresh); err != nil {
t.Fatal(err)
}
// One made from the start, as the server's default makes it, and stuck behind its history.
old := fresh
old.Name, old.FromNow = "old", false
if err := js.EnsureConsumer(old); err != nil {
t.Fatal(err)
}
if info, _ = js.js.ConsumerInfo(stream, "old"); info.NumPending != 6 {
t.Fatalf("the default should replay all six: %d", info.NumPending)
}
before, after, err := js.ResetConsumer(stream, "old")
if err != nil {
t.Fatal(err)
}
info, _ = js.js.ConsumerInfo(stream, "old")
if before.Pending != 6 || after.Pending != 0 || info.Config.MaxAckPending != 1 || info.Config.MaxDeliver != 5 ||
info.Config.AckWait != 30*time.Second || info.Config.DeliverSubject == "" {
t.Fatalf("before %+v after %+v config %+v", before, after, info.Config)
}
// Never a work queue: what is pending there is work.
const queue = "QUEUE_TEST"
_ = js.js.DeleteStream(queue)
if _, err := js.js.AddStream(&nats.StreamConfig{Name: queue, Subjects: []string{"queue.>"}, Retention: nats.WorkQueuePolicy, Storage: nats.MemoryStorage}); err != nil {
t.Fatal(err)
}
defer func() { _ = js.js.DeleteStream(queue) }()
if _, err := js.js.AddConsumer(queue, &nats.ConsumerConfig{Durable: "w", AckPolicy: nats.AckExplicitPolicy}); err != nil {
t.Fatal(err)
}
if _, _, err := js.ResetConsumer(queue, "w"); err == nil {
t.Fatal("a work queue's consumer was reset")
}
}
+45 -1
View File
@@ -47,7 +47,13 @@ type Consumer struct {
// and a merge that came back rebuilt what it had just built, five times over on 2026-09-30.
// With one outstanding, the server holds the rest, and the heartbeat is keeping the message.
MaxAckPending int
Why string
// FromNow makes a consumer that does not exist yet start at the stream's end rather than its
// beginning (novox/hq issue 248). For a consumer that reacts to announcements — a merge, a build's
// outcome — on a stream that keeps a week of them: the server's default, everything the stream
// holds, replays every merge and every build of that week as if it had just happened. A consumer
// that exists keeps where it is, whatever this says; only its making is decided here.
FromNow bool
Why string
}
// seatStreamName is the stream holding a seat's inbound work. Named after the seat rather than
@@ -144,6 +150,44 @@ func ConsumerFor(p Principal) (Consumer, bool) {
}, true
}
// ModuleConsumer is one module's durable consumer, with the module and node it is for.
type ModuleConsumer struct {
Node, Module string
Consumer Consumer
}
// ConsumersOf is every module's durable consumer the composed users imply: each module user's, and
// each module a runtime carries — a carried module with no account of its own is no user (novox/hq
// issue 195), and its consumer is still the controller's to make, since the runtime reads it on the
// module's behalf (ADR 0198). One per module and node, whichever of the two named it first.
func ConsumersOf(users []Principal) []ModuleConsumer {
var out []ModuleConsumer
seen := map[string]bool{}
add := func(p Principal) {
c, needed := ConsumerFor(p)
if !needed || seen[p.Node+"/"+p.Module] {
return
}
seen[p.Node+"/"+p.Module] = true
out = append(out, ModuleConsumer{Node: p.Node, Module: p.Module, Consumer: c})
}
for _, p := range users {
if p.Kind == KindModule {
add(p)
}
}
for _, p := range users {
if p.Kind != KindNodeTools {
continue
}
for _, d := range p.Carries {
add(Principal{Kind: KindModule, Node: p.Node, Module: d.Module, Emits: d.Emits,
Consumes: d.Consumes, Serves: d.Serves, Holds: d.Holds, Uses: d.Uses, Watches: d.Watches})
}
}
return out
}
// HolderConsumerFor is the worker a seat's holders share on that seat's work queue.
//
// **One worker for every holder, and each holder pulls one ask when it is idle** (novox/hq ADR
+55
View File
@@ -86,6 +86,13 @@ func pinnedTo(path string) (*tls.Config, error) {
return PinnedToFingerprint(want), nil
}
// OnConn is the JetStream handle over a connection the caller already holds — the control plane's
// link — for asserting what the bus holds without dialling a second time.
func OnConn(conn *nats.Conn) *JetStream {
js, _ := conn.JetStream()
return &JetStream{conn: conn, js: js}
}
// DialPinned is Dial with the server's certificate pinned by a fingerprint the caller already holds
// — a module or a build machine that was handed one beside its credential, and has no file.
func DialPinned(url, fingerprint string, opts ...nats.Option) (*JetStream, error) {
@@ -301,6 +308,9 @@ func (j *JetStream) EnsureConsumer(c Consumer) error {
}
return nil
case errors.Is(err, nats.ErrConsumerNotFound):
if c.FromNow {
want.DeliverPolicy = nats.DeliverNewPolicy
}
if _, err := j.js.AddConsumer(c.Stream, want); err != nil {
return fmt.Errorf("creating consumer %s on %s: %w", c.Name, c.Stream, err)
}
@@ -310,6 +320,51 @@ func (j *JetStream) EnsureConsumer(c Consumer) error {
}
}
// ConsumerState is what a reset says about a consumer, before and after.
type ConsumerState struct {
DeliverPolicy string
Delivered uint64
AckFloor uint64
Pending uint64
AckPending int
}
func stateOf(info *nats.ConsumerInfo) ConsumerState {
policy, _ := info.Config.DeliverPolicy.MarshalJSON()
return ConsumerState{DeliverPolicy: strings.Trim(string(policy), `"`), Delivered: info.Delivered.Stream,
AckFloor: info.AckFloor.Stream, Pending: info.NumPending, AckPending: info.NumAckPending}
}
// ResetConsumer re-makes a consumer to start from now, its configuration otherwise unchanged (novox/hq
// issue 248): what it had not yet delivered or acknowledged is dropped, which is the point — on a stream
// that keeps history, a consumer replaying a week of announcements does nothing anyone wants. Refused on
// a work queue, where what is pending is work nobody else will do.
func (j *JetStream) ResetConsumer(stream, name string) (before, after ConsumerState, err error) {
info, err := j.js.StreamInfo(stream)
if err != nil {
return before, after, fmt.Errorf("asking about stream %s: %w", stream, err)
}
if info.Config.Retention == nats.WorkQueuePolicy {
return before, after, fmt.Errorf("%s is a work queue: what its consumer has pending is work, and a reset would drop it", stream)
}
have, err := j.js.ConsumerInfo(stream, name)
if err != nil {
return before, after, fmt.Errorf("asking about consumer %s on %s: %w", name, stream, err)
}
before = stateOf(have)
want := have.Config
want.DeliverPolicy = nats.DeliverNewPolicy
want.OptStartSeq, want.OptStartTime = 0, nil
if err := j.js.DeleteConsumer(stream, name); err != nil {
return before, after, fmt.Errorf("removing consumer %s on %s: %w", name, stream, err)
}
made, err := j.js.AddConsumer(stream, &want)
if err != nil {
return before, after, fmt.Errorf("re-making consumer %s on %s — it is gone until the controller asserts it at its next start: %w", name, stream, err)
}
return before, stateOf(made), nil
}
func retentionOf(r Retention) nats.RetentionPolicy {
switch r {
case RetentionWorkQueue:
+36 -3
View File
@@ -18,6 +18,7 @@ import (
"regexp"
"sort"
"strings"
"time"
)
// A Kind is what a principal is, which decides the shape of its authority rather than its
@@ -124,6 +125,10 @@ type Principal struct {
// goes with the retired seat row.
var seatsTheControllerAsks = []string{"node-build-agent", "mesh-build-machine"}
// perMachineEvents are a node-scoped seat's events about the holder itself, whose last token is the
// holder's machine (novox/hq ADR 0219): `paused.<node>`, the build agent saying whether it takes work.
var perMachineEvents = map[string]bool{"paused.*": true}
// enrolmentPrefix is the space every enrolling node's user and inbox live under, so the one place the
// controller may answer an enrolment is derived from the same constant the user is named from.
const enrolmentPrefix = "enrol"
@@ -185,6 +190,13 @@ type Permissions struct {
AllowResponses bool
}
// ResponseTTL is how long the bus lets a principal answer a request it received. Its one answer has
// to come inside this, and a seat's holder answers within link.AnswerWithin — inside it by design.
// **A broker reloading its user list forgets every answer it was about to permit**, whatever this
// says (novox/hq issue 265): a call that is still running when the list reloads has its answer
// refused, which is why a holder answers before it does what can reload it.
const ResponseTTL = time.Minute
// PermissionsFor derives a principal's authority. Pure, and the only place authority is decided:
// a permission that cannot be derived from a declaration is a permission nobody can explain.
func PermissionsFor(p Principal) (Permissions, error) {
@@ -221,6 +233,10 @@ func PermissionsFor(p Principal) (Permissions, error) {
// the role, and whichever machine holding it is idle takes it.
for _, seat := range seatsTheControllerAsks {
pub = append(pub, "mesh.seat."+seat+".accept.>")
// **And its holders' verbs, on every machine** (novox/hq ADR 0219): what a holder is
// building, kill it, pause it, resume it. The queue is the controller's to show and to
// change, and what one machine is doing with an ask it took only that machine can say.
pub = append(pub, "mesh.seat."+seat+".tool.>")
}
// **And what the mesh says it did** (novox/hq ADR 0134). The control plane states its own
// facts under the seat it holds, because a role's events belong to the role and keep their
@@ -245,10 +261,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
// And says so (novox/hq ADR 0197): it answers discovery for the seat it serves.
sub = append(sub, announcing(ControllerSeat)...)
// The two events it reacts to, and its ack subject on the stream they arrive from
// The events it reacts to, and its ack subject on the stream they arrive from
// (streams.go). **Each named, not a pattern**: `mesh.mod.*.event.>` would make the
// controller a subscriber to every event in the mesh, and its permission list would stop
// saying what it is for. The ack grant below is scoped per stream because the controller's
// saying what it is for. The one wildcard is the emitter of a provider's standing (ADR
// 0224) — still two named events, from whichever module provides. The ack grant below is scoped per stream because the controller's
// consumer name is the same on both and `$JS.ACK.CONTROL.controller.>` does not cover a
// delivery from EVENTS — a consumer that cannot ack has every message redelivered for
// ever, refused by the list it already has.
@@ -404,10 +421,26 @@ func PermissionsFor(p Principal) (Permissions, error) {
"$JS.API.CONSUMER.INFO."+stream+"."+worker,
"$JS.API.CONSUMER.MSG.NEXT."+stream+"."+worker,
"$JS.ACK."+stream+"."+worker+".>")
// **And whether an ask it took was cancelled** (novox/hq ADR 0219): one key of the
// seat's cancelled set, read directly by its id, so a holder that fetched an ask in the
// moment the controller cancelled it ends it instead of building it. Read, never written:
// the set is the controller's, and only the work queues the controller asks — and so may
// cancel from — have one.
if hasCancelledSet(s.Name) {
set := CancelledSetName(s.Name)
pub = append(pub, "$JS.API.DIRECT.GET.KV_"+set+".$KV."+set+".>")
}
for _, a := range s.Accepts {
sub = append(sub, seatSubject(s, "accept", a))
}
for _, e := range s.Emits {
// **A machine says its own state and no other's** (novox/hq ADR 0219): on a node-scoped
// seat, an event about the holder itself carries the machine as its last token, and
// each holder is granted its own machine's alone.
if s.Scope == "node" && p.Node != "" && perMachineEvents[e] {
pub = append(pub, seatSubject(s, "event", strings.TrimSuffix(e, "*")+p.Node))
continue
}
pub = append(pub, seatSubject(s, "event", e))
}
for _, t := range s.Serves {
@@ -753,7 +786,7 @@ func ComposeAccounts(principals []Principal) (string, error) {
fmt.Fprintf(&b, " publish: { allow: [%s] }\n", quoted(perms.Publish))
fmt.Fprintf(&b, " subscribe: { allow: [%s] }\n", quoted(perms.Subscribe))
if perms.AllowResponses {
b.WriteString(" allow_responses: { max: 1, ttl: \"1m\" }\n")
fmt.Fprintf(&b, " allow_responses: { max: 1, ttl: \"%dm\" }\n", int(ResponseTTL/time.Minute))
}
b.WriteString(" } }\n")
}
+8 -8
View File
@@ -5,16 +5,16 @@ import "testing"
// A node-scoped seat's tool carries the node (novox/hq ADR 0132, design 33 §4): two nodes holding one
// node-scoped seat derive two addresses, and a user of the seat may publish any node's.
func TestTwoNodesHoldingOneNodeSeatDeriveTwoToolAddresses(t *testing.T) {
seat := Seat{Name: "node-dns-resolver", Scope: "node", Serves: []string{"lookup"}}
one, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "dnsmasq", Holds: []Seat{seat}, PasswordHash: "x"})
two, _ := PermissionsFor(Principal{Kind: KindModule, Node: "two", Module: "dnsmasq", Holds: []Seat{seat}, PasswordHash: "x"})
has(t, one.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup.one")
has(t, two.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup.two")
hasNot(t, one.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup")
hasNot(t, one.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup.two")
seat := Seat{Name: "node-hostname", Scope: "node", Serves: []string{"entries"}}
one, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "hostname", Holds: []Seat{seat}, PasswordHash: "x"})
two, _ := PermissionsFor(Principal{Kind: KindModule, Node: "two", Module: "hostname", Holds: []Seat{seat}, PasswordHash: "x"})
has(t, one.Subscribe, "mesh.seat.node-hostname.tool.entries.one")
has(t, two.Subscribe, "mesh.seat.node-hostname.tool.entries.two")
hasNot(t, one.Subscribe, "mesh.seat.node-hostname.tool.entries")
hasNot(t, one.Subscribe, "mesh.seat.node-hostname.tool.entries.two")
user, _ := PermissionsFor(Principal{Kind: KindModule, Node: "three", Module: "asker", Uses: []Seat{seat}, PasswordHash: "x"})
has(t, user.Publish, "mesh.seat.node-dns-resolver.tool.lookup.*")
has(t, user.Publish, "mesh.seat.node-hostname.tool.entries.*")
}
// A mesh-scoped seat's tool stays flat: nothing about it changes.
+2 -1
View File
@@ -160,7 +160,8 @@ func RaiseBuckets(a BucketAsserter, buckets []Bucket) (undeclared []string, err
return nil, fmt.Errorf("listing the bus's state: %w", err)
}
for _, n := range names {
if !declared[n] {
// A seat's cancelled set is the mesh's own (novox/hq ADR 0219), not a module's state.
if !declared[n] && !IsCancelledSet(n) {
undeclared = append(undeclared, n)
}
}
+14
View File
@@ -164,3 +164,17 @@ func TestABucketIsAssertedInPlace(t *testing.T) {
}
}
}
// Against a real server: the handle over a connection the control plane already holds asserts a
// bucket as Dial's does — what a push uses, so a module registered since the last start has its
// bucket before its membership names it.
func TestABucketIsAssertedOverAHeldConnection(t *testing.T) {
js := aLiveBus(t)
held := OnConn(js.Conn())
if _, err := RaiseBuckets(held, []Bucket{{Module: "statetest", Name: "held"}}); err != nil {
t.Fatalf("asserting over a held connection failed: %v", err)
}
if _, err := js.Context().KeyValue("statetest_held"); err != nil {
t.Fatalf("the bucket is not there: %v", err)
}
}
+17 -1
View File
@@ -226,8 +226,23 @@ var ControllerFollows = []string{
// registers build-agent itself comes from there. Appended, for the same reason as above; goes
// with the retired seat row.
seatEventSubject("mesh-build-machine", "built"),
// **Every provider's standing** (novox/hq ADR 0224): a consumer it has failed for minutes, and
// that consumer recovered. The one pattern on this list, and a narrow one — two named events,
// from whichever module provides — because the rule is about every provider, and a list of
// providers here would be a list somebody forgets to extend. On 2026-10-05 the identity provider
// failed every consumer for a day and only its journal said so (issue 179). Appended, because
// the index is a name.
moduleEventSubject("*", ProvisionerFailing),
moduleEventSubject("*", ProvisionerRecovered),
}
// The provider standing events, by their local names. Written here as well as in the catalogue
// (catalogue.ProvisionerEvents), which this package cannot import; a test keeps them agreeing.
const (
ProvisionerFailing = "provisioner.failing"
ProvisionerRecovered = "provisioner.recovered"
)
// moduleEventSubject is where one module's event lands. The same derivation PermissionsFor uses, so
// what the controller subscribes and what the emitter is permitted to publish cannot drift apart.
func moduleEventSubject(module, event string) string {
@@ -270,7 +285,8 @@ func MeshConsumers() []Consumer {
// announcement handed over behind it must wait on the server, not time out on the
// client and come back to be acted on again.
MaxAckPending: 1,
Why: "the two events the mesh's own controller reacts to, one at a time; after " +
FromNow: true,
Why: "the events the mesh's own controller reacts to, one at a time; after " +
"max-deliver it dead-letters, because an announcement it cannot act on will not " +
"become actionable",
},
+2 -2
View File
@@ -24,8 +24,8 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused", "mesh.seat.node-build-agent.accept.>"] }
subscribe: { allow: ["$JS.API.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>"] }
subscribe: { allow: ["$JS.API.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "enrol.one", password: "$2a$11$eeeeeeeeeeeeeeeeeeeeee", permissions: {
+13
View File
@@ -37,6 +37,10 @@ type Declared struct {
State []Bucket
// Reads are other modules' state it reads, each `<module>.<name>` (novox/hq ADR 0201).
Reads []string
// NoAccount says the module declares no own secret named broker, so no account could ever be
// delivered to it and nothing can connect as it (novox/hq issue 195). Said in the negative so a
// record that does not say is composed as it always was.
NoAccount bool
}
// Records is what composing a user list needs to know about the mesh, and nothing more.
@@ -81,6 +85,15 @@ func Users(r Records) ([]Principal, error) {
if runtimeHere && d.Module == RuntimeModule {
continue
}
// **A module with nowhere to read an account is no user** (novox/hq issue 195). `module
// issue` refuses it one (issue 078: an account nothing reads is an orphan), so its user
// could only ever be left out of the file for want of a password — and every such module
// was named, on every status and plan, as a credential the mesh had not minted. Where the
// runtime is, it speaks for the module; where it is not, the module cannot speak at all,
// and a user would not change that.
if d.NoAccount {
continue
}
out = append(out, Principal{
Kind: KindModule, Node: node, Module: d.Module,
Emits: d.Emits, Consumes: d.Consumes, Serves: d.Serves,
+54
View File
@@ -296,3 +296,57 @@ func TestTheRuntimeModuleBecomesTheMachinesRuntimePrincipal(t *testing.T) {
t.Error("telegram lost its own principal when the runtime arrived on its node")
}
}
// A module that declares nowhere to read an account is no user: it could never be issued one, so it
// was only ever named as a credential the mesh had not minted (novox/hq issue 195). Where the runtime
// is, the runtime still carries it — its grants are the runtime's.
func TestAModuleThatCannotReadAnAccountIsNoUser(t *testing.T) {
r := someRecords()
r.Assigned["one"] = append(r.Assigned["one"],
Declared{Module: "packet-filter", NoAccount: true, Emits: []string{"rule.changed"}},
Declared{Module: RuntimeModule})
users, err := Users(r)
if err != nil {
t.Fatal(err)
}
for _, u := range users {
if u.Username() == "one.packet-filter" {
t.Fatalf("packet-filter declares no broker secret and was composed as a user: %v", namesOf(t, r))
}
if u.Kind == KindNodeTools {
carried := false
for _, d := range u.Carries {
carried = carried || d.Module == "packet-filter"
}
if !carried {
t.Error("the runtime stopped carrying a module that has no account of its own")
}
}
}
if names := strings.Join(namesOf(t, r), ","); !strings.Contains(names, "one.telegram") {
t.Errorf("a module that does read an account lost its user: %s", names)
}
}
// A carried module with no account still has its consumer made: the runtime reads it on the module's
// behalf (ADR 0198), and the consumer was derived from the module's own user until issue 195 took
// that user away.
func TestACarriedModuleWithNoAccountStillHasItsConsumer(t *testing.T) {
r := someRecords()
r.Assigned["one"] = append(r.Assigned["one"],
Declared{Module: "listener", NoAccount: true, Consumes: []string{"shop.order.placed"}},
Declared{Module: RuntimeModule})
r.Assigned["two"] = append(r.Assigned["two"],
Declared{Module: "auditor", Consumes: []string{"shop.order.placed"}})
users, err := Users(r)
if err != nil {
t.Fatal(err)
}
got := map[string]int{}
for _, c := range ConsumersOf(users) {
got[c.Node+"/"+c.Module]++
}
if got["one/listener"] != 1 || got["two/auditor"] != 1 || len(got) != 2 {
t.Fatalf("consumers: %v", got)
}
}
+3
View File
@@ -761,6 +761,9 @@ func Command(ctx context.Context, dir, name string, args ...string) (string, err
tell("run", "$ (%s) %s %s", short(filepath.Base(dir)), name, strings.Join(args, " "))
cmd := exec.CommandContext(ctx, name, args...)
cmd.Dir = dir
// **Ended whole when the build is** (novox/hq ADR 0219): its own process group, killed as one
// when the context ends, so a build killed by hand leaves no `git` or `docker` child running.
inItsOwnGroup(cmd)
out, err := cmd.CombinedOutput()
if err != nil {
tell("run", "! %s %s failed after %s", name, args[0], since(started))
+78
View File
@@ -0,0 +1,78 @@
package builder
import (
"context"
"os/exec"
"strings"
"syscall"
"time"
)
// A build killed by hand (novox/hq ADR 0219).
//
// A build is a tree of commands — git, then docker, and docker's own children — and a container the
// docker client started keeps running when the client dies. So ending one by hand is three things:
// the build's context is cancelled, which kills each command's whole process group rather than the
// one process the context knows; every container the build started carries the build's id as a
// label, so what outlived its client is found and removed by that label; and the holder announces
// the outcome itself, since nothing else will.
// BuildLabel is the label every container a build starts carries, valued with the build's id.
const BuildLabel = "mesh.build"
// KillWait is how long a command killed with its build may take to let go of its output before it
// is abandoned: a grandchild holding the pipe open must not hold the build open with it.
const KillWait = 10 * time.Second
// inItsOwnGroup makes a command the leader of its own process group, killed as a group when its
// context ends.
func inItsOwnGroup(cmd *exec.Cmd) {
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error {
if cmd.Process == nil {
return nil
}
// The negative pid is the group: the command and everything it started.
if err := syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL); err != nil {
return cmd.Process.Kill()
}
return nil
}
cmd.WaitDelay = KillWait
}
// Labelled is a Runner that marks every container a build starts with the build's id, so a kill
// finds what outlived the docker client (novox/hq ADR 0219). Applied at the one place every command
// passes rather than at each `docker run` the builder composes: a run added later is labelled too.
func Labelled(run Runner, id string) Runner {
return func(ctx context.Context, dir string, name string, args ...string) (string, error) {
return run(ctx, dir, name, LabelledArgs(name, args, id)...)
}
}
// LabelledArgs is a command's arguments with the build's label added, when it is a `docker run`.
func LabelledArgs(name string, args []string, id string) []string {
if name != "docker" || len(args) == 0 || args[0] != "run" || id == "" {
return args
}
out := make([]string, 0, len(args)+2)
out = append(out, "run", "--label", BuildLabel+"="+id)
return append(out, args[1:]...)
}
// RemoveContainersOf removes every container labelled with the build's id, running or not, and
// says how many. Run with a context of its own: the build's is the one that was just cancelled.
func RemoveContainersOf(ctx context.Context, run Runner, id string) (int, error) {
out, err := run(ctx, "", "docker", "ps", "-aq", "--filter", "label="+BuildLabel+"="+id)
if err != nil {
return 0, err
}
ids := strings.Fields(out)
if len(ids) == 0 {
return 0, nil
}
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f"}, ids...)...); err != nil {
return 0, err
}
return len(ids), nil
}
+103
View File
@@ -0,0 +1,103 @@
package builder
import (
"context"
"errors"
"os"
"path/filepath"
"reflect"
"strconv"
"strings"
"syscall"
"testing"
"time"
)
// A build killed by hand (novox/hq ADR 0219) ends every command it started: the context's end kills
// the command's whole process group, not the one process the context knows about.
func TestAKilledBuildEndsItsWholeProcessGroup(t *testing.T) {
dir := t.TempDir()
child := filepath.Join(dir, "child")
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
began := time.Now()
go func() {
// A shell that starts a grandchild and waits on it: killing the shell alone would leave the
// grandchild running, holding the output open.
_, err := Command(ctx, dir, "sh", "-c", `sleep 60 & echo $! > child; wait`)
done <- err
}()
var pid int
for deadline := time.Now().Add(5 * time.Second); time.Now().Before(deadline); time.Sleep(20 * time.Millisecond) {
if raw, err := os.ReadFile(child); err == nil && strings.TrimSpace(string(raw)) != "" {
pid, _ = strconv.Atoi(strings.TrimSpace(string(raw)))
break
}
}
if pid == 0 {
t.Fatal("the command never started its child")
}
cancel()
select {
case err := <-done:
if err == nil {
t.Fatal("a killed command reported success")
}
case <-time.After(KillWait + 5*time.Second):
t.Fatal("the killed command never returned")
}
if took := time.Since(began); took > KillWait {
t.Errorf("ending the command took %s: the group was not killed, the wait ran out", took)
}
for deadline := time.Now().Add(2 * time.Second); time.Now().Before(deadline); time.Sleep(20 * time.Millisecond) {
if err := syscall.Kill(pid, 0); errors.Is(err, syscall.ESRCH) {
return
}
}
_ = syscall.Kill(pid, syscall.SIGKILL)
t.Fatalf("the grandchild %d outlived the kill", pid)
}
// Every `docker run` a build starts carries its id as a label; nothing else is touched.
func TestEveryContainerABuildStartsCarriesItsID(t *testing.T) {
got := LabelledArgs("docker", []string{"run", "--rm", "img", "sh"}, "build-1")
if want := []string{"run", "--label", "mesh.build=build-1", "--rm", "img", "sh"}; !reflect.DeepEqual(got, want) {
t.Errorf("docker run became %v", got)
}
for _, c := range []struct {
name string
args []string
}{{"docker", []string{"build", "."}}, {"git", []string{"run"}}, {"docker", nil}} {
if got := LabelledArgs(c.name, c.args, "build-1"); !reflect.DeepEqual(got, c.args) {
t.Errorf("%s %v became %v", c.name, c.args, got)
}
}
var ran [][]string
run := Labelled(func(_ context.Context, _ string, name string, args ...string) (string, error) {
ran = append(ran, append([]string{name}, args...))
return "", nil
}, "build-2")
_, _ = run(context.Background(), "", "docker", "run", "img")
if want := [][]string{{"docker", "run", "--label", "mesh.build=build-2", "img"}}; !reflect.DeepEqual(ran, want) {
t.Errorf("ran %v", ran)
}
}
// What a kill removes is found by the label, and only what it finds.
func TestAKillRemovesTheContainersLabelledWithTheBuild(t *testing.T) {
var ran []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
if args[0] == "ps" {
return "c1\nc2\n", nil
}
return "", nil
}
n, err := RemoveContainersOf(context.Background(), run, "build-3")
if err != nil || n != 2 {
t.Fatalf("%d %v", n, err)
}
if want := []string{"docker ps -aq --filter label=mesh.build=build-3", "docker rm -f c1 c2"}; !reflect.DeepEqual(ran, want) {
t.Errorf("ran %v", ran)
}
}
+29 -7
View File
@@ -7,6 +7,8 @@ import (
"io"
"net/http"
"strings"
"github.com/novox/mesh-controller/internal/artifacts"
)
// Where built artifacts go.
@@ -66,22 +68,32 @@ func (r Registry) PublishImage(ctx context.Context, localTag, repository string)
return pinned, nil
}
// PublishArchive stores bytes as a blob and returns where to fetch them from.
// PublishArchive stores bytes as a blob, holds it by a manifest, and returns where to fetch them
// from.
//
// Two steps, which is the registry's own protocol: ask for somewhere to put it, then put it there
// naming the digest. The registry verifies the digest itself, so a blob that arrived corrupted is
// refused by the thing storing it rather than by the machine unpacking it a week later.
// Two steps for the blob, which is the registry's own protocol: ask for somewhere to put it, then
// put it there naming the digest. The registry verifies the digest itself, so a blob that arrived
// corrupted is refused by the thing storing it rather than by the machine unpacking it a week
// later.
//
// **Then a manifest that names it** (novox/hq issue 253, ADR 0189). The store's own collector
// marks only from manifests, and a blob no manifest names is collected however much the mesh
// means to keep it — so an archive published bare is an archive the first nightly collection
// deletes. The holder is composed from the digest and size alone (artifacts.Holder), which is
// what lets the sweep recompute it to backfill or let go without anything being recorded here.
// What a machine is told to fetch is the blob, exactly as before.
func (r Registry) PublishArchive(ctx context.Context, repository string, body []byte, digest string) (string, error) {
base := "http://" + r.Address + "/v2/" + repository
final := base + "/blobs/" + digest
// Already there. Blobs are immutable and named by their content, so this is not an
// optimisation — re-uploading would be asking the registry to store what it already has under
// the name it already has.
// the name it already has. It is still held: a blob published before holders existed is
// exactly the one a rebuild of the same source finds already there.
if there, err := r.has(ctx, final); err != nil {
return "", err
} else if there {
return final, nil
return final, r.hold(ctx, repository, digest, len(body))
}
start, err := http.NewRequestWithContext(ctx, http.MethodPost, base+"/blobs/uploads/", nil)
@@ -119,7 +131,17 @@ func (r Registry) PublishArchive(ctx context.Context, repository string, body []
said, _ := io.ReadAll(io.LimitReader(done.Body, 4096))
return "", fmt.Errorf("%s refused the blob: %s %s", base, done.Status, strings.TrimSpace(string(said)))
}
return final, nil
return final, r.hold(ctx, repository, digest, len(body))
}
// hold puts the manifest holding an archive beside it. A build whose archive could not be held is
// a failed build: recorded as published, it would be an archive the store's collector takes.
func (r Registry) hold(ctx context.Context, repository, digest string, size int) error {
store := artifacts.Store{Address: r.Address, HTTP: r.client()}
if _, err := store.HoldBlob(ctx, repository, digest, int64(size)); err != nil {
return fmt.Errorf("published %s/blobs/%s and could not hold it by a manifest: %w", repository, digest, err)
}
return nil
}
// has is whether this registry already holds what is at that URL.
+86 -6
View File
@@ -9,6 +9,8 @@ import (
"net/http/httptest"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/artifacts"
)
// An OCI registry as a content-addressed blob store, which is what it is.
@@ -18,9 +20,10 @@ import (
// there is not sent again, and that a tag is never accepted as a pin.
type fakeRegistry struct {
blobs map[string][]byte
uploads int
location string
blobs map[string][]byte
manifests map[string][]byte // "<repository>@<digest>"
puts map[string]int // blob uploads, by digest
location string
}
func (f *fakeRegistry) serve(t *testing.T) *httptest.Server {
@@ -28,6 +31,8 @@ func (f *fakeRegistry) serve(t *testing.T) *httptest.Server {
if f.blobs == nil {
f.blobs = map[string][]byte{}
}
f.manifests = map[string][]byte{}
f.puts = map[string]int{}
mux := http.NewServeMux()
server := httptest.NewServer(mux)
mux.HandleFunc("/v2/", func(w http.ResponseWriter, r *http.Request) {
@@ -39,8 +44,28 @@ func (f *fakeRegistry) serve(t *testing.T) *httptest.Server {
return
}
w.WriteHeader(http.StatusNotFound)
case strings.Contains(r.URL.Path, "/manifests/"):
// The archive's holder (novox/hq issue 253): asked for with an Accept, put by digest.
repository, digest, _ := strings.Cut(strings.TrimPrefix(r.URL.Path, "/v2/"), "/manifests/")
switch r.Method {
case http.MethodHead:
if _, ok := f.manifests[repository+"@"+digest]; ok &&
strings.Contains(r.Header.Get("Accept"), "application/vnd.oci.image.manifest.v1+json") {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
case http.MethodPut:
body, _ := io.ReadAll(r.Body)
sum := sha256.Sum256(body)
if digest != "sha256:"+hex.EncodeToString(sum[:]) {
w.WriteHeader(http.StatusBadRequest)
return
}
f.manifests[repository+"@"+digest] = body
w.WriteHeader(http.StatusCreated)
}
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/blobs/uploads/"):
f.uploads++
where := f.location
if where == "" {
where = "/v2/upload/" + hex.EncodeToString([]byte("session"))
@@ -58,6 +83,7 @@ func (f *fakeRegistry) serve(t *testing.T) *httptest.Server {
return
}
f.blobs[digest] = body
f.puts[digest]++
w.WriteHeader(http.StatusCreated)
default:
w.WriteHeader(http.StatusNotFound)
@@ -92,6 +118,57 @@ func TestAnArchiveIsStoredAndFetchableByItsDigest(t *testing.T) {
}
}
func TestAnArchiveIsPublishedWithAManifestHoldingIt(t *testing.T) {
// The store's collector marks only from manifests, so a bare blob is one the first collection
// deletes, kept or not (novox/hq issue 253, ADR 0189). Every archive goes out held, by the
// holder the sweep can compute for itself from the digest and size.
f := &fakeRegistry{}
r := registryFor(t, f)
body := []byte("a theme")
sum := sha256.Sum256(body)
digest := "sha256:" + hex.EncodeToString(sum[:])
where, err := r.PublishArchive(context.Background(), "shell/config", body, digest)
if err != nil {
t.Fatal(err)
}
if !strings.HasSuffix(where, "/v2/shell/config/blobs/"+digest) {
t.Fatalf("a machine is told to fetch %q; the blob is still what is fetched", where)
}
manifest, holder := artifacts.Holder(digest, int64(len(body)))
if got := f.manifests["shell/config@"+holder]; string(got) != string(manifest) {
t.Fatalf("the store holds %v; want the holder %s in the archive's own repository", f.manifests, holder)
}
if !strings.Contains(string(manifest), `"digest":"`+digest+`"`) {
t.Fatalf("the holder does not name the archive: %s", manifest)
}
empty := sha256.Sum256([]byte("{}"))
if _, ok := f.blobs["sha256:"+hex.EncodeToString(empty[:])]; !ok {
t.Fatal("the holder's empty config was never put, and a registry refuses a manifest without it")
}
}
func TestAnArchiveAlreadyThereIsStillHeld(t *testing.T) {
// A rebuild of the same source finds its archive already there — often one published bare,
// before holders. It is not sent again, and it is held.
f := &fakeRegistry{}
r := registryFor(t, f)
body := []byte("published bare")
sum := sha256.Sum256(body)
digest := "sha256:" + hex.EncodeToString(sum[:])
f.blobs[digest] = body
if _, err := r.PublishArchive(context.Background(), "shell/config", body, digest); err != nil {
t.Fatal(err)
}
if f.puts[digest] != 0 {
t.Fatal("a blob already there was sent again")
}
if _, holder := artifacts.Holder(digest, int64(len(body))); f.manifests["shell/config@"+holder] == nil {
t.Fatal("a blob already there was left bare")
}
}
func TestABlobAlreadyThereIsNotSentAgain(t *testing.T) {
// Not an optimisation: blobs are named by their content, so re-uploading is asking the
// registry to store what it already has under the name it already has.
@@ -107,8 +184,11 @@ func TestABlobAlreadyThereIsNotSentAgain(t *testing.T) {
if _, err := r.PublishArchive(context.Background(), "shell/config", body, digest); err != nil {
t.Fatal(err)
}
if f.uploads != 1 {
t.Fatalf("the blob was uploaded %d times", f.uploads)
if f.puts[digest] != 1 {
t.Fatalf("the blob was uploaded %d times", f.puts[digest])
}
if len(f.manifests) != 1 {
t.Fatalf("publishing twice left %d holders; want the one", len(f.manifests))
}
}
+72
View File
@@ -0,0 +1,72 @@
package catalogue
import (
"strings"
"testing"
)
// A store provider says how its data is backed up (novox/hq ADR 0214); a provider of something that
// holds nothing does not have to.
func TestAStoreProviderWithoutABackupIsRefusedByTheCheck(t *testing.T) {
bare, err := ParseManifest([]byte(`{"module":"pg","version":"1",
"provides":[{"name":"postgres-database","scope":"mesh"}]}`))
if err != nil {
t.Fatalf("parsing refused it, and the rule is the check's: %v", err)
}
if problems := CheckBackup(bare); len(problems) != 1 || !strings.Contains(problems[0], "node-backup") {
t.Fatalf("a store with no backup passed the check: %v", problems)
}
backed, err := ParseManifest([]byte(`{"module":"pg","version":"1",
"provides":[{"name":"postgres-database","scope":"mesh"}],
"resources":[{"id":"dumps","type":"directory","mode":"0700"}],
"contributions":[{"seat":"node-backup","kind":"backup","content":"path ${dir:dumps}"}]}`))
if err != nil {
t.Fatal(err)
}
if problems := CheckBackup(backed); len(problems) != 0 {
t.Fatalf("a store that contributes a backup was refused: %v", problems)
}
route, _ := ParseManifest([]byte(`{"module":"r","version":"1","provides":[{"name":"route","scope":"mesh"}]}`))
if problems := CheckBackup(route); len(problems) != 0 {
t.Fatalf("a route was asked for a backup: %v", problems)
}
}
// A backup contribution names its module's own directories; one it does not declare is refused
// where it is written rather than reaching the holder as the literal text.
func TestABackupNamingAnUndeclaredDirectoryIsRefused(t *testing.T) {
_, err := ParseManifest([]byte(`{"module":"pg","version":"1",
"contributions":[{"seat":"node-backup","kind":"backup","content":"path ${dir:nowhere}"}]}`))
if err == nil || !strings.Contains(err.Error(), "nowhere") {
t.Fatalf("a backup of an undeclared directory was accepted: %v", err)
}
}
// The holder receives every module's backup lines with each module's own directories filled, each
// module's under a comment naming it — and a kind written in a shell's grammar is left untouched.
func TestBackupContributionsArePlacedWithTheirModulesDirectories(t *testing.T) {
pg, err := ParseManifest([]byte(`{"module":"pg","version":"1",
"resources":[{"id":"dumps","type":"directory","path":"/srv/pg/dumps","mode":"0700"}],
"contributions":[{"seat":"node-backup","kind":"backup","content":"run pg-dump-all\npath ${dir:dumps}"}]}`))
if err != nil {
t.Fatal(err)
}
mail, err := ParseManifest([]byte(`{"module":"mail","version":"1",
"resources":[{"id":"spool","type":"directory","mode":"0700"}],
"contributions":[{"seat":"node-backup","kind":"backup","content":"path ${dir:spool}"}]}`))
if err != nil {
t.Fatal(err)
}
placed, err := seatContributions([]Manifest{pg, mail}, BackupSeat, "backup", Rendering{})
if err != nil {
t.Fatal(err)
}
for _, want := range []string{"# pg\nrun pg-dump-all\npath /srv/pg/dumps\n", "# mail\npath /var/lib/mail/spool\n"} {
if !strings.Contains(placed, want) {
t.Errorf("placed contributions lack %q:\n%s", want, placed)
}
}
if strings.Contains(placed, "${dir:") {
t.Errorf("a directory reached the holder unfilled:\n%s", placed)
}
}
+4 -3
View File
@@ -21,8 +21,9 @@ import (
// formats and gains no fields.
//
// **It stays name-agnostic** ([ADR 0027]). The mesh does not learn what a `postgres-database` is:
// `at`, `as` and `from` are facts about any provision at all, and everything else comes from what
// the provider said it serves — whose keys are agreed by the requirement's name, not by this file.
// `at`, `as`, `from` and `address` (the providing machine's private address, novox/hq ADR 0194) are
// facts about any provision at all, and everything else comes from what the provider said it
// serves — whose keys are agreed by the requirement's name, not by this file.
// bound is where a module says a value from one of its bindings belongs:
// ${bound:<provision>.<key>}.
@@ -61,7 +62,7 @@ func knownFor(m Manifest, needs []Needed, node string) (map[string]map[string]st
"as": as,
}
// What the provider derives for this consumer rather than for all of them
// (novox/hq ADR 0202). Filled here, the one place a provision and the module
// (novox/hq ADR 0201). Filled here, the one place a provision and the module
// requiring it are both in hand.
served, err := ServedTo(n.Serves, as)
if err != nil {
@@ -232,3 +232,34 @@ func TestTwoModulesOnOneNodeAreTwoIdentities(t *testing.T) {
t.Fatal("one module on two machines shares an identity")
}
}
// A machine's resolver configuration must name its resolver by address — it cannot resolve the name
// of the thing it resolves names with (novox/hq ADR 0194). So a binding offers the providing
// machine's private address beside its name, and only when the machine has one.
func TestABindingOffersTheProvidersAddress(t *testing.T) {
consumer := func() Resolution {
return Resolution{
Node: "workstation",
Modules: []Manifest{{
Module: "resolv-conf",
Requires: []string{"wildcard-resolution"},
Resources: []map[string]any{{
"id": "resolv", "type": "file", "path": "/etc/resolv.conf", "mode": "0644",
"content": "nameserver ${bound:wildcard-resolution:address}\n",
}},
}},
Needs: []Needed{{Name: "wildcard-resolution", From: "anchor", At: "anchor.internal", For: "resolv-conf"}},
}
}
out, err := consumer().Declaration(Rendering{Machines: map[string]string{"anchor.internal": "10.77.0.1"}})
if err != nil {
t.Fatal(err)
}
if got := fileNamed(out, "resolv-conf.resolv")["content"]; got != "nameserver 10.77.0.1\n" {
t.Fatalf("the resolver is not named by its address: %q", got)
}
// A machine with no address yet: refused, never written with a blank where the address belongs.
if _, err := consumer().Declaration(Rendering{}); err == nil {
t.Fatal("a file naming an address the mesh does not have was composed")
}
}
@@ -86,3 +86,36 @@ func TestNoCatalogueManifestNamesAnInstallation(t *testing.T) {
t.Fatalf("%d value(s) name an installation:\n %s", len(named), strings.Join(named, "\n "))
}
}
// TestEveryCatalogueStoreSaysHowItIsBackedUp is ADR 0214's check over the real catalogue: a module
// providing a store contributes a backup to node-backup, so a store added is a store backed up.
func TestEveryCatalogueStoreSaysHowItIsBackedUp(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
stores := 0
for _, p := range found {
raw, err := os.ReadFile(p)
if err != nil {
t.Fatalf("%s: %v", p, err)
}
m, err := ParseManifest(raw)
if err != nil {
continue // TestEveryCatalogueManifestParses says why
}
for _, o := range m.Provides {
if storeProvisions[o.Name] {
stores++
break
}
}
for _, problem := range CheckBackup(m) {
t.Error(problem)
}
}
if stores == 0 {
t.Fatal("no module in the catalogue provides a store, so this proved nothing")
}
}
+2 -1
View File
@@ -159,7 +159,8 @@ func TestTwoWaysToBeOnAPrivateNetworkRefuseAndNameBoth(t *testing.T) {
}
}
// reloading answers the runtime's trust as the networking module does (novox/hq ADR 0102): a file
// reloading answers the runtime's trust as the networking module once did (novox/hq ADR 0102; it no
// longer does, ADR 0222 — and beside the runtime's module it is refused, generated_collision_test): a file
// written into, and the runtime reloaded on it.
type reloading struct{}
+4 -4
View File
@@ -8,7 +8,7 @@ import (
)
// What a provider derives for one consumer, said once in the provider's definition and delivered
// to both ends (novox/hq ADR 0202, issue 124).
// to both ends (novox/hq ADR 0201, issue 124).
//
// A `serves` block is otherwise literal: the same values for every consumer. Where the provider
// *names the resource* — a bucket, a database, a vhost — the name is derived from who is asking,
@@ -165,7 +165,7 @@ func sortedAnyKeys(values map[string]any) []string {
}
// derivedFor is what the provider on this machine derives for one consumer of one provision
// (novox/hq ADR 0202).
// (novox/hq ADR 0201).
//
// Settled first, then derived: an operator may set a prefix on what the provider serves and the
// mesh still fills the consumer's half of it ([ADR 0174]). Only the keys that actually name the
@@ -213,7 +213,7 @@ func (r Resolution) derivedFor(provision, as, consumer, local string, settings S
"%s keeps several holders of %s (this one is %q), and %s derives %s for each "+
"consumer from the login the mesh minted. Each holder has its own login, and a "+
"consumer is told one value per requirement — so the two ends would name "+
"different things and nothing would compare them (novox/hq ADR 0202)",
"different things and nothing would compare them (novox/hq ADR 0201)",
consumer, local, provision, m.Module, orNothing(sortedAnyKeys(names)))
}
settled, err := Settle(names, settings[m.Module])
@@ -230,7 +230,7 @@ func (r Resolution) derivedFor(provision, as, consumer, local string, settings S
}
// notTranscribed refuses a consumer's file that writes out the value its provider derives for it,
// instead of asking for it (novox/hq ADR 0202, issue 124).
// instead of asking for it (novox/hq ADR 0201, issue 124).
//
// **What would have caught the one wrong instance.** The object store's three consumers each wrote
// their bucket into their own configuration by hand. One of them named a predecessor's bucket, and
@@ -0,0 +1,121 @@
package catalogue
import (
"reflect"
"strings"
"testing"
)
// Defends novox/hq ADR 0210 §3: a contribution is a dependency on the seat that receives it.
func TestAContributionDependsOnTheSeatThatPlacesIt(t *testing.T) {
env := mod("theme", nil, nil, nil)
env.Environment = &Environment{Variables: map[string]string{"GTK_THEME": "Adwaita:dark"}}
path := mod("toolchain", nil, nil, nil)
path.Environment = &Environment{Path: []PathEntry{{Entry: "/opt/x/bin"}}}
shell := mod("prompt", nil, nil, nil)
shell.Shell = []ShellCode{{For: "zsh", Slot: "first", Code: "true"}}
session := mod("wallpaper", nil, nil, nil)
session.Shell = []ShellCode{{For: "xinitrc", Slot: "normal", Code: "true"}}
resources := mod("bar", nil, nil, nil)
resources.Shell = []ShellCode{{For: "xresources", Slot: "normal", Code: "x: y"}}
empty := mod("nothing", nil, nil, nil)
empty.Environment = &Environment{}
cases := map[string]struct {
m Manifest
want []string
}{
"a variable": {env, []string{EnvironmentSeat}},
"a path entry": {path, []string{EnvironmentSeat}},
"shell code": {shell, []string{LoginShellSeat}},
"the session's start": {session, []string{DisplayServerSeat}},
"the session's X resources": {resources, []string{DisplayServerSeat}},
"an empty environment": {empty, nil},
}
for name, c := range cases {
got := DependsOn(c.m)
if len(got) == 0 {
got = nil
}
if !reflect.DeepEqual(got, c.want) {
t.Errorf("%s depends on %v, want %v", name, got, c.want)
}
}
}
func TestAContributionIsMetByAHolderOnTheNodeAndRefusedWithout(t *testing.T) {
holder := mod("node-env", nil, nil, nil, Claim{Name: EnvironmentSeat})
contributor := mod("theme", nil, nil, nil)
contributor.Environment = &Environment{Variables: map[string]string{"GTK_THEME": "Adwaita:dark"}}
catalogue := map[string]Manifest{"node-env": holder, "theme": contributor}
if _, err := AssignRefusal(catalogue, "laptop", []string{"node-env"}, []string{"theme"}); err != nil {
t.Fatalf("a contributor beside the holder is refused: %v", err)
}
_, err := AssignRefusal(catalogue, "laptop", nil, []string{"theme"})
if err == nil {
t.Fatal("a contributor on a node without the holder was accepted, and its contribution would be written nowhere")
}
if !strings.Contains(err.Error(), EnvironmentSeat) || !strings.Contains(err.Error(), "node-env") {
t.Errorf("the refusal names neither the seat nor its holder: %v", err)
}
if _, err := AssignRefusal(catalogue, "laptop", nil, []string{"theme", "node-env"}); err != nil {
t.Errorf("the contributor and the holder assigned together are refused: %v", err)
}
}
func TestAHolderMeetsItsOwnContribution(t *testing.T) {
// The display server's module contributes nothing to its own seat today, but the zsh module's
// environment contributions do go to another seat: a claim meets only the seat it names.
zsh := mod("zsh", nil, nil, nil, Claim{Name: LoginShellSeat})
zsh.Shell = []ShellCode{{For: "zsh", Slot: "normal", Code: "true"}}
zsh.Environment = &Environment{Variables: map[string]string{"EDITOR": "vim"}}
catalogue := map[string]Manifest{"zsh": zsh,
"node-env": mod("node-env", nil, nil, nil, Claim{Name: EnvironmentSeat})}
unheld := UnheldDependencies(catalogue, "laptop", []Manifest{zsh}, nil)
if len(unheld) != 1 || unheld[0].Seat != EnvironmentSeat {
t.Errorf("zsh alone: unheld %v, want only %s (its shell code is its own seat's)", unheld, EnvironmentSeat)
}
}
func TestTwoModulesDeclaringOnePackageAreRefusedBeforeAnythingIsRecorded(t *testing.T) {
pacman := withResources(mod("pacman", nil, nil, nil, Claim{Name: PackageManagerSeat}),
res("package", "pacman-contrib"))
bar := withResources(mod("bar", nil, nil, nil), res("package", "bar"), res("package", "pacman-contrib"))
other := withResources(mod("other", nil, nil, nil), res("package", "other"))
catalogue := map[string]Manifest{"pacman": pacman, "bar": bar, "other": other}
err := CollisionRefusal(catalogue, "laptop", []string{"pacman"}, []string{"bar"})
if err == nil || !strings.Contains(err.Error(), "pacman-contrib") || !strings.Contains(err.Error(), "bar") {
t.Fatalf("the second owner of a package was not refused by name: %v", err)
}
if err := CollisionRefusal(catalogue, "laptop", []string{"pacman"}, []string{"other"}); err != nil {
t.Errorf("a module declaring nothing shared is refused: %v", err)
}
// A collision already on the node is status's, not a reason to refuse an unrelated assignment.
if err := CollisionRefusal(catalogue, "laptop", []string{"pacman", "bar"}, []string{"other"}); err != nil {
t.Errorf("an unrelated assignment is refused for a collision already there: %v", err)
}
}
func TestCodeForAPowerMomentDependsOnThePowerSeat(t *testing.T) {
for _, moment := range powerMoments {
m := mod("laptop", nil, nil, nil)
m.Shell = []ShellCode{{For: moment, Slot: "normal", Code: "true"}}
if got := DependsOn(m); !reflect.DeepEqual(got, []string{PowerSeat}) {
t.Errorf("code for %s depends on %v, want %s", moment, got, PowerSeat)
}
if p := m.shellProblems(); len(p) != 0 {
t.Errorf("code for %s is refused: %v", moment, p)
}
}
m := mod("laptop", nil, nil, nil)
m.Shell = []ShellCode{{For: "after-lunch", Slot: "normal", Code: "true"}}
if p := m.shellProblems(); len(p) == 0 {
t.Error("code for a moment that does not exist was accepted")
}
if s, ok := SeatNamed(PowerSeat); !ok || s.Scope != ScopeNode {
t.Errorf("%s is not a node seat of the mesh's own: %+v %v", PowerSeat, s, ok)
}
}
+119 -17
View File
@@ -155,6 +155,15 @@ type Rendering struct {
// standing beside the machines and looking as real as they do.
Machines map[string]string
// Zones is every zone a module in the mesh answers itself, where it is answered (novox/hq ADR
// 0199): the mesh's resolver forwards each one there.
Zones []ZoneAt
// Holders is, for each replicated mesh seat, every machine holding it, by internal name and
// private address (novox/hq ADR 0223) — the same shape as Machines. What a machine's resolver
// file lists: every holder of the mesh's resolver, this machine first if it is one.
Holders map[string]map[string]string
Settings SettingsBy
Generators map[string]Generator
// Grants are the credentials this node must create, for the provisions it offers. Passed in
@@ -197,6 +206,11 @@ type Rendering struct {
// stored (novox/hq 04-ISSUES/102).
ArtifactStore string
// SeatReach is where this machine reaches the holder of each mesh-scoped seat it may be asked
// about (host:port), by seat: what ${seat:<seat>:reach} answers with (novox/hq ADR 0222). Absent
// when no holder is reachable yet; see seat_into.go for which seats are answered.
SeatReach map[string]string
// Built is every `<module>/<artifact>` the mesh has built. What tells a reference recorded
// with an address — before references were kept without one — from an image a module runs
// straight from a public registry.
@@ -415,6 +429,18 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
filtering := AsNftables(rules, with.Mesh, r.PublicDomain != "", with.Foundation,
with.OutwardLinks, with.TunnelInterface)
// **A variable two modules set is refused whether or not anything places it** (novox/hq ADR
// 0203 §5): the account has one environment, and a machine whose holder arrives later should not
// be the moment two modules are found to disagree about it.
if err := variablesSetOnce(r.Modules); err != nil {
return nil, err
}
generated, err := r.generatedHere(with)
if err != nil {
return nil, err
}
var out []map[string]any
for _, m := range r.Modules {
if with.Adopted && m.Filtering != nil {
@@ -603,14 +629,14 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// and nothing would say so.
continue
}
first = append(first, map[string]any{
first = append(first, ownedBy(r.provisionsAs(m), map[string]any{
// One file per holder — the consumer's module with its local name after it
// where it keeps several (ADR 0094); the lab found two files with one id.
"id": GrantID(to, g.Consumer+"."+holderAs(g.From, g.Local)),
"type": "file",
"path": grantPath(m.Grants[to], g.Consumer, holderAs(g.From, g.Local)),
"sealed": g.Sealed,
})
}))
}
}
for _, to := range sortedKeys(m.Binds) {
@@ -647,7 +673,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
}
as := ConsumerIdentity(r.Node, IdentitySource(m.Slug, m.Module))
// What the provider derives for THIS consumer, filled here where the consumer is
// known (novox/hq ADR 0202). The same fill knownFor does below, so the binding file
// known (novox/hq ADR 0201). The same fill knownFor does below, so the binding file
// and the module's `${bound:…}` substitutions cannot say different things.
told := *found
told.Serves, err = ServedTo(told.Serves, as)
@@ -681,23 +707,14 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
first = append(first, file)
}
if m.Computed != "" {
generator, known := with.Generators[m.Computed]
if !known {
return nil, fmt.Errorf(
"%s says its resources are computed by %q, and this control plane has no %q",
m.Module, m.Computed, m.Computed)
}
generated, part, err := generator.Resources(r.Node)
if err != nil {
return nil, err
}
mine, part := generated[m.Module]
if !part {
// Assigned, and not yet part of what this generates. Nothing to put on the
// machine, which is different from an error: a node given the network module
// before it has an address is in exactly that state, briefly.
continue
}
resources = generated
resources = mine
}
// Now, and not before: a module whose resources are computed replaces them wholesale, and
@@ -756,6 +773,16 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
known[provision] = values
}
}
// And the providing machine's private address, beside its name (novox/hq ADR 0194). A name is
// what nearly every consumer wants; the one that cannot use it is a machine's resolver
// configuration, which must reach the resolver before it can resolve anything — the resolver's
// own name included. Absent when the machine has no address yet, so a file naming it is refused
// rather than written with a blank where an address belongs.
for _, values := range known {
if address := with.Machines[values["at"]]; address != "" {
values["address"] = address
}
}
// And what the module is called through each requirement it contributes to (novox/hq
// 04-ISSUES/122) — the same composition its binding file carries.
for provision, values := range known {
@@ -777,7 +804,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// And the machine underneath, which no binding of its own can tell it.
thisMachine := machineFacts(r, with.Names, with.MeshRange)
// **A definition that already holds the answer transcribed it** (novox/hq ADR 0202).
// **A definition that already holds the answer transcribed it** (novox/hq ADR 0201).
// Judged over what the module itself declares, and before anything is substituted: the
// mesh's own generated files — the binding, the contributions — legitimately carry the
// derived value, and after substitution so does every consumer's file, so this is the one
@@ -889,6 +916,13 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
return nil, err
}
publishedOn(copied, m.Module, with)
// The account's environment and every module's shell code, where this module holds the
// seat that places them (novox/hq ADR 0203, ADR 0204). Gathered from every module on
// the node, as the jails are, and **last of every placeholder pass**: shell code is a
// shell's own syntax, full of `${…}` no pass above should ever be shown.
if err := contributionsInto(copied, m, r.Modules, thisMachine, with); err != nil {
return nil, err
}
copied["id"] = m.Module + "." + fmt.Sprint(resource["id"])
// A service saying what it reflects names resources within its own module, so those
// are prefixed too or they would point at nothing.
@@ -951,7 +985,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// plane's; making a name resolve is the module's software. Emitted as ordinary files under
// this module's name, so they are applied, reported and removed exactly as anything else
// it declares.
given, err := FactsInto(m, r, with.Names, with.Machines, with.Accounts, with.Suffix)
given, err := FactsFrom(m, r, with)
if err != nil {
return nil, err
}
@@ -1212,7 +1246,7 @@ type Contribution struct {
// is what makes swapping one for another cost nothing.
Values map[string]any `json:"values"`
// Derived is what this provider's own definition said it derives for this consumer, already
// derived (novox/hq ADR 0202).
// derived (novox/hq ADR 0201).
//
// **The provider is told, rather than recomputing it.** A served value may name the consumer's
// identity — a bucket named for who is asking, a database prefixed with it — and before this
@@ -1226,6 +1260,28 @@ type Contribution struct {
Derived map[string]any `json:"derived,omitempty"`
}
// provisionsAs is the account that reads what the mesh writes for this provider: the one secret
// per consumer it must open to set that consumer's password (novox/hq issue 225).
//
// **A root-owned 0600 file is one that process cannot read**, which is the same sentence already
// written above for a module's own secrets — and the grant secret is the other kind of secret
// the mesh writes for a module, so it is the same rule.
//
// Which account depends on where the module's code runs. A module whose code is a bundle is run
// by the node's tool runtime, as the node's account ([ADR 0198](0198)); one still in a container
// is whatever it declares as its secrets owner. Nothing names these paths, so the rule that
// claims a bundle's other files by the words that name them (givenTo) cannot reach them: the
// harness composes a grant secret's path from the contributions file, not from a word.
//
// Empty is root, which is what it was and what a module with no bundle and no declared owner
// still wants.
func (r Resolution) provisionsAs(m Manifest) string {
if len(m.Bundles) > 0 && r.Account != "" {
return r.Account
}
return m.SecretsOwner
}
// grantPath is where one consumer's sealed credential lands on the providing machine.
//
// Suffixed, so the directory can also hold whatever the module writing it keeps there and so a
@@ -2298,3 +2354,49 @@ func accountsFirst(resources []map[string]any) (accounts, rest []map[string]any)
}
return accounts, rest
}
// generatedHere is what each computed module's generator answers for this machine, by module —
// absent for a module whose machine is not yet part of what it generates — held to the rule every
// module is held to: no two modules on a machine declare one path, unit, name or package (novox/hq
// issue 190, step 5; ADR 0222).
//
// Resolution checks the catalogue's manifests, and a computed module's manifest has none of the
// resources it will declare — they exist only once its generator has answered for this machine,
// here — so a generated resource writing into another module's file was never seen. That is how
// the private network came to declare the container runtime's daemon file and service beside the
// runtime's own module. Asked once, so what is checked is exactly what is declared.
func (r Resolution) generatedHere(with Rendering) (map[string][]map[string]any, error) {
generated := map[string][]map[string]any{}
placed := make([]Manifest, 0, len(r.Modules))
for _, m := range r.Modules {
if m.Computed == "" {
placed = append(placed, m)
continue
}
generator, known := with.Generators[m.Computed]
if !known {
return nil, fmt.Errorf(
"%s says its resources are computed by %q, and this control plane has no %q",
m.Module, m.Computed, m.Computed)
}
resources, part, err := generator.Resources(r.Node)
if err != nil {
return nil, err
}
computed := m
computed.Resources = nil
if part {
generated[m.Module] = resources
computed.Resources = resources
}
placed = append(placed, computed)
}
if len(generated) == 0 {
return generated, nil // nothing resolution has not already judged
}
if problems := checkResources(placed); len(problems) > 0 {
return nil, fmt.Errorf("what the mesh computes for this machine collides with a module's own: %s",
strings.Join(problems, "; "))
}
return generated, nil
}
@@ -7,7 +7,7 @@ import (
)
// What a provider derives for each consumer, said once and delivered to both ends
// (novox/hq ADR 0202, issue 124).
// (novox/hq ADR 0201, issue 124).
//
// The failure these are written against: the object store's provisioner derived each consumer's
// bucket from the login the mesh minted, in its own code, and the mesh had no channel to tell the
+593
View File
@@ -0,0 +1,593 @@
package catalogue
import (
"fmt"
"regexp"
"sort"
"strings"
)
// The account's environment and the login shell's code, composed from the modules a node runs
// (novox/hq ADR 0203, ADR 0204).
//
// **The same shape as the jails.** Every module may contribute — a toolchain its directory on PATH,
// a version manager a variable naming its home, a prompt the code that loads it — naming no node, no
// path and no file of the shell's (ADR 0112). The one module holding the matching seat places the
// result with a placeholder in its own file, and the controller fills it from every module on the
// node. A node not running a module has none of its contribution, and unassigning one takes its
// lines away at the next composition.
//
// **Two kinds of contribution, kept apart on purpose.** The environment is facts, which the
// controller writes in two standard formats — POSIX assignment and the service manager's
// environment.d — so a terminal, a script, the login shell's `execute` and a graphical session all
// read the same values (ADR 0203). Shell code is not a fact: it is text in one shell's syntax, which
// the controller sorts into a slot and pastes without reading, as it pastes a jail's stanza (ADR
// 0204).
// EnvironmentSeat and LoginShellSeat are the seats whose holders may place what the modules
// contributed: the account's environment, and the login shell's code.
const (
EnvironmentSeat = "node-environment"
LoginShellSeat = "node-login-shell"
// PowerSeat is the seat whose holder places code for the power moments (novox/hq ADR 0211).
PowerSeat = "node-power"
)
// Where an environment entry on PATH goes: before the account's existing PATH, or after it.
const (
PathAtStart = "start"
PathAtEnd = "end"
)
// Environment is what one module adds to the account's environment (novox/hq ADR 0203).
type Environment struct {
// Variables are names and literal values. A value may name the machine's own facts with
// ${machine:…}, resolved before anything is written, and nothing else that expands.
Variables map[string]string `json:"variables,omitempty"`
// Path is entries on the account's PATH, each at its start or its end, in the order declared.
Path []PathEntry `json:"path,omitempty"`
}
// PathEntry is one directory a module puts on the account's PATH.
type PathEntry struct {
Entry string `json:"entry"`
At string `json:"at"`
}
// ShellCode is one piece of code a module adds to a shell's startup (novox/hq ADR 0204).
type ShellCode struct {
// For is the shell the code is written in.
For string `json:"for"`
// Slot is where it runs among the other modules' code: first, normal or last. Named rather
// than numbered, because every contributor would guess a number and a collision says nothing.
Slot string `json:"slot"`
// Code is never interpreted — it is the shell's syntax, and only the shell reads it.
Code string `json:"code"`
}
// The shells and slots a contribution may name (novox/hq ADR 0204). Closed, so a typo is a refusal
// at the check rather than code that silently lands in no placeholder.
var (
knownShells = []string{"zsh", "bash", "fish"}
knownSlots = []string{"first", "normal", "last"}
// sessionFiles are the two files of the graphical session's start that read no directory, so a
// contribution to them is a slot rather than a drop-in (novox/hq ADR 0208 §4): `xinitrc` is POSIX
// code the session's start runs, `xresources` X resources merged at its start. Placed by the
// display server's holder, as a shell's slots are placed by the login shell's.
sessionFiles = []string{"xinitrc", "xresources"}
// powerMoments are the moments of a machine's power a module may run code at (novox/hq ADR
// 0211 §3): POSIX code run as root by node-power's holder, in module order, each piece bounded.
powerMoments = []string{"after-boot", "before-sleep", "after-wake", "before-shutdown", "on-mains", "on-battery"}
)
// contributionTargets is every name a contribution's `for` may take.
func contributionTargets() []string {
out := append(append([]string(nil), knownShells...), sessionFiles...)
return append(out, powerMoments...)
}
// placerOf is the seat whose holder places a contribution for this target (novox/hq ADR 0204,
// ADR 0208 §4).
func placerOf(target string) string {
if oneOf(sessionFiles, target) {
return DisplayServerSeat
}
if oneOf(powerMoments, target) {
return PowerSeat
}
return LoginShellSeat
}
// The two renderings of the environment a holder may place (novox/hq ADR 0203, decision 3).
const (
EnvironmentPOSIX = "posix"
EnvironmentSystemd = "systemd"
)
// ofEnvironment and ofShell are where a holder places what was contributed: ${environment:posix},
// ${environment:systemd} and ${shell:<shell>:<slot>}. Loose inside the braces on purpose, so a
// misspelt key is found and refused rather than left in a file as a literal nobody reads.
var (
ofEnvironment = regexp.MustCompile(`\$\{environment:([^}]*)\}`)
ofShell = regexp.MustCompile(`\$\{shell:([^}]*)\}`)
// ofContributed is either kind of contributed text, matched together so both fill in one pass.
ofContributed = regexp.MustCompile(`\$\{(shell|contribution):([^}]*)\}`)
)
// variableName is a POSIX shell variable name, which is also what environment.d accepts.
var variableName = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
// environmentProblems is what is wrong with this module's environment contribution, from the
// manifest alone.
func (m Manifest) environmentProblems() []string {
if m.Environment == nil {
return nil
}
var problems []string
for _, n := range sortedKeys(m.Environment.Variables) {
switch {
case !variableName.MatchString(n):
problems = append(problems, fmt.Sprintf(
"%s sets the variable %q, which is not a name a shell accepts: a letter or an "+
"underscore, then letters, digits and underscores", m.Module, n))
continue
case n == "PATH":
// PATH is the one variable every module shares, so no module may set it whole: a second
// setter would replace the first's entries, and the account's own PATH with them.
problems = append(problems, fmt.Sprintf(
"%s sets PATH as a variable; a module adds an entry under environment.path, at the "+
"start or the end, and PATH is composed from every module's (novox/hq ADR 0203)", m.Module))
continue
}
if why := literalProblem(m.Environment.Variables[n]); why != "" {
problems = append(problems, fmt.Sprintf(
"%s sets %s to %q, which %s — %s", m.Module, n, m.Environment.Variables[n], why, literalRule))
}
}
seen := map[string]bool{}
for i, p := range m.Environment.Path {
switch {
case p.Entry == "":
problems = append(problems, fmt.Sprintf("%s's PATH entry %d names no directory", m.Module, i+1))
case strings.Contains(ofMachine.ReplaceAllString(p.Entry, ""), ":"):
// A colon is PATH's own separator, so an entry holding one is two entries, and the
// check that it is already present would look for the wrong thing.
problems = append(problems, fmt.Sprintf(
"%s puts %q on PATH, which holds a colon, PATH's own separator", m.Module, p.Entry))
case seen[p.Entry]:
problems = append(problems, fmt.Sprintf("%s puts %q on PATH twice", m.Module, p.Entry))
default:
if why := literalProblem(p.Entry); why != "" {
problems = append(problems, fmt.Sprintf(
"%s puts %q on PATH, which %s — %s", m.Module, p.Entry, why, literalRule))
}
}
seen[p.Entry] = true
if p.At != PathAtStart && p.At != PathAtEnd {
problems = append(problems, fmt.Sprintf(
"%s puts %q on PATH at %q; an entry goes at %q or %q of the account's PATH",
m.Module, p.Entry, p.At, PathAtStart, PathAtEnd))
}
}
return problems
}
// literalRule is why a value must be literal, said with every refusal of one.
const literalRule = "a value is literal, so a POSIX shell and the service manager read it alike, and " +
"names the machine only through the mesh's own ${machine:…} facts (novox/hq ADR 0203)"
// literalProblem is why a value cannot be written, unquoted by either reader, as the same string in
// both formats — or nothing. A `$` would expand differently in each; a quote or a backslash is
// quoting in one and a character in the other; a line break ends the line in both.
func literalProblem(v string) string {
switch {
case strings.ContainsAny(v, `'"`):
return "holds a quote"
case strings.Contains(v, `\`):
return "holds a backslash"
case strings.ContainsAny(v, "\n\r"):
return "holds a line break"
case strings.ContainsRune(v, 0):
return "holds a NUL"
case strings.Contains(ofMachine.ReplaceAllString(v, ""), "$"):
return "holds a $ that is not one of the machine's ${machine:…} facts"
}
return ""
}
// shellProblems is what is wrong with this module's shell code, from the manifest alone. The code
// itself is not judged: it is the shell's syntax, which the controller does not read.
func (m Manifest) shellProblems() []string {
var problems []string
for i, c := range m.Shell {
if !oneOf(contributionTargets(), c.For) {
problems = append(problems, fmt.Sprintf(
"%s's shell code %d is for %q; the shells are %s, the session's files %s, and the power "+
"moments %s", m.Module, i+1, c.For, strings.Join(knownShells, ", "),
strings.Join(sessionFiles, ", "), strings.Join(powerMoments, ", ")))
}
if !oneOf(knownSlots, c.Slot) {
problems = append(problems, fmt.Sprintf(
"%s's shell code %d goes in the slot %q; the slots are %s", m.Module, i+1, c.Slot,
strings.Join(knownSlots, ", ")))
}
if strings.TrimSpace(c.Code) == "" {
problems = append(problems, fmt.Sprintf("%s's shell code %d has no code", m.Module, i+1))
}
}
return problems
}
// contributionPlaceholderProblems is every place this module's resources name the environment or
// the shell's code and may not — judged from the manifest, so the catalogue check refuses it before
// a mesh does, and again at composition in the same words.
func (m Manifest) contributionPlaceholderProblems() []string {
var problems []string
for _, r := range m.Resources {
problems = append(problems, placeholderProblems(m, r)...)
problems = append(problems, seatPlaceholderProblems(m, r)...)
}
return problems
}
// placeholderProblems is what is wrong with one resource's ${environment:…} and ${shell:…}.
//
// **The seat authorises it, not the placeholder** (novox/hq ADR 0203 §5, ADR 0204 §3), as the seat
// authorises the bus's user list: a module that does not hold the account's environment writing it
// would be a second writer of a file there is one of, and a module that does not hold the login
// shell writing every module's shell code would be a second shell.
func placeholderProblems(m Manifest, r map[string]any) []string {
var problems []string
for _, field := range sortedKeys(r) {
s, ok := r[field].(string)
if !ok {
continue
}
env := ofEnvironment.FindAllStringSubmatch(s, -1)
code := ofShell.FindAllStringSubmatch(s, -1)
if len(env)+len(code) == 0 {
continue
}
if field != "content" {
// Placed only where a file's bytes are, which is where every one of them is meant to go:
// a path or an owner holding several lines of shell is nothing the host could act on.
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s in its %s; the environment and the shell's code are placed "+
"only in a file's content", m.Module, r["id"], placeholderOf(env, code), field))
continue
}
for _, e := range env {
if e[1] != EnvironmentPOSIX && e[1] != EnvironmentSystemd {
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s; the environment is ${environment:%s} or ${environment:%s}",
m.Module, r["id"], e[0], EnvironmentPOSIX, EnvironmentSystemd))
}
}
if len(env) > 0 && !m.ClaimsSeat(EnvironmentSeat) {
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s and %s does not claim %s; the account's environment is "+
"written by that seat's holder alone (novox/hq ADR 0203)",
m.Module, r["id"], env[0][0], m.Module, EnvironmentSeat))
}
// Each placeholder judged by its own target: a shell's code is the login shell's holder's to
// place (ADR 0204), the session's files the display server's (ADR 0208 §4) — and a holder of
// one placing the other's would be a second writer of a file there is one of.
refusedFor := map[string]bool{}
for _, c := range code {
target, slot, two := strings.Cut(c[1], ":")
if !two || !oneOf(contributionTargets(), target) || !oneOf(knownSlots, slot) {
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s; shell code is ${shell:<shell>:<slot>}, the shell one of "+
"%s, the session's file one of %s or the power moment one of %s, and the slot one of %s",
m.Module, r["id"], c[0], strings.Join(knownShells, ", "), strings.Join(sessionFiles, ", "),
strings.Join(powerMoments, ", "), strings.Join(knownSlots, ", ")))
continue
}
seat := placerOf(target)
if m.ClaimsSeat(seat) || refusedFor[seat] {
continue
}
refusedFor[seat] = true
if seat == PowerSeat {
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s and %s does not claim %s; every module's code for a power "+
"moment is placed by the power seat's holder alone (novox/hq ADR 0211)",
m.Module, r["id"], c[0], m.Module, seat))
continue
}
if seat == DisplayServerSeat {
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s and %s does not claim %s; every module's %s is placed by "+
"the display server's holder alone (novox/hq ADR 0208)",
m.Module, r["id"], c[0], m.Module, seat, target))
continue
}
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s and %s does not claim %s; every module's shell code is "+
"placed by the login shell's holder alone (novox/hq ADR 0204)",
m.Module, r["id"], c[0], m.Module, seat))
}
}
return problems
}
func placeholderOf(env, code [][]string) string {
if len(env) > 0 {
return env[0][0]
}
return code[0][0]
}
// contributedEnvironment is one node's environment, gathered and in the order it is written.
type contributedEnvironment struct {
// variables is by module in name order, each module's sorted by name.
variables []setBy
// start and end are PATH's entries in their final order, each once.
start, end []placedOn
}
type setBy struct {
module string
names []string
values map[string]string
}
type placedOn struct {
module, entry string
}
// inModuleOrder is the modules sorted by name — the order contributions are written in (novox/hq
// ADR 0203, ADR 0204), so the same set composes byte for byte whatever order they were assigned in.
func inModuleOrder(modules []Manifest) []Manifest {
out := append([]Manifest(nil), modules...)
sort.SliceStable(out, func(a, b int) bool { return out[a].Module < out[b].Module })
return out
}
// variablesSetOnce refuses a variable two modules on one node both set (novox/hq ADR 0203 §5),
// naming both. Neither is chosen: whichever was written last would win in one reader and not
// necessarily in the other, and the module that lost would not be told.
func variablesSetOnce(modules []Manifest) error {
setter := map[string]string{}
for _, m := range inModuleOrder(modules) {
if m.Environment == nil {
continue
}
for _, n := range sortedKeys(m.Environment.Variables) {
if first, taken := setter[n]; taken {
return fmt.Errorf(
"%s and %s both set %s on this machine; the account has one environment, so one "+
"of them must stop setting it (novox/hq ADR 0203)", first, m.Module, n)
}
setter[n] = m.Module
}
}
return nil
}
// environmentOn gathers every module's environment on a node, with the machine's facts in place.
//
// A PATH entry two modules both add is written once, where the first puts it: two toolchains
// sharing ~/.local/bin is ordinary, and nothing about it is in conflict.
func environmentOn(modules []Manifest, facts map[string]string) (contributedEnvironment, error) {
var env contributedEnvironment
if err := variablesSetOnce(modules); err != nil {
return env, err
}
placed := map[string]bool{}
for _, m := range inModuleOrder(modules) {
if m.Environment == nil {
continue
}
if len(m.Environment.Variables) > 0 {
set := setBy{module: m.Module, values: map[string]string{}}
for _, n := range sortedKeys(m.Environment.Variables) {
v, err := factsIn(m.Environment.Variables[n], facts, m.Module, n)
if err != nil {
return env, err
}
set.names = append(set.names, n)
set.values[n] = v
}
env.variables = append(env.variables, set)
}
for _, p := range m.Environment.Path {
entry, err := factsIn(p.Entry, facts, m.Module, "a PATH entry")
if err != nil {
return env, err
}
if strings.Contains(entry, ":") {
return env, fmt.Errorf("%s puts %q on PATH on this machine, which holds a colon, PATH's own separator",
m.Module, entry)
}
if placed[entry] {
continue
}
placed[entry] = true
if p.At == PathAtEnd {
env.end = append(env.end, placedOn{m.Module, entry})
} else {
env.start = append(env.start, placedOn{m.Module, entry})
}
}
}
return env, nil
}
// factsIn resolves a contributed value's ${machine:…} facts with this machine's — first, before
// either format is written, so both say the same thing (novox/hq ADR 0203).
func factsIn(v string, facts map[string]string, module, what string) (string, error) {
for _, key := range machineUsed(v) {
value, has := facts[key]
if !has {
return "", fmt.Errorf("%s sets %s to a value that says ${machine:%s}, and this machine says %s",
module, what, key, orNothing(namesOfFacts(facts)))
}
v = strings.ReplaceAll(v, fmt.Sprintf("${machine:%s}", key), value)
}
// Judged again once filled: a fact is the mesh's, and still has to be a literal both readers
// take alike.
if why := literalProblem(v); why != "" {
return "", fmt.Errorf("%s sets %s to %q on this machine, which %s — %s", module, what, v, why, literalRule)
}
return v, nil
}
// posix is the environment as lines a POSIX shell sources (novox/hq ADR 0203 §3): every variable
// exported, every PATH entry added only when it is missing, so sourcing the file twice — a login
// shell that starts another — changes nothing. POSIX sh only, because sh, bash and zsh all read it.
//
// The start entries are written last-first: each is put in front of PATH, so the last written ends
// up first, and the result reads in module order, then the order each module declared.
func (e contributedEnvironment) posix() string {
var b strings.Builder
for _, set := range e.variables {
fmt.Fprintf(&b, "# %s\n", set.module)
for _, n := range set.names {
fmt.Fprintf(&b, "export %s='%s'\n", n, set.values[n])
}
}
named := ""
for i := len(e.start) - 1; i >= 0; i-- {
p := e.start[i]
if p.module != named {
fmt.Fprintf(&b, "# %s\n", p.module)
named = p.module
}
fmt.Fprintf(&b, "case \":${PATH}:\" in *':%s:'*) ;; *) PATH='%s'\"${PATH:+:${PATH}}\" ;; esac\n",
p.entry, p.entry)
}
named = ""
for _, p := range e.end {
if p.module != named {
fmt.Fprintf(&b, "# %s\n", p.module)
named = p.module
}
fmt.Fprintf(&b, "case \":${PATH}:\" in *':%s:'*) ;; *) PATH=\"${PATH:+${PATH}:}\"'%s' ;; esac\n",
p.entry, p.entry)
}
if len(e.start)+len(e.end) > 0 {
b.WriteString("export PATH\n")
}
return b.String()
}
// systemd is the same environment as the service manager's environment.d reads it (novox/hq ADR
// 0203 §3), for the account's user manager and so for everything a graphical session starts. Read
// once per manager start, so it needs no guard against running twice; the account's existing PATH
// sits between the start and the end entries.
func (e contributedEnvironment) systemd() string {
var b strings.Builder
for _, set := range e.variables {
fmt.Fprintf(&b, "# %s\n", set.module)
for _, n := range set.names {
fmt.Fprintf(&b, "%s=%s\n", n, set.values[n])
}
}
if len(e.start) > 0 {
fmt.Fprintf(&b, "# %s\nPATH=%s${PATH:+:$PATH}\n", modulesOf(e.start), entriesOf(e.start))
}
if len(e.end) > 0 {
fmt.Fprintf(&b, "# %s\nPATH=${PATH:+$PATH:}%s\n", modulesOf(e.end), entriesOf(e.end))
}
return b.String()
}
// modulesOf names who contributed a line holding several modules' entries, in the order they appear.
func modulesOf(entries []placedOn) string {
var names []string
seen := map[string]bool{}
for _, p := range entries {
if !seen[p.module] {
seen[p.module] = true
names = append(names, p.module)
}
}
return strings.Join(names, ", ")
}
func entriesOf(entries []placedOn) string {
out := make([]string, len(entries))
for i, p := range entries {
out[i] = p.entry
}
return strings.Join(out, ":")
}
// shellCode is every module's code for one shell and one slot (novox/hq ADR 0204 §3): in module
// order, each module's pieces in the order it declared them, each preceded by a line naming the
// module, and empty when nothing is contributed.
func shellCode(modules []Manifest, shell, slot string) string {
var b strings.Builder
for _, m := range inModuleOrder(modules) {
named := false
for _, c := range m.Shell {
if c.For != shell || c.Slot != slot {
continue
}
if !named {
fmt.Fprintf(&b, "# %s\n", m.Module)
named = true
}
b.WriteString(c.Code)
if !strings.HasSuffix(c.Code, "\n") {
b.WriteString("\n")
}
}
}
return b.String()
}
// contributionsInto fills a holder's file with the node's environment and its shell code.
//
// **Last, after every other placeholder pass, and in one pass each.** Shell code is contributed text
// in a shell's own syntax — `${XDG_CACHE_HOME:-$HOME/.cache}`, `${(%):-%n}` — and the rendered
// environment holds `${PATH:+…}`: a scanner for the mesh's own placeholders that ran after these
// were in place would read the shell's expansions as the mesh's and refuse them, or fill a
// `${machine:…}` some module wrote for its shell to see. So nothing runs after them, the environment
// is filled before the shell's code is, and each is replaced in a single pass over what the holder
// wrote, so a contributed piece is never scanned again.
func contributionsInto(resource map[string]any, m Manifest, modules []Manifest, facts map[string]string, with Rendering) error {
if problems := append(placeholderProblems(m, resource), seatPlaceholderProblems(m, resource)...); len(problems) > 0 {
return fmt.Errorf("%s", problems[0])
}
content, ok := resource["content"].(string)
if !ok {
return nil
}
if ofEnvironment.MatchString(content) {
env, err := environmentOn(modules, facts)
if err != nil {
return err
}
content = ofEnvironment.ReplaceAllStringFunc(content, func(placeholder string) string {
if ofEnvironment.FindStringSubmatch(placeholder)[1] == EnvironmentSystemd {
return env.systemd()
}
return env.posix()
})
}
// Shell code and seat contributions in one pass (novox/hq ADR 0212): both are contributed text
// the controller does not read, so neither may be scanned after the other is in place — a
// contributed line that happened to spell the other's placeholder would be filled.
if ofContributed.MatchString(content) {
var failed error
content = ofContributed.ReplaceAllStringFunc(content, func(placeholder string) string {
found := ofContributed.FindStringSubmatch(placeholder)
first, second, _ := strings.Cut(found[2], ":")
if found[1] == "contribution" {
placed, err := seatContributions(modules, first, second, with)
if err != nil && failed == nil {
failed = err
}
return placed
}
return shellCode(modules, first, second)
})
if failed != nil {
return failed
}
}
resource["content"] = content
return nil
}
+471
View File
@@ -0,0 +1,471 @@
package catalogue
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// Defends novox/hq ADR 0203 (the account's environment is one module's, and every module
// contributes to it) and ADR 0204 (shell code in named slots, placed by the login shell's holder).
// contributors is a fixed set of contributions, in no particular order: what the renderings are
// asserted against byte for byte. go-toolchain and zsh both put ~/.local/bin on PATH, which is the
// ordinary case of two modules sharing a directory, and is written once.
func contributors() []Manifest {
return []Manifest{
{Module: "zsh", Environment: &Environment{
Variables: map[string]string{"XDG_CONFIG_HOME": "${machine:account-home}/.config", "EDITOR": "vim"},
Path: []PathEntry{
{Entry: "${machine:account-home}/.local/bin", At: PathAtStart},
{Entry: "${machine:account-home}/bin", At: PathAtStart},
{Entry: "/opt/scripts", At: PathAtEnd},
},
}},
{Module: "go-toolchain", Environment: &Environment{
Variables: map[string]string{"GOPATH": "${machine:account-home}/go"},
Path: []PathEntry{
{Entry: "${machine:account-home}/go/bin", At: PathAtStart},
{Entry: "/usr/local/go/bin", At: PathAtStart},
{Entry: "${machine:account-home}/.local/bin", At: PathAtStart},
},
}},
{Module: "agent", Environment: &Environment{
Variables: map[string]string{"DISABLE_AUTOUPDATER": "1"},
Path: []PathEntry{{Entry: "/opt/agent/bin", At: PathAtEnd}},
}},
// A module contributing nothing is in the set and writes nothing.
{Module: "postgres"},
}
}
var operatorFacts = map[string]string{"name": "workstation", "account": "op", "account-home": "/home/op"}
// The final PATH this set composes, around whatever the account had: the start entries in module
// order and then declared order, the account's own, then the end entries.
const composedPOSIX = `# agent
export DISABLE_AUTOUPDATER='1'
# go-toolchain
export GOPATH='/home/op/go'
# zsh
export EDITOR='vim'
export XDG_CONFIG_HOME='/home/op/.config'
# zsh
case ":${PATH}:" in *':/home/op/bin:'*) ;; *) PATH='/home/op/bin'"${PATH:+:${PATH}}" ;; esac
# go-toolchain
case ":${PATH}:" in *':/home/op/.local/bin:'*) ;; *) PATH='/home/op/.local/bin'"${PATH:+:${PATH}}" ;; esac
case ":${PATH}:" in *':/usr/local/go/bin:'*) ;; *) PATH='/usr/local/go/bin'"${PATH:+:${PATH}}" ;; esac
case ":${PATH}:" in *':/home/op/go/bin:'*) ;; *) PATH='/home/op/go/bin'"${PATH:+:${PATH}}" ;; esac
# agent
case ":${PATH}:" in *':/opt/agent/bin:'*) ;; *) PATH="${PATH:+${PATH}:}"'/opt/agent/bin' ;; esac
# zsh
case ":${PATH}:" in *':/opt/scripts:'*) ;; *) PATH="${PATH:+${PATH}:}"'/opt/scripts' ;; esac
export PATH
`
const composedSystemd = `# agent
DISABLE_AUTOUPDATER=1
# go-toolchain
GOPATH=/home/op/go
# zsh
EDITOR=vim
XDG_CONFIG_HOME=/home/op/.config
# go-toolchain, zsh
PATH=/home/op/go/bin:/usr/local/go/bin:/home/op/.local/bin:/home/op/bin${PATH:+:$PATH}
# agent, zsh
PATH=${PATH:+$PATH:}/opt/agent/bin:/opt/scripts
`
func TestTheEnvironmentRendersForAPOSIXShellByteForByte(t *testing.T) {
env, err := environmentOn(contributors(), operatorFacts)
if err != nil {
t.Fatal(err)
}
if got := env.posix(); got != composedPOSIX {
t.Fatalf("the POSIX rendering is\n%s\nnot\n%s", got, composedPOSIX)
}
}
func TestTheEnvironmentRendersForTheServiceManagerByteForByte(t *testing.T) {
env, err := environmentOn(contributors(), operatorFacts)
if err != nil {
t.Fatal(err)
}
if got := env.systemd(); got != composedSystemd {
t.Fatalf("the environment.d rendering is\n%s\nnot\n%s", got, composedSystemd)
}
}
// Sourcing twice changes nothing (ADR 0203 §3): a login shell that starts another reads the file
// again, and a PATH that grew each time would be the symptom. Run by a real `sh`, because the claim
// is about what a shell does with the file, not about what the file looks like.
func TestThePOSIXEnvironmentSourcedTwiceLeavesPATHAsOnce(t *testing.T) {
sh, err := exec.LookPath("sh")
if err != nil {
t.Skip("no sh on this machine")
}
script := "PATH=/usr/bin:/bin\n" + composedPOSIX + "once=$PATH\n" + composedPOSIX +
`[ "$PATH" = "$once" ] || { echo "changed: $once -> $PATH"; exit 1; }` + "\n" +
`echo "$PATH"; echo "$GOPATH"`
out, err := exec.Command(sh, "-c", script).CombinedOutput()
if err != nil {
t.Fatalf("sourcing twice: %v\n%s", err, out)
}
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
want := "/home/op/go/bin:/usr/local/go/bin:/home/op/.local/bin:/home/op/bin:/usr/bin:/bin:/opt/agent/bin:/opt/scripts"
if lines[0] != want {
t.Fatalf("PATH is %s, not %s", lines[0], want)
}
if lines[1] != "/home/op/go" {
t.Fatalf("GOPATH was not exported: %q", lines[1])
}
// And an entry the account already has stays where it is, and once.
out, err = exec.Command(sh, "-c", "PATH=/opt/scripts:/usr/bin\n"+composedPOSIX+`echo "$PATH"`).CombinedOutput()
if err != nil {
t.Fatalf("%v\n%s", err, out)
}
if got := strings.TrimSpace(string(out)); got !=
"/home/op/go/bin:/usr/local/go/bin:/home/op/.local/bin:/home/op/bin:/opt/scripts:/usr/bin:/opt/agent/bin" {
t.Fatalf("an entry already on PATH was added again or moved: %s", got)
}
}
// The environment.d rendering, read by the service manager's own generator where this machine has
// one — the same reader an account's user manager runs, so the PATH it composes is the one asserted.
func TestTheServiceManagerReadsTheSystemdRenderingAsMeant(t *testing.T) {
generator := "/usr/lib/systemd/user-environment-generators/30-systemd-environment-d-generator"
if _, err := os.Stat(generator); err != nil {
t.Skip("no environment.d generator on this machine")
}
config := t.TempDir()
if err := os.MkdirAll(filepath.Join(config, "environment.d"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(config, "environment.d", "50-mesh.conf"), []byte(composedSystemd), 0o644); err != nil {
t.Fatal(err)
}
cmd := exec.Command(generator)
cmd.Env = []string{"PATH=/usr/bin:/bin", "HOME=" + config, "XDG_CONFIG_HOME=" + config}
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("%v\n%s", err, out)
}
want := "PATH=/home/op/go/bin:/usr/local/go/bin:/home/op/.local/bin:/home/op/bin:/usr/bin:/bin:/opt/agent/bin:/opt/scripts"
if !strings.Contains(string(out), want+"\n") || !strings.Contains(string(out), "GOPATH=/home/op/go\n") {
t.Fatalf("the service manager read\n%s", out)
}
}
// Nothing contributed renders nothing, in both formats — not an empty `export PATH`.
func TestNoContributionsRenderNothing(t *testing.T) {
env, err := environmentOn([]Manifest{{Module: "postgres"}}, operatorFacts)
if err != nil {
t.Fatal(err)
}
if env.posix() != "" || env.systemd() != "" {
t.Fatalf("an empty environment rendered %q and %q", env.posix(), env.systemd())
}
}
// A ${machine:…} fact the machine does not have is refused naming the module, as a file's is.
func TestAContributedFactTheMachineLacksIsRefused(t *testing.T) {
_, err := environmentOn(contributors(), map[string]string{"name": "server"})
if err == nil || !strings.Contains(err.Error(), "go-toolchain sets GOPATH") ||
!strings.Contains(err.Error(), "${machine:account-home}") {
t.Fatalf("a missing account home was not refused by name: %v", err)
}
}
// ADR 0203 §5: two modules setting one variable are refused, both named — neither silently wins.
func TestAVariableTwoModulesSetIsRefusedNamingBoth(t *testing.T) {
modules := append(contributors(), Manifest{Module: "neovim", Environment: &Environment{
Variables: map[string]string{"EDITOR": "nvim"}}})
_, err := environmentOn(modules, operatorFacts)
if err == nil || err.Error() != "neovim and zsh both set EDITOR on this machine; the account has one "+
"environment, so one of them must stop setting it (novox/hq ADR 0203)" {
t.Fatalf("a variable set twice was not refused naming both: %v", err)
}
// And at composition, whether or not the node holds the environment.
r := Resolution{Node: "workstation", Account: "op", Modules: modules}
if _, err := r.Declaration(Rendering{}); err == nil || !strings.Contains(err.Error(), "neovim and zsh both set EDITOR") {
t.Fatalf("composition accepted a variable set twice: %v", err)
}
}
// shells contributes code for several shells and slots, in no order.
func shells() []Manifest {
return []Manifest{
{Module: "zsh-syntax-highlighting", Shell: []ShellCode{
{For: "zsh", Slot: "last", Code: "source /usr/share/zsh/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh"},
}},
{Module: "powerlevel10k", Shell: []ShellCode{
{For: "zsh", Slot: "first", Code: "if [[ -r \"${XDG_CACHE_HOME:-$HOME/.cache}/p10k-instant-prompt-${(%):-%n}.zsh\" ]]; then\n" +
" source \"${XDG_CACHE_HOME:-$HOME/.cache}/p10k-instant-prompt-${(%):-%n}.zsh\"\nfi\n"},
{For: "zsh", Slot: "normal", Code: "source ~/.local/share/powerlevel10k/powerlevel10k.zsh-theme"},
{For: "zsh", Slot: "normal", Code: "[[ -f ~/.local/share/powerlevel10k/p10k.zsh ]] && source ~/.local/share/powerlevel10k/p10k.zsh"},
}},
{Module: "zsh-autosuggestions", Shell: []ShellCode{
{For: "zsh", Slot: "normal", Code: "source /usr/share/zsh/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh"},
{For: "bash", Slot: "normal", Code: "echo not for zsh"},
}},
{Module: "direnv", Shell: []ShellCode{
{For: "fish", Slot: "last", Code: "direnv hook fish | source"},
{For: "bash", Slot: "last", Code: "eval \"$(direnv hook bash)\""},
}},
}
}
// ADR 0204 §3: a slot holds that shell's code only, in module order, each module's pieces in the
// order it declared them under a line naming it; empty when nothing is contributed.
func TestShellCodeLandsInItsSlotInModuleOrderForItsShellOnly(t *testing.T) {
if got, want := shellCode(shells(), "zsh", "normal"), "# powerlevel10k\n"+
"source ~/.local/share/powerlevel10k/powerlevel10k.zsh-theme\n"+
"[[ -f ~/.local/share/powerlevel10k/p10k.zsh ]] && source ~/.local/share/powerlevel10k/p10k.zsh\n"+
"# zsh-autosuggestions\n"+
"source /usr/share/zsh/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh\n"; got != want {
t.Fatalf("zsh's normal slot is\n%s\nnot\n%s", got, want)
}
if got, want := shellCode(shells(), "zsh", "last"), "# zsh-syntax-highlighting\n"+
"source /usr/share/zsh/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh\n"; got != want {
t.Fatalf("zsh's last slot is\n%s\nnot\n%s", got, want)
}
if got, want := shellCode(shells(), "bash", "last"), "# direnv\neval \"$(direnv hook bash)\"\n"; got != want {
t.Fatalf("bash's last slot is %q, not %q", got, want)
}
if got := shellCode(shells(), "fish", "first"); got != "" {
t.Fatalf("a slot nobody contributed to holds %q", got)
}
}
// The holder of node-login-shell, as WP3's zsh module writes its block, with its own zsh around the
// slots — which holds `${…}` of the shell's own that no mesh pass may touch either.
func zshHolder() Manifest {
return Manifest{Module: "zsh", Claims: []Claim{{Name: LoginShellSeat, Scope: ScopeNode}},
Resources: []map[string]any{
{"id": "zshrc", "type": "file", "path": "${machine:account-home}/.zshrc", "content": "" +
"${shell:zsh:first}" +
"PROMPT='%n@%m ${PWD/#$HOME/~} '\n" +
"${shell:zsh:normal}" +
"alias ll='ls -l'\n" +
"${shell:zsh:last}"},
}}
}
// The case the ordering exists for: contributed zsh code full of `${…}` reaches the file byte for
// byte, because the shell's code is placed after every other placeholder pass and in one pass — a
// scanner for the mesh's placeholders that ran after it would read `${XDG_CACHE_HOME:-…}` and
// `${(%):-%n}` as the mesh's, or fill a `${machine:…}` some module wrote for its shell to see.
func TestShellCodeReachesTheHoldersFileByteForByte(t *testing.T) {
modules := append(shells(), zshHolder(), Manifest{Module: "sly", Shell: []ShellCode{
{For: "zsh", Slot: "last", Code: "echo ${machine:account-home} ${secret:x} ${shell:zsh:first} ${environment:posix}"},
}})
r := Resolution{Node: "workstation", Account: "op", Modules: modules}
out, err := r.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
var zshrc map[string]any
for _, res := range out {
if res["id"] == "zsh.zshrc" {
zshrc = res
}
}
if zshrc == nil {
t.Fatalf("the holder's file was not composed: %v", out)
}
if zshrc["path"] != "/home/op/.zshrc" {
t.Fatalf("the holder's own placeholders were not filled first: %v", zshrc["path"])
}
want := "# powerlevel10k\n" +
"if [[ -r \"${XDG_CACHE_HOME:-$HOME/.cache}/p10k-instant-prompt-${(%):-%n}.zsh\" ]]; then\n" +
" source \"${XDG_CACHE_HOME:-$HOME/.cache}/p10k-instant-prompt-${(%):-%n}.zsh\"\nfi\n" +
"PROMPT='%n@%m ${PWD/#$HOME/~} '\n" +
"# powerlevel10k\n" +
"source ~/.local/share/powerlevel10k/powerlevel10k.zsh-theme\n" +
"[[ -f ~/.local/share/powerlevel10k/p10k.zsh ]] && source ~/.local/share/powerlevel10k/p10k.zsh\n" +
"# zsh-autosuggestions\n" +
"source /usr/share/zsh/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh\n" +
"alias ll='ls -l'\n" +
"# sly\n" +
"echo ${machine:account-home} ${secret:x} ${shell:zsh:first} ${environment:posix}\n" +
"# zsh-syntax-highlighting\n" +
"source /usr/share/zsh/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh\n"
if got := zshrc["content"]; got != want {
t.Fatalf("the holder's .zshrc is\n%s\nnot\n%s", got, want)
}
}
// The holder of node-environment places both renderings, and they are the same as rendered alone.
func TestTheEnvironmentHolderPlacesBothRenderings(t *testing.T) {
holder := Manifest{Module: "node-env", Claims: []Claim{{Name: EnvironmentSeat, Scope: ScopeNode}},
Resources: []map[string]any{
{"id": "posix", "type": "file", "path": "${machine:account-home}/.config/mesh/environment.sh",
"content": "# The mesh's environment.\n${environment:posix}"},
{"id": "systemd", "type": "file", "path": "${machine:account-home}/.config/environment.d/50-mesh.conf",
"content": "${environment:systemd}"},
}}
r := Resolution{Node: "workstation", Account: "op", Modules: append(contributors(), holder)}
out, err := r.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
by := map[string]any{}
for _, res := range out {
by[res["id"].(string)] = res["content"]
}
if by["node-env.posix"] != "# The mesh's environment.\n"+composedPOSIX {
t.Fatalf("the POSIX file is\n%v", by["node-env.posix"])
}
if by["node-env.systemd"] != composedSystemd {
t.Fatalf("the environment.d file is\n%v", by["node-env.systemd"])
}
}
// ADR 0203 §5 and ADR 0204 §3: a placeholder outside the seat's holder is refused — by the parser,
// which is what the catalogue check and registration run, and again at composition, in the same words.
func TestAPlaceholderOutsideTheHolderIsRefused(t *testing.T) {
for _, c := range []struct{ content, want string }{
{"${environment:posix}", "toolchain's resource rc names ${environment:posix} and toolchain does not claim node-environment"},
{"${shell:zsh:normal}", "toolchain's resource rc names ${shell:zsh:normal} and toolchain does not claim node-login-shell"},
} {
raw := `{"module":"toolchain","resources":[{"id":"rc","type":"file","path":"/etc/rc","content":"` + c.content + `"}]}`
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("the catalogue check accepted %s outside its holder: %v", c.content, err)
}
m := Manifest{Module: "toolchain", Resources: []map[string]any{
{"id": "rc", "type": "file", "path": "/etc/rc", "content": c.content}}}
r := Resolution{Node: "workstation", Account: "op", Modules: []Manifest{m}}
if _, err := r.Declaration(Rendering{}); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("composition accepted %s outside its holder: %v", c.content, err)
}
}
}
// A key nobody renders is refused, not left in the file as a literal.
func TestAnUnknownPlaceholderKeyIsRefused(t *testing.T) {
for _, c := range []struct{ content, want string }{
{"${environment:foo}", "names ${environment:foo}; the environment is ${environment:posix} or ${environment:systemd}"},
{"${shell:zsh:middle}", "names ${shell:zsh:middle}; shell code is ${shell:<shell>:<slot>}"},
{"${shell:tcsh:first}", "names ${shell:tcsh:first}; shell code is ${shell:<shell>:<slot>}"},
{"${shell:zsh}", "names ${shell:zsh}; shell code is ${shell:<shell>:<slot>}"},
} {
raw := `{"module":"holder","claims":[{"name":"node-environment","scope":"node"},{"name":"node-login-shell","scope":"node"}],` +
`"resources":[{"id":"rc","type":"file","path":"/etc/rc","content":"` + c.content + `"}]}`
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s was accepted: %v", c.content, err)
}
}
// And outside a file's content, where nothing could be placed.
raw := `{"module":"holder","claims":[{"name":"node-environment","scope":"node"}],` +
`"resources":[{"id":"rc","type":"file","path":"/etc/${environment:posix}","content":"x"}]}`
if _, err := ParseManifest([]byte(raw)); err == nil ||
!strings.Contains(err.Error(), "names ${environment:posix} in its path; the environment and the shell's code are placed only in a file's content") {
t.Errorf("a placeholder in a path was accepted: %v", err)
}
}
// What ADR 0203 §2 allows a contribution to say, refused at parse when it says anything else.
func TestAMalformedEnvironmentIsRefusedAtParse(t *testing.T) {
for _, c := range []struct{ environment, want string }{
{`{"variables":{"1X":"a"}}`, `tool sets the variable "1X", which is not a name a shell accepts`},
{`{"variables":{"MY-VAR":"a"}}`, `tool sets the variable "MY-VAR", which is not a name a shell accepts`},
{`{"variables":{"PATH":"/bin"}}`, `tool sets PATH as a variable; a module adds an entry under environment.path`},
{`{"variables":{"A":"$HOME/x"}}`, `tool sets A to "$HOME/x", which holds a $ that is not one of the machine's ${machine:…} facts`},
{`{"variables":{"A":"${HOME}/x"}}`, `tool sets A to "${HOME}/x", which holds a $`},
{`{"variables":{"A":"it's"}}`, `tool sets A to "it's", which holds a quote`},
{`{"variables":{"A":"say \"hi\""}}`, `which holds a quote`},
{`{"variables":{"A":"a\\b"}}`, `which holds a backslash`},
{`{"variables":{"A":"a\nb"}}`, `which holds a line break`},
{`{"variables":{"A":"a\u0000b"}}`, `which holds a NUL`},
{`{"path":[{"entry":"","at":"start"}]}`, `tool's PATH entry 1 names no directory`},
{`{"path":[{"entry":"/a:/b","at":"start"}]}`, `tool puts "/a:/b" on PATH, which holds a colon`},
{`{"path":[{"entry":"$HOME/bin","at":"start"}]}`, `tool puts "$HOME/bin" on PATH, which holds a $`},
{`{"path":[{"entry":"/a","at":"middle"}]}`, `tool puts "/a" on PATH at "middle"; an entry goes at "start" or "end"`},
{`{"path":[{"entry":"/a"}]}`, `tool puts "/a" on PATH at ""`},
{`{"path":[{"entry":"/a","at":"start"},{"entry":"/a","at":"end"}]}`, `tool puts "/a" on PATH twice`},
{`{"variables":{"A":"x"},"paths":[]}`, `unknown field "paths"`},
} {
_, err := ParseManifest([]byte(`{"module":"tool","environment":` + c.environment + `}`))
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: want %q, got %v", c.environment, c.want, err)
}
}
// What is allowed: a literal, and the machine's own facts.
if _, err := ParseManifest([]byte(`{"module":"tool","environment":{` +
`"variables":{"GOPATH":"${machine:account-home}/go","DISABLE_X":"1","ANSWER":"a b+c=d"},` +
`"path":[{"entry":"${machine:account-home}/go/bin","at":"start"},{"entry":"/opt/x","at":"end"}]}}`)); err != nil {
t.Fatalf("a well-formed environment was refused: %v", err)
}
}
func TestMalformedShellCodeIsRefusedAtParse(t *testing.T) {
for _, c := range []struct{ shell, want string }{
{`[{"for":"tcsh","slot":"normal","code":"x"}]`, `tool's shell code 1 is for "tcsh"; the shells are zsh, bash, fish`},
{`[{"for":"zsh","slot":"middle","code":"x"}]`, `tool's shell code 1 goes in the slot "middle"; the slots are first, normal, last`},
{`[{"for":"zsh","slot":"last","code":"x"},{"for":"zsh","slot":"last","code":" \n"}]`, `tool's shell code 2 has no code`},
{`[{"for":"zsh","slot":"last","code":"x","order":1}]`, `unknown field "order"`},
} {
_, err := ParseManifest([]byte(`{"module":"tool","shell":` + c.shell + `}`))
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: want %q, got %v", c.shell, c.want, err)
}
}
// The code itself is never judged: a shell's own `${…}` is not the mesh's.
if _, err := ParseManifest([]byte(`{"module":"tool","shell":[{"for":"zsh","slot":"first",` +
`"code":"source \"${XDG_CACHE_HOME:-$HOME/.cache}/p10k-instant-prompt-${(%):-%n}.zsh\""}]}`)); err != nil {
t.Fatalf("shell code was judged as if it were the mesh's: %v", err)
}
}
// ADR 0203 §1 and ADR 0204 §1: both seats are the mesh's own, held once per machine; the login
// shell's contract is `execute`, described and with a schema an agent can call.
func TestTheSeatTableCarriesTheEnvironmentAndTheLoginShell(t *testing.T) {
env, ok := SeatNamed("node-environment")
if !ok || env.Scope != ScopeNode || env.Decision != "novox/hq ADR 0203" ||
len(env.Serves)+len(env.Accepts)+len(env.Emits) != 0 || env.Delivers != "" {
t.Fatalf("node-environment is not a node seat with no protocol: %+v (defined %v)", env, ok)
}
shell, ok := SeatNamed("node-login-shell")
if !ok || shell.Scope != ScopeNode || shell.Decision != "novox/hq ADR 0204" {
t.Fatalf("node-login-shell is not a node seat: %+v (defined %v)", shell, ok)
}
if len(shell.Serves) != 1 || shell.Serves[0].Name != "execute" || shell.Serves[0].Description == "" {
t.Fatalf("the login shell serves %+v, not execute alone", shell.Serves)
}
props, _ := shell.Serves[0].Input["properties"].(map[string]any)
required, _ := shell.Serves[0].Input["required"].([]string)
if _, has := props["command"]; !has || len(required) != 1 || required[0] != "command" {
t.Fatalf("execute does not require a command: %v", shell.Serves[0].Input)
}
if _, has := props["timeout_seconds"]; !has {
t.Fatalf("execute takes no timeout: %v", props)
}
}
// ADR 0204 §1: the login shell is the mesh's, so no module declares it — neither under the mesh's
// name nor under the name a module gave it before.
func TestNoModuleMayDeclareTheLoginShell(t *testing.T) {
for _, n := range []string{"login-shell", "node-login-shell", "node-environment"} {
raw := `{"module":"zsh","seats":[{"name":"` + n + `","scope":"node","serves":["execute"]}],"tools":["execute"]}`
_, err := ParseManifest([]byte(raw))
if err == nil {
t.Errorf("a module declaring %q was accepted", n)
}
}
got := strings.Join(declaredSeatProblems(Manifest{Module: "zsh",
DefinesSeats: []SeatDeclaration{{Name: "login-shell", Scope: ScopeNode}}}), "; ")
if !strings.Contains(got, `zsh declares a seat named "login-shell"; the login shell is the mesh's own seat node-login-shell`) {
t.Fatalf("declaring login-shell was not refused by name: %q", got)
}
// And a shell module claiming the mesh's seat, serving execute, is what the seat is for.
m, err := ParseManifest([]byte(`{"module":"zsh","tools":["execute"],` +
`"claims":[{"name":"node-login-shell","scope":"node"}]}`))
if err != nil {
t.Fatal(err)
}
if err := CanHold(m, Seat{Name: LoginShellSeat, Scope: ScopeNode, Serves: loginShellVerbs()}); err != nil {
t.Fatalf("a shell module claiming the seat cannot hold it: %v", err)
}
}
+32
View File
@@ -3,6 +3,7 @@ package catalogue
import (
"fmt"
"regexp"
"slices"
"strings"
)
@@ -159,3 +160,34 @@ func consumePattern(pattern string) error {
}
return nil
}
// The events a provider says about its consumers (novox/hq ADR 0224): a consumer it has failed
// without one success for minutes, and that consumer succeeding again or being withdrawn. The
// controller follows them from every module and `status` names a consumer failing until it recovers.
const (
ProvisionerFailing = "provisioner.failing"
ProvisionerRecovered = "provisioner.recovered"
)
// ProvisionerEvents are both, in the order they are said.
var ProvisionerEvents = []string{ProvisionerFailing, ProvisionerRecovered}
// EmitsAll is every event a module may publish: what it declares and, for a module that receives
// contributions — a provider, running a provisioner over them — the provider's standing events.
//
// **Derived, not declared**, because they are the mesh's rule about every provider rather than
// anything one module chose to say: a provider whose manifest forgot them would fail its consumers
// as silently as on 2026-10-05, with its announcement refused by the bus (novox/hq issue 179). Every
// grant of a module's publishing reads this, never the declared list alone.
func (m Manifest) EmitsAll() []string {
out := append([]string(nil), m.Emits...)
if len(m.Receives) == 0 {
return out
}
for _, e := range ProvisionerEvents {
if !slices.Contains(out, e) {
out = append(out, e)
}
}
return out
}
@@ -0,0 +1,84 @@
package catalogue
import (
"strings"
"testing"
)
// novox/hq issue 190, step 5 (ADR 0222): what the mesh computes for a module is held to the rule
// every module is — no two modules on a machine declare one path, unit, name or package. The
// private network once declared the container runtime's file and service beside the runtime's own
// module, and nothing refused it, because the collision check only ever saw catalogue manifests.
func runtimesOwn() Manifest {
return Manifest{Module: "docker", Resources: []map[string]any{
{"id": "daemon", "type": "file", "path": "/etc/docker/daemon.json", "into": "json",
"content": `{"live-restore": true}`},
{"id": "runtime", "type": "service", "unit": "docker.service", "state": "running",
"reload-on": []any{"daemon"}},
}}
}
func TestAGeneratedResourceCollidingWithAModulesIsRefused(t *testing.T) {
r := Resolution{Node: "workstation", Modules: []Manifest{
{Module: "mesh-network", Computed: "mesh-network", Provides: Offers("private-network")},
runtimesOwn(),
}}
_, err := r.Declaration(Rendering{Generators: map[string]Generator{"mesh-network": reloading{}}})
if err == nil {
t.Fatal("a generated resource declaring the runtime's file beside the runtime's module was accepted")
}
for _, want := range []string{"mesh-network", "docker", "/etc/docker/daemon.json", "docker.service"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not name %s: %v", want, err)
}
}
}
func TestAGeneratedResourceBesideAModulesOwnIsComposed(t *testing.T) {
r := Resolution{Node: "workstation", Modules: []Manifest{
{Module: "mesh-network", Computed: "mesh-network", Provides: Offers("private-network")},
runtimesOwn(),
}}
gen := &fake{on: map[string]bool{"workstation": true}}
out, err := r.Declaration(Rendering{Generators: map[string]Generator{"mesh-network": gen}})
if err != nil {
t.Fatalf("disjoint resources were refused: %v", err)
}
if fileNamed(out, "docker.daemon") == nil {
t.Fatalf("the runtime's own file is missing: %v", out)
}
// And a machine not on the network yet generates nothing, which collides with nothing.
if _, err := r.Declaration(Rendering{Generators: map[string]Generator{"mesh-network": &fake{}}}); err != nil {
t.Fatalf("a machine off the network was refused: %v", err)
}
}
// The catalogue's container runtime module states the mesh's registry itself (ADR 0222).
func TestTheRuntimesModuleTrustsTheMeshsRegistry(t *testing.T) {
docker := catalogueManifest(t, "docker")
r := Resolution{Node: "workstation", Modules: []Manifest{docker}}
out, err := r.Declaration(Rendering{
SeatReach: map[string]string{"mesh-artifact-store": "anchor.internal:5100"},
})
if err != nil {
t.Fatal(err)
}
daemon := fileNamed(out, "docker.daemon")
if daemon == nil || daemon["into"] != "json" {
t.Fatalf("the runtime's file is not written into: %v", daemon)
}
content, _ := daemon["content"].(string)
if !strings.Contains(content, `"insecure-registries": ["anchor.internal:5100"]`) ||
!strings.Contains(content, `"live-restore": true`) {
t.Fatalf("the runtime's file says %q", content)
}
out, err = r.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
if content, _ := fileNamed(out, "docker.daemon")["content"].(string); strings.Contains(content, "insecure-registries") {
t.Fatalf("with no store on the network, the runtime is told %q", content)
}
}
@@ -0,0 +1,103 @@
package catalogue
import (
"strings"
"testing"
)
// A grant secret is read by whatever provisions, and that stopped being root (novox/hq issue 225).
//
// The mesh seals one credential per consumer beside the provider's contributions file. The
// provider's harness reads both: the file to learn who asked, the secret to set their password.
// While a module's own code ran in a container as root, a root-owned 0600 file was readable by
// the thing that needed it. ADR 0198 moved that code under the node's runtime, which runs as the
// operator's account — and the secret stayed root's.
//
// **The cost was silence.** The harness says `secret not readable yet`, which is true and
// ordinary on the first pass, so four thousand refusals in three hours read as patience. No user
// was ever created, and two consumers crash-looped against a database that had never heard of
// them.
//
// The same reasoning is already written for a module's *own* secrets, three hundred lines above:
// "a root-owned 0600 file is one that process cannot read". This is that rule reaching the other
// kind of secret the mesh writes for a module.
// aProviderWithABundle is a provider whose code is a bundle the node's runtime runs — the shape
// every TypeScript provisioner has since ADR 0198.
func aProviderWithABundle() Manifest {
return Manifest{
Module: "mongodb", Version: "1",
Provides: FromAnywhere("mongodb-database"),
Receives: map[string]string{"mongodb-database": "/var/lib/mongodb/grants/mesh.json"},
Grants: map[string]string{"mongodb-database": "/var/lib/mongodb/grants"},
Bundles: []Bundle{{Name: "code", Language: "typescript"}},
Resources: []map[string]any{{
"id": "server", "type": "container", "name": "mongodb-server",
"image": "mongo@sha256:" + strings.Repeat("a", 64),
}},
}
}
func TestAGrantSecretIsOwnedByTheAccountThatProvisions(t *testing.T) {
r, err := Resolve(shelf(aProviderWithABundle()), []string{"mongodb"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
r.Account = "operator"
out, err := r.Declaration(Rendering{Grants: []Grant{{
Provision: "mongodb-database", Consumer: "workstation", From: "photos", Slug: "photos",
Values: map[string]any{}, Sealed: "c2VhbGVk",
}}})
if err != nil {
t.Fatal(err)
}
var secret map[string]any
for _, res := range out {
if res["type"] == "file" && strings.HasSuffix(fmtPath(res), ".secret") {
secret = res
}
}
if secret == nil {
t.Fatalf("no grant secret was composed at all: %v", out)
}
if got := secret["owner"]; got != "operator" {
t.Fatalf("the grant secret at %v belongs to %v; the provisioner runs as %q and a "+
"root-owned 0600 file is one it cannot read — which is silent, because the harness "+
"calls it \"not readable yet\"", fmtPath(secret), got, "operator")
}
}
// And a provider whose code still runs in a container keeps the owner it declares, so this
// changes nothing for the modules the runtime has not taken.
func TestAContainerProvidersGrantSecretKeepsItsDeclaredOwner(t *testing.T) {
m := aProviderWithABundle()
m.Bundles = nil
m.SecretsOwner = "65534:65534"
r, err := Resolve(shelf(m), []string{"mongodb"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
r.Account = "operator"
out, err := r.Declaration(Rendering{Grants: []Grant{{
Provision: "mongodb-database", Consumer: "workstation", From: "photos", Slug: "photos",
Values: map[string]any{}, Sealed: "c2VhbGVk",
}}})
if err != nil {
t.Fatal(err)
}
for _, res := range out {
if res["type"] == "file" && strings.HasSuffix(fmtPath(res), ".secret") {
if got := res["owner"]; got != "65534:65534" {
t.Fatalf("a container provider's grant secret belongs to %v, not what it declares", got)
}
return
}
}
t.Fatal("no grant secret was composed")
}
func fmtPath(r map[string]any) string {
p, _ := r["path"].(string)
return p
}
+109
View File
@@ -0,0 +1,109 @@
package catalogue
// The graphical session's seats (novox/hq ADR 0208): one module per piece of software, each piece's
// role a node seat in the mesh's own set, so i3 and sway, xterm and foot, rofi and dmenu compete for
// a role rather than each inventing one — and a machine running two of one role is refused at
// assignment instead of found by two bars on one screen.
const (
LoginManagerSeat = "node-login-manager"
DisplayServerSeat = "node-display-server"
DisplaySessionSeat = "node-display-session"
TerminalEmulatorSeat = "node-terminal-emulator"
LauncherSeat = "node-launcher"
NotifierSeat = "node-notifier"
LockScreenSeat = "node-lock-screen"
ClipboardSeat = "node-clipboard"
BarSeat = "node-bar"
CompositorSeat = "node-compositor"
SecretServiceSeat = "node-secret-service"
)
// graphicalSessionSeats are the eleven, in the order ADR 0208's table reads, each with the verbs
// research 026/05 starts it with. Three have none yet: the bar, the compositor and the secret
// service are roles a second holder competes for, and nothing has needed to ask them anything.
func graphicalSessionSeats() []Seat {
const decided = "novox/hq ADR 0208"
return []Seat{
{Name: LoginManagerSeat, Scope: ScopeNode, Decision: decided, Serves: []Verb{
{Name: "sessions", Description: "The sessions the login manager offers on this machine, and which " +
"one the operator account starts by default.",
Input: schema(map[string]string{}, nil)},
}},
{Name: DisplayServerSeat, Scope: ScopeNode, Decision: decided, Serves: []Verb{
{Name: "displays", Description: "The monitors connected now, each with its identity, its modes and " +
"where it is placed; and the layout profile in force, if one matches.",
Input: schema(map[string]string{}, nil)},
// Profiles are keyed by the monitors' identities and are the operator's data (ADR 0208 §6).
{Name: "layout", Description: "The monitor layout profiles, keyed by the connected monitors' " +
"identities: list them, save the current arrangement under a name, or apply one.",
Input: withEnum(schema(map[string]string{
"action": "list, save or apply",
"name": "the profile to save or apply (save and apply only)",
}, []string{"action"}), "action", "list", "save", "apply")},
}},
// It receives window-manager configuration lines from every other module (novox/hq ADR 0212):
// bindings, start-up commands, rules — placed by the session's holder, never written into
// its directory by the contributor.
{Name: DisplaySessionSeat, Scope: ScopeNode, Decision: decided,
Receives: []Receivable{{Kind: "config", Comment: "#"}}, Serves: []Verb{
{Name: "reload", Description: "Reload the session's configuration in place, keeping its windows.",
Input: schema(map[string]string{}, nil)},
{Name: "workspaces", Description: "The session's workspaces: each one's name, output, and whether " +
"it is visible or focused.",
Input: schema(map[string]string{}, nil)},
{Name: "windows", Description: "The session's windows: each one's title, class, workspace and " +
"whether it has focus; narrowed to one workspace when named.",
Input: schema(map[string]string{"workspace": "one workspace (optional)"}, nil)},
}},
{Name: TerminalEmulatorSeat, Scope: ScopeNode, Decision: decided, Serves: []Verb{
{Name: "open", Description: "Open a terminal window in the operator's session, running a command " +
"or the login shell, in a directory or the account's home.",
Input: schema(map[string]string{
"command": "what to run in it (optional; the login shell when absent)",
"directory": "where it starts (optional; the account's home when absent)",
}, nil)},
}},
{Name: LauncherSeat, Scope: ScopeNode, Decision: decided, Serves: []Verb{
{Name: "menu", Description: "Put a menu of choices in front of the operator and answer with the " +
"one chosen, or nothing when the menu was dismissed — the dmenu-compatible contract.",
Input: map[string]any{"type": "object", "required": []string{"choices"},
"properties": map[string]any{
"choices": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": "the lines to choose between, in order"},
"prompt": map[string]any{"type": "string", "description": "what the menu asks (optional)"},
}}},
}},
{Name: NotifierSeat, Scope: ScopeNode, Decision: decided, Serves: []Verb{
{Name: "send", Description: "Show the operator a notification.",
Input: withEnum(schema(map[string]string{
"title": "the notification's summary",
"body": "its text (optional)",
"urgency": "low, normal (the default) or critical",
}, []string{"title"}), "urgency", "low", "normal", "critical")},
{Name: "history", Description: "The notifications shown lately, newest first.",
Input: schema(map[string]string{"limit": "how many (optional, default 20)"}, nil)},
}},
{Name: LockScreenSeat, Scope: ScopeNode, Decision: decided, Serves: []Verb{
{Name: "lock", Description: "Lock the operator's session now.",
Input: schema(map[string]string{}, nil)},
}},
{Name: ClipboardSeat, Scope: ScopeNode, Decision: decided, Serves: []Verb{
{Name: "history", Description: "What the clipboard held lately, newest first.",
Input: schema(map[string]string{"limit": "how many (optional, default 20)"}, nil)},
{Name: "copy", Description: "Put text on the operator's clipboard.",
Input: schema(map[string]string{"text": "the text"}, []string{"text"})},
}},
{Name: BarSeat, Scope: ScopeNode, Decision: decided},
{Name: CompositorSeat, Scope: ScopeNode, Decision: decided},
{Name: SecretServiceSeat, Scope: ScopeNode, Decision: decided},
}
}
// withEnum narrows one string property of a schema to the values it may take, so a caller is told
// the choices by the schema rather than by a refusal.
func withEnum(s map[string]any, property string, values ...string) map[string]any {
props := s["properties"].(map[string]any)
p := props[property].(map[string]any)
p["enum"] = values
return s
}
@@ -0,0 +1,213 @@
package catalogue
import (
"encoding/json"
"reflect"
"strings"
"testing"
)
// Defends novox/hq ADR 0208: the graphical session is one module per piece, on the mesh's seats.
// §2: the eleven roles are the mesh's own node seats, each with the verbs it starts with.
func TestTheGraphicalSessionsSeatsAreTheMeshsOwnWithTheirVerbs(t *testing.T) {
want := map[string][]string{
LoginManagerSeat: {"sessions"},
DisplayServerSeat: {"displays", "layout"},
DisplaySessionSeat: {"reload", "workspaces", "windows"},
TerminalEmulatorSeat: {"open"},
LauncherSeat: {"menu"},
NotifierSeat: {"send", "history"},
LockScreenSeat: {"lock"},
ClipboardSeat: {"history", "copy"},
BarSeat: nil,
CompositorSeat: nil,
SecretServiceSeat: nil,
}
for name, verbs := range want {
s, ok := SeatNamed(name)
if !ok {
t.Errorf("%s is not in the mesh's set", name)
continue
}
if s.Scope != ScopeNode || s.Decision != "novox/hq ADR 0208" {
t.Errorf("%s is %s-scoped under %q", name, s.Scope, s.Decision)
}
var got []string
for _, v := range s.Serves {
got = append(got, v.Name)
if v.Description == "" || v.Input["type"] != "object" {
t.Errorf("%s.%s has no description or no object schema", name, v.Name)
}
}
if !reflect.DeepEqual(got, verbs) {
t.Errorf("%s serves %v, want %v", name, got, verbs)
}
}
// The launcher's menu takes a list, and the layout verb says its actions.
menu, _ := SeatNamed(LauncherSeat)
choices := menu.Serves[0].Input["properties"].(map[string]any)["choices"].(map[string]any)
if choices["type"] != "array" {
t.Errorf("menu's choices are %v, not a list", choices["type"])
}
display, _ := SeatNamed(DisplayServerSeat)
action := display.Serves[1].Input["properties"].(map[string]any)["action"].(map[string]any)
if !reflect.DeepEqual(action["enum"], []string{"list", "save", "apply"}) {
t.Errorf("layout's actions are %v", action["enum"])
}
// And they survive the store's JSON, which is where the live set comes from.
if _, err := json.Marshal(graphicalSessionSeats()); err != nil {
t.Fatal(err)
}
}
// §2: a module may claim one of them, and may not declare it as its own.
func TestNoModuleMayDeclareAGraphicalSessionSeat(t *testing.T) {
raw := `{"module":"xorg","seats":[{"name":"node-display-server","scope":"node"}]}`
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), "mesh's own namespace") {
t.Fatalf("a module declared node-display-server as its own: %v", err)
}
}
func displayServer(name, display string, claims ...string) Manifest {
m := Manifest{Module: name, Provides: []Offer{{Name: display, Reach: ReachMachine}}}
for _, c := range claims {
m.Claims = append(m.Claims, Claim{Name: c})
}
return m
}
func windowManager() Manifest {
return Manifest{Module: "i3", Requires: []string{"x11-display"}}
}
// §3: a display is resolved on the requiring module's own node.
func TestAMachineReachRequirementResolvesToTheProviderOnItsOwnNode(t *testing.T) {
cat := shelf(windowManager(), displayServer("xorg", "x11-display", DisplayServerSeat))
got, err := Resolve(cat, []string{"xorg", "i3"}, workstation(), World{})
if err != nil {
t.Fatalf("i3 beside xorg did not resolve: %v", err)
}
if !reflect.DeepEqual(names(got), []string{"xorg", "i3"}) && !reflect.DeepEqual(names(got), []string{"i3", "xorg"}) {
t.Errorf("resolved %v", names(got))
}
}
// §3: never answered by installing a provider, and never by another machine's.
func TestAMachineReachRequirementIsNotPulledInNorAnsweredFromAnotherNode(t *testing.T) {
cat := shelf(windowManager(),
displayServer("xorg", "x11-display", DisplayServerSeat),
displayServer("xwayland", "x11-display"))
// Another machine runs xorg and says so to the world; it does not count.
world := World{Offered: map[string][]Provider{
"x11-display": {{Node: "laptop", At: "laptop.mesh", Module: "xorg"}}}}
_, err := Resolve(cat, []string{"i3"}, workstation(), world)
if err == nil {
t.Fatal("i3 resolved on a machine with no display of its own")
}
for _, want := range []string{
`"x11-display" is wanted by i3`, "usable only on the machine that provides it",
"assign one to workstation", "xorg (holds node-display-server)", "xwayland",
} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q:\n%v", want, err)
}
}
// With a single provider in the catalogue too: one candidate is still not a choice to make
// for somebody, unlike a node-scoped provision without the machine's reach.
_, err = Resolve(shelf(windowManager(), displayServer("xorg", "x11-display", DisplayServerSeat)),
[]string{"i3"}, workstation(), World{})
if err == nil {
t.Fatal("xorg was pulled in for i3")
}
// Not in the first pass, whose refusals take the machine off the network.
if _, err := Resolve(cat, []string{"i3"}, workstation(), World{Unchecked: true}); err != nil {
t.Errorf("the first pass refused: %v", err)
}
}
func TestTheMachinesReachIsAProvisionsOnlyReachAndIsNodeScoped(t *testing.T) {
for _, c := range []struct{ provides, want string }{
{`{"name":"x11-display","reach":"internal"}`, `with reach "internal"; a provision's reach is "machine" or nothing`},
{`{"name":"x11-display","scope":"mesh","reach":"machine"}`, `at scope "mesh" with the machine's reach`},
} {
_, err := ParseManifest([]byte(`{"module":"xorg","provides":[` + c.provides + `]}`))
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: want %q, got %v", c.provides, c.want, err)
}
}
m, err := ParseManifest([]byte(`{"module":"xorg","provides":[{"name":"x11-display","reach":"machine"}]}`))
if err != nil {
t.Fatal(err)
}
if !m.Provides[0].MachineReach() {
t.Fatal("the reach was not read")
}
back, _ := json.Marshal(m.Provides[0])
if string(back) != `{"name":"x11-display","reach":"machine"}` {
t.Errorf("written back as %s", back)
}
}
func TestACatalogueDisagreeingAboutAProvisionsReachIsRefused(t *testing.T) {
cat := shelf(windowManager(), displayServer("xorg", "x11-display"),
Manifest{Module: "fake-x", Provides: Offers("x11-display")})
_, err := Resolve(cat, []string{"xorg", "i3"}, workstation(), World{})
if err == nil || !strings.Contains(err.Error(), `the catalogue disagrees about "x11-display"`) {
t.Fatalf("a provision with and without the machine's reach gave %v", err)
}
}
// §4: xinitrc and xresources slots, placed by the display server's holder alone.
func TestTheSessionsFilesArePlacedByTheDisplayServersHolderAlone(t *testing.T) {
xorg := Manifest{Module: "xorg", Claims: []Claim{{Name: DisplayServerSeat}},
Shell: []ShellCode{{For: "xinitrc", Slot: "first", Code: "xset s off"}},
Resources: []map[string]any{
{"id": "xinitrc", "type": "file", "path": "/home/op/.xinitrc",
"content": "${shell:xinitrc:first}${shell:xinitrc:normal}${shell:xinitrc:last}"},
{"id": "xresources", "type": "file", "path": "/home/op/.Xresources",
"content": "${shell:xresources:normal}"},
}}
i3 := Manifest{Module: "i3", Shell: []ShellCode{{For: "xinitrc", Slot: "last", Code: "exec i3"}}}
theme := Manifest{Module: "theme", Shell: []ShellCode{
{For: "xresources", Slot: "normal", Code: "Xft.dpi: 96"},
{For: "zsh", Slot: "normal", Code: "not for the session"},
}}
r := Resolution{Node: "workstation", Account: "op", Modules: []Manifest{xorg, i3, theme}}
out, err := r.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
by := map[string]any{}
for _, res := range out {
by[res["id"].(string)] = res["content"]
}
if got := by["xorg.xinitrc"]; got != "# xorg\nxset s off\n# i3\nexec i3\n" {
t.Errorf("the .xinitrc is %q", got)
}
if got := by["xorg.xresources"]; got != "# theme\nXft.dpi: 96\n" {
t.Errorf("the .Xresources is %q", got)
}
// The contributions parse; the placeholders parse only in the holder.
if _, err := ParseManifest([]byte(`{"module":"i3","shell":[{"for":"xinitrc","slot":"last","code":"exec i3"},` +
`{"for":"xresources","slot":"normal","code":"i3.font: x"}]}`)); err != nil {
t.Fatalf("a session contribution was refused: %v", err)
}
for _, c := range []struct{ claims, content, want string }{
{``, "${shell:xinitrc:normal}", "does not claim node-display-server; every module's xinitrc is placed by the display server's holder alone"},
{`{"name":"node-login-shell"}`, "${shell:xresources:normal}", "does not claim node-display-server"},
{`{"name":"node-display-server"}`, "${shell:zsh:normal}", "does not claim node-login-shell"},
{`{"name":"node-display-server"}`, "${shell:xsession:normal}", "the session's file one of xinitrc, xresources"},
} {
raw := `{"module":"holder","claims":[` + c.claims + `],"resources":[{"id":"rc","type":"file","path":"/etc/rc","content":"` +
c.content + `"}]}`
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s with claims [%s]: want %q, got %v", c.content, c.claims, c.want, err)
}
}
if _, err := ParseManifest([]byte(`{"module":"xorg","claims":[{"name":"node-display-server"}],` +
`"resources":[{"id":"rc","type":"file","path":"/home/op/.xinitrc","content":"${shell:xinitrc:last}"}]}`)); err != nil {
t.Errorf("the display server's holder could not place the session's slots: %v", err)
}
}
+98
View File
@@ -0,0 +1,98 @@
package catalogue
import (
"strings"
"testing"
)
// novox/hq ADR 0223 part 3: a machine's names are one seat's. The `hosts` module holding
// `node-hosts-file` became `hostname` holding `node-hostname`, which writes /etc/hostname beside the
// machine's own lines in /etc/hosts. The seat was renamed (ADR 0122), so what claims the old name — a
// manifest registered before the rename — still holds the one seat.
func withHostnameAlias(t *testing.T) {
t.Helper()
was := aliases
t.Cleanup(func() { aliases = was })
UseAliases(map[string]string{"node-hosts-file": "node-hostname"})
}
func TestTheHostsFilesFormerNameResolvesToTheHostnameSeat(t *testing.T) {
if _, known := SeatNamed("node-hostname"); !known {
t.Fatal("node-hostname is not in the mesh's set")
}
withHostnameAlias(t)
seat, known := SeatNamed("node-hosts-file")
if !known || seat.Name != "node-hostname" || seat.Scope != ScopeNode {
t.Fatalf("the former name did not resolve: %+v %v", seat, known)
}
var verbs []string
for _, v := range seat.Serves {
verbs = append(verbs, v.Name)
}
if strings.Join(verbs, " ") != "entries add remove" {
t.Errorf("the renamed seat serves %v; its verbs are unchanged", verbs)
}
}
// One seat under either name: the module registered before the rename and the one after cannot both
// hold it on one machine.
func TestTheOldAndTheNewClaimantAreOneSeatOnAMachine(t *testing.T) {
withHostnameAlias(t)
cat := shelf(
mod("hosts", nil, nil, nil, Claim{Name: "node-hosts-file"}),
mod("hostname", nil, nil, nil, Claim{Name: "node-hostname"}),
)
_, err := Resolve(cat, []string{"hosts", "hostname"}, workstation(), World{})
if err == nil || !strings.Contains(err.Error(), "node-hostname") {
t.Errorf("hosts and hostname both held the machine's names on one machine: %v", err)
}
if _, err := Resolve(cat, []string{"hosts"}, workstation(), World{}); err != nil {
t.Errorf("a machine still assigned hosts under the old name does not resolve: %v", err)
}
}
// The catalogue's module: /etc/hostname is the operator's `hostname` setting. Without it the module is
// left out, naming the key — the mesh never renames a machine on its own — and a mesh-wide setting
// naming ${machine:name} gives every machine its mesh name.
func TestTheMachinesNameIsItsSetting(t *testing.T) {
cat := map[string]Manifest{"hostname": catalogueManifest(t, "hostname")}
got, err := Resolve(cat, []string{"hostname"}, Node{Name: "ace", At: "ace.internal"}, World{})
if err != nil {
t.Fatal(err)
}
machines := map[string]string{"ace.internal": "10.42.0.2"}
nameOf := func(settings SettingsBy) (string, string) {
t.Helper()
composed, err := got.Compose(Rendering{Names: machines, Machines: machines, Suffix: "internal",
Settings: settings})
if err != nil {
t.Fatal(err)
}
if why := composed.LeftOut["hostname"]; why != "" {
return "", why
}
for _, r := range composed.Resources {
if r["path"] == "/etc/hostname" {
return r["content"].(string), ""
}
}
t.Fatal("no /etc/hostname composed")
return "", ""
}
if _, why := nameOf(nil); !strings.Contains(why, "hostname") {
t.Errorf("with no setting the machine's name was written, or left out for another reason: %q", why)
}
if name, why := nameOf(SettingsBy{"hostname": {{From: "ace", Values: map[string]any{"hostname": "Ace"}}}}); name != "Ace\n" {
t.Errorf("the operator's name for the machine gave %q (%s)", name, why)
}
mesh := Layer{From: "the mesh", Values: map[string]any{"hostname": "${machine:name}"}}
if name, why := nameOf(SettingsBy{"hostname": {mesh}}); name != "ace\n" {
t.Errorf("a mesh-wide ${machine:name} gave %q (%s)", name, why)
}
node := Layer{From: "ace", Values: map[string]any{"hostname": "Ace"}}
if name, why := nameOf(SettingsBy{"hostname": {mesh, node}}); name != "Ace\n" {
t.Errorf("a machine's own name over the mesh-wide one gave %q (%s)", name, why)
}
}
-106
View File
@@ -1,106 +0,0 @@
package catalogue_test
import (
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/overlay"
)
// novox/hq issue 128, held where the host will see it: the shipped networking module's names,
// through the whole composition, and not only through FactsInto.
//
// Composition prefixes a fact's id with the module that asked for it and passes everything else
// through; a step that dropped `into` on the way would send the region as a whole file, and the
// host would write the machine's hosts file over again with every unit test above still green.
// onTheNetwork stands in for the overlay's generator: the node is part of the private network,
// and what the generator writes is not what is under test here.
type onTheNetwork struct{}
func (onTheNetwork) Resources(string) ([]map[string]any, bool, error) {
return []map[string]any{{"id": "overlay-config", "type": "file",
"path": "/etc/wireguard/mesh0.conf", "mode": "0600", "content": "[Interface]\n"}}, true, nil
}
func TestTheHostsRegionArrivesAsTheHostWillReadIt(t *testing.T) {
shelf := provided(t)
// A resolver restarting on the names another module put on the machine, and one resource it
// only runs at start — neither of which composition has any business changing.
resolver, err := catalogue.ParseManifest([]byte(`{
"module": "resolver", "version": "1", "requires": ["mesh-addressing"],
"resources": [
{"id": "seed", "type": "file", "path": "/etc/resolver/seed", "mode": "0644",
"content": "seed\n", "at": "start"},
{"id": "daemon", "type": "service", "unit": "resolver.service", "state": "running",
"restart-on": ["seed", "mesh-wireguard.fact-node-names"]}
]}`))
if err != nil {
t.Fatal(err)
}
shelf[resolver.Module] = resolver
got, err := catalogue.Resolve(shelf, []string{overlay.Domain, "resolver"},
catalogue.Node{Name: "homer", At: "homer.internal"}, catalogue.World{})
if err != nil {
t.Fatal(err)
}
names := map[string]string{"homer.internal": "10.42.0.1", "marge.internal": "10.42.0.2"}
out, err := got.Declaration(catalogue.Rendering{
Names: names, Machines: names, Suffix: "internal",
Generators: map[string]catalogue.Generator{overlay.Name: onTheNetwork{}},
})
if err != nil {
t.Fatal(err)
}
ids := map[string]map[string]any{}
for _, r := range out {
ids[r["id"].(string)] = r
}
hosts := ids[overlay.Name+".fact-node-names"]
if hosts == nil {
t.Fatalf("no names reached the machine; the declaration has %v", keys(ids))
}
if hosts["path"] != "/etc/hosts" || hosts["into"] != "block" {
t.Fatalf("the hosts file is not written into as a region: %v", hosts)
}
content := hosts["content"].(string)
if !strings.Contains(content, "10.42.0.1\thomer.internal\thomer\t# this machine\n") {
t.Errorf("the region does not name the machine:\n%s", content)
}
for _, floor := range []string{"Generated by the mesh", "localhost", "127.0.1.1"} {
if strings.Contains(content, floor) {
t.Errorf("the region carries %q, which is the machine's:\n%s", floor, content)
}
}
// The resolver's reference to it still names a resource the host will be sent.
daemon := ids["resolver.daemon"]
if daemon == nil {
t.Fatalf("the resolver's service was not composed: %v", keys(ids))
}
for _, named := range daemon["restart-on"].([]any) {
if ids[named.(string)] == nil {
t.Errorf("the resolver restarts on %v, which is nothing the host is sent", named)
}
}
if !reflect.DeepEqual(daemon["restart-on"], []any{"resolver.seed", overlay.Name + ".fact-node-names"}) {
t.Errorf("restart-on is %v", daemon["restart-on"])
}
// And a resource's own `at` passes through as the manifest wrote it.
if seed := ids["resolver.seed"]; seed == nil || seed["at"] != "start" {
t.Errorf("a resource's at did not survive composition: %v", seed)
}
}
func keys(m map[string]map[string]any) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
return out
}
+57 -5
View File
@@ -147,8 +147,17 @@ type Offer struct {
// shared by every consumer (novox/hq ADR 0158): software that holds one password or one key
// cannot give each consumer a login of its own. The named secret must say how it is taken.
Credential *OfferCredential `json:"credential,omitempty"`
// Reach is ReachMachine for a provision usable only on the provider's own machine — a display
// (novox/hq ADR 0208 §3). Node scope already keeps a provision off other machines; what this adds
// is that a requirement for it is never answered by installing a provider: the display server is
// a seat's holder gated by the machine's graphical session, and pulling one in for whatever asked
// is the misassignment research 026 found. Unmet, the requirement is refused naming who could.
Reach string `json:"reach,omitempty"`
}
// MachineReach is whether a provision is usable only on its provider's own machine.
func (o Offer) MachineReach() bool { return o.Reach == ReachMachine }
// OfferCredential names which of the provider's own secrets a provision's consumers receive.
type OfferCredential struct {
Own string `json:"own"`
@@ -196,27 +205,29 @@ func (o *Offer) UnmarshalJSON(raw []byte) error {
Name string `json:"name"`
Scope string `json:"scope,omitempty"`
Credential *OfferCredential `json:"credential,omitempty"`
Reach string `json:"reach,omitempty"`
}
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&full); err != nil {
return fmt.Errorf("a provided name is either a string or {name, scope, credential}: %w", err)
return fmt.Errorf("a provided name is either a string or {name, scope, credential, reach}: %w", err)
}
o.Name, o.Scope, o.Credential = full.Name, full.Scope, full.Credential
o.Name, o.Scope, o.Credential, o.Reach = full.Name, full.Scope, full.Credential, full.Reach
return nil
}
// MarshalJSON writes back the short form when there is nothing else to say, so a manifest that
// went through the mesh comes out looking like the one that went in.
func (o Offer) MarshalJSON() ([]byte, error) {
if o.Scope == "" && o.Credential == nil {
if o.Scope == "" && o.Credential == nil && o.Reach == "" {
return json.Marshal(o.Name)
}
return json.Marshal(struct {
Name string `json:"name"`
Scope string `json:"scope,omitempty"`
Credential *OfferCredential `json:"credential,omitempty"`
}{o.Name, o.Scope, o.Credential})
Reach string `json:"reach,omitempty"`
}{o.Name, o.Scope, o.Credential, o.Reach})
}
// Manifest is everything a module says about itself.
@@ -517,6 +528,22 @@ type Manifest struct {
// holder. Like Filtering: one module per node gathers what every module declared and writes it.
Jailing *Jailing `json:"jailing,omitempty"`
// Environment is what this module adds to the operator account's environment: variables, and
// entries on PATH (novox/hq ADR 0203). Facts, not lines of one shell's syntax — the holder of
// node-environment places them, and the controller writes them in each reader's format. Like
// Jails: any module contributes, gathered from every module on the node, written by the holder.
Environment *Environment `json:"environment,omitempty"`
// Shell is code this module adds to the login shell's startup, for a named shell in a named
// slot (novox/hq ADR 0204). The controller never reads it: it is placed, in module order, where
// the holder of node-login-shell put the slot's placeholder.
Shell []ShellCode `json:"shell,omitempty"`
// Contributions are configuration this module gives the holder of a seat it does not hold, in
// that tool's own grammar (novox/hq ADR 0212): a seat, a kind the seat receives, and the text. The
// holder places them; the module depends on the seat (ADR 0210 §3).
Contributions []SeatContribution `json:"contributions,omitempty"`
// Guards are ports of this module's the mesh refuses on an adopted node except from the
// private network and from the machine itself (novox/hq ADR 0100) — the store's port and the
// broker's management port. The ports the software uses; the mesh guards where the machine
@@ -549,6 +576,10 @@ type Manifest struct {
// `restart-on` names to restart when the roster changes.
Facts map[string]RosterFile `json:"facts,omitempty"`
// Zone is the zone of names this module answers itself, and the listen that answers it (novox/hq
// ADR 0199). The mesh's resolver forwards the zone to it; nothing here names an address.
Zone *Zone `json:"zone,omitempty"`
// Certificate is where this module wants a certificate for its machine's name inside the
// mesh, and where the key that goes with it can be found.
//
@@ -1306,6 +1337,19 @@ func ParseManifest(raw []byte) (Manifest, error) {
"%s provides %q at scope %q; a provision is %q or %q",
m.Module, p, s, ScopeNode, ScopeMesh))
}
switch {
case offer.Reach == "":
case offer.Reach != ReachMachine:
// The one reach a provision has (novox/hq ADR 0208): a provision reached over the private
// network is mesh scope, and the world reaches nothing but a name.
problems = append(problems, fmt.Sprintf(
"%s provides %q with reach %q; a provision's reach is %q or nothing",
m.Module, p, offer.Reach, ReachMachine))
case offer.At() != ScopeNode:
problems = append(problems, fmt.Sprintf(
"%s provides %q at scope %q with the machine's reach; a provision usable only on its own "+
"machine is node-scoped (novox/hq ADR 0208)", m.Module, p, offer.At()))
}
if p == m.Module {
// Harmless and worth saying: a module always provides its own name, so writing it
// suggests the author expected it not to.
@@ -1439,7 +1483,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
"%s serves %q to whoever requires it, and does not provide it", m.Module, to))
}
}
// A served value may be derived for the consumer it is served to (novox/hq ADR 0202). Read
// A served value may be derived for the consumer it is served to (novox/hq ADR 0201). Read
// here, where the definition is, rather than when somebody first requires it: a rule that
// would be refused at the first consumer is wrong from the moment it is written.
problems = append(problems, CheckServes(m)...)
@@ -1805,6 +1849,13 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, m.unknownDirRefs()...)
problems = append(problems, m.unknownAccessRefs()...)
problems = append(problems, m.jailProblems()...)
// What a module adds to the account's environment and to the login shell, and the holder's
// placeholders for them (novox/hq ADR 0203, ADR 0204) — here, so the catalogue check refuses
// them in the words registration does.
problems = append(problems, m.environmentProblems()...)
problems = append(problems, m.shellProblems()...)
problems = append(problems, m.contributionPlaceholderProblems()...)
problems = append(problems, m.seatContributionProblems()...)
for i, r := range m.Resources {
id, _ := r["id"].(string)
@@ -1816,6 +1867,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
problems = append(problems, zoneProblems(m)...)
if len(problems) > 0 {
sort.Strings(problems)
return Manifest{}, fmt.Errorf("this manifest cannot be used:\n - %s",
+54 -27
View File
@@ -19,7 +19,7 @@ func provided(t *testing.T) map[string]catalogue.Manifest {
t.Helper()
out := map[string]catalogue.Manifest{}
for _, raw := range []map[string]any{
overlay.Manifest(), overlay.DomainManifest(),
overlay.Manifest(),
} {
b, err := json.Marshal(raw)
if err != nil {
@@ -34,42 +34,69 @@ func provided(t *testing.T) map[string]catalogue.Manifest {
return out
}
func TestTheShippedNetworkingModulesResolveOnTheirOwn(t *testing.T) {
got, err := catalogue.Resolve(provided(t), []string{overlay.Domain}, catalogue.Node{Name: "workstation", Site: "house"}, catalogue.World{})
func TestTheShippedPrivateNetworkResolvesOnItsOwn(t *testing.T) {
// **Assigned directly** (novox/hq ADR 0226): the `networking` bundle that required it is
// retired, so the private network's own module is what a machine is given, and it must need
// nothing the control plane does not ship beside it.
got, err := catalogue.Resolve(provided(t), []string{overlay.Name}, catalogue.Node{Name: "workstation", Site: "house"}, catalogue.World{})
if err != nil {
t.Fatalf("assigning %s does not work out of the box: %v", overlay.Domain, err)
t.Fatalf("assigning %s does not work out of the box: %v", overlay.Name, err)
}
var have []string
for _, m := range got.Modules {
have = append(have, m.Module)
}
for _, want := range []string{overlay.Domain, overlay.Name} {
if !strings.Contains(strings.Join(have, " "), want) {
t.Fatalf("%s did not bring in %s: %v", overlay.Domain, want, have)
}
if len(got.Modules) != 1 || got.Modules[0].Module != overlay.Name {
t.Fatalf("assigning %s brought %v", overlay.Name, got.Modules)
}
// **The names come WITH the network now, not from a third module.** Being on the private
// network is what gives a machine a name, so the provider asks for the node-names fact and
// there is nothing else to bring in. A module that ran nothing used to be here.
// **The network writes no names** (novox/hq ADR 0199): /etc/hosts is the hosts seat holder's
// file, and the mesh's resolver answers the machines' names. No fact of the network's module
// may name that file.
for _, m := range got.Modules {
if m.Module == overlay.Name && m.Facts["node-names"].Path == "" {
t.Fatalf("the network's provider does not ask for the names: %+v", m.Facts)
for name, f := range m.Facts {
if f.Path == "/etc/hosts" {
t.Fatalf("the network's provider still writes /etc/hosts, as its %q fact", name)
}
}
}
}
func TestAnotherVPNSatisfiesNetworkingWithoutDraggingWireGuardIn(t *testing.T) {
// **This inverted, and the inversion is the improvement.** The names used to be a module that
// required the mesh's own addressing, which only WireGuard provided — so choosing another VPN
// dragged WireGuard in anyway, and the node-scoped claim existed to at least make that
// collision loud. With the names a fact rather than a provision, a person who chose tailscale
// gets tailscale, and there is nothing left to collide.
func TestTheControlPlaneShipsNoNetworkingBundle(t *testing.T) {
// ADR 0226: one module for the one private network. A bundle shipped beside it again would be
// a second name every machine carries for the same thing.
shipped := provided(t)
got, err := catalogue.Resolve(
withTailscale(shipped),
[]string{overlay.Domain, "tailscale"},
if len(shipped) != 1 {
t.Fatalf("the control plane ships %d modules, want only %s", len(shipped), overlay.Name)
}
if _, ok := shipped["networking"]; ok {
t.Fatal("the retired networking bundle is shipped again")
}
}
func TestARefusalForWantOfThePrivateNetworkNamesItsModule(t *testing.T) {
// The hint in a refusal is a string in the resolver rather than an import of this package. It
// named `networking` until ADR 0226; a hint naming a module nobody ships sends a person to
// assign something that does not exist.
shelf := map[string]catalogue.Manifest{
"app": {Module: "app", Version: "1", Requires: []string{"postgres-database"}},
"postgres": {Module: "postgres", Version: "1", Provides: catalogue.FromAnywhere("postgres-database")},
}
_, err := catalogue.Resolve(shelf, []string{"app"}, catalogue.Node{Name: "workstation"},
catalogue.World{Offered: map[string][]catalogue.Provider{
"postgres-database": {{Node: "anchor", At: "anchor.internal"}}}})
if err == nil {
t.Fatal("an app off the private network was pointed at a database on it")
}
if !strings.Contains(err.Error(), "assign "+overlay.Name) {
t.Fatalf("the refusal does not name the private network's module %s: %v", overlay.Name, err)
}
}
func TestAnotherVPNIsChosenByAssigningItInstead(t *testing.T) {
// What the bundle was for, kept without it: a machine given another VPN answers
// private-network from that VPN, and WireGuard is not dragged in by anything.
shipped := provided(t)
shelf := withTailscale(shipped)
shelf["needs-network"] = catalogue.Manifest{Module: "needs-network", Version: "1",
Requires: []string{overlay.Requirement}}
got, err := catalogue.Resolve(shelf, []string{"needs-network", "tailscale"},
catalogue.Node{Name: "workstation", Site: "house"}, catalogue.World{})
if err != nil {
t.Fatalf("choosing another VPN was refused: %v", err)
+119
View File
@@ -0,0 +1,119 @@
package catalogue
import (
"os"
"testing"
)
// The public issuer is the proxy's own fact (novox/hq ADR 0226), and the folding of it must not
// move the proxy's account.
//
// route-proxy keeps each ACME authority's account and certificates in a directory named after a
// digest of the directory URL and of the root bundle its `trust` container copies in
// (examples/route-proxy, forThisAuthority). Until ADR 0226 the URL was rendered from a binding to
// `public-acme`'s `acme-ca`, spelled with the port. A URL spelled any other way — even the same
// authority without `:443` — or a root bundle from another image is a new authority to the proxy:
// a new account, and every routed name ordered again, on two machines at once, against Let's
// Encrypt's rate limits. So what the module now states is held to exactly what the binding rendered.
// renderedBeforeTheFold is route-proxy's acme.env as the binding to public-acme rendered it on every
// machine running the proxy, read from the controller's plan on 2026-10-06.
const renderedBeforeTheFold = "ACME_DIRECTORY=https://acme-v02.api.letsencrypt.org:443/directory\n" +
"ACME_ROOTS=https://acme-v02.api.letsencrypt.org:443\n" +
"ACME_ROOTS_PATH=\n"
// trustImage is the image whose system bundle becomes the public root the account directory is
// named after. Moving it renames that directory.
const trustImage = "alpine@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b"
func TestRouteProxyKeepsItsPublicAccountDirectory(t *testing.T) {
m := catalogueManifest(t, "route-proxy")
for _, r := range m.Requires {
if r == "acme-ca" {
t.Fatal("route-proxy still asks for acme-ca; the public issuer is its own fact (ADR 0226)")
}
}
// As a build would leave it: each artifact a container names resolved to an image. Which image
// does not matter to the file checked here.
for _, res := range m.Resources {
if artifact, ok := res["artifact"].(string); ok {
delete(res, "artifact")
res["image"] = "registry.invalid/route-proxy/" + artifact +
"@sha256:1111111111111111111111111111111111111111111111111111111111111111"
}
}
r := Resolution{
Node: "anchor",
Modules: []Manifest{m},
Needs: []Needed{{
Name: "internal-acme-ca", From: "home", At: "home.internal", For: "route-proxy",
Serves: map[string]any{
"port": float64(9000), "path": "/acme/acme/directory", "roots": "/roots.pem",
},
}},
}
// Its own broker credential, sealed as the mesh would; what it is does not matter here.
out, err := r.Declaration(Rendering{Needed: map[string]map[string]string{
"route-proxy": {"broker": "sealed"}}})
if err != nil {
t.Fatalf("route-proxy could not be composed for a machine: %v", err)
}
env := fileNamed(out, "route-proxy.acme-env")
if env == nil {
t.Fatalf("nothing writes the public issuer's environment: %v", out)
}
if got, _ := env["content"].(string); got != renderedBeforeTheFold {
t.Fatalf("acme.env changed, which moves the proxy's account and reorders every certificate:\n"+
"got %q\nwant %q", got, renderedBeforeTheFold)
}
if fileNamed(out, "route-proxy.bound-acme-ca") != nil {
t.Error("a binding to acme-ca is still written")
}
var trust string
if m.Build != nil {
for _, a := range m.Build.Artifacts {
if a.Name == "trust" {
trust = a.From
}
}
}
if trust != trustImage {
t.Errorf("the trust image is %q, not %q: its system bundle is the public root the account "+
"directory is named after, so moving it reorders every certificate. Move it only with a "+
"plan for the account (ADR 0226)", trust, trustImage)
}
}
// The modules ADR 0226 retired stay retired, and nothing asks for what they provided.
func TestTheRetiredNetworkingModulesAreNotInTheCatalogue(t *testing.T) {
if _, err := os.Stat("../../../mesh-catalog/modules"); err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err)
}
for _, gone := range []string{"public-acme", "dhcpcd", "cloudflare-dns"} {
if _, err := os.Stat("../../../mesh-catalog/modules/" + gone); err == nil {
t.Errorf("%s is in the catalogue again; ADR 0226 retired it", gone)
}
}
entries, err := os.ReadDir("../../../mesh-catalog/modules")
if err != nil {
t.Fatal(err)
}
for _, e := range entries {
raw, err := os.ReadFile("../../../mesh-catalog/modules/" + e.Name() + "/module.json")
if err != nil {
continue
}
m, err := ParseManifest(raw)
if err != nil {
continue // judged by the catalogue's own tests
}
for _, r := range m.Requires {
if r == "acme-ca" || r == "public-dns" {
t.Errorf("%s requires %q, which nothing in the catalogue provides since ADR 0226",
m.Module, r)
}
}
}
}
@@ -0,0 +1,75 @@
package catalogue
import (
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
)
// A container publishes only a port its module declares (novox/hq issue 227).
//
// **The short form is a question the mesh answers.** `"80"` means *publish what the software
// calls 80*, and the mesh fills in the machine's half from the port it assigned
// ([ADR 0038](0038)). It can only assign one for a port the module declared in `listens` — so a
// container publishing a number that appears nowhere in `listens` gets no assignment, and
// `publishedOn` falls back to the number as written. It escapes to the machine.
//
// That is how the photo module asked for port 80 on the control node, where the reverse proxy
// holds it: it declared its web endpoint at 4001, published a bare 80, and the container never
// started. Four other modules publish 80 quite safely — because they declare 80, so the mesh
// gives them a machine port for it. The difference is the declaration, not the number.
//
// A mapping written the long way is a module pinning both halves on purpose and is left alone.
func TestEveryPublishedPortIsOneItsModuleDeclares(t *testing.T) {
root := catalogueRoot(t)
entries, err := os.ReadDir(filepath.Join(root, "modules"))
if err != nil {
t.Fatal(err)
}
var escaped []string
for _, entry := range entries {
if !entry.IsDir() {
continue
}
raw, err := os.ReadFile(filepath.Join(root, "modules", entry.Name(), "module.json"))
if err != nil {
continue
}
m, err := ParseManifest(raw)
if err != nil {
// Whether every manifest parses is TestEveryCatalogueManifestParses's question.
continue
}
declared := map[int]bool{}
for _, l := range m.Listens {
declared[l.Port] = true
}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
listed, _ := r["ports"].([]any)
for _, p := range listed {
written := strings.Split(fmt.Sprint(p), "/")[0]
if strings.Contains(written, ":") {
continue // pinned by hand, both halves, on purpose
}
port, err := strconv.Atoi(strings.TrimSpace(written))
if err != nil || declared[port] {
continue
}
escaped = append(escaped, fmt.Sprintf(
"%s's %v publishes %d, and %s declares no such port — the mesh has nothing "+
"to assign, so %d reaches the machine as written",
m.Module, r["id"], port, m.Module, port))
}
}
}
if len(escaped) > 0 {
t.Fatalf("a container may publish only a port its module declares:\n - %s",
strings.Join(escaped, "\n - "))
}
}
+224 -13
View File
@@ -147,6 +147,10 @@ type Resolution struct {
// dropped nor fatal to the rest. A module that is *required* by something running here is a
// different case — that set is incoherent and is refused (see checkCapabilities).
Unhostable []Unhostable
// Unheld is every dependency of this node's modules on a seat nothing here holds (novox/hq ADR
// 0207) — reported rather than refused while enforceSeatDependencies is off, so a node short of a
// holder still converges and `status` says what it is short of.
Unheld []Unheld
}
// Unhostable is one directly-assigned module the machine cannot run.
@@ -229,6 +233,27 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
local[o.Name] = true
}
}
// Which names are usable only on their provider's own machine (novox/hq ADR 0208 §3). Also a
// property of the name: a display one provider says is the machine's and another says is not
// would be pulled in for one consumer and refused for the next.
machineReach := map[string]bool{}
plainLocal := map[string]bool{}
for _, m := range catalogue {
for _, o := range m.Provides {
if o.MachineReach() {
machineReach[o.Name] = true
} else if o.At() == ScopeNode {
plainLocal[o.Name] = true
}
}
}
for want := range machineReach {
if plainLocal[want] {
problems = append(problems, fmt.Sprintf(
"the catalogue disagrees about %q: some modules provide it with the machine's reach and "+
"others without, so a requirement for it would be met differently by each", want))
}
}
for want := range brokered {
if local[want] {
problems = append(problems, fmt.Sprintf(
@@ -316,7 +341,7 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
// trust boundary the moment both ends are containers**, and treating it as one gave the
// commonest arrangement of all — a service and its database on one node — the weakest
// handling, silently.
if satisfied[want] && !isModule(catalogue, want) {
if satisfied[want] && !isModule(catalogue, want) && !answeredElsewhere(want, node, world, brokered) {
here := func(name string) bool { return chosen[name] || assignedHere[name] }
local := providersHere(catalogue, here, want)
// Which of them it matters to choose between. A plain capability — a shell, a display
@@ -495,6 +520,23 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
continue
}
// Usable only on its provider's own machine, and not here: refused, never answered by
// installing a provider (novox/hq ADR 0208 §3). The display server is the machine's own role,
// gated by its graphical session; one pulled in for a window manager is the misassignment
// research 026 found. Another node's provider never counts — node scope is never brokered.
if machineReach[want] && !isModule(catalogue, want) {
reported[want] = true
if world.Unchecked {
// The first pass's refusals take a machine off the network; the second says it.
continue
}
problems = append(problems, fmt.Sprintf(
"%q is wanted by %s and is usable only on the machine that provides it, and nothing "+
"assigned to %s does — %s", want, because[want], node.Name,
machineReachRemedy(catalogue, want, node.Name)))
continue
}
candidates := offers[want]
switch len(candidates) {
case 0:
@@ -628,6 +670,22 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
problems = append(problems, checkResources(resolution.Modules)...)
resolution.Claims = claims
// Judged over the closure — what this node will actually run — so a holder pulled in by a
// requirement counts, and the holders' mutual dependence resolves (novox/hq ADR 0207 §3).
// Reported until the switch; refused after it, though never in the first pass, whose refusals
// make a machine vanish from the network rather than report anything.
resolution.Unheld = UnheldDependencies(catalogue, node.Name, resolution.Modules, nil)
// Only a dependency some module in the catalogue could meet is refused: one with no possible
// holder has no remedy to name, and stays a report in `status` (novox/hq ADR 0207 §4, read with
// the assign rule above it).
if enforceSeatDependencies && !world.Unchecked {
for _, u := range resolution.Unheld {
if len(u.Holders) > 0 {
problems = append(problems, u.String())
}
}
}
if len(problems) > 0 {
sort.Strings(problems)
return Resolution{}, &Refusal{Problems: problems}
@@ -709,16 +767,27 @@ func checkClaims(modules []Manifest, node Node, elsewhere []Held, holdings []Hel
var problems []string
var held []Held
// onRecord is the recorded holder of a seat, if a handover ever named one.
onRecord := func(claim, scope string) (Held, bool) {
// onRecord is every recorded holder of a seat, if a handover ever named one: one for most seats,
// and as many as were added for a replicated one (novox/hq ADR 0223).
onRecord := func(claim, scope string) []Held {
var out []Held
for _, h := range holdings {
hs, ok := SeatNamed(h.Claim)
cs, cok := SeatNamed(claim)
if ok && cok && hs.Name == cs.Name && h.Scope == scope {
return h, true
out = append(out, h)
}
}
return Held{}, false
return out
}
// recordedHere says this node's module is one of a seat's holders on record.
recordedHere := func(claim, scope, module string) bool {
for _, rec := range onRecord(claim, scope) {
if rec.Node == node.Name && rec.Module == module {
return true
}
}
return false
}
byScope := map[string]map[string]string{} // scope → claim → module
@@ -729,21 +798,35 @@ func checkClaims(modules []Manifest, node Node, elsewhere []Held, holdings []Hel
// the seat but is not the one on record is eligible, and that is all: it is not a second
// holder, so it is not refused, and it does not hold (novox/hq ADR 0131). This is what
// lets the next holder stand beside the current one until the seat is handed over.
if rec, recorded := onRecord(c.Name, scope); recorded {
if rec.Node != node.Name || rec.Module != m.Module {
if recs := onRecord(c.Name, scope); len(recs) > 0 {
// **A seat held once is on record once** (novox/hq ADR 0223). Only a replicated seat
// may have several holders on record; several for any other seat is a store that
// disagrees with the mesh's definition of the role, and it is refused, named, rather
// than letting two machines answer for what the mesh has one of.
if s, known := SeatNamed(c.Name); known && !s.Replicated && len(recs) > 1 {
problems = append(problems, fmt.Sprintf(
"%q is on record as held by %d assignments, and it is held once per %s — "+
"`seat %s --to <node>/<module>` records one", c.Name, len(recs), scope, c.Name))
continue
}
if !recordedHere(c.Name, scope, m.Module) {
continue
}
}
if byScope[scope] == nil {
byScope[scope] = map[string]string{}
}
if other, taken := byScope[scope][c.Name]; taken {
// **One seat under either of its names** (novox/hq ADR 0122): a manifest registered before
// a rename claims the former name, and one written after it the current — two claimants of
// one seat, compared by the seat they resolve to and not by how each spelled it.
seat := canonicalSeat(c.Name)
if other, taken := byScope[scope][seat]; taken {
problems = append(problems, fmt.Sprintf(
"%s and %s both claim %q, and only one thing may hold it per %s",
other, m.Module, c.Name, scope))
other, m.Module, seat, scope))
continue
}
byScope[scope][c.Name] = m.Module
byScope[scope][seat] = m.Module
held = append(held, Held{Claim: c.Name, Scope: scope, Node: node.Name,
Module: m.Module, Site: node.Site})
}
@@ -752,11 +835,17 @@ func checkClaims(modules []Manifest, node Node, elsewhere []Held, holdings []Hel
// And against the rest of the mesh, for the scopes that reach past this machine.
for _, h := range held {
for _, e := range elsewhere {
if e.Node == node.Name || e.Claim != h.Claim || e.Scope != h.Scope {
if e.Node == node.Name || canonicalSeat(e.Claim) != canonicalSeat(h.Claim) || e.Scope != h.Scope {
continue
}
switch h.Scope {
case ScopeMesh:
// **A holder on record is never a second claimant** (novox/hq ADR 0223). Records are
// the mesh's settled answer: one for most seats, several only for a replicated seat,
// each added by an act. Two holders here are two records, and both hold.
if recordedHere(h.Claim, h.Scope, h.Module) {
continue
}
// Both claim and nobody is on record, or this refusal could not have happened.
// The remedy is the handover that records the holder (novox/hq ADR 0131,
// 04-ISSUES/170), so it is named here rather than left to be found.
@@ -777,6 +866,15 @@ func checkClaims(modules []Manifest, node Node, elsewhere []Held, holdings []Hel
return held, problems
}
// canonicalSeat is the seat a claimed name refers to, by its current name: itself for a name the mesh
// does not know (a module's own seat), its seat's name for a former one.
func canonicalSeat(name string) string {
if s, known := SeatNamed(name); known {
return s.Name
}
return name
}
// checkResources refuses two modules writing the same thing.
//
// This costs no manifest field: the mesh already holds every resource of every module, so two
@@ -804,6 +902,28 @@ func checkResources(modules []Manifest) []string {
// does not exist is the manifest's own problem, refused where it was made.
dirs := dirsFor(m, Rendering{})
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "user" {
// **An account is shared; what it is set to is not.** Several modules may need one
// login: the shell's module sets its shell, the container runtime's puts it in the
// `docker` group. The host only ever adds groups — it never takes the account out of
// one, not even when the resource that named it is undeclared — so groups from
// several modules cannot contradict each other and are not owned. A shell or a home
// is one value, and two modules setting it would each be undone by the other's
// apply: each stays one module's per node, and two are refused naming both.
name, _ := r["name"].(string)
for _, field := range []string{"shell", "home"} {
if v, ok := r[field].(string); !ok || v == "" || name == "" {
continue
}
key := "user " + field + " " + name
if other, taken := owner[key]; taken && other != m.Module {
problems = append(problems, fmt.Sprintf(
"%s and %s both set the %s of the user %q", other, m.Module, field, name))
}
owner[key] = m.Module
}
continue
}
for _, field := range []string{"path", "unit", "name", "package"} {
value, ok := r[field].(string)
if !ok || value == "" {
@@ -832,6 +952,23 @@ func checkResources(modules []Manifest) []string {
}
}
}
// **A file the mesh renders for a module is that module's path too** (novox/hq ADR 0223). The
// machine's resolver file is a fact the uplink's holder asks for, and a second module asking
// for it, or declaring it, would have the host write one path twice in every apply, the last
// one winning. A fact under the operator's home is placed per account and compared nowhere.
for _, name := range sortedFacts(m.Facts) {
fact := m.Facts[name]
if fact.Home || !strings.HasPrefix(fact.Path, "/") {
continue
}
key := "path " + fact.Path
if other, taken := owner[key]; taken && other != m.Module {
problems = append(problems, fmt.Sprintf(
"%s and %s both declare the path %q", other, m.Module, fact.Path))
}
owner[key] = m.Module
ownedPath[fact.Path] = m.Module
}
}
// An accessed path is the operator's, so no module may declare it as one of its own
@@ -882,8 +1019,10 @@ func FromAnywhere(names ...string) []Offer {
//
// A string here rather than an import, because the private network is a module the control plane
// ships and this package must not depend on the thing it resolves. The name being wrong would
// show up as a refusal naming a module nobody can assign, which a test checks.
const meshNetwork = "networking"
// show up as a refusal naming a module nobody can assign, which a test checks
// (provided_test.go). The private network's own module since novox/hq ADR 0226 retired the
// `networking` bundle that required it.
const meshNetwork = "mesh-wireguard"
// providersFirst orders a node's modules so that what answers a requirement comes before what
// asked for it.
@@ -1019,3 +1158,75 @@ func eachLocal(needs []Needed, catalogue map[string]Manifest, n Needed) []Needed
}
return needs
}
// machineReachRemedy names what would meet a requirement with the machine's reach: every module in
// the catalogue that provides it, each with the node seats it holds — for a display, the holders of
// node-display-server (novox/hq ADR 0208 §3), named by the seat because that is the role being
// asked for, without this code knowing which seat any provision belongs to.
func machineReachRemedy(catalogue map[string]Manifest, want, node string) string {
var named []string
for _, name := range sortedKeys(catalogue) {
m := catalogue[name]
provides := false
for _, o := range m.Provides {
if o.Name == want {
provides = true
}
}
if !provides {
continue
}
var held []string
for _, c := range m.Claims {
if c.At() == ScopeNode {
held = append(held, c.Name)
}
}
if len(held) > 0 {
named = append(named, fmt.Sprintf("%s (holds %s)", name, strings.Join(held, ", ")))
} else {
named = append(named, name)
}
}
if len(named) == 0 {
return "and nothing in the catalogue provides it"
}
return fmt.Sprintf("assign one to %s: %s", node, strings.Join(named, "; "))
}
// answeredElsewhere says that a mesh-wide provision this machine could answer itself is answered by
// another machine all the same: one this machine was pinned to, or the holder of the mesh seat that
// delivers it (novox/hq ADR 0110, ADR 0194).
//
// **A provider on the machine was taken before either was asked.** The branch for a provision
// answered here bound the consumer to the local provider and consulted a pin only between two local
// ones, and the seat's holder never; both were read only for a provider on another machine. So when
// every machine ran a provider of `wildcard-resolution` — each machine's own resolver, still assigned
// while the mesh moved to one — every machine bound its resolver configuration to itself, with the
// mesh's one resolver recorded, held and pinned (novox/hq issue 258). The seat is the mesh's choice of
// who answers, made once; a provider that merely runs here does not overrule it, and neither does it
// overrule a pin somebody set on this machine.
//
// Not in the first pass, which asks only what this machine offers and has no providers to read.
func answeredElsewhere(want string, node Node, world World, brokered map[string]bool) bool {
if world.Unchecked || !brokered[want] {
return false
}
if c, pinned := world.Pinned[want]; pinned {
return c.Node != node.Name
}
// **A machine holding the seat answers itself** (novox/hq ADR 0223). With a replicated seat
// another machine holds it too, and the first holder in the providers' order may be that one; a
// holder is still where this machine's own requirement is answered, so its resolver file lists
// itself first.
if seat, delivered := SeatDelivering(want); delivered {
for _, h := range append(append([]Held(nil), world.Holdings...), world.Held...) {
if hs, ok := SeatNamed(h.Claim); ok && hs.Name == seat.Name && h.Scope == seat.Scope &&
h.Node == node.Name {
return false
}
}
}
holder, held := HolderAmong(want, world.Offered[want], world.Held)
return held && holder.Node != node.Name
}
@@ -0,0 +1,57 @@
package catalogue
import (
"reflect"
"testing"
)
// novox/hq ADR 0223 part 2: /etc/resolv.conf belongs to the module holding node-uplink. The seat that
// wrote it, node-resolver-config, and ADR 0220's dependency of it on the uplink retire: one owner for
// the file, and it is the program that would otherwise rewrite it.
func TestTheResolverConfigSeatIsGone(t *testing.T) {
if _, known := SeatNamed("node-resolver-config"); known {
t.Error("node-resolver-config is still in the mesh's set; the uplink's holder writes the resolver file")
}
// An uplink's holder needs no seat beside it for the file: it is its own.
if got := DependsOn(mod("networkmanager", nil, nil, nil, Claim{Name: "node-uplink"})); len(got) != 0 {
t.Errorf("an uplink holder with nothing declared depends on %v", got)
}
}
// The catalogue as it is: nothing claims the retired seat, and every manager the mesh knows holds the
// uplink and writes the resolver file.
func TestTheCataloguesUplinksWriteTheResolverFileAndNothingElseDoes(t *testing.T) {
cat := map[string]Manifest{}
for _, m := range theCatalogue(t) {
cat[m.Module] = m
}
if got := PossibleHolders(cat, "node-uplink"); !reflect.DeepEqual(got, uplinks) {
t.Errorf("node-uplink can be held by %v, not %v", got, uplinks)
}
for name, m := range cat {
for _, c := range m.Claims {
if c.Name == "node-resolver-config" {
t.Errorf("%s still claims node-resolver-config", name)
}
}
_, writes := m.Facts["resolvers"]
isUplink := false
for _, u := range uplinks {
isUplink = isUplink || u == name
}
for _, f := range m.Facts {
if f.Path == "/etc/resolv.conf" && !isUplink {
t.Errorf("%s writes /etc/resolv.conf and does not hold the uplink", name)
}
}
for _, r := range m.Resources {
if r["path"] == "/etc/resolv.conf" {
t.Errorf("%s declares /etc/resolv.conf as a file of its own", name)
}
}
if isUplink && !writes {
t.Errorf("%s holds the uplink and does not write the resolver file", name)
}
}
}
+92 -49
View File
@@ -15,7 +15,8 @@ import (
// same arrangement, and to the two things a resolver here must never do — read resolv.conf for
// its upstreams, or take an address systemd-resolved holds.
// resolverShelf is the three resolver modules beside something that answers `mesh-addressing`.
// resolverShelf is the resolver, an uplink module that writes what the machine asks (novox/hq ADR
// 0223), and the container runtime beside something that answers `mesh-addressing`.
// The networking module that really does is composed in the controller and cannot be imported
// here, so a stand-in offers the same word; what is under test is the manifests, not the network.
func resolverShelf(t *testing.T) map[string]Manifest {
@@ -23,7 +24,7 @@ func resolverShelf(t *testing.T) map[string]Manifest {
shelf := map[string]Manifest{
"net": {Module: "net", Version: "1", Provides: []Offer{{Name: "mesh-addressing"}}},
}
for _, name := range []string{"dnsmasq", "resolv-conf", "resolved-split-dns"} {
for _, name := range []string{"dnsmasq", "systemd-networkd", "docker"} {
shelf[name] = catalogueManifest(t, name)
}
return shelf
@@ -48,9 +49,12 @@ func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T
}
for _, want := range []string{
"\nno-resolv\n", "\nserver=1.1.1.1\n", "\nserver=8.8.8.8\n",
// Loopback is the mesh-wide setting's default; a machine answering its own LAN adds its
// address there (novox/hq issue 198).
"\nlisten-address=${setting:listen-addresses}\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
// The private address and loopback, never a LAN's (novox/hq ADR 0194): a device that is not a
// member cannot reach what the mesh's names point at.
"\nlisten-address=127.0.0.1\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
// No hosts file and no operator's files: the mesh's resolver answers every node (ADR 0199).
"\nno-hosts\n",
"\nconf-file=" + m.Facts["zones"].Path + "\n",
"\ndomain-needed\n", "\nbogus-priv\n",
"\nconf-file=" + m.Facts["node-zones"].Path + "\n",
} {
@@ -73,37 +77,45 @@ func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T
t.Errorf("the resolver listens on %s", taken)
}
}
// And the file that decides what the machine asks names it there, alone.
var resolv string
for _, r := range catalogueManifest(t, "resolv-conf").Resources {
if r["path"] == "/etc/resolv.conf" {
resolv, _ = r["content"].(string)
// Never a directory or a file the operator keeps: a line written for one machine's programs would
// become an answer for every node (ADR 0199).
for _, never := range []string{"conf-dir=", "addn-hosts=", "listen-address=${setting:"} {
if strings.Contains(config, never) {
t.Errorf("the mesh's resolver still reads or listens on %q", never)
}
}
var nameservers []string
for _, line := range strings.Split(resolv, "\n") {
if strings.HasPrefix(line, "nameserver ") {
nameservers = append(nameservers, strings.TrimPrefix(line, "nameserver "))
// And the file that decides what the machine asks names every one of the mesh's resolvers, by
// address, from the seat's holders, and no public one (ADR 0223): a resolver library that asks
// every listed server at once takes the first reply, and a public "no such name" for a mesh name
// won it. Written by every uplink module, since the uplink's holder owns the file.
for _, uplink := range uplinks {
fact, ok := catalogueManifest(t, uplink).Facts["resolvers"]
if !ok || fact.Path != "/etc/resolv.conf" {
t.Fatalf("%s's resolver file is not rendered from the roster: %+v", uplink, fact)
}
}
if len(nameservers) != 1 || nameservers[0] != "127.0.0.1" {
t.Errorf("resolv.conf names %v; the predecessor's names the mesh's resolver alone at 127.0.0.1", nameservers)
}
// The split-DNS alternative points at the same address, or a machine that keeps
// systemd-resolved in charge would route the mesh's suffix to nothing.
for _, r := range catalogueManifest(t, "resolved-split-dns").Resources {
if content, _ := r["content"].(string); content != "" && !strings.Contains(content, "DNS=127.0.0.1\n") {
t.Errorf("resolved-split-dns does not point at the resolver's address:\n%s", content)
if !strings.Contains(fact.Template, `{{range index .Holders "mesh-dns-resolver"}}nameserver {{.Address}}`) {
t.Errorf("%s's resolv.conf does not list every holder of the mesh's resolver:\n%s", uplink, fact.Template)
}
for _, line := range strings.Split(fact.Template, "\n") {
if strings.HasPrefix(line, "nameserver ") && !strings.Contains(line, "{{") {
t.Errorf("%s's resolv.conf names a resolver of its own beside the mesh's: %s", uplink, line)
}
}
if !strings.Contains(fact.Template, "options timeout:1 attempts:2 edns0\n") {
t.Errorf("%s: a silent resolver is not passed over after one short wait:\n%s", uplink, fact.Template)
}
}
}
// The resolver and what points the machine at it compose on one machine, and what arrives is the
// mesh's account of every machine as a wildcard, the suffix kept local, the daemon restarting on
// that file, and the runtime pointed at this machine's own address.
// that file, the machine pointed at the resolver by address, and the runtime given no resolver of
// its own but kept running across a restart (ADR 0196).
func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
got, err := Resolve(resolverShelf(t), []string{"dnsmasq", "resolv-conf"},
Node{Name: "anchor", At: "anchor.internal"}, World{})
got, err := Resolve(resolverShelf(t), []string{"dnsmasq", "systemd-networkd", "docker"},
Node{Name: "anchor", At: "anchor.internal", Capabilities: map[string]bool{
"package-manager": true, "service-manager": true, "privileged": true,
"uplink-systemd-networkd": true}}, World{})
if err != nil {
t.Fatal(err)
}
@@ -115,6 +127,8 @@ func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
// issue 111) — the resolver's zones read only the second, and in this scenario the two
// happen to be the same map, since nothing routed is part of it.
Names: twoMachines, Machines: twoMachines, Suffix: "internal",
Zones: []ZoneAt{{Zone: "incus", Address: "10.42.0.2", Port: 5353}},
Holders: map[string]map[string]string{"mesh-dns-resolver": {"anchor.internal": "10.42.0.1"}},
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Settings: SettingsBy{"dnsmasq": {{From: "the mesh", Values: map[string]any{"listen-addresses": "127.0.0.1"}}}},
})
@@ -144,24 +158,35 @@ func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
for _, id := range service["restart-on"].([]any) {
reflects[id.(string)] = true
}
if !reflects["dnsmasq.config"] || !reflects["dnsmasq.fact-node-zones"] {
t.Errorf("the daemon does not restart on its configuration and the machines file both: %v", service["restart-on"])
if !reflects["dnsmasq.config"] || !reflects["dnsmasq.fact-node-zones"] || !reflects["dnsmasq.fact-zones"] {
t.Errorf("the daemon does not restart on its configuration, the machines and the zones: %v", service["restart-on"])
}
if z, _ := ids["dnsmasq.fact-zones"]["content"].(string); !strings.Contains(z, "server=/incus/10.42.0.2#5353\n") {
t.Errorf("the resolver was not told to forward the zone to its answerer:\n%s", z)
}
// The runtime's own file, written into (novox/hq ADR 0102) with the keys this module states:
// where containers resolve, and that a restart keeps them running — because the runtime reads
// `dns` only when it starts, and the one restart that needs is the operator's (issue 110).
runtime := ids["dnsmasq.runtime-dns"]
// The runtime's own file, written into (novox/hq ADR 0102) with one key, by the runtime's own
// module — a module does not write another software's configuration (issue 190): a restart keeps
// every container running. No `dns` — a container copies its machine's resolvers (ADR 0196), and
// neither the resolver nor what decides how the machine resolves writes the runtime's file.
if ids["dnsmasq.runtime-dns"] != nil {
t.Errorf("the resolver still writes the runtime's dns: %v", ids["dnsmasq.runtime-dns"])
}
for id := range ids {
if strings.HasPrefix(id, "systemd-networkd.runtime") {
t.Errorf("what the machine asks still writes the runtime's file: %s", id)
}
}
runtime := ids["docker.daemon"]
if runtime == nil || runtime["path"] != "/etc/docker/daemon.json" || runtime["into"] != "json" {
t.Fatalf("the runtime's dns is not written into its file: %v", runtime)
t.Fatalf("live-restore is not written into the runtime's file: %v", runtime)
}
var keys map[string]any
if err := json.Unmarshal([]byte(runtime["content"].(string)), &keys); err != nil {
t.Fatalf("the runtime's keys are not JSON: %v", err)
}
dns, _ := keys["dns"].([]any)
if len(keys) != 2 || len(dns) != 1 || dns[0] != "10.42.0.1" || keys["live-restore"] != true {
t.Errorf("the runtime is given %v; containers resolve at this machine's own private-network address, a restart keeps them, and nothing else is written", keys)
if len(keys) != 1 || keys["live-restore"] != true {
t.Errorf("the runtime is given %v; live-restore and nothing else", keys)
}
// The runtime is reloaded when that file changes, and never restarted: a restart stops every
// container on the machine (ADR 0102), and a reload is what turns live-restore on.
@@ -171,10 +196,10 @@ func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
continue
}
if _, restarts := r["restart-on"]; restarts {
t.Errorf("the resolver orders the runtime restarted, which stops every container (ADR 0102): %v", r)
t.Errorf("the runtime is ordered restarted, which stops every container (ADR 0102): %v", r)
}
for _, on := range asStrings(r["reload-on"]) {
if on == "dnsmasq.runtime-dns" {
if on == "docker.daemon" {
reloaded = true
}
}
@@ -183,22 +208,40 @@ func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
t.Errorf("the runtime is not reloaded when its file changes, so live-restore never takes effect")
}
resolv := ids["resolv-conf.resolv"]
if resolv == nil || !strings.Contains(resolv["content"].(string), "\nnameserver 127.0.0.1\n") {
t.Fatalf("the machine is not pointed at the resolver: %v", resolv)
resolv := ids["systemd-networkd.fact-resolvers"]
if resolv == nil || !strings.Contains(resolv["content"].(string), "\nnameserver 10.42.0.1\noptions ") {
t.Fatalf("the machine is not pointed at the resolver by address, and only at it: %v", resolv)
}
}
// Two modules deciding what a machine asks are refused on one machine, as before — the claim
// exists so they never take turns overwriting each other.
// Two modules deciding what a machine asks are refused on one machine, as before — now that the file
// is the uplink's (novox/hq ADR 0223), by two things: a machine runs one network manager, and no other
// module may write the resolver file beside the uplink's, as a file or as a rendered fact.
func TestTwoThingsDecidingWhatAMachineAsksAreRefused(t *testing.T) {
_, err := Resolve(resolverShelf(t), []string{"dnsmasq", "resolv-conf", "resolved-split-dns"},
Node{Name: "anchor", At: "anchor.internal"}, World{})
if err == nil {
t.Fatal("resolv-conf and resolved-split-dns were both assigned to one machine")
shelf := resolverShelf(t)
shelf["networkmanager"] = catalogueManifest(t, "networkmanager")
node := Node{Name: "anchor", At: "anchor.internal", Capabilities: map[string]bool{
"package-manager": true, "service-manager": true,
"uplink-systemd-networkd": true, "uplink-networkmanager": true}}
_, err := Resolve(shelf, []string{"dnsmasq", "systemd-networkd", "networkmanager"}, node, World{})
if err == nil || !strings.Contains(err.Error(), "node-uplink") {
t.Fatalf("two network managers were both assigned to one machine: %v", err)
}
if !strings.Contains(err.Error(), "node-resolver-config") {
t.Fatalf("the refusal does not say what was claimed: %v", err)
// The second is made up: what is under test is that the resolver file has one owner, so any
// module asking the mesh to render it — or declaring it — will do.
for name, m := range map[string]Manifest{
"a-rendered-one": {Module: "a-rendered-one", Version: "1",
Facts: map[string]RosterFile{"mine": {Path: "/etc/resolv.conf", Template: "nameserver 10.42.0.1\n"}}},
"a-declared-one": {Module: "a-declared-one", Version: "1", Resources: []map[string]any{
{"id": "mine", "type": "file", "path": "/etc/resolv.conf", "content": "nameserver 10.42.0.1\n"}}},
} {
shelf := resolverShelf(t)
shelf[name] = m
_, err := Resolve(shelf, []string{"dnsmasq", "systemd-networkd", name}, node, World{})
if err == nil || !strings.Contains(err.Error(), "/etc/resolv.conf") {
t.Errorf("%s wrote the resolver file beside the uplink's: %v", name, err)
}
}
}
+70 -8
View File
@@ -61,6 +61,22 @@ type rosterView struct {
Suffix string
Names []rosterEntry
Machines []rosterEntry
// Zones is every zone a module in the mesh answers itself, with where its answerer is (novox/hq
// ADR 0199) — what the mesh's resolver forwards. Ordered by zone.
Zones []rosterZone
// Holders is the machines holding each replicated mesh seat, by seat (novox/hq ADR 0223) — what
// a machine's resolver file lists for `mesh-dns-resolver`. **This machine first when it is one
// of them**, then the rest by name: the nearest holder is asked first, and two renderings of
// one mesh on one machine are one file. A template reads one seat's with
// `index .Holders "<seat>"`; a seat nobody holds ranges over nothing.
Holders map[string][]rosterEntry
}
// rosterZone is one zone as a template sees it: the zone, and the address and port answering it.
type rosterZone struct {
Zone string
Address string
Port int
}
// rosterEntry is one machine as a template sees it: its bare name, its full mesh name, its address,
@@ -80,20 +96,31 @@ type rosterEntry struct {
// `machines` is only the machines — the two must not be confused (novox/hq 04-ISSUES/111), so both
// are given and the template chooses.
func FactsInto(m Manifest, r Resolution, every, machines, accounts map[string]string, suffix string) ([]map[string]any, error) {
return FactsWithZonesInto(m, r, every, machines, accounts, suffix, nil)
}
// FactsWithZonesInto is FactsInto with the mesh's zones in the view, for a template that ranges them.
func FactsWithZonesInto(m Manifest, r Resolution, every, machines, accounts map[string]string, suffix string,
zones []ZoneAt) ([]map[string]any, error) {
return FactsFrom(m, r, Rendering{Names: every, Machines: machines, Accounts: accounts, Suffix: suffix,
Zones: zones})
}
// FactsFrom renders the roster files a module asked for from everything the mesh composed for this
// machine: its machines, zones and the holders of each replicated seat.
func FactsFrom(m Manifest, r Resolution, with Rendering) ([]map[string]any, error) {
if len(m.Facts) == 0 {
return nil, nil
}
names := make([]string, 0, len(m.Facts))
for name := range m.Facts {
names = append(names, name)
}
sort.Strings(names)
names := sortedFacts(m.Facts)
view := rosterView{
Node: r.Node,
Suffix: strings.TrimPrefix(suffixOr(suffix), "."),
Names: entriesFrom(every, accounts, suffix),
Machines: entriesFrom(machines, accounts, suffix),
Suffix: strings.TrimPrefix(suffixOr(with.Suffix), "."),
Names: entriesFrom(with.Names, with.Accounts, with.Suffix),
Machines: entriesFrom(with.Machines, with.Accounts, with.Suffix),
Zones: zonesFrom(with.Zones),
Holders: holdersFrom(with.Holders, with.Accounts, with.Suffix, r.Node),
}
out := make([]map[string]any, 0, len(names))
@@ -223,3 +250,38 @@ func sortedNames(addresses map[string]string) []string {
sort.Strings(out)
return out
}
// zonesFrom is the zones a template ranges, ordered by zone so two renderings of one mesh are one file.
func zonesFrom(zones []ZoneAt) []rosterZone {
out := make([]rosterZone, 0, len(zones))
for _, z := range zones {
out = append(out, rosterZone{Zone: z.Zone, Address: z.Address, Port: z.Port})
}
sort.Slice(out, func(i, j int) bool { return out[i].Zone < out[j].Zone })
return out
}
// holdersFrom is each replicated seat's holders as a template ranges them: this machine first when
// it holds the seat, then the others by name (novox/hq ADR 0223). A machine with no address is left
// out, as everywhere in the roster — a resolver named at nothing is a lookup that hangs.
func holdersFrom(holders map[string]map[string]string, accounts map[string]string, suffix, node string) map[string][]rosterEntry {
out := make(map[string][]rosterEntry, len(holders))
for seat, at := range holders {
entries := entriesFrom(at, accounts, suffix)
sort.SliceStable(entries, func(i, j int) bool {
return entries[i].Name == node && entries[j].Name != node
})
out[seat] = entries
}
return out
}
// sortedFacts is a module's fact names in order, so what is said about them is said the same way twice.
func sortedFacts(facts map[string]RosterFile) []string {
out := make([]string, 0, len(facts))
for name := range facts {
out = append(out, name)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,67 @@
package catalogue
import "testing"
// A mesh-wide provision whose seat is held on another machine is answered by that holder, even on a
// machine that runs a provider of its own (novox/hq issue 258). Every machine ran its own resolver
// while the mesh moved to one; each bound its resolver configuration to itself, with the mesh's
// resolver held elsewhere and pinned.
func resolverMesh() map[string]Manifest {
return map[string]Manifest{
"resolver": {Module: "resolver", Version: "1",
Provides: []Offer{{Name: "wildcard-resolution", Scope: ScopeMesh}},
Claims: []Claim{{Name: "mesh-dns-resolver", Scope: ScopeMesh}}},
"asker": {Module: "asker", Version: "1", Requires: []string{"wildcard-resolution"}},
}
}
func resolverWorld(held string, pin *Chosen) World {
w := World{
Offered: map[string][]Provider{"wildcard-resolution": {
{Node: "anchor", At: "anchor.internal", Module: "resolver"},
{Node: "laptop", At: "laptop.internal", Module: "resolver"},
}},
}
// Held is who holds it across the mesh; Holdings is the same holder on record, which lets this
// machine's own claimant stand beside it (ADR 0131).
w.Held = []Held{{Claim: "mesh-dns-resolver", Scope: ScopeMesh, Node: held, Module: "resolver"}}
w.Holdings = w.Held
if pin != nil {
w.Pinned = map[string]Chosen{"wildcard-resolution": *pin}
}
return w
}
func answeredFrom(t *testing.T, world World) string {
t.Helper()
laptop := Node{Name: "laptop", At: "laptop.internal"}
got, err := Resolve(resolverMesh(), []string{"resolver", "asker"}, laptop, world)
if err != nil {
t.Fatal(err)
}
for _, n := range got.Needs {
if n.Name == "wildcard-resolution" {
return n.From
}
}
t.Fatalf("no binding for wildcard-resolution: %+v", got.Needs)
return ""
}
func TestTheSeatsHolderElsewhereAnswersBeforeThisMachinesOwnProvider(t *testing.T) {
if from := answeredFrom(t, resolverWorld("anchor", nil)); from != "anchor" {
t.Fatalf("bound to %s; the seat is held on anchor", from)
}
}
func TestAPinElsewhereAnswersBeforeThisMachinesOwnProvider(t *testing.T) {
if from := answeredFrom(t, resolverWorld("laptop", &Chosen{Node: "anchor", Module: "resolver"})); from != "anchor" {
t.Fatalf("bound to %s; this machine was pinned to anchor", from)
}
}
func TestTheHolderOnThisMachineStillAnswersHere(t *testing.T) {
if from := answeredFrom(t, resolverWorld("laptop", nil)); from != "laptop" {
t.Fatalf("bound to %s; the seat is held here", from)
}
}
+226
View File
@@ -0,0 +1,226 @@
package catalogue
import (
"fmt"
"regexp"
"strings"
)
// A module contributes to a seat it does not hold (novox/hq ADR 0212).
//
// ADR 0210 made a tool's configuration its seat holder's, and every other module's way in a
// contribution to the seat. The environment, the shell's slots and the power moments each became a
// field of their own; this is the general form, so a new seat that takes contributions is a row in
// the seat table rather than a change to the manifest: a contribution names a seat, a kind that
// seat receives, and text in the tool's own grammar, which the controller never reads.
// HotkeysSeat is the machine's hotkey daemon (novox/hq ADR 0212 §5).
const HotkeysSeat = "node-hotkeys"
// MessageBusSeat is the machine's D-Bus (novox/hq ADR 0215).
const MessageBusSeat = "node-message-bus"
// BackupSeat is the machine's backups (novox/hq ADR 0214, to-be 43).
const BackupSeat = "node-backup"
// storeProvisions are the provisions whose provider keeps its consumers' data (novox/hq ADR 0214):
// a module providing one must say how to back it up, or the data the mesh hands out is the data it
// cannot restore — issue 241's seven databases. A provision that holds nothing worth keeping (a
// route, a cache, a name) is not here; adding one is adding a store.
var storeProvisions = map[string]bool{
"postgres-database": true,
"mssql-database": true,
"mongodb-database": true,
"s3-bucket": true,
"influxdb-api": true,
"secret": true,
}
// CheckBackup is a store provider that contributes no backup (novox/hq ADR 0214, "How it is
// checked").
//
// **The catalogue check's, not parsing's.** A manifest already registered and running was written
// before the rule; refusing it on read would make the controller refuse the very providers whose
// data the rule protects. New definitions meet it in the catalogue check, where they are written.
func CheckBackup(m Manifest) []string {
backs := false
for _, c := range m.Contributions {
if s, known := SeatNamed(c.Seat); known && s.Name == BackupSeat && c.Kind == "backup" {
backs = true
}
}
if backs {
return nil
}
var problems []string
for _, o := range m.Provides {
if storeProvisions[o.Name] {
problems = append(problems, fmt.Sprintf(
"%s provides %s, which keeps its consumers' data, and contributes no backup to %s; a "+
"store says how its data is copied (novox/hq ADR 0214)", m.Module, o.Name, BackupSeat))
}
}
return problems
}
// SeatContribution is one piece of configuration a module gives a seat's holder to place.
type SeatContribution struct {
// Seat is the seat whose holder places it.
Seat string `json:"seat"`
// Kind is which of the seat's receivable kinds it is.
Kind string `json:"kind"`
// Content is the text, in the tool's own grammar. Never interpreted.
Content string `json:"content"`
}
// ofContribution is where a holder places a kind: ${contribution:<seat>:<kind>}. Loose inside the
// braces, so a misspelt seat or kind is found and refused rather than written out as text.
var ofContribution = regexp.MustCompile(`\$\{contribution:([^}]*)\}`)
// receivable is what a seat receives of a kind, with the seat's canonical name; false when the seat
// is unknown or does not receive it.
func receivable(seat, kind string) (Seat, Receivable, bool) {
s, known := SeatNamed(seat)
if !known {
return Seat{}, Receivable{}, false
}
for _, r := range s.Receives {
if r.Kind == kind {
return s, r, true
}
}
return s, Receivable{}, false
}
// kindsOf names a seat's receivable kinds for a refusal.
func kindsOf(s Seat) string {
if len(s.Receives) == 0 {
return "it receives no contributions"
}
var kinds []string
for _, r := range s.Receives {
kinds = append(kinds, r.Kind)
}
return "it receives " + strings.Join(kinds, ", ")
}
// seatContributionProblems is what is wrong with this module's contributions, from the manifest
// alone (novox/hq ADR 0212 §2).
func (m Manifest) seatContributionProblems() []string {
var problems []string
for i, c := range m.Contributions {
s, r, ok := receivable(c.Seat, c.Kind)
// A directory a contribution names must be one of this module's own, here rather than on the
// machine — where a `${dir:x}` nobody declared would reach the holder as the literal text.
if ok && r.Dirs {
declared := map[string]string{}
for _, res := range m.Resources {
if fmt.Sprint(res["type"]) == "directory" {
declared[fmt.Sprint(res["id"])] = ""
}
}
if _, err := dirFill(c.Content, declared, m.Module); err != nil {
problems = append(problems, fmt.Sprintf("%s's contribution %d: %v", m.Module, i+1, err))
}
}
switch {
case s.Name == "":
problems = append(problems, fmt.Sprintf(
"%s's contribution %d is to the seat %q, which the mesh does not define", m.Module, i+1, c.Seat))
case !ok:
problems = append(problems, fmt.Sprintf(
"%s's contribution %d to %s is of the kind %q; %s (novox/hq ADR 0212)",
m.Module, i+1, s.Name, c.Kind, kindsOf(s)))
}
if strings.TrimSpace(c.Content) == "" {
problems = append(problems, fmt.Sprintf("%s's contribution %d has no content", m.Module, i+1))
}
}
return problems
}
// seatPlaceholderProblems is what is wrong with one resource's ${contribution:…}: placed only in a
// file's content, naming a seat and a kind it receives, and only by a module that claims that seat
// — another would be a second writer of a file there is one of (novox/hq ADR 0212 §3).
func seatPlaceholderProblems(m Manifest, r map[string]any) []string {
var problems []string
for _, field := range sortedKeys(r) {
v, ok := r[field].(string)
if !ok {
continue
}
found := ofContribution.FindAllStringSubmatch(v, -1)
if len(found) == 0 {
continue
}
if field != "content" {
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s in its %s; contributions are placed only in a file's content",
m.Module, r["id"], found[0][0], field))
continue
}
for _, f := range found {
seat, kind, two := strings.Cut(f[1], ":")
s, _, ok := receivable(seat, kind)
if !two || !ok {
detail := "the mesh defines no seat " + fmt.Sprintf("%q", seat)
if s.Name != "" {
detail = s.Name + ": " + kindsOf(s)
}
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s; a contribution is ${contribution:<seat>:<kind>} (%s)",
m.Module, r["id"], f[0], detail))
continue
}
if !m.ClaimsSeat(s.Name) {
problems = append(problems, fmt.Sprintf(
"%s's resource %v names %s and %s does not claim %s; every module's contributions to a "+
"seat are placed by its holder alone (novox/hq ADR 0212)",
m.Module, r["id"], f[0], m.Module, s.Name))
}
}
}
return problems
}
// seatContributions is every module's contribution of one kind to one seat (novox/hq ADR 0212 §3):
// in module order, each module's pieces in the order it declared them, each module's preceded by a
// comment line naming it in the tool's grammar, and empty when nothing is contributed. A kind that
// takes directories has each contributor's `${dir:<id>}` filled with where that module's directory
// is on this machine (novox/hq to-be 43).
func seatContributions(modules []Manifest, seat, kind string, with Rendering) (string, error) {
s, r, ok := receivable(seat, kind)
if !ok {
return "", nil
}
var failed error
var b strings.Builder
for _, m := range inModuleOrder(modules) {
named := false
for _, c := range m.Contributions {
if c.Kind != kind {
continue
}
if cs, known := SeatNamed(c.Seat); !known || cs.Name != s.Name {
continue
}
if !named {
fmt.Fprintf(&b, "%s %s\n", r.Comment, m.Module)
named = true
}
content := c.Content
if r.Dirs {
filled, err := dirFill(content, dirsFor(m, with), m.Module)
if err != nil && failed == nil {
failed = err
}
content = filled
}
b.WriteString(content)
if !strings.HasSuffix(content, "\n") {
b.WriteString("\n")
}
}
}
return b.String(), failed
}

Some files were not shown because too many files have changed in this diff Show More