Compare commits

..
Author SHA1 Message Date
jochen f0a62c1b47 Store optional requirements beside a plain requires list (hq issue 166, review of #281)
mesh/merge-gate head f0a62c1b473f merged into trunk 454a3db69a1e — fail: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; head f…
mesh/repo-check head f0a62c1b473f merged into trunk 454a3db69a1e — fail: head f0a62c1b473f does not merge cleanly into trunk 454a3db69a1e: in conflict test…
mesh/delivery rejected: the gate failed or could not run, or the repository's own check failed
A controller older than optional requirements reads `requires` as names, and Inventory.Catalogue
stops at the first manifest it cannot read: the long form stored inside `requires` would have left
a rollback below this controller composing no machine. Stored, `requires` stays plain and the
optional names sit in `optional-requires`, a key an older controller sets aside, leaving out only
the module carrying it. A refusal of a hard requirement now names a module that wants it hard.
2026-10-13 01:16:32 +02:00
jochen bc81029a03 Let a requirement be optional, binding only a provider that already runs (hq issue 166, ADR 0330)
mesh/merge-gate out of date: judged against trunk 56d6313262e8, the trunk is now cbac703e817d; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 56d6313262e8, the trunk is now cbac703e817d; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Making every soft dependency a provision made it hard: a mesh without influxdb could not assign
grafana, and one with influxdb in its catalogue would install it for grafana. A requirement may now
be written {"name": ..., "optional": true}. It resolves as a hard one when a provider is assigned or
reachable, and is otherwise absent: never refused, nothing installed for it. Its binding, credential
and only-when-bound files are left out, its ${bound:...|default:...} placeholders take their
defaults, the module check refuses one that could render empty, and plan and node list it unbound.
2026-10-13 00:39:58 +02:00
mesh-admin 21ac8fd858 Merge pull request 'Let a mergeable file name the setting keys it takes (hq issue 168, ADR 0326)' (#272) from fix/168-a-file-names-the-settings-it-takes into main 2026-10-12 22:07:08 +00:00
mesh-admin e63af0df17 Merge pull request 'End a walk's apply phase at each machine's own report of its send (hq issue 582)' (#278) from fix/582-apply-ends-at-the-machines-own-report into main 2026-10-12 21:36:33 +00:00
mesh-admin 961f2e82ea Merge pull request 'Hand a delivery build out before pull-request checks (issue 558)' (#276) from fix/558-a-delivery-build-goes-first into main 2026-10-12 21:20:19 +00:00
jschoubben e727589e48 End a walk's apply phase at each machine's own report of its send (hq issue 582)
mesh/merge-gate head e727589e483b merged into trunk 961f2e82ea1c — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head e727589e483b merged into trunk 961f2e82ea1c — pass: its merge-check.sh passed
mesh/delivery delivered
The walk records its send to the rest only after every machine was sent,
which takes 20-28 s, so the machine sent first fell outside the 2 s slack,
its report was missed and the phase ended at the next walk's send to it.
Keep each machine's own send time with what it was sent, match its apply
duration by that send and digest, and show each machine's report in the
walk's phases (ADR 0282 decision 6).
2026-10-12 23:18:28 +02:00
mesh-admin 2f157148c6 Merge pull request 'node shows what a machine said it is made of, with its age (issue 160)' (#269) from fix/160-node-shows-what-a-machine-is into main 2026-10-12 20:37:06 +00:00
mesh-admin 13be75eb65 Merge pull request 'Replay issue 201 with only what the controller had before its fix (for mesh-lab R201)' (#275) from test/201-replay into main 2026-10-12 20:22:37 +00:00
mesh-admin 13711fb5b9 Merge pull request 'Mark that the build agent takes work, so its health check can see it ready (hq issue 544)' (#277) from fix/544-the-build-agent-says-it-is-ready into main 2026-10-12 19:58:09 +00:00
jschoubben fc573f951a Remove only a move's proven second copy, and never delete a handed-out original (hq issue 558)
mesh/merge-gate head fc573f951a47 merged into trunk 2f157148c6c8 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head fc573f951a47 merged into trunk 2f157148c6c8 — pass: its merge-check.sh passed
mesh/delivery delivered
Second review of #276: a dead letter delivered again beside its original
shares its request id and was removed as a copy, losing the ask; and a
publish that outlived the pause could delete an original a holder had
taken. A copy now names the message it copies, removals are said, the
publish waits inside the pause's margin, and an original handed out
meanwhile keeps its place while the copy goes.
2026-10-12 21:46:49 +02:00
jschoubben 1ded7dff34 Make a queue move lossless and run its passes on the acting controller alone (hq issue 558)
mesh/merge-gate out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Review of #276: a move deleted before it published, so a failed publish
lost an ask; a verb's own process ran a pass beside the serving one, so
one could resume the queue while the other moved asks and a build ran
twice. The copy is now published first and second copies removed by id,
asks only signal the acting controller's loop, moves stop with 5 s of
the pause left, a cancel finds a moved ask by id, and an unreadable ask
keeps its age.
2026-10-12 21:38:11 +02:00
jschoubben 50c55a773e Mark that the build agent takes work, so its health check can see it ready (novox/hq issue 544)
mesh/merge-gate head 50c55a773eb8 merged into trunk c0c379af2b3c — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 50c55a773eb8 merged into trunk c0c379af2b3c — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/544-the-build-agent-says-it-is-ready delivered: every member is delivered
The agent listens on nothing, so the node-engine could judge it only alive and
held it starting for the whole grace on every machine a walk moved it to. It
now writes a mark once it is bound to its worker, after its seat verbs are
served, and 'mesh-builder ready' passes only while the mark is there.
2026-10-12 21:33:12 +02:00
jschoubben c6e234455f Hand a delivery build out before pull-request checks (hq issue 558)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
A merged change's walk waited minutes on its build behind checks in a
first-come queue. The queue now hands deliveries out first, a check that
has waited ten minutes before deliveries asked after it, and says each
ask's kind; the order is kept in the work queue itself.
2026-10-12 21:25:46 +02:00
jochen 45a1783ba9 Replay issue 201 with only what the controller had before its fix
mesh/merge-gate head 45a1783ba9a4 merged into trunk 13711fb5b91b — pass: the change touches no module of the mesh's graph
mesh/repo-check head 45a1783ba9a4 merged into trunk 13711fb5b91b — pass: its merge-check.sh passed
mesh/delivery delivered
The fix's own TestReplay201 reads what #273 added (sendScope.puttingBack,
recordedKept's core keep, the condition), so it cannot be laid over the
commit before. TestReplay201ThePlansSend composes a plan's send to the
control node through keepingRecorded, planFor and declarationFor and holds
that it carries the controller the machine was last sent, not the older
record; it fails on 84deb53^1 and passes on 84deb53. For mesh-lab's R201
(hq issue 201).
2026-10-12 21:17:31 +02:00
jochen b538397055 Name the issue that holds the refusal back (hq issue 574)
mesh/merge-gate head b5383970550b merged into trunk e63af0df17f5 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head b5383970550b merged into trunk e63af0df17f5 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 18:00:49 +02:00
jochen dd10889209 Hold back refusing a declaring module's silent mergeable file (hq issue 168)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
betterbird declares a preference and owns a mergeable file that names no
keys, and the catalogue can only name them once this controller runs: the
refusal made the catalogue check fail against catalogue main. It follows
once mesh-catalog names the keys.
2026-10-12 18:00:33 +02:00
jochen 286257d802 Let a mergeable file name the setting keys it takes (hq issue 168, ADR 0326)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestModuleCheckPassesTheCatalogue (0.89s); and TestEveryCatalogueManifestParses, TestNoCatalogue…
mesh/delivery superseded: a newer head of the same pull request
Every key of a module's layers merged into every mergeable file it owns:
the mesh's own words, a key meant for a ${setting:} in another file, a
key for a second configuration file. A module could not own two
configurable files. A file now names its keys in takes-settings and
takes only those; a key no file names is stray where it is set; a
module that declares a setting names the keys of every mergeable file,
and one that declares none keeps today's form (ADR 0164 §15).
2026-10-12 17:47:05 +02:00
jochen 67d54d9d09 Say an account without its time as age unknown, not as two thousand years (issue 160)
mesh/merge-gate head 67d54d9d09b2 merged into trunk 13be75eb65c0 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 67d54d9d09b2 merged into trunk 13be75eb65c0 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 17:43:38 +02:00
jochen 7953d2889d Show what a machine said it is made of in node, with its age (issue 160)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The architecture and kernel a machine reported were kept and read by nothing, and the facts the
node-engine now says (memory, filesystems, processor, distribution, virtual or physical, uptime,
timezone) would have joined them. node shows every one in one block with the age of the account,
and a fact the machine could not determine as unknown with its why, never as a zero.
2026-10-12 17:23:07 +02:00
76 changed files with 3262 additions and 5266 deletions
+13 -1
View File
@@ -66,6 +66,11 @@ broker to take work from or a registry to publish into:
Without --registry the artifacts stay in this machine's container runtime, named by the
digest of their own configuration. The result is printed as JSON.
mesh-builder ready
exits 0 once the agent running in this container is bound to its build work and pulls it, and
1 before: its container's health check.
`
func run() error {
@@ -76,12 +81,18 @@ func run() error {
return nil
case "build":
return buildOnce(context.Background(), os.Args[2:])
case "ready":
// The container's health check (novox/hq issue 544): exits non-zero until the agent takes work.
return ready()
default:
fmt.Print(usage)
return nil
}
}
if err := notReady(); err != nil {
return err
}
credential, err := brokerFrom()
if err != nil {
return err
@@ -134,7 +145,8 @@ func run() error {
// **What an earlier holder here left goes before anything is taken** (novox/hq issue 306): a holder
// recreated mid-check left the check's store and bus running, and the ask went to another machine.
h.removeLeftBehind()
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused})
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused,
Taking: func() { taking(seat) }})
defer machine.Close()
// The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired
+44
View File
@@ -0,0 +1,44 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
)
// **The build agent says it is ready** (novox/hq issue 544). It listens on nothing, so no http or tcp
// check can see it work; it writes a mark once it is bound to its worker and pulls build work, and its
// container's health check runs `mesh-builder ready`, which passes only while the mark is there. Without
// it the node-engine could judge the agent only alive, and every walk that moved it waited out the whole
// grace on each machine.
//
// The mark lives in the container's own temporary directory, never in the workspace the machine keeps:
// a new container starts without it, and a restarted one removes it before it binds again.
func readyMark() string { return filepath.Join(os.TempDir(), "mesh-builder.ready") }
// notReady removes the mark: a holder starting is not taking work until it says so again.
func notReady() error {
if err := os.Remove(readyMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove the mark that this agent takes build work (%s): %w", readyMark(), err)
}
return nil
}
// taking writes the mark. A mark that cannot be written is said, and the agent goes on building: its
// check then says it is not ready, which is the truth the gate should hear.
func taking(seat string) {
if err := os.WriteFile(readyMark(), []byte(seat+"\n"), 0o644); err != nil {
fmt.Fprintf(os.Stderr, "cannot mark that this agent takes build work (%s): %v\n", readyMark(), err)
return
}
fmt.Fprintf(os.Stderr, "taking build work from %s\n", seat)
}
// ready is the health check: nil when this container's agent is bound and pulling build work.
func ready() error {
if _, err := os.Stat(readyMark()); err != nil {
return fmt.Errorf("not taking build work yet: %s is not there", readyMark())
}
return nil
}
+32
View File
@@ -0,0 +1,32 @@
package main
import (
"os"
"testing"
)
// The health check passes only between the agent saying it takes work and its next start (novox/hq
// issue 544).
func TestTheAgentIsReadyOnlyOnceItTakesWork(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := ready(); err == nil {
t.Fatal("an agent that never took work is ready")
}
taking("mesh-build-machine")
if err := ready(); err != nil {
t.Fatalf("an agent taking work is not ready: %v", err)
}
// Started again, it is not ready until it binds again.
if err := notReady(); err != nil {
t.Fatal(err)
}
if err := ready(); err == nil {
t.Fatal("an agent starting again is ready before it takes work")
}
if err := notReady(); err != nil {
t.Fatalf("removing a mark that is not there failed: %v", err)
}
if _, err := os.Stat(readyMark()); err == nil {
t.Fatal("the mark is still there")
}
}
+2 -23
View File
@@ -117,13 +117,10 @@ type asked struct {
// Proposal is a settings layer proposed through a verb (proposals.go, novox/hq ADR 0277): about no
// condition, set on Approve by the serving controller itself, and left alone by the reconciling of conditions.
Proposal *settingsProposal `json:"proposal,omitempty"`
// Take is a library the operator is asked to let the mesh take (kept_libraries.go, novox/hq ADR 0328 §7): about no
// condition, taken on Approve by the serving controller itself, asked once per library.
Take *libraryTake `json:"take,omitempty"`
}
// ofACondition says an ask is one of a condition's: not a rehearsal, not a proposal.
func (r asked) ofACondition() bool { return !r.Rehearsal && r.Proposal == nil && r.Take == nil }
func (r asked) ofACondition() bool { return !r.Rehearsal && r.Proposal == nil }
// partKey is an ask's place among what is asked: its condition and its part.
func partKey(condition, part string) string { return condition + "#" + part }
@@ -195,11 +192,6 @@ type asker struct {
call func(ctx context.Context, a conditions.Action, args map[string]string) error
// setLayer sets a proposed settings layer on the operator's warrant (novox/hq ADR 0277); nil cannot.
setLayer setOnWarrant
// folders is every library waiting to be taken (novox/hq ADR 0328 §7), and take what takes one on the operator's
// warrant; nil asks about none. machines are the mesh's machine names, allowed in an ask's words.
folders func(ctx context.Context) ([]libraryTake, error)
take takeOnWarrant
machines func(ctx context.Context) ([]string, error)
// record writes the hand-act log.
record func(ctx context.Context, act link.HandAct) error
// routerRecord reads the router's record of an ask for a warrant missed; nil reads nothing.
@@ -562,10 +554,6 @@ func (a *asker) reconcile(ctx context.Context) error {
return err
}
}
// And every library waiting to be taken, once per library (novox/hq ADR 0328 §7).
if err := a.askFolders(ctx); err != nil {
a.logf("the operator could not be asked about the libraries waiting to be taken: %v", err)
}
why := "the router refused the ask"
if len(refusedWords) > 0 {
why += ": " + refusedWords[0]
@@ -830,7 +818,7 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
if err != nil {
return err
}
stillOpen := r.Rehearsal || r.Proposal != nil || r.Take != nil // a rehearsal, a proposal and a take are about no condition
stillOpen := r.Rehearsal || r.Proposal != nil // a rehearsal and a proposal are about no condition
for _, c := range open {
stillOpen = stillOpen || c.Key == r.Condition
}
@@ -888,15 +876,6 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
if acted == nil && outcome != "" && !strings.HasPrefix(outcome, "nothing") {
outcome = "done: " + outcome
}
case r.Take != nil && act.Verb == takeVerb:
// A library taken on the operator's warrant (novox/hq ADR 0328 §7): the act is composed again from the
// record and held to what the option bound before the engine is asked.
outcome, acted = a.decideTake(ctx, *r, act, w)
if acted == nil && !strings.HasPrefix(outcome, "nothing") {
outcome = "done: " + outcome
}
case r.Take != nil:
acted = fmt.Errorf("the take %s offers no act %s", r.ID, act.Verb)
case act.Arguments["silence"] != "":
acted = a.silence(ctx, act.Arguments["silence"], conditions.MaxSilence, byWords(w), why)
default:
-21
View File
@@ -572,27 +572,6 @@ func servingAsker(open *stores, conn *nats.Conn, js jetstream.JetStream, keeper
call: callAction(conn),
// A proposed settings layer is set by this controller itself on the warrant (novox/hq ADR 0277).
setLayer: setLayerIn(open),
// A shared folder waiting to be kept is asked about once, and taken on the warrant (novox/hq ADR 0325 §8).
folders: foldersIn(open),
take: takeFolderIn(open, func(ctx context.Context, act link.AccessActAsk) (link.HandOverAnswer, error) {
ident, err := open.Identity(ctx)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("the mesh's signing key cannot be read, so nothing was asked of "+
"%s: %w", act.Node, err)
}
return link.AskTake(ctx, conn, ident, act, link.TakeWithin)
}),
machines: func(ctx context.Context) ([]string, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, err
}
names := make([]string, 0, len(nodes))
for _, n := range nodes {
names = append(names, n.Name)
}
return names, nil
},
record: func(ctx context.Context, act link.HandAct) error {
_, err := link.RecordHandAct(ctx, conn, act)
return err
-8
View File
@@ -152,14 +152,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
}
}
// **A module that uses several libraries is warned** (novox/hq ADR 0328 §4): an image that drops added groups when
// it changes user reaches only the first. Printed, never counted.
for _, name := range names {
for _, w := range catalogue.LibraryWarnings(shelf[name]) {
fmt.Fprintf(out, "warning: %s\n", w)
}
}
// **A file that asks for a setting and does not say whether it is trusted counts as trusted** (novox/hq issue
// 339): listed and counted, never refused, so an author can opt out a file nothing trusts.
unsaid := 0
+5 -5
View File
@@ -1042,13 +1042,13 @@ func deliveryCommand(ctx context.Context, args []string) error {
return fmt.Errorf("delivery %s: plan, order, check, go, stop or walks", sub)
}
// appliedFrom answers a machine's first report after a send from the controller's `apply` durations; a lookup
// that fails is a report not read, which leaves the walk's end unknown rather than wrong.
// appliedFrom answers a machine's own report of the send a walk made to it from the controller's `apply` durations
// (novox/hq issue 582); a lookup that fails is a report not read, which leaves the walk's end unknown rather than wrong.
func appliedFrom(ctx context.Context, inv *inventory.Inventory) inventory.AppliedLookup {
return func(node string, sent time.Time) (inventory.AppliedReport, bool) {
r, ok, err := inv.FirstAppliedAfter(ctx, node, sent)
return func(node string, send inventory.SentDeclaration, walkSent time.Time) (inventory.AppliedReport, bool) {
r, ok, err := inv.FirstAppliedAfter(ctx, node, send, walkSent)
if err != nil {
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, sent.Format(time.RFC3339), err)
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, walkSent.Format(time.RFC3339), err)
return inventory.AppliedReport{}, false
}
return r, ok
-56
View File
@@ -154,10 +154,6 @@ type gateFacts struct {
// groupsAdded is, per module, whether the move judged puts an account in a group its previous build did
// not (issue 318 review): the only move whose wait for a new login is excused.
groupsAdded map[string]bool
// accessesChanged is, per module, whether the move judged changes its accesses (novox/hq ADR 0325): an access the
// engine cannot judge this look (unknown), or entries it left while a disk was away, of a module whose move leaves
// its accesses as they were is no fault of the move. A module absent here is taken to change them: judged as before.
accessesChanged map[string]bool
// waits is what the controller holds to check a module's wait for the operator (novox/hq ADR 0283): the
// manifest of each module's build judged, and the secrets given on each machine.
waits operatorWaitFacts
@@ -532,7 +528,6 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
return "", err
}
facts.groupsAdded = movesAddingGroups(ctx, open.inventory, g, pairs, shelf)
facts.accessesChanged = movesChangingAccesses(ctx, open.inventory, g, pairs, shelf)
facts.waits = gateWaitFacts(ctx, open.inventory, g, pairs, shelf, facts.health)
facts.sent = g.Sent
facts.commits, facts.sentBuilds = judgedCommits(g, pairs), map[string]map[string]string{}
@@ -788,57 +783,6 @@ func joinSaid(a, b string) string {
return a + "; " + b
}
// movesChangingAccesses is, per module a gate judges, whether its move changes its accesses — one added, dropped,
// or given another use — read from the builds' manifests (novox/hq ADR 0325). A move not known changes them: an
// access it cannot judge then holds the gate to its bound, and never passes unseen.
func movesChangingAccesses(ctx context.Context, inv *inventory.Inventory, g *inventory.PlanGate, pairs []judged,
shelf map[string]catalogue.Manifest) map[string]bool {
out := map[string]bool{}
for _, j := range pairs {
if _, done := out[j.module]; done {
continue
}
from, to := g.From, g.To
for _, c := range g.Carried {
if c.Module == j.module {
from, to = c.From, c.To
break
}
}
target, found, err := inv.ManifestAt(ctx, j.module, to)
if err != nil || !found {
target = shelf[j.module]
}
before, had, err := inv.ManifestAt(ctx, j.module, from)
if from == "" || err != nil || !had {
out[j.module] = true
continue
}
out[j.module] = !sameAccessUse(before, target)
}
return out
}
// sameAccessUse says a move leaves a module's accesses and libraries as they were (the review of e1c2b744, novox/hq
// ADR 0328): the same accesses at the same use, the same library entries, and whether it joins — a module that starts
// joining runs as another user, and changes who uses its libraries.
func sameAccessUse(a, b catalogue.Manifest) bool {
return sameAccesses(a.Accesses, b.Accesses) && catalogue.LibraryUseShape(a) == catalogue.LibraryUseShape(b)
}
// sameAccesses says two manifests declare the same accesses, at the same use, in any order.
func sameAccesses(a, b []catalogue.Access) bool {
key := func(as []catalogue.Access) string {
var out []string
for _, x := range as {
out = append(out, x.ID+"\x00"+x.Path+"\x00"+x.At())
}
sort.Strings(out)
return strings.Join(out, "\x01")
}
return key(a) == key(b)
}
// movesAddingGroups is, per module a gate judges, whether its move puts an account in a group the build it
// moved from did not: read from the builds' manifests. A module whose move is not known adds none.
func movesAddingGroups(ctx context.Context, inv *inventory.Inventory, g *inventory.PlanGate, pairs []judged,
-164
View File
@@ -1,164 +0,0 @@
package main
import (
"context"
"fmt"
"sort"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A module that joins keeps the user it ran as until it may join (novox/hq ADR 0328 §8, ADR 0334).
//
// **Held at the build it ran before, composed exactly as then.** A module whose registered build joins — declares
// libraries or uses its identity — on a machine that was last sent another build of it, is held there while
//
// - the machine's engine is older than link.OwnGroupContract: it cannot put the tool runner in the module's own group
// nor keep its own directories' group, so the module's tools would lose their read (the re-review of 96e15f86, 2);
// - or a library it uses is not yet taken (§8).
//
// Held, it is composed at that build — the manifest the build records hold, in place of the registered one, as a
// recorded module is kept (recorded_kept.go) — and with the settings it last ran with (SettingsAsOf the machine's last
// send): its resources, its ports in the filter, its contributions and bindings are what they were, and nothing of the
// new contract is sent for it. The send records that the machine still carries that build. A delivery that brings the
// joining build does not send it while it is held: the module waits for a person, the plan says so and judges its
// other modules, and sends it once nothing holds it (release_plan.go, heldIn). A module the machine never ran has
// nothing to keep, and is composed as registered; a build the records no longer hold cannot be kept.
// heldModule is one module held on a machine: the build it keeps and why, in words with no path.
type heldModule struct {
commit string
// library is the first library it uses that is not taken, empty when none.
library string
why string
}
// holdJoining is every module held on a machine, by module; shelf, when given, has each one's manifest replaced by
// the build it keeps.
func holdJoining(ctx context.Context, inv *inventory.Inventory, node string,
shelf map[string]catalogue.Manifest) (map[string]heldModule, error) {
registered := shelf
if registered == nil {
var err error
if registered, err = inv.Catalogue(ctx); err != nil {
return nil, err
}
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
return nil, err
}
var joining []string
for m := range sent {
if man, has := registered[m]; has && catalogue.Joins(man) {
joining = append(joining, m)
}
}
if len(joining) == 0 {
return nil, nil
}
sort.Strings(joining)
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return nil, err
}
engineJoins := had && stated.Contract >= link.OwnGroupContract
libs, err := inv.Libraries(ctx)
if err != nil {
return nil, err
}
taken, err := librariesTaken(ctx, inv, libs)
if err != nil {
return nil, err
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
out := map[string]heldModule{}
for _, m := range joining {
was := sent[m]
if was == "" || sameCommit(was, current[m].Commit) {
continue // the machine already runs the registered build: nothing before it to keep
}
h := heldModule{commit: was}
for _, u := range registered[m].Libraries {
if !taken[u.Library] {
h.library = u.Library
break
}
}
switch {
case !engineJoins:
h.why = fmt.Sprintf("%s keeps the build it runs on %s until that machine's node-engine keeps a module's own "+
"group (novox/hq ADR 0334)", m, node)
case h.library != "":
h.why = fmt.Sprintf("%s waits for the operator to let the mesh keep the library %s (novox/hq ADR 0328 §8)",
m, h.library)
default:
continue
}
before, found, err := inv.ManifestAt(ctx, m, was)
if err != nil {
return nil, err
}
if !found || (engineJoins && usesLibrary(before, h.library)) {
continue
}
if shelf != nil {
shelf[m] = before
}
out[m] = h
}
return out, nil
}
// heldIn is why a module is held on any of the machines given, or "": what a delivery's first send waits on.
func heldIn(ctx context.Context, inv *inventory.Inventory, module string, machines []string) (string, error) {
for _, node := range machines {
held, err := holdJoining(ctx, inv, node, nil)
if err != nil {
return "", err
}
if h, is := held[module]; is {
return h.why, nil
}
}
return "", nil
}
// heldSettings puts, for each module held on a machine, the settings it last ran with in place of today's: the layers
// as they stood at the machine's last send. Today's stand where no send is recorded.
func heldSettings(ctx context.Context, inv *inventory.Inventory, node string, held map[string]heldModule,
settings catalogue.SettingsBy) error {
if len(held) == 0 {
return nil
}
last, found, err := inv.LastSend(ctx, node)
if err != nil || !found {
return err
}
for m := range held {
layers, err := inv.SettingsAsOf(ctx, node, m, last.SentAt)
if err != nil {
return err
}
if len(layers) == 0 {
delete(settings, m)
continue
}
settings[m] = layers
}
return nil
}
func usesLibrary(m catalogue.Manifest, library string) bool {
for _, u := range m.Libraries {
if library != "" && u.Library == library {
return true
}
}
return false
}
-144
View File
@@ -1,144 +0,0 @@
package main
import (
"encoding/json"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
const heldImage = "x@sha256:2222222222222222222222222222222222222222222222222222222222222222"
// radarrBefore is radarr as it ran before it joined: its own port, its own data, PUID from a setting.
func radarrBefore() catalogue.Manifest {
return catalogue.Manifest{Module: "radarr", Version: "1", Capabilities: []string{"container-runtime"},
Listens: []catalogue.Listening{{Port: 7878, From: catalogue.FromMesh, Name: "web", Why: "its web UI"}},
Resources: []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "app", "type": "container", "name": "radarr", "image": heldImage, "ports": []any{"7878"},
"env": map[string]any{"PUID": "1000"}, "volumes": []any{"${dir:config}:/config"}},
}}
}
// radarrJoined is the same module's next build, joining the media library.
func radarrJoined() catalogue.Manifest {
m := radarrBefore()
m.Libraries = []catalogue.LibraryUse{{ID: "movies", Library: "media", Path: "movies", Use: catalogue.AccessReadWrite}}
m.Resources = []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "app", "type": "container", "name": "radarr", "image": heldImage, "ports": []any{"7878"},
"env": map[string]any{"PUID": "${identity:uid}"},
"volumes": []any{"${dir:config}:/config", "${library:movies}:/movies"}},
}
return m
}
// novox/hq ADR 0328 §8 (the review of 7ea6d36e, item 1): a joining module that ran on a machine before is held until
// its libraries are taken — composed EXACTLY as before it joined, at the build the machine runs: its resources, its
// port in the filter, its contributions; only the new build waits. The record says the machine still carries that build.
func TestAHeldModuleIsComposedAsBeforeItJoined(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
controller := catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"}
register(t, open, controller)
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
before, joined := radarrBefore(), radarrJoined()
raw, _ := json.Marshal(before)
b1 := inventory.Build{ID: "b1", Repository: "https://forge.invalid/media.git", Ref: "main", Module: "radarr",
Commit: "c1c1c1c1", On: "builder", Manifest: raw}
if err := inv.RecordBuild(ctx, b1); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, before, inventory.Source{Repository: b1.Repository, Ref: "main", BuiltFrom: b1.Commit}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "radarr"); err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
// What anchor composes for radarr before it joined.
plan, settings, err := planFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
was, err := declarationWith(ctx, open, "anchor", plan, settings, gens, Reading)
if err != nil {
t.Fatal(err)
}
wasRules, err := plan.Rules(catalogue.Rendering{Settings: settings})
if err != nil {
t.Fatal(err)
}
record, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": b1.Commit}, 1, 1,
inventory.Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
// Anchor's engine keeps libraries; the next build joins media, which is not taken.
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.LibraryContract,
SaidAt: time.Now(), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
raw, _ = json.Marshal(joined)
b2 := b1
b2.ID, b2.Commit, b2.Manifest = "b2", "c2c2c2c2", raw
if err := inv.RecordBuild(ctx, b2); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, joined, inventory.Source{Repository: b2.Repository, Ref: "main", BuiltFrom: b2.Commit}); err != nil {
t.Fatal(err)
}
plan, settings, err = planFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
held, err := declarationWith(ctx, open, "anchor", plan, settings, gens, Reading)
if err != nil {
t.Fatal(err)
}
if why, left := held.leftOutWhy["radarr"]; left {
t.Fatalf("radarr was left out while held: %s", why)
}
if !reflect.DeepEqual(radarrResources(was.Resources), radarrResources(held.Resources)) {
t.Fatalf("radarr held is composed otherwise than before it joined:\n %v\n %v", radarrResources(was.Resources),
radarrResources(held.Resources))
}
rules, err := plan.Rules(catalogue.Rendering{Settings: settings})
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(rules, wasRules) || !strings.Contains(asJSON(rules), "7878") {
t.Fatalf("radarr's port is not in the filter as before: %s", asJSON(rules))
}
}
func radarrResources(rs []map[string]any) []map[string]any {
var out []map[string]any
for _, r := range rs {
if strings.HasPrefix(r["id"].(string), "radarr.") {
out = append(out, r)
}
}
return out
}
func asJSON(v any) string {
raw, _ := json.Marshal(v)
return string(raw)
}
-329
View File
@@ -1,329 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
"github.com/novox/mesh-controller/internal/link"
)
// The re-review of 96e15f86, item 1 (novox/hq ADR 0328 §8): a delivery that brings a joining build of a module whose
// library is not taken does not send it — the module waits for a person, the plan stays open saying so, and the other
// modules of the plan are sent and judged as usual. Once the library is taken, the waiting build is sent and judged,
// with no new merge.
func TestADeliveryWaitsForALibraryAndJudgesTheRest(t *testing.T) {
judgeLeavesAsTheCore(t)
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
keeper, _ := withConditionsInMemory(t)
was := doctorFrom
doctorFrom = &doctor{open: open, keeper: keeper, teller: &conditions.Told{}}
t.Cleanup(func() { doctorFrom = was })
register(t, open, catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"})
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
app1 := catalogue.Manifest{Module: "app", Version: "c1"}
app2 := catalogue.Manifest{Module: "app", Version: "c2"}
builds := []struct {
id, module, commit string
m catalogue.Manifest
at time.Time
}{
{"build-app-1", "app", "c1", app1, time.Now().Add(-2 * time.Hour)},
{"build-radarr-1", "radarr", "r1", radarrBefore(), time.Now().Add(-2 * time.Hour)},
{"build-app-2", "app", "c2", app2, time.Now().Add(-time.Minute)},
{"build-radarr-2", "radarr", "r2", radarrJoined(), time.Now().Add(-time.Minute)},
}
for _, b := range builds {
raw, _ := json.Marshal(b.m)
if err := inv.RecordBuild(ctx, inventory.Build{ID: b.id, Module: b.module, Commit: b.commit,
Repository: "novox/mesh-media-catalog", Path: "modules/" + b.module, Ref: "main", Manifest: raw,
Asked: b.at, At: b.at}); err != nil {
t.Fatal(err)
}
}
registerAt := func(m catalogue.Manifest, commit string, asked time.Time) {
if err := inv.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-media-catalog", Seat: "git",
Path: "modules/" + m.Module, Ref: "main", BuiltFrom: commit, Head: commit, Asked: asked}); err != nil {
t.Fatal(err)
}
}
registerAt(app1, "c1", time.Now().Add(-2*time.Hour))
registerAt(radarrBefore(), "r1", time.Now().Add(-2*time.Hour))
if _, err := inv.Assign(ctx, "anchor", "app"); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "anchor", "radarr"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor-0", map[string]string{"app": "c1", "radarr": "r1"}); err != nil {
t.Fatal(err)
}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.OwnGroupContract,
SaidAt: time.Now(), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
registerAt(app2, "c2", time.Now().Add(-time.Minute))
registerAt(radarrJoined(), "r2", time.Now().Add(-time.Minute))
// Every send records what it carries: the registered builds, a held module at the build it runs.
var sent []string
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
for _, node := range names {
held, err := holdJoining(ctx, open.inventory, node, nil)
if err != nil {
return nil, err
}
carried := map[string]string{"app": current["app"].Commit, "radarr": current["radarr"].Commit}
for m, h := range held {
carried[m] = h.commit
}
sent = append(sent, node+":app="+carried["app"]+",radarr="+carried["radarr"])
digest := fmt.Sprintf("d-%s-%d", node, len(sent))
if err := open.inventory.RecordSent(ctx, nodeID(t, open, node), digest, carried); err != nil {
return nil, err
}
if _, err := open.inventory.RecordDoing(ctx, nodeID(t, open, node), inventory.Doing{Node: node,
Outcome: inventory.OutcomeApplied, Declared: digest, Applied: 1, At: time.Now()}); err != nil {
return nil, err
}
}
return names, nil
}
t.Cleanup(func() { sendRollout = wasSend })
wasGather := gatherGateFacts
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
f := gateFacts{now: time.Now(), reports: map[string]inventory.Reported{}, engines: map[string]string{},
rolledBack: map[string][]lease.Rollback{}, served: map[string]served{}}
reports, err := open.inventory.LastReports(ctx)
for _, r := range reports {
f.reports[r.Node] = r
}
return f, err
}
t.Cleanup(func() { gatherGateFacts = wasGather })
wasSettle, wasEvery := gateSettle, gateEvery
gateSettle, gateEvery = 0, 0
t.Cleanup(func() { gateSettle, gateEvery = wasSettle, wasEvery })
built := time.Now().UTC()
plan := inventory.Plan{ID: "plan-held", Repository: "novox/mesh-media-catalog", Branch: "main", Commit: "c2",
Created: built, State: inventory.PlanBuilding, Tiers: [][]string{{"app", "radarr"}},
Modules: map[string]*inventory.PlanModule{
"app": {State: "built", BuiltAt: &built, Commit: "c2", Build: "build-app-2"},
"radarr": {State: "built", BuiltAt: &built, Commit: "r2", Build: "build-radarr-2"}}}
if err := inv.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
for i := 0; i < 4; i++ {
advancePlans(ctx, open)
}
p, err := inv.PlanByID(ctx, "plan-held")
if err != nil {
t.Fatal(err)
}
if p.State == inventory.PlanSuperseded || p.State == inventory.PlanDone || p.State == inventory.PlanFailed {
t.Fatalf("the plan is %s (%s); want it open, waiting for the library", p.State, p.Note)
}
if g := p.Modules["app"].Gate; g == nil || g.Verdict != inventory.GatePassed {
t.Fatalf("app was not judged and passed beside the held module: %+v (%s)", p.Modules["app"], p.Note)
}
if p.Modules["radarr"].FirstAt != nil || !strings.Contains(p.Note, "library media") {
t.Fatalf("radarr was sent, or the plan does not say it waits for the library: %+v, %q", p.Modules["radarr"], p.Note)
}
if slices.ContainsFunc(sent, func(s string) bool { return strings.Contains(s, "radarr=r2") }) {
t.Fatalf("radarr's joining build was sent while held: %v", sent)
}
// The library is taken: the waiting build is sent, with no new merge.
if err := inv.RecordTakeStarted(ctx, inventory.TakenLibrary{Library: "media", Home: "anchor", Root: "/storage/media",
By: "the operator", Ask: "c01", Warrant: "c01"}); err != nil {
t.Fatal(err)
}
if err := inv.TakeDone(ctx, "media", "c01"); err != nil {
t.Fatal(err)
}
for i := 0; i < 4; i++ {
advancePlans(ctx, open)
}
if !slices.ContainsFunc(sent, func(s string) bool { return strings.Contains(s, "radarr=r2") }) {
t.Fatalf("radarr's joining build was not sent once its library was taken: %v", sent)
}
}
// The re-review of 96e15f86, item 2: a module joins only on an engine stating link.OwnGroupContract (7). On an engine
// at 6 a module that ran there before is held at the build it ran, whatever its library — so its tools never lose
// their read of its directories.
func TestAJoiningModuleIsHeldOnAnEngineOlderThanOwnGroups(t *testing.T) {
open, record := heldMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RecordTakeStarted(ctx, inventory.TakenLibrary{Library: "media", Home: "anchor", Root: "/storage/media",
By: "the operator", Ask: "c01", Warrant: "c01"}); err != nil {
t.Fatal(err)
}
if err := inv.TakeDone(ctx, "media", "c01"); err != nil {
t.Fatal(err)
}
_ = record
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.LibraryContract,
SaidAt: time.Now(), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
held, err := holdJoining(ctx, inv, "anchor", nil)
if err != nil {
t.Fatal(err)
}
if held["radarr"].commit != "c1c1c1c1" {
t.Fatalf("on an engine at contract 6 radarr is not held at the build it ran: %+v", held)
}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.OwnGroupContract,
SaidAt: time.Now().Add(time.Second), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
if held, err = holdJoining(ctx, inv, "anchor", nil); err != nil || len(held) > 0 {
t.Fatalf("on an engine at contract 7 with its library taken radarr is still held: %+v (%v)", held, err)
}
}
// heldMesh is anchor running radarr's build from before it joined, the joining build registered, media defined.
func heldMesh(t *testing.T) (*stores, inventory.Node) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"})
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
for _, b := range []struct {
id, commit string
m catalogue.Manifest
}{{"b1", "c1c1c1c1", radarrBefore()}, {"b2", "c2c2c2c2", radarrJoined()}} {
raw, _ := json.Marshal(b.m)
if err := inv.RecordBuild(ctx, inventory.Build{ID: b.id, Repository: "https://forge.invalid/media.git", Ref: "main",
Module: "radarr", Commit: b.commit, On: "builder", Manifest: raw}); err != nil {
t.Fatal(err)
}
}
if err := inv.RegisterModule(ctx, radarrBefore(), inventory.Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c1c1c1c1"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "radarr"); err != nil {
t.Fatal(err)
}
record, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": "c1c1c1c1"}, 1, 1,
inventory.Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, radarrJoined(), inventory.Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c2c2c2c2"}); err != nil {
t.Fatal(err)
}
return open, record
}
// The re-review of 96e15f86, item 3: a held module is composed with the settings it last ran with — a layer changed
// for the joining build (its puid removed) does not reach the build it still runs.
func TestAHeldModuleKeepsTheSettingsItRanWith(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"})
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
before := radarrBefore()
before.Resources = append(before.Resources, map[string]any{"id": "identity", "type": "file",
"path": "${dir:config}/identity.env", "mode": "0644", "content": "PUID=${setting:puid}\n"})
joined := radarrJoined()
for _, b := range []struct {
id, commit string
m catalogue.Manifest
}{{"b1", "c1c1c1c1", before}, {"b2", "c2c2c2c2", joined}} {
raw, _ := json.Marshal(b.m)
if err := inv.RecordBuild(ctx, inventory.Build{ID: b.id, Repository: "https://forge.invalid/media.git", Ref: "main",
Module: "radarr", Commit: b.commit, On: "builder", Manifest: raw}); err != nil {
t.Fatal(err)
}
}
if err := inv.RegisterModule(ctx, before, inventory.Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c1c1c1c1"}); err != nil {
t.Fatal(err)
}
if err := inv.SetSettings(ctx, "anchor", "radarr", map[string]any{"puid": "1000"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "radarr"); err != nil {
t.Fatal(err)
}
record, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": "c1c1c1c1"}, 1, 1,
inventory.Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
time.Sleep(20 * time.Millisecond)
if err := inv.RegisterModule(ctx, joined, inventory.Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c2c2c2c2"}); err != nil {
t.Fatal(err)
}
if err := inv.ClearSettings(ctx, "anchor", "radarr"); err != nil { // the joining build reads no puid
t.Fatal(err)
}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.OwnGroupContract,
SaidAt: time.Now(), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
plan, settings, err := planFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
held, err := declarationWith(ctx, open, "anchor", plan, settings, gens, Reading)
if err != nil {
t.Fatal(err)
}
if why, left := held.leftOutWhy["radarr"]; left {
t.Fatalf("radarr held was left out: %s", why)
}
for _, r := range held.Resources {
if r["id"] == "radarr.identity" {
if r["content"] != "PUID=1000\n" {
t.Fatalf("radarr held reads %q; want the puid it ran with", r["content"])
}
return
}
}
t.Fatal("radarr held has no identity file")
}
-401
View File
@@ -1,401 +0,0 @@
package main
// Existing data joins a library once, on the operator's word (novox/hq ADR 0328 §7, §8).
//
// **What waits.** A node-engine that keeps libraries states a library waiting, with an access wait of reason `keep`,
// while its root on its home holds data not yet in its group; a module that uses it waits the same way. The
// controller excuses such a wait only for a defined library that reaches the machine and is not yet taken on its home,
// on the filesystem the wait names (checkLibraryWait), and the module is said as its needs-operator condition, in words
// with no path.
//
// **One ask per library.** The serving controller's asker asks the operator once for each such library — however many
// modules and machines wait on it — at the level approve (ADR 0259), in the clause's words: "Let the mesh keep library
// <name> at <root> on <machine>: …". It is kept in the controller's asks beside a proposal (asked.Take), never a
// condition's. While it is open no second is asked; one that expired unanswered is asked again while the library
// waits; one about a library that no longer waits is taken back; one whose library moved to another filesystem is
// asked again for the one it is on now. None is asked for a library already taken.
//
// **On the warrant, and only on it**, the controller asks the library's home engine to take it (link.AskTake), naming
// "library:<name>" and the warrant — after checking the library still waits on the filesystem the operator was asked
// about. On the engine's word that it started, the library is recorded taken (kept_library), and from the next send
// the library says `taken`.
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/outward"
)
// libraryTake is one library the operator is asked to let the mesh take, as the controller's ask keeps it
// (asked.Take). Every field the ask is composed from is here, so the act on the warrant is composed from the record.
type libraryTake struct {
Library string `json:"library"`
// Node is the library's home, where it is taken; Root its root there.
Node string `json:"node"`
Root string `json:"root"`
// Modules are the modules waiting on it, sorted.
Modules []string `json:"modules"`
// Placed is the filesystem the root is on, as the home's engine names it without a path: the warrant is for it.
Placed string `json:"placed,omitempty"`
At time.Time `json:"at"`
}
// takeVerb is the verb a take's answer binds: the controller's own, performed by itself on the warrant.
const takeVerb = askerName + ".take"
// key is the library on its home, at its root, on its filesystem: what one ask is kept for. A library that moved is
// asked again.
func (t libraryTake) key() string {
return t.Library + "\x00" + t.Node + "\x00" + t.Root + "\x00" + t.Placed
}
// digest binds everything the take does and was said for.
func (t libraryTake) digest() string {
raw, _ := json.Marshal(struct {
Library, Node, Root, Placed string
Modules []string
}{t.Library, t.Node, t.Root, t.Placed, t.Modules})
sum := sha256.Sum256(raw)
return "sha256:" + hex.EncodeToString(sum[:])
}
// actions is the take's one answer, at the level approve, binding the exact act through boundAct.
func (t libraryTake) actions(id string) []conditions.Action {
return []conditions.Action{{Label: "Let the mesh keep it", Verb: takeVerb, Machine: t.Node,
Level: conditions.LevelApprove, Arguments: map[string]string{"take": id, "library": t.Library,
"node": t.Node, "access": libraryAccess + t.Library, "digest": t.digest()}}}
}
// takeWords is the clause's ask (ADR 0328 §7), with the root as shown.
func takeWords(library, shown, machine string) string {
return fmt.Sprintf("Let the mesh keep library %s at %s on %s: its group becomes %s, the group may read and write, "+
"folders keep the group; owners and everyone else's rights stay as they are", library, shown, machine,
catalogue.MeshName(library))
}
// ask is the take's ask, composed from it alone. The root is said whole where the router's content rule lets it leave
// the mesh, and otherwise as "its root".
func (t libraryTake) ask(id string, machines []string) (asks.Ask, map[string]int) {
shown := t.Root
if _, ok := outward.Check(shown, machines...); !ok {
shown = "its root"
}
headline := fmt.Sprintf("Let the mesh keep library %s?", t.Library)
if len([]rune(headline)) > asks.HeadlineLength {
headline = "Let the mesh keep a library?"
}
explanation := takeWords(t.Library, shown, t.Node) + "."
if len(t.Modules) > 0 {
explanation += " Waiting on it: " + strings.Join(t.Modules, ", ") + "."
}
if _, ok := outward.Check(explanation, machines...); !ok {
explanation = sayable(explanation, machines)
}
q := asks.Ask{ID: id, Headline: headline, Explanation: explanation,
Who: asks.Operator, Expires: t.At.Add(askApproveFor),
OnExpiry: "nothing is changed, and you are asked again while the library waits", About: "take." + t.Library}
options := map[string]int{}
for i, act := range t.actions(id) {
binds, _ := asks.ActDigest(boundAct(act))
oid := optionID(act.Label)
options[oid] = i
q.Options = append(q.Options, asks.Option{ID: oid, Label: act.Label, Level: asks.Level(act.Level), Binds: binds,
Does: "the library's data joins its group, and is kept from then on without asking"})
}
return q, options
}
// takeOnWarrant performs a take the operator approved: the engine is asked, and the library recorded taken on its
// word. It answers what the engine said.
type takeOnWarrant func(ctx context.Context, t libraryTake, askID, by string) (string, error)
// askFolders asks the operator about every library waiting to be taken that is not asked about yet, and takes back an
// open ask about one that no longer waits. Called at the end of each reconcile, after the conditions' asks, within the
// asks the router holds open at once.
func (a *asker) askFolders(ctx context.Context) error {
if a.folders == nil {
return nil
}
wanted, err := a.folders(ctx)
if err != nil {
return err
}
now := a.now()
all, err := a.store.All(ctx)
if err != nil {
return err
}
open, openNow := map[string]asked{}, 0
for _, r := range all {
if r.State != askOpen || !now.Before(r.Ask.Expires) {
continue
}
openNow++
if r.Take != nil {
open[r.Take.key()] = r
}
}
want := map[string]bool{}
for _, t := range wanted {
want[t.key()] = true
}
for key, r := range open {
if !want[key] {
if err := a.cancel(ctx, r, "the library no longer waits to be taken there"); err != nil {
return err
}
openNow--
}
}
var machines []string
if a.machines != nil {
if machines, err = a.machines(ctx); err != nil {
return err
}
}
for _, t := range wanted {
if _, asking := open[t.key()]; asking {
continue // one ask per library (§7)
}
if openNow >= askMostOpen {
return nil // asked when one of the open ones ends
}
t.At = now
id := newAskID()
q, options := t.ask(id, machines)
if err := q.Check(now); err != nil {
a.logf("the operator could not be asked to let the mesh keep the library %s: %v", t.Library, err)
continue
}
body, err := json.Marshal(q)
if err != nil {
return err
}
take := t
if err := a.store.Create(ctx, asked{ID: id, Condition: q.About, Ask: q, Actions: t.actions(id),
Options: options, State: askOpen, Opened: now, Take: &take}); err != nil {
return fmt.Errorf("the take could not be kept in the controller's asks, so it was not asked: %w", err)
}
if err := a.publish(ctx, asks.AskSubject(askerName), body, "ask."+id); err != nil {
_, _ = a.store.Change(ctx, id, func(x *asked) bool {
if x.State != askOpen || x.Acted != "" {
return false
}
x.State, x.Ended, x.Acted = askUnsent, a.now(), "nothing: it could not be published: "+err.Error()
return true
})
a.logf("the operator could not be asked to let the mesh keep the library %s: %v", t.Library, err)
continue
}
openNow++
a.logf("asked the operator to let the mesh keep the library %s on %s (%s)", t.Library, t.Node, id)
}
return nil
}
// decideTake is what the asker does with a warrant for a take (asker.Decided): the act only when it is the one the
// option bound — composed again from the record — and the library still waits on the filesystem the operator was
// asked about; then the engine is asked.
func (a *asker) decideTake(ctx context.Context, r asked, act conditions.Action, w asks.Warrant) (string, error) {
t := *r.Take
recomposed := t.actions(r.ID)
option, offered := r.Ask.Option(w.Option)
if !offered || len(recomposed) != 1 {
return "", fmt.Errorf("the take %s offers no answer %q: nothing is taken", r.ID, w.Option)
}
if err := option.Performs(boundAct(recomposed[0])); err != nil {
return "", fmt.Errorf("the take kept for %s is not the one the operator was shown: %v", r.ID, err)
}
if err := option.Performs(boundAct(act)); err != nil {
return "", fmt.Errorf("the act chosen for %s is not the one the operator was shown: %v", r.ID, err)
}
if a.take == nil {
return "", errors.New("this controller cannot take a library on a warrant")
}
// **The library must still wait where the operator was asked about it** (the review of e1c2b744): a take is about
// no condition, so nothing else rechecks it. A library that moved to another home or filesystem, or no longer
// waits, is not taken: the ask is spent, and one for where it waits now is asked.
if a.folders != nil {
waiting, err := a.folders(ctx)
if err != nil {
return "", fmt.Errorf("whether the library %s still waits could not be read, so nothing is taken: %w",
t.Library, err)
}
still := false
for _, now := range waiting {
still = still || now.key() == t.key()
}
if !still {
return "nothing: the library no longer waits to be taken where the operator was asked about it (it " +
"moved, or was taken meanwhile); asked again where it waits now", nil
}
}
by := fmt.Sprintf("approved by %s via %s at %s (ask %s)", byWords(w), viaWords(w),
w.At.Local().Format("2006-01-02 15:04"), r.ID)
return a.take(ctx, t, r.ID, by)
}
// takeFolderIn is takeOnWarrant on this controller's stores and bus: the home's engine is asked with the mesh's key,
// naming the library and the ask the warrant answered; on its word that it started, the library is recorded taken. A
// refusal records nothing.
func takeFolderIn(open *stores, ask func(ctx context.Context, act link.AccessActAsk) (link.HandOverAnswer, error)) takeOnWarrant {
return func(ctx context.Context, t libraryTake, askID, by string) (string, error) {
// Who asked, as one short line the engine takes (at most link.MaxHandOverBy): the ask names the rest.
short := by
if len(short) > 180 {
short = "the operator, on the warrant for ask " + askID
}
answer, err := ask(ctx, link.AccessActAsk{Node: t.Node, Path: t.Root, By: short,
Access: libraryAccess + t.Library, Warrant: askID})
if err != nil {
return "", err
}
if answer.Refused != "" {
return "", fmt.Errorf("%s refused to take the library %s: %s", t.Node, t.Library, answer.Refused)
}
// Started, not taken: the library is taken when the engine states the act done (stateHealth, TakeDone).
if err := open.inventory.RecordTakeStarted(ctx, inventory.TakenLibrary{Library: t.Library, Home: t.Node,
Root: t.Root, Placed: t.Placed, By: by, Ask: askID, Warrant: askID, Said: answer.Said}); err != nil {
return "", fmt.Errorf("%s started the take of %s (%s), and it could NOT be recorded: %w", t.Node, t.Library,
answer.Said, err)
}
return fmt.Sprintf("%s started taking the library %s: %s; it is taken when the engine says the act is done",
t.Node, t.Library, answer.Said), nil
}
}
// waitingFolder is one library wait excused, as foldersWaiting gathers them.
type waitingFolder struct {
module, machine string
// placed is the filesystem the library's root is on, as its home's engine names it without a path; empty from
// another machine.
placed string
}
// foldersWaiting is every library waiting to be taken across the mesh (ADR 0328 §7, §8): from each machine's newest
// statement, each wait of reason keep that is excused (checkWaiting), by library.
func foldersWaiting(ctx context.Context, inv *inventory.Inventory, healths map[string]inventory.NodeHealth) (
map[string][]waitingFolder, operatorWaitFacts) {
var f operatorWaitFacts
out := map[string][]waitingFolder{}
machines := make([]string, 0, len(healths))
for m := range healths {
machines = append(machines, m)
}
sort.Strings(machines)
for _, machine := range machines {
rs := healths[machine].Resources
var modules []string
for _, r := range rs {
for _, w := range r.Waits {
if r.State == link.StateWaiting && w.Access != "" && !w.Disk {
modules = append(modules, r.Module)
}
}
}
if len(modules) == 0 {
continue
}
readWaitFacts(ctx, inv, machine, modules, nil, &f)
for _, r := range checkWaiting(machine, rs, f) {
if r.State != link.StateWaiting {
continue
}
for _, w := range r.Waits {
if w.Access == "" || w.Disk || w.Reason == link.WaitDisk {
continue
}
lib, isLibrary := strings.CutPrefix(w.Access, libraryAccess)
if !isLibrary {
lib = catalogue.LibraryOfEntry(f.manifests[r.Module], w.Access)
}
if lib == "" {
continue
}
placed := ""
if l := f.libraries[lib]; l.Home == machine {
placed = w.Placed
}
out[lib] = append(out[lib], waitingFolder{module: r.Module, machine: machine, placed: placed})
}
}
}
return out, f
}
// foldersIn is the asker's folders on this controller's stores: every library waiting to be taken, with its home, its
// root and the modules waiting on it.
func foldersIn(open *stores) func(ctx context.Context) ([]libraryTake, error) {
return func(ctx context.Context) ([]libraryTake, error) {
healths, err := open.inventory.Healths(ctx)
if err != nil {
return nil, err
}
waiting, f := foldersWaiting(ctx, open.inventory, healths)
return withoutTakesUnderWay(takesOf(waiting, f.libraries), f.taken), nil
}
}
// withoutTakesUnderWay is the takes not already under way: a take started on the same home, root and filesystem, which
// the engine has not yet stated done or failed, is not asked about again (the review of 7ea6d36e, item 5).
func withoutTakesUnderWay(takes []libraryTake, recorded map[string]inventory.TakenLibrary) []libraryTake {
var out []libraryTake
for _, t := range takes {
r, was := recorded[t.Library]
if was && !r.Done && r.Home == t.Node && r.Root == t.Root && (t.Placed == "" || r.Placed == "" || r.Placed == t.Placed) {
continue
}
out = append(out, t)
}
return out
}
// takesOf is one take per library and filesystem from the waits gathered, sorted. A library no longer defined is
// asked about nowhere.
func takesOf(waiting map[string][]waitingFolder, libraries map[string]catalogue.Library) []libraryTake {
var out []libraryTake
for lib, ws := range waiting {
l, defined := libraries[lib]
if !defined {
continue
}
// The home's word on the filesystem stands; waits from other machines name none.
placed := map[string]bool{}
for _, w := range ws {
if w.placed != "" {
placed[w.placed] = true
}
}
if len(placed) == 0 {
placed[""] = true
}
for p := range placed {
t := libraryTake{Library: lib, Node: l.Home, Root: l.Root, Placed: p}
seen := map[string]bool{}
for _, w := range ws {
if w.module != "" && !seen[w.module] {
seen[w.module] = true
t.Modules = append(t.Modules, w.module)
}
}
sort.Strings(t.Modules)
out = append(out, t)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].key() < out[j].key() })
return out
}
-585
View File
@@ -1,585 +0,0 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq ADR 0328 §7, §8 and §9 on the controller's side: a library's wait is excused only for a defined library
// that reaches the machine and is not yet taken, or whose disk is not there; the gate passes it as a wait for a person
// and the operator reads it as needs-operator in words with no path; the operator is asked once per library, never
// about one taken; and the library is taken only on the warrant, where it still waits.
var radarrManifest = catalogue.Manifest{Module: "radarr", Version: "1",
Libraries: []catalogue.LibraryUse{{ID: "movies", Library: "media", Path: "movies", Use: catalogue.AccessReadWrite}}}
var plexManifest = catalogue.Manifest{Module: "plex", Version: "1",
Libraries: []catalogue.LibraryUse{{ID: "films", Library: "media", Path: "movies", Use: catalogue.AccessRead}}}
var mediaLib = catalogue.Library{Name: "media", Home: "ace", Root: "/storage/media", Reach: []string{"novox"}}
func keepWait(access string) inventory.Wait {
return inventory.Wait{Part: "radarr", Access: access, Reason: link.WaitKeep,
What: "let the mesh keep the library media"}
}
func placedWait(access, placed string) inventory.Wait {
w := keepWait(access)
w.Placed = placed
return w
}
func diskWait(access string) inventory.Wait {
return inventory.Wait{Part: "radarr", Access: access, Disk: true, Reason: link.WaitDisk,
What: "mount the disk that holds media"}
}
// libraryFacts is the facts with media defined, and each library given taken on ace on fs-000000000001.
func libraryFactsWith(taken ...string) operatorWaitFacts {
t := map[string]inventory.TakenLibrary{}
for _, l := range taken {
t[l] = inventory.TakenLibrary{Library: l, Home: "ace", Root: "/storage/media", Placed: "fs-000000000001", Done: true}
}
return operatorWaitFacts{
manifests: map[string]catalogue.Manifest{"radarr": radarrManifest, "plex": plexManifest},
given: map[string]map[string]time.Time{"radarr@ace": {}, "plex@ace": {}},
libraries: map[string]catalogue.Library{"media": mediaLib},
taken: t,
librariesRead: true,
}
}
// Rows 8 and 9: excused for a defined library reaching the machine and not yet taken — by the module's entry or the
// library's own wait — and for one whose disk is not there, taken or not; judged unhealthy when undefined, not
// reaching the machine, already taken on the filesystem it is on, for an access that is not a library, or for a
// reason the controller does not know.
func TestALibraryWaitIsExcusedOnlyForALibraryNotYetTaken(t *testing.T) {
for _, c := range []struct {
name string
machine string
w inventory.Wait
f operatorWaitFacts
says string
}{
{"the module's entry, not taken", "ace", keepWait("movies"), libraryFactsWith(), ""},
{"the library's own wait, not taken", "ace", keepWait("library:media"), libraryFactsWith(), ""},
{"on a machine it reaches", "novox", keepWait("movies"), libraryFactsWith(), ""},
{"its disk not there", "ace", diskWait("movies"), libraryFactsWith(), ""},
{"its disk not there, taken before", "ace", diskWait("movies"), libraryFactsWith("media"), ""},
{"taken already", "ace", keepWait("movies"), libraryFactsWith("media"), "taken already"},
{"taken on the filesystem it is on", "ace", placedWait("library:media", "fs-000000000001"),
libraryFactsWith("media"), "taken already"},
{"taken on another filesystem: it moved", "ace", placedWait("library:media", "fs-0000000000ff"),
libraryFactsWith("media"), ""},
{"taken at another root: its root changed", "ace", keepWait("movies"), func() operatorWaitFacts {
f := libraryFactsWith("media")
f.taken["media"] = inventory.TakenLibrary{Library: "media", Home: "ace", Root: "/storage/old-media", Done: true,
Placed: "fs-000000000001"}
return f
}(), ""},
{"a machine it does not reach", "far", keepWait("movies"), libraryFactsWith(), "does not reach far"},
{"undefined", "ace", keepWait("library:books"), libraryFactsWith(), "not defined"},
{"not a library", "ace", keepWait("downloads"), libraryFactsWith(), "only a library waits"},
{"a reason not known", "ace", inventory.Wait{Part: "radarr", Access: "movies", Reason: "later", What: "w"},
libraryFactsWith(), "take it or for its disk"},
{"libraries unread", "ace", keepWait("movies"), operatorWaitFacts{manifests: libraryFactsWith().manifests},
"could not be read"},
} {
err := checkWait("radarr", c.machine, c.w, c.f)
switch {
case c.says == "" && err != nil:
t.Errorf("%s: refused: %v", c.name, err)
case c.says != "" && (err == nil || !strings.Contains(err.Error(), c.says)):
t.Errorf("%s: %v; want a refusal saying %q", c.name, err, c.says)
}
}
}
func libraryWaiting(module string, w inventory.Wait) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: module, Resource: module + ".app", Kind: "container", Target: module,
State: link.StateWaiting, Waits: []inventory.Wait{w}}
}
// Rows 8 and 9 at the gate: both excused waits pass the first-node gate as a wait for a person, and the module is said
// as needs-operator in words with no path; a wait on a library taken already does not pass.
func TestALibraryWaitPassesTheGateAndAsksTheOperatorWithoutAPath(t *testing.T) {
now := time.Now()
for _, w := range []inventory.Wait{keepWait("movies"), diskWait("movies")} {
w.What += " at /storage/media" // an engine's words carrying a path never reach the operator's
f := gateFacts{now: now, waits: libraryFactsWith(), groupsAdded: map[string]bool{"radarr": false},
health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now,
Resources: []inventory.ResourceHealth{libraryWaiting("radarr", w)}}}}
h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f)
if h != healthPerson || !strings.Contains(why, "waits for the operator") || strings.Contains(why, "/") {
t.Fatalf("an excused library wait reads %v %q; want a wait for a person, with no path", h, why)
}
o := needsOperatorObservation("radarr", "ace", []inventory.Wait{w}, nil)
for _, words := range []string{o.Summary, o.Headline, o.Explanation, o.Needs} {
if strings.Contains(words, "/") {
t.Errorf("the operator reads a path: %q", words)
}
}
want := "let the mesh keep the library entry movies when you are asked"
if w.Disk {
want = "mount the disk that holds the library entry movies on ace"
}
if o.Kind != kindNeedsOperator || !strings.Contains(o.Needs, want) {
t.Errorf("needs-operator %s asks %q; want %q", o.Kind, o.Needs, want)
}
}
f := gateFacts{now: now, waits: libraryFactsWith("media"), groupsAdded: map[string]bool{"radarr": false},
health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now,
Resources: []inventory.ResourceHealth{libraryWaiting("radarr", keepWait("movies"))}}}}
if h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthNotYet || !strings.Contains(why, "taken") {
t.Fatalf("a wait on a taken library reads %v %q", h, why)
}
}
func takeRig(t *testing.T, waiting func() map[string][]waitingFolder) (*askerRig, *[]string) {
r := newAskerRig(t)
var taken []string
libs := map[string]catalogue.Library{"media": mediaLib}
r.a.folders = func(context.Context) ([]libraryTake, error) { return takesOf(waiting(), libs), nil }
r.a.machines = func(context.Context) ([]string, error) { return []string{"ace", "novox"}, nil }
r.a.take = func(_ context.Context, lt libraryTake, askID, by string) (string, error) {
taken = append(taken, lt.Library+" "+lt.Node+" "+lt.Root+" "+lt.Placed+" "+askID+" "+by)
return "taking", nil
}
return r, &taken
}
func takeAsks(r *askerRig) []asked {
var out []asked
for _, a := range r.store {
if a.Take != nil {
out = append(out, a)
}
}
return out
}
func openTakes(r *askerRig) []asked {
var out []asked
for _, a := range takeAsks(r) {
if a.State == askOpen {
out = append(out, a)
}
}
return out
}
// Row 7: one ask per library however many modules and machines wait on it, in the clause's words, at approve; none
// again while it is open; taken back when the library no longer waits.
func TestTheOperatorIsAskedOncePerLibrary(t *testing.T) {
current := map[string][]waitingFolder{"media": {
{module: "radarr", machine: "ace", placed: "fs-000000000001"},
{module: "plex", machine: "novox"},
{module: "", machine: "ace", placed: "fs-000000000001"}}}
r, _ := takeRig(t, func() map[string][]waitingFolder { return current })
ctx := context.Background()
for i := 0; i < 3; i++ {
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
}
kept := takeAsks(r)
if len(kept) != 1 {
t.Fatalf("%d asks for one library", len(kept))
}
q := kept[0].Ask
if err := q.Check(r.now); err != nil {
t.Fatal(err)
}
if q.Highest() != asks.Approve || len(q.Options) != 1 {
t.Fatalf("the ask offers %+v; want one answer at approve", q.Options)
}
// The root may not leave the mesh (the router's content rule): it is said as "its root".
want := "Let the mesh keep library media at its root on ace: its group becomes mesh-media, the group may read " +
"and write, folders keep the group; owners and everyone else's rights stay as they are. Waiting on it: plex, radarr."
if q.Explanation != want {
t.Errorf("the ask says\n %q\nwant\n %q", q.Explanation, want)
}
if kept[0].Take.Node != "ace" || kept[0].Take.Placed != "fs-000000000001" {
t.Errorf("the take is for %+v; want the home and the filesystem its home names", kept[0].Take)
}
current = nil
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
if got := takeAsks(r); len(got) != 1 || got[0].State != askCancelled {
t.Fatalf("the ask about a library that no longer waits: %+v", got)
}
}
// Row 7: nothing is taken without the operator's warrant — a warrant for another asker, an ask that expired, or a
// record changed after the ask takes nothing; the warrant takes it once, on its home, naming library:<name>.
func TestALibraryIsTakenOnlyOnTheWarrant(t *testing.T) {
waits := func() map[string][]waitingFolder {
return map[string][]waitingFolder{"media": {{module: "radarr", machine: "ace", placed: "fs-000000000001"}}}
}
ctx := context.Background()
r, taken := takeRig(t, waits)
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
a := takeAsks(r)[0]
other := warrantOn(a, "let-the-mesh-keep-it", r.now)
other.Asker = "someone-else"
answerWith(t, r, other)
changed := r.store[a.ID]
changed.Take = &libraryTake{Library: "media", Node: "ace", Root: "/", Modules: a.Take.Modules, Placed: a.Take.Placed}
r.store[a.ID] = changed
answerWith(t, r, warrantOn(a, "let-the-mesh-keep-it", r.now))
if len(*taken) != 0 || !strings.HasPrefix(r.store[a.ID].Acted, "failed") {
t.Fatalf("a take whose record changed after the ask: %v, %q", *taken, r.store[a.ID].Acted)
}
r2, taken2 := takeRig(t, waits)
if err := r2.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
b := takeAsks(r2)[0]
answerWith(t, r2, asks.Warrant{Ask: b.ID, Asker: "mesh-controller", Outcome: asks.OutcomeExpired, At: r2.now})
if len(*taken2) != 0 {
t.Fatal("taken on an ask that expired")
}
r3, taken3 := takeRig(t, waits)
if err := r3.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
c := takeAsks(r3)[0]
w := warrantOn(c, "let-the-mesh-keep-it", r3.now)
answerWith(t, r3, w)
answerWith(t, r3, w)
if len(*taken3) != 1 || !strings.HasPrefix((*taken3)[0], "media ace /storage/media fs-000000000001 "+c.ID) ||
!strings.Contains((*taken3)[0], "telegram") {
t.Fatalf("the warrant took %v", *taken3)
}
if !strings.HasPrefix(r3.store[c.ID].Acted, "done") {
t.Fatalf("the take was recorded %q", r3.store[c.ID].Acted)
}
}
// A warrant for a library on one filesystem is never carried out on another: when it comes and the library waits on
// another, nothing is sent, the ask is spent, and the operator is asked again for the filesystem it is on now.
func TestAWarrantForOneFilesystemIsNotCarriedOutOnAnother(t *testing.T) {
placed := "fs-000000000001"
r, taken := takeRig(t, func() map[string][]waitingFolder {
return map[string][]waitingFolder{"media": {{module: "radarr", machine: "ace", placed: placed}}}
})
ctx := context.Background()
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
a := takeAsks(r)[0]
placed = "fs-0000000000ff"
answerWith(t, r, warrantOn(a, "let-the-mesh-keep-it", r.now))
if len(*taken) != 0 {
t.Fatalf("a warrant for %s was carried out on %s: %v", a.Take.Placed, placed, *taken)
}
if spent := r.store[a.ID]; spent.State == askOpen || !strings.HasPrefix(spent.Acted, "nothing") {
t.Fatalf("the ask after the library moved: %s, %q", spent.State, spent.Acted)
}
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
if again := openTakes(r); len(again) != 1 || again[0].Take.Placed != placed {
t.Fatalf("not asked again for the filesystem it is on now: %+v", again)
}
}
// The take act names library:<name> and the warrant, is signed over its own context, and is recorded only on the
// engine's word.
func TestATakeAskNamesTheLibraryAndItsWarrant(t *testing.T) {
_, key, _ := ed25519.GenerateKey(nil)
signer := keySigner{key}
ask := link.AccessActAsk{Node: "ace", Path: "/storage/media", By: "the operator", Access: "library:media"}
if _, err := link.SignTake(context.Background(), signer, ask); err == nil {
t.Fatal("a take with no warrant was signed")
}
ask.Warrant = "c0123456789abcdef"
body, err := link.SignTake(context.Background(), signer, ask)
if err != nil {
t.Fatal(err)
}
var signed link.SignedHandOver
if err := json.Unmarshal(body, &signed); err != nil {
t.Fatal(err)
}
if !ed25519.Verify(key.Public().(ed25519.PublicKey), append([]byte(link.AccessActContext), signed.Ask...), signed.Signature) {
t.Fatal("the take is not signed over its own context")
}
var got map[string]any
if err := json.Unmarshal(signed.Ask, &got); err != nil {
t.Fatal(err)
}
if got["act"] != "take" || got["warrant"] != "c0123456789abcdef" || got["access"] != "library:media" ||
got["path"] != "/storage/media" {
t.Fatalf("the ask carries %v", got)
}
var asked link.AccessActAsk
refusing := takeFolderIn(nil, func(_ context.Context, a link.AccessActAsk) (link.HandOverAnswer, error) {
asked = a
return link.HandOverAnswer{Refused: "the disk is not there"}, nil
})
if _, err := refusing(context.Background(), libraryTake{Library: "media", Node: "ace", Root: "/storage/media"},
"c01", "the operator"); err == nil {
t.Fatal("a refused take was answered as done")
}
if asked.Access != "library:media" || asked.Node != "ace" || asked.Path != "/storage/media" || asked.Warrant != "c01" {
t.Fatalf("the engine was asked %+v", asked)
}
unreachable := takeFolderIn(nil, func(context.Context, link.AccessActAsk) (link.HandOverAnswer, error) {
return link.HandOverAnswer{}, errors.New("nothing on ace answers")
})
if _, err := unreachable(context.Background(), libraryTake{Library: "media", Node: "ace", Root: "/storage/media"},
"c01", "the operator"); err == nil {
t.Fatal("an unanswered take was answered as done")
}
}
type keySigner struct{ key ed25519.PrivateKey }
func (s keySigner) Sign(_ context.Context, b []byte) ([]byte, error) {
return ed25519.Sign(s.key, b), nil
}
// The envelope carries groups, identities and libraries only when composed — for an engine at link.LibraryContract —
// and an older engine's body is what it was.
func TestTheDeclarationSaysItsLibrariesOnlyWhenComposed(t *testing.T) {
older, err := sendable{Resources: []map[string]any{{"id": "a"}}}.Body()
if err != nil {
t.Fatal(err)
}
for _, key := range []string{"groups", "identities", "libraries"} {
if strings.Contains(string(older), key) {
t.Fatalf("an older engine's body names %s: %s", key, older)
}
}
newer, err := sendable{Resources: []map[string]any{{"id": "a"}},
Groups: []catalogue.GroupEntry{{Name: "mesh-media", GID: 50010}},
Identities: []catalogue.IdentityEntry{{Module: "radarr", Name: "mesh-radarr", UID: 50020, GID: 50010}},
Libraries: []catalogue.LibraryEntry{{Library: "media", Root: "/storage/media", Group: "mesh-media", GID: 50010,
Home: true, Paths: []string{"movies"}, Runner: "jochen"}}}.Body()
if err != nil {
t.Fatal(err)
}
for _, want := range []string{`"groups":[{"name":"mesh-media","gid":50010}]`,
`"identities":[{"module":"radarr","name":"mesh-radarr","uid":50020,"gid":50010}]`,
`"libraries":[{"library":"media","root":"/storage/media","group":"mesh-media","gid":50010,"home":true,` +
`"taken":false,"paths":["movies"],"runner":"jochen"}]`} {
if !strings.Contains(string(newer), want) {
t.Errorf("the body lacks %s: %s", want, newer)
}
}
if link.LibraryContract != 6 {
t.Fatalf("the contract the engine states for ADR 0328 is 6, and this controller reads %d", link.LibraryContract)
}
}
// The gate: an access or library the engine cannot judge this look, or entries left while a disk was away, of a module
// whose move leaves its accesses and libraries as they were, are not counted; of one whose move changes them — a
// library added, or joining — or is not known, they hold the gate to its bound.
func TestAnUnknownLibraryIsJudgedByWhetherTheMoveChangesIt(t *testing.T) {
now := time.Now()
since := now.Add(-time.Minute)
unknown := inventory.ResourceHealth{Module: "radarr", Resource: "radarr.library-movies", Kind: link.KindAccess,
Target: "/storage/media/movies", State: link.StateUnknown, Since: now}
left := inventory.ResourceHealth{Module: "radarr", Resource: "radarr.access-gone~entries", Kind: link.KindKeptEntries,
Target: "/storage/media", State: link.StateUnhealthy, Since: since.Add(-time.Hour)}
facts := func(changes map[string]bool, rs ...inventory.ResourceHealth) gateFacts {
return gateFacts{now: now, waits: libraryFactsWith(), groupsAdded: map[string]bool{"radarr": false},
accessesChanged: changes, health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now,
Resources: rs}}}
}
unchanged := map[string]bool{"radarr": false}
if h, why := moduleHealthWord("radarr", "ace", since, facts(unchanged, unknown)); h != healthGood {
t.Errorf("an unknown library of a move that changes nothing reads %v %q", h, why)
}
if h, why := moduleHealthWord("radarr", "ace", since, facts(unchanged, left)); h != healthPerson ||
!strings.Contains(why, "access-gone~entries") {
t.Errorf("entries left, a move that changes nothing: %v %q", h, why)
}
for name, changes := range map[string]map[string]bool{"changes them": {"radarr": true}, "not known": nil} {
for _, r := range []inventory.ResourceHealth{unknown, left} {
if h, _ := moduleHealthWord("radarr", "ace", since, facts(changes, r)); h != healthNotYet {
t.Errorf("%s %s of a move that %s reads %v; want held to the bound", r.Kind, r.State, name, h)
}
}
}
plain := catalogue.Manifest{Module: "radarr"}
if sameAccessUse(plain, radarrManifest) || !sameAccessUse(radarrManifest, radarrManifest) {
t.Fatal("a move that adds a library, and joins, reads unchanged — or an unchanged one reads changed")
}
}
// The engine states a library once, of no module; each module assigned to the machine that uses it is judged and said
// by a copy of that verdict — its wait excused and passed by the gate as a wait for a person. A library unknown or
// starting (a take walking under it) of a move that changes no library is not counted.
func TestALibrarysVerdictIsSaidForTheModulesThatUseIt(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "ace"); err != nil {
t.Fatal(err)
}
for _, m := range []catalogue.Manifest{radarrManifest, plexManifest, {Module: "plain", Version: "1"}} {
if err := inv.RegisterModule(ctx, m, inventory.Source{}); err != nil {
t.Fatal(err)
}
if m.Module == "plex" {
continue // pulled in by a requirement, not assigned: it is in what the machine was sent (item 4)
}
if _, err := inv.Assign(ctx, "ace", m.Module); err != nil {
t.Fatal(err)
}
}
record, err := inv.NodeByName(ctx, "ace")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": "c1", "plex": "c2", "plain": "c3"},
1, 1, inventory.Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
lib := inventory.ResourceHealth{Resource: "library:media", Kind: link.KindLibrary, Target: "/storage/media",
State: link.StateWaiting, Waits: []inventory.Wait{{Part: "media", Access: "library:media", Reason: link.WaitKeep,
Placed: "fs-000000000001", What: "let the mesh keep the library media"}}}
copies := libraryVerdictsFor(ctx, inv, "ace", []inventory.ResourceHealth{lib})
var modules []string
for _, c := range copies {
modules = append(modules, c.Module)
}
if strings.Join(modules, ",") != "plex,radarr" {
t.Fatalf("the library's verdict is said for %v; want plex and radarr, the modules using it", modules)
}
paths, err := inv.LibraryPaths(ctx)
if err != nil {
t.Fatal(err)
}
if strings.Join(paths["media"], ",") != "movies" {
t.Fatalf("the paths kept of media are %v", paths["media"])
}
now := time.Now()
f := gateFacts{now: now, waits: libraryFactsWith(), groupsAdded: map[string]bool{"radarr": false},
accessesChanged: map[string]bool{"radarr": false},
health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now, Resources: copies}}}
if h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthPerson {
t.Fatalf("radarr waiting on media reads %v %q; want a wait for a person", h, why)
}
for _, state := range []string{link.StateUnknown, link.StateStarting} {
c := copies[1]
c.State, c.Waits = state, nil
f.health["ace"] = inventory.NodeHealth{Node: "ace", HeardAt: now, Resources: []inventory.ResourceHealth{c}}
if h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthGood {
t.Errorf("media %s, a move that changes no library: %v %q", state, h, why)
}
f.accessesChanged = map[string]bool{"radarr": true}
if h, _ := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthNotYet {
t.Errorf("media %s, a move that changes a library: %v; want held to the bound", state, h)
}
f.accessesChanged = map[string]bool{"radarr": false}
}
}
// The review of 7ea6d36e, item 5: the engine saying it started a take does not make the library taken. While the act
// runs the library still waits and the operator is not asked again; it is taken when the engine states the act done
// under that warrant.
func TestALibraryIsTakenOnlyWhenTheEngineSaysTheActIsDone(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"})
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
take := takeFolderIn(open, func(context.Context, link.AccessActAsk) (link.HandOverAnswer, error) {
return link.HandOverAnswer{Said: "the take of library:media started"}, nil
})
lt := libraryTake{Library: "media", Node: "anchor", Root: "/storage/media", Placed: "fs-000000000001"}
if _, err := take(ctx, lt, "c01", "the operator"); err != nil {
t.Fatal(err)
}
libs, err := inv.Libraries(ctx)
if err != nil {
t.Fatal(err)
}
taken, err := librariesTaken(ctx, inv, libs)
if err != nil {
t.Fatal(err)
}
if taken["media"] {
t.Fatal("a take only started made the library taken")
}
// While it runs: the wait is still excused, and no new ask is made for it.
var f operatorWaitFacts
readWaitFacts(ctx, inv, "anchor", nil, nil, &f)
if err := checkWait("radarr", "anchor", placedWait("library:media", "fs-000000000001"), f); err != nil {
t.Fatalf("the wait while the take runs: %v", err)
}
if got := withoutTakesUnderWay([]libraryTake{lt}, f.taken); len(got) != 0 {
t.Fatalf("a library whose take runs is asked about again: %+v", got)
}
// The engine states the act done under that warrant.
act, _ := json.Marshal(map[string]any{"act": "take", "access": "library:media", "warrant": "c01", "state": "done"})
if err := stateHealth(ctx, inv, nil, "anchor", link.Health{Contract: link.LibraryContract, At: time.Now(),
Resources: []link.ResourceHealth{{Resource: "library:media", Kind: link.KindLibrary, Target: "/storage/media",
State: link.StateHealthy, Since: time.Now(), Act: act}}}, time.Now()); err != nil {
t.Fatal(err)
}
if taken, err = librariesTaken(ctx, inv, libs); err != nil || !taken["media"] {
t.Fatalf("the act done did not make the library taken (%v)", err)
}
}
// ADR 0334: the engine states the tool runner's membership as resources of kind runner — `runner:<module>` of that
// module, `runner:library:<name>` of no module. A library's is said for the modules that use it; neither fails the
// first-node gate of a move that leaves the module's libraries as they were, and both hold one that changes them.
func TestARunnersVerdictIsSaidAndJudgedByWhetherTheMoveChangesIt(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "ace"); err != nil {
t.Fatal(err)
}
for _, m := range []catalogue.Manifest{radarrManifest, plexManifest} {
if err := inv.RegisterModule(ctx, m, inventory.Source{}); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "ace", m.Module); err != nil {
t.Fatal(err)
}
}
lib := inventory.ResourceHealth{Resource: "runner:library:media", Kind: link.KindRunner, Target: "jochen",
State: link.StateUnhealthy, Reason: "the tool runner is not in mesh-media yet", Since: time.Now()}
copies := libraryVerdictsFor(ctx, inv, "ace", []inventory.ResourceHealth{lib})
if len(copies) != 2 || copies[0].Module != "plex" || copies[1].Module != "radarr" {
t.Fatalf("the runner's library verdict is said for %+v; want plex and radarr", copies)
}
own := inventory.ResourceHealth{Module: "radarr", Resource: "runner:radarr", Kind: link.KindRunner, Target: "jochen",
State: link.StateUnhealthy, Reason: "the tool runner is not in mesh-radarr yet", Since: time.Now()}
now := time.Now()
for _, r := range []inventory.ResourceHealth{copies[1], own} {
f := gateFacts{now: now, waits: libraryFactsWith(), groupsAdded: map[string]bool{"radarr": false},
accessesChanged: map[string]bool{"radarr": false},
health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now, Resources: []inventory.ResourceHealth{r}}}}
if h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthGood {
t.Errorf("%s, a move that changes no library: %v %q; want not counted", r.Resource, h, why)
}
f.accessesChanged = map[string]bool{"radarr": true}
if h, _ := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthNotYet {
t.Errorf("%s, a move that changes a library: %v; want held to the bound", r.Resource, h)
}
}
}
+135
View File
@@ -0,0 +1,135 @@
package main
import (
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// machineLines is what a machine said it is made of (novox/hq issue 160), as `node` shows it: every fact in
// one block with its age, the architecture and kernel the profile always carried among them, and each fact
// the machine could not determine said as unknown with its why — never as a zero, and never left out.
func machineLines(a inventory.MachineAccount, had bool, now time.Time) []string {
if !had {
return []string{" what it is made of: never said — this machine has not reported a profile"}
}
m := a.Machine
if m == nil {
return []string{
" what it is made of: not said — its node-engine is older than the one that says it",
" " + factLine("architecture", orUnsaid(a.Architecture)),
" " + factLine("kernel", orUnsaid(a.Kernel)),
}
}
age := roughly(now.Sub(m.ObservedAt)) + " ago"
if m.ObservedAt.IsZero() {
// Said with no time: its age is not known, and is never read as two thousand years.
age = "at a time it did not say (age unknown)"
}
out := []string{fmt.Sprintf(" what it is made of, as it said %s:", age)}
add := func(name, value string) { out = append(out, " "+factLine(name, value)) }
unknown := map[string]string{}
for k, v := range m.Unknown {
unknown[k] = v
}
// said is the value, or the fact's own reason it is unknown; a fact neither said nor unknown is a
// node-engine that broke its own rule, and is said so rather than shown blank.
said := func(fact, value string) string {
if value != "" {
delete(unknown, fact)
return value
}
if why, ok := unknown[fact]; ok {
delete(unknown, fact)
return "unknown: " + why
}
return "unknown: the machine said neither it nor why"
}
add("architecture", orUnsaid(a.Architecture))
release := said("kernel release", m.KernelRelease)
if m.KernelRelease == "" {
release = "(release " + release + ")"
}
add("kernel", orUnsaid(a.Kernel)+" "+release)
dist := said("distribution", m.Distribution)
if m.Distribution != "" {
dist += " " + said("version", m.Version)
}
add("distribution", dist)
add("processor", said("processor", m.Processor))
add("cores", said("cores", countWords(m.Cores)))
add("threads", said("threads", countWords(m.Threads)))
add("memory", said("memory", bytesWords(m.MemoryBytes)))
add("virtual", said("virtual or physical", m.Virtual))
up := ""
if m.BootedAt != nil {
up = fmt.Sprintf("%s, since %s", roughly(now.Sub(*m.BootedAt)), m.BootedAt.Local().Format("2006-01-02 15:04"))
}
add("up", said("uptime", up))
add("timezone", said("timezone", m.Timezone))
if len(m.Filesystems) == 0 {
add("filesystems", said("filesystems", ""))
} else {
for i, f := range m.Filesystems {
name := ""
if i == 0 {
name = "filesystems"
}
add(name, fmt.Sprintf("%s %s, %s free (%s %s)", f.Mount, bytesWords(f.SizeBytes),
bytesWords(f.FreeBytes), f.Type, f.Device))
}
if why, ok := unknown["filesystems"]; ok {
add("", "and some unknown: "+why)
delete(unknown, "filesystems")
}
}
add("public address", said("public address", ""))
// Any other fact a newer node-engine named unknown, so nothing it said is dropped.
rest := make([]string, 0, len(unknown))
for k := range unknown {
rest = append(rest, k)
}
sort.Strings(rest)
for _, k := range rest {
add(k, "unknown: "+unknown[k])
}
return out
}
func factLine(name, value string) string { return fmt.Sprintf("%-15s %s", name, value) }
func orUnsaid(s string) string {
if strings.TrimSpace(s) == "" {
return "not said"
}
return s
}
// countWords is a number a machine said, or nothing when it said none (its unknown says why).
func countWords(n int) string {
if n <= 0 {
return ""
}
return fmt.Sprint(n)
}
// bytesWords is an amount of bytes a person reads, in binary units; nothing when none was said.
func bytesWords(n int64) string {
if n <= 0 {
return ""
}
units := []string{"B", "KiB", "MiB", "GiB", "TiB", "PiB"}
v, u := float64(n), 0
for v >= 1024 && u < len(units)-1 {
v /= 1024
u++
}
if u == 0 {
return fmt.Sprintf("%d B", n)
}
return fmt.Sprintf("%.1f %s", v, units[u])
}
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// Defends novox/hq issue 160: `node` shows every fact a machine said it is made of, in one block with its
// age, the architecture and kernel among them; and a fact it could not determine as unknown with why.
func TestNodeShowsWhatAMachineIsMadeOfWithItsAge(t *testing.T) {
now := time.Date(2026, 10, 12, 15, 0, 0, 0, time.UTC)
booted := now.Add(-75 * time.Hour)
a := inventory.MachineAccount{Architecture: "amd64", Kernel: "linux", Machine: &inventory.MadeOf{
ObservedAt: now.Add(-3 * time.Hour), KernelRelease: "7.2.8-arch1-2", Distribution: "Arch Linux",
Version: "rolling", Processor: "AMD Ryzen 9 8945HS", Cores: 8, Threads: 16, MemoryBytes: 32891494400,
Virtual: "physical", BootedAt: &booted, Timezone: "Europe/Brussels",
Filesystems: []inventory.MountedFilesystem{{Mount: "/", Device: "/dev/nvme0n1p3", Type: "ext4",
SizeBytes: 987978522624, FreeBytes: 31830110208}},
Unknown: map[string]string{"public address": "how it learns it is not decided yet"},
}}
got := strings.Join(machineLines(a, true, now), "\n")
t.Log("\n" + got)
for _, want := range []string{
"what it is made of, as it said 3h ago:",
"architecture amd64",
"kernel linux 7.2.8-arch1-2",
"distribution Arch Linux rolling",
"processor AMD Ryzen 9 8945HS",
"cores 8",
"threads 16",
"memory 30.6 GiB",
"virtual physical",
"up 3d, since",
"timezone Europe/Brussels",
"filesystems / 920.1 GiB, 29.6 GiB free (ext4 /dev/nvme0n1p3)",
"public address unknown: how it learns it is not decided yet",
} {
if !strings.Contains(got, want) {
t.Errorf("missing %q", want)
}
}
}
func TestAFactAMachineCouldNotReadIsShownUnknownNeverZero(t *testing.T) {
now := time.Date(2026, 10, 12, 15, 0, 0, 0, time.UTC)
a := inventory.MachineAccount{Architecture: "arm64", Kernel: "linux", Machine: &inventory.MadeOf{
ObservedAt: now, Unknown: map[string]string{"memory": "MemTotal not found in /proc/meminfo",
"cores": "/proc/cpuinfo says no core id", "kernel release": "permission denied"}}}
got := strings.Join(machineLines(a, true, now), "\n")
t.Log("\n" + got)
for _, want := range []string{
"memory unknown: MemTotal not found in /proc/meminfo",
"cores unknown: /proc/cpuinfo says no core id",
"kernel linux (release unknown: permission denied)",
// Neither said nor named unknown: the node-engine broke its rule, and that is shown.
"timezone unknown: the machine said neither it nor why",
} {
if !strings.Contains(got, want) {
t.Errorf("missing %q", want)
}
}
if strings.Contains(got, " 0 ") || strings.Contains(got, "0 B") {
t.Error("a fact the machine could not read was shown as a zero")
}
}
func TestAnOlderNodeEngineStillShowsArchitectureAndKernel(t *testing.T) {
got := strings.Join(machineLines(inventory.MachineAccount{Architecture: "amd64", Kernel: "linux"}, true,
time.Now()), "\n")
if !strings.Contains(got, "older than the one that says it") || !strings.Contains(got, "architecture amd64") ||
!strings.Contains(got, "kernel linux") {
t.Errorf("%s", got)
}
if got := strings.Join(machineLines(inventory.MachineAccount{}, false, time.Now()), "\n"); !strings.Contains(got, "never said") {
t.Errorf("%s", got)
}
}
func TestAnAccountWithoutItsTimeSaysAgeUnknown(t *testing.T) {
got := machineLines(inventory.MachineAccount{Architecture: "amd64", Kernel: "linux",
Machine: &inventory.MadeOf{KernelRelease: "7.2"}}, true, time.Now())[0]
if !strings.Contains(got, "age unknown") || strings.Contains(got, "d ago") {
t.Errorf("%s", got)
}
}
+2 -138
View File
@@ -2,9 +2,7 @@ package main
import (
"context"
"encoding/json"
"fmt"
"slices"
"sort"
"strings"
"sync/atomic"
@@ -42,96 +40,6 @@ const (
moduleUnhealthyAfter = 2
)
// runnerLibrary prefixes the engine's verdict on the tool runner's membership of a library's group (ADR 0334).
const runnerLibrary = "runner:library:"
// libraryVerdictsFor is, for each library verdict in a statement (kind library, no module), one copy for each module
// on the machine — assigned, or pulled in by a requirement — whose manifest uses that library: what the module is judged and said by. None when the
// assignments or the catalogue cannot be read: the library's own verdict stands alone, and no module is excused by it.
func libraryVerdictsFor(ctx context.Context, inv *inventory.Inventory, node string,
rs []inventory.ResourceHealth) []inventory.ResourceHealth {
var libraries []inventory.ResourceHealth
for _, r := range rs {
if r.Module == "" && ((r.Kind == link.KindLibrary && strings.HasPrefix(r.Resource, libraryAccess)) ||
(r.Kind == link.KindRunner && strings.HasPrefix(r.Resource, runnerLibrary))) {
libraries = append(libraries, r)
}
}
if len(libraries) == 0 || inv == nil {
return nil
}
assigned, err := inv.Assigned(ctx, node)
if err != nil {
// empty-on-error: the library's own verdict is kept and judged; no module is excused by a copy it could not make
return nil
}
// And the modules a requirement pulled in: what the machine was last sent (the review of 7ea6d36e, item 4).
sent, _, err := inv.SentBuilds(ctx, node)
if err != nil {
// empty-on-error: the library's own verdict is kept and judged; no module is excused by a copy it could not make
return nil
}
for m := range sent {
if !slices.Contains(assigned, m) {
assigned = append(assigned, m)
}
}
sort.Strings(assigned)
shelf, err := inv.Catalogue(ctx)
if err != nil {
// empty-on-error: the library's own verdict is kept and judged; no module is excused by a copy it could not make
return nil
}
var out []inventory.ResourceHealth
for _, lib := range libraries {
name := strings.TrimPrefix(strings.TrimPrefix(lib.Resource, runnerLibrary), libraryAccess)
for _, module := range assigned {
for _, u := range shelf[module].Libraries {
if u.Library == name {
copied := lib
copied.Module = module
out = append(out, copied)
break
}
}
}
}
return out
}
// takesStated keeps what a statement says of each library's take act: done under a warrant, the library is taken;
// failed, the take is forgotten and the operator asked again. Running or stopped (it goes on) changes nothing.
func takesStated(ctx context.Context, inv *inventory.Inventory, rs []link.ResourceHealth) error {
if inv == nil {
return nil
}
for _, r := range rs {
lib, isLibrary := strings.CutPrefix(r.Resource, libraryAccess)
if !isLibrary || r.Kind != link.KindLibrary || len(r.Act) == 0 {
continue
}
var act struct {
Act string `json:"act"`
Warrant string `json:"warrant"`
State string `json:"state"`
}
if err := json.Unmarshal(r.Act, &act); err != nil || act.Act != link.ActTake || act.Warrant == "" {
continue
}
var err error
switch act.State {
case "done":
err = inv.TakeDone(ctx, lib, act.Warrant)
case "failed":
err = inv.TakeFailed(ctx, lib, act.Warrant)
}
if err != nil {
return err
}
}
return nil
}
// healthRefused counts the statements refused as older than the one kept, for the log and a test.
var healthRefused atomic.Int64
@@ -169,22 +77,8 @@ func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Ke
for _, w := range r.Waits {
kept.Waits = append(kept.Waits, inventory.Wait{Part: w.Part, Secret: w.Secret, Setting: w.Setting, What: w.What})
}
// An access's waits (novox/hq ADR 0325 §7), kept beside the others as one more thing the operator gives: the
// access by its module's id, and whether it waits for a disk rather than for the operator's word.
for _, w := range r.AccessWaits {
kept.Waits = append(kept.Waits, inventory.Wait{Part: w.Part, Access: w.Access, Disk: w.Reason == link.WaitDisk,
Placed: w.Placed, Reason: w.Reason, What: w.What})
}
resources = append(resources, kept)
}
// **A library's take is done, or failed, when its engine says so** (novox/hq ADR 0328 §7): only then is it taken,
// or asked about again.
if err := takesStated(ctx, inv, h.Resources); err != nil {
return err
}
// **A library's verdict is said for the modules that use it** (novox/hq ADR 0328 §8): the engine states each
// library once, as `library:<name>`, of no module; a module that uses it waits with it, and is said so.
resources = append(resources, libraryVerdictsFor(ctx, inv, node, resources)...)
// **A wait for the operator is checked before it is excused** (novox/hq ADR 0283): a waiting resource whose
// wait does not check out is judged unhealthy, saying why; one that does is kept beside the unhealthy ones, so
// judgeModuleHealth can say it as needs-operator. What is stored is what the machine said.
@@ -750,7 +644,7 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
if waits && !f.groupsAdded[module] {
waits = false
}
var found, onWaiting, leftover []string
var found, onWaiting []string
var forOperator []inventory.Wait
for _, r := range resources {
if r.Module != module {
@@ -773,32 +667,6 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
found = append(found, r.Resource)
continue
}
// **The mesh's entries left on a folder whose disk was away are a condition this build found** when its move
// leaves the module's accesses and their users as they were (novox/hq ADR 0325 §8): they may name an access no
// longer declared, they wait for the disk, and a delivery that did not bring them is not held for them — said in
// the verdict, as a directory used as found is. A move that changes the accesses is held to them, whenever they
// were stated (the review of e1c2b744).
changed, known := f.accessesChanged[module]
changes := !known || changed
if r.Kind == link.KindKeptEntries && r.State == link.StateUnhealthy && !changes {
leftover = append(leftover, r.Resource)
continue
}
// **An access the engine could not judge this look** (its filesystem could not be named — a hung mount) carries
// no wait. Of a module whose move leaves its accesses as they were it is a condition the move found, not
// counted; of one whose move changes them it is not yet healthy, held to the gate's bound like anything still
// unsettled, so the gate fails safe at the bound and never waits for ever.
// The same for a library (ADR 0328): unknown this look, or starting while a take walks under it, of a move that
// leaves the module's libraries as they were, is not this move's.
if (r.Kind == link.KindAccess || r.Kind == link.KindLibrary) && !changes &&
(r.State == link.StateUnknown || (r.Kind == link.KindLibrary && r.State == link.StateStarting)) {
continue
}
// The tool runner's membership (ADR 0334 §2), of a move that changes no library of the module, is not this
// move's: it is said as the module's condition, and holds a move that changes them to the gate's bound.
if r.Kind == link.KindRunner && !changes {
continue
}
switch r.State {
case link.StateHealthy:
case link.StateStarting:
@@ -822,7 +690,7 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
reasonAfter(r.Reason))
}
}
if waits || len(found) > 0 || len(forOperator) > 0 || len(onWaiting) > 0 || len(leftover) > 0 {
if waits || len(found) > 0 || len(forOperator) > 0 || len(onWaiting) > 0 {
said := onWaiting
if waits {
said = append(said, wait)
@@ -834,10 +702,6 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
said = append(said, fmt.Sprintf("on %s, %s uses %s as found and waits for the operator to hand it over "+
"(`nox node hand-over %s <directory>` on the control-node)", machine, module, strings.Join(found, ", "), machine))
}
if len(leftover) > 0 {
said = append(said, fmt.Sprintf("on %s, the mesh's entries %s could not be taken out while their disk was "+
"away, stated before this send: they are taken out when it is back", machine, strings.Join(leftover, ", ")))
}
return healthPerson, strings.Join(said, "; ")
}
return healthGood, ""
+9 -1
View File
@@ -237,7 +237,15 @@ func moduleCommand(ctx context.Context, args []string) error {
says = append(says, "provides "+describeOffers(m.Provides))
}
if len(m.Requires) > 0 {
says = append(says, "requires "+strings.Join(m.Requires, ", "))
// An optional requirement says so (novox/hq ADR 0330): it is not refused when absent.
required := make([]string, 0, len(m.Requires))
for _, r := range m.Requires {
if m.IsOptional(r) {
r += " (optional)"
}
required = append(required, r)
}
says = append(says, "requires "+strings.Join(required, ", "))
}
for _, c := range m.Claims {
says = append(says, "claims "+c.At()+"/"+c.Name)
+18
View File
@@ -875,6 +875,17 @@ func showNode(ctx context.Context, stored *stores, name string) error {
}
}
// What it is made of, in one block with its age (novox/hq issue 160): collected and shown in the same
// breath, so no fact the machine says is kept and read by nothing.
if account, had, err := inv.MachineOf(ctx, name); err != nil {
fmt.Printf("\n what it is made of could NOT be read: %v\n", err)
} else {
fmt.Println()
for _, line := range machineLines(account, had, time.Now()) {
fmt.Println(line)
}
}
held, err := inv.Profile(ctx, name)
if err != nil {
return err
@@ -918,6 +929,13 @@ func showNode(ctx context.Context, stored *stores, name string) error {
for _, line := range notAppliedLines(name, leftOutOf(plan, settings, node.Adopted)) {
fmt.Println(line)
}
// And which optional requirements of its modules nothing assigned answers (novox/hq ADR 0330 §7).
if lines := unboundLines(plan); len(lines) > 0 {
fmt.Println()
for _, line := range lines {
fmt.Println(line)
}
}
}
return nil
}
+2 -102
View File
@@ -44,34 +44,17 @@ func needsOperatorKey(module, node string) string {
type operatorWaitFacts struct {
manifests map[string]catalogue.Manifest
given map[string]map[string]time.Time
// libraries is every library the operator defined, and taken every one taken (novox/hq ADR 0328 §7, §8);
// librariesRead says both were read — when not, no access wait is excused.
libraries map[string]catalogue.Library
taken map[string]inventory.TakenLibrary
librariesRead bool
}
// checkWait is nil when a wait is excused, and otherwise why not, in words. Pure.
func checkWait(module, machine string, w inventory.Wait, f operatorWaitFacts) error {
// A library's own wait names it, and no module's manifest (novox/hq ADR 0328 §7).
if lib, ok := strings.CutPrefix(w.Access, libraryAccess); ok && w.Secret == "" && w.Setting == "" {
return checkLibraryWait(machine, lib, w, f)
}
m, known := f.manifests[module]
if !known {
return fmt.Errorf("says it waits for %s, and the mesh holds no manifest of %s to check it against", waitNames(w), module)
}
named := 0
for _, n := range []string{w.Secret, w.Setting, w.Access} {
if n != "" {
named++
}
}
switch {
case named != 1:
return fmt.Errorf("says it waits, naming %s, where a wait names one secret or one setting, or one access", waitNames(w))
case w.Access != "":
return checkAccessWait(module, machine, m, w, f)
case w.Secret != "" && w.Setting != "", w.Secret == "" && w.Setting == "":
return fmt.Errorf("says it waits, naming %s, where a wait names one secret or one setting", waitNames(w))
case w.Setting != "":
if _, declared := m.Settings[w.Setting]; !declared {
return fmt.Errorf("says it waits for the setting %s, which %s does not declare", w.Setting, module)
@@ -97,63 +80,6 @@ func checkWait(module, machine string, w inventory.Wait, f operatorWaitFacts) er
return nil
}
// libraryAccess prefixes the access a library's own wait and take name: "library:<name>".
const libraryAccess = "library:"
// checkAccessWait is nil when a module's access wait is excused (novox/hq ADR 0328 §8, §9): the access is one of its
// library entries, and its library's wait is excused. Any other access waits for nothing the operator gives: an
// access that is not a library keeps ADR 0051's meaning.
func checkAccessWait(module, machine string, m catalogue.Manifest, w inventory.Wait, f operatorWaitFacts) error {
lib := catalogue.LibraryOfEntry(m, w.Access)
if lib == "" {
return fmt.Errorf("says it waits for %s, which is no library entry %s declares: only a library waits for "+
"the operator (novox/hq ADR 0328 §8)", w.Access, module)
}
return checkLibraryWait(machine, lib, w, f)
}
// checkLibraryWait is nil when a wait on a library is excused (novox/hq ADR 0328 §8, §9): the library is defined and
// reaches the machine, and either its disk is not there (reason disk) or it has not yet been taken on its home, on the
// filesystem the wait names (reason keep). Any other is judged unhealthy, saying why: a module that uses a library
// already taken is never asked about it.
func checkLibraryWait(machine, lib string, w inventory.Wait, f operatorWaitFacts) error {
if !f.librariesRead {
return fmt.Errorf("says it waits for the library %s, and the libraries could not be read", lib)
}
l, defined := f.libraries[lib]
switch {
case !defined:
return fmt.Errorf("says it waits for the library %s, which is not defined", lib)
case !l.Reaches(machine):
return fmt.Errorf("says it waits for the library %s, which does not reach %s", lib, machine)
case w.Reason == link.WaitDisk || (w.Reason == "" && w.Disk):
return nil
case w.Reason != link.WaitKeep:
return fmt.Errorf("says it waits for the library %s for %q, and a library waits for the operator's word to "+
"take it or for its disk", lib, w.Reason)
}
// Taken on another home or filesystem (it moved) is not taken by that warrant: excused, and asked again (§7).
// Only the home's engine names the root's filesystem; another machine's wait names its mount, and says nothing of it.
placed := w.Placed
if machine != l.Home {
placed = ""
}
if t, was := f.taken[lib]; was && t.Done && t.Home == l.Home && t.Root == l.Root &&
(placed == "" || t.Placed == "" || t.Placed == placed) {
return fmt.Errorf("says it waits for the library %s, which is taken already: a module that uses a taken "+
"library is never asked about it", lib)
}
return nil
}
// accessWords is an access a wait names, in words with no path: "the library media", or "its library entry movies".
func accessWords(access string) string {
if lib, ok := strings.CutPrefix(access, libraryAccess); ok {
return "the library " + lib
}
return "the library entry " + access
}
// waitRefusedPrefix opens every reason checkWait gives, so the words of a refused wait are told from a check's own.
const waitRefusedPrefix = "says it waits"
@@ -162,10 +88,6 @@ func waitNames(w inventory.Wait) string {
switch {
case w.Secret != "" && w.Setting != "":
return "the secret " + w.Secret + " and the setting " + w.Setting
case w.Access != "" && (w.Secret != "" || w.Setting != ""):
return "the access " + w.Access + " and a secret or a setting"
case w.Access != "":
return "the access " + w.Access
case w.Secret != "":
return "the secret " + w.Secret
case w.Setting != "":
@@ -225,14 +147,6 @@ func operatorWaitSaid(module, machine string, waits []inventory.Wait) string {
var parts []string
for _, w := range waits {
part := fmt.Sprintf("%s (%s", w.What, waitNames(w))
// An access in the controller's own words, never the engine's: words with no path (novox/hq ADR 0325 §7).
switch {
case w.Access != "" && w.Disk:
part = fmt.Sprintf("the disk that holds %s mounted (%s", accessWords(w.Access), waitNames(w))
case w.Access != "":
part = fmt.Sprintf("leave to keep %s (%s, asked of you once for the library", accessWords(w.Access),
waitNames(w))
}
if w.Secret != "" {
part += fmt.Sprintf(", given with `nox secret ask %s %s %s`", machine, module, w.Secret)
}
@@ -266,11 +180,6 @@ func needsOperatorWords(module, node string, waits []inventory.Wait) words {
act, secret = fmt.Sprintf("type %s at %s's desk prompt", w.What, node), true
case w.Setting != "":
act = fmt.Sprintf("approve %s of %s on %s when it is proposed to you", w.Setting, module, node)
// An access, in words with no path (novox/hq ADR 0325 §7): the folder by the access's name.
case w.Access != "" && w.Disk:
act = fmt.Sprintf("mount the disk that holds %s on %s", accessWords(w.Access), node)
case w.Access != "":
act = fmt.Sprintf("let the mesh keep %s when you are asked", accessWords(w.Access))
}
if act != "" && !seen[act] {
seen[act] = true
@@ -308,7 +217,6 @@ func readWaitFacts(ctx context.Context, inv *inventory.Inventory, machine string
if f.given == nil {
f.given = map[string]map[string]time.Time{}
}
if inv == nil {
return
}
@@ -337,14 +245,6 @@ func readWaitFacts(ctx context.Context, inv *inventory.Inventory, machine string
f.given[module+"@"+machine] = given
}
}
// The libraries and which are taken (ADR 0328 §7, §8): read once.
if !f.librariesRead {
libs, lerr := inv.Libraries(ctx)
taken, terr := inv.TakenLibraries(ctx)
if lerr == nil && terr == nil {
f.libraries, f.taken, f.librariesRead = libs, taken, true
}
}
}
// waitingModules is every module with a waiting resource in a statement.
+15 -160
View File
@@ -9,7 +9,6 @@ import (
"fmt"
"io"
"os"
"path/filepath"
"slices"
"sort"
"strings"
@@ -134,11 +133,6 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
if _, err := keepRecorded(ctx, open, nodeName, shelf); err != nil {
return catalogue.Resolution{}, nil, err
}
// And a module that may not join yet, at the build it ran before (novox/hq ADR 0328 §8, ADR 0334).
held, err := holdJoining(ctx, inv, nodeName, shelf)
if err != nil {
return catalogue.Resolution{}, nil, err
}
resolved, err := catalogue.Resolve(shelf, assigned,
catalogue.Node{Name: nodeName, Site: site, Capabilities: capabilities,
@@ -224,10 +218,6 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
}
settings[m.Module] = layers
}
// A held module with the settings it last ran with (ADR 0328 §8).
if err := heldSettings(ctx, inv, nodeName, held, settings); err != nil {
return catalogue.Resolution{}, nil, err
}
return resolved, settings, nil
}
@@ -457,8 +447,7 @@ func declarationWith(ctx context.Context, open *stores, node string,
if err != nil {
return sendable{}, err
}
out := sendable{Resources: composed.Resources, Adoption: adoption, Groups: composed.Groups,
Identities: composed.Identities, Libraries: composed.Libraries,
out := sendable{Resources: composed.Resources, Adoption: adoption,
Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut, withheld: with.Withheld,
unbound: with.Unbound, foreseen: composed.Foreseen, unplaced: composed.Unplaced,
@@ -482,22 +471,11 @@ func declarationWith(ctx context.Context, open *stores, node string,
names = append(names, m.Module)
}
out.Builds = carriedBuilds(names, composed.LeftOut, current, before)
// A recorded module kept at the build the machine runs is recorded as carrying that one (ADR 0245), and so
// is a joining module held at the build it ran before (ADR 0328 §8).
// A recorded module kept at the build the machine runs is recorded as carrying that one (ADR 0245).
kept, err := recordedKept(ctx, open, node)
if err != nil {
return sendable{}, err
}
held, err := holdJoining(ctx, open.inventory, node, nil)
if err != nil {
return sendable{}, err
}
if kept == nil && len(held) > 0 {
kept = map[string]string{}
}
for m, h := range held {
kept[m] = h.commit
}
for m, was := range kept {
if _, carried := out.Builds[m]; carried {
out.Builds[m] = was
@@ -997,17 +975,9 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// The libraries, the numbers of the modules that join and of the libraries' groups, and which libraries are taken
// (novox/hq ADR 0328), to an engine that keeps libraries.
lf, err := libraryFacts(ctx, inv, node, plan, choosing)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
return catalogue.Rendering{
ReadsHealth: readsHealth,
JudgesRoot: judgesRoot,
KeepsLibraries: lf.keeps, Identities: lf.identities, Libraries: lf.libraries, LibraryGroups: lf.groups,
TakenLibraries: lf.taken, LibraryPaths: lf.paths, RunnerJoinsOwnGroups: lf.ownGroups,
ReadsHealth: readsHealth,
JudgesRoot: judgesRoot,
BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Foreseen: foreseen, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
@@ -1040,132 +1010,6 @@ func engineJudgesRoot(ctx context.Context, inv *inventory.Inventory, node string
return had && stated.Contract >= link.RootContract, nil
}
// engineKeepsLibraries says whether a machine's node-engine makes the mesh's users and library groups, keeps
// libraries, and keeps a joining module's own groups (novox/hq ADR 0328, ADR 0334: its statement's contract is
// link.OwnGroupContract or later), by its own newest statement. On an older one no module joins: the libraries
// alone are not enough, since its tools would lose their read of a joining module's directories.
func engineKeepsLibraries(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.OwnGroupContract, nil
}
// libraryRendering is what a machine's composition needs of ADR 0328.
type libraryRendering struct {
keeps bool
identities map[string]catalogue.ModuleIdentity
libraries map[string]catalogue.Library
groups map[string]int
taken map[string]bool
paths map[string][]string
ownGroups bool
}
// libraryFacts reads what a machine's composition needs of ADR 0328: the libraries and their group numbers, the
// numbers of the modules here that join, which libraries are taken, and the sub-paths every module asks of each. **A number is allocated only on the send path** (Allocating),
// as a port is: a question about what would be sent reads what was allocated. The libraries are read for every engine,
// so a module that joins is left out, naming why, where its engine cannot be told it does.
func libraryFacts(ctx context.Context, inv *inventory.Inventory, node string, plan catalogue.Resolution,
choosing Choosing) (libraryRendering, error) {
var lf libraryRendering
joining := false
for _, m := range plan.Modules {
joining = joining || catalogue.Joins(m)
}
libs, err := inv.Libraries(ctx)
if err != nil {
return lf, err
}
reached := false
for _, l := range libs {
reached = reached || l.Reaches(node)
}
if !joining && !reached {
return lf, nil // nothing here joins, and no library reaches it: composed exactly as before (§14)
}
lf.libraries = libs
if lf.keeps, err = engineKeepsLibraries(ctx, inv, node); err != nil || !lf.keeps {
return lf, err
}
lf.ownGroups = true // a module joins only on an engine that also keeps its own groups (ADR 0334)
lf.identities = map[string]catalogue.ModuleIdentity{}
for _, m := range plan.Modules {
if !catalogue.Joins(m) {
continue
}
var id catalogue.ModuleIdentity
had := true
if choosing == Allocating {
id, err = inv.AllocateIdentity(ctx, m.Module)
} else {
id, had, err = inv.IdentityOf(ctx, m.Module)
}
if err != nil {
return lf, err
}
if had {
lf.identities[m.Module] = id
}
}
if lf.groups, err = inv.LibraryGroups(ctx); err != nil {
return lf, err
}
if choosing == Allocating {
for name, l := range libs {
if _, has := lf.groups[name]; has || !l.Reaches(node) {
continue
}
if lf.groups[name], err = inv.AllocateLibraryGroup(ctx, name); err != nil {
return lf, err
}
}
}
if lf.paths, err = inv.LibraryPaths(ctx); err != nil {
return lf, err
}
if lf.taken, err = librariesTaken(ctx, inv, libs); err != nil {
return lf, err
}
return lf, nil
}
// librariesTaken is which libraries are taken now (§7): recorded taken on the home they have, on the filesystem their
// home's newest statement names — or stated kept by their home's engine, which keeps one it found empty without asking.
func librariesTaken(ctx context.Context, inv *inventory.Inventory, libs map[string]catalogue.Library) (map[string]bool, error) {
recorded, err := inv.TakenLibraries(ctx)
if err != nil {
return nil, err
}
out := map[string]bool{}
for name, l := range libs {
t, was := recorded[name]
taken := was && t.Done && t.Home == l.Home && t.Root == l.Root
stated, had, err := inv.HealthOf(ctx, l.Home)
if err != nil {
return nil, err
}
if had {
for _, r := range stated.Resources {
kept := (r.Resource == libraryAccess+name || filepath.Clean(r.Target) == l.Root) &&
(r.Kind == link.KindAccess || r.Kind == link.KindLibrary)
if kept && r.State == link.StateHealthy {
taken = true
}
for _, w := range r.Waits {
if w.Access == libraryAccess+name && w.Reason == link.WaitKeep && w.Placed != "" && was &&
t.Placed != "" && w.Placed != t.Placed {
taken = false // it moved to another filesystem: not taken there until its new warrant
}
}
}
}
out[name] = taken
}
return out, nil
}
// zonesInTheMesh is every zone a module in the mesh declares, where the mesh placed it (novox/hq ADR
// 0199): the zone settled from that node's settings, the node's private address, the port the
// answering listen is published on there.
@@ -1471,6 +1315,14 @@ func planCommand(ctx context.Context, args []string) error {
fmt.Printf("%s is assigned nothing\n", args[0])
return nil
}
// Which optional requirements are unbound (novox/hq ADR 0330 §7): beside the document on the
// error stream where the answer is the declaration itself, which carries no such field — the
// node-engine refuses an envelope key it does not know.
if *asDiff || *asJSON {
for _, line := range unboundLines(plan) {
fmt.Fprintf(os.Stderr, "%s:%s\n", args[0], line)
}
}
if *asDiff {
declared, err := declarationFor(ctx, open, args[0], plan, settings)
if err != nil {
@@ -1552,6 +1404,9 @@ func planCommand(ctx context.Context, args []string) error {
for _, n := range plan.Needs {
fmt.Printf(" needs %s from %s, for %s\n", n.Name, n.From, n.For)
}
for _, line := range unboundLines(plan) {
fmt.Println(line)
}
declared, err := declarationFor(ctx, open, args[0], plan, settings)
if err != nil {
return err
+56 -14
View File
@@ -14,7 +14,6 @@ import (
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
@@ -81,6 +80,30 @@ func listQueue(ctx context.Context, args []string, w io.Writer) error {
return nil
}
// keepingTheQueueInOrder runs the build queue's passes while this controller acts (novox/hq issue 558):
// the one process that puts the queue in order, so no two passes pause and resume it over each other.
// The build seat is read from the catalogue at most every half minute.
func keepingTheQueueInOrder(ctx context.Context, open *stores, js *broker.JetStream) {
if js == nil {
return
}
seat, readAt := link.TheBuildMachine, time.Time{}
seatNow := func() string {
if time.Since(readAt) < watchEvery {
return seat
}
if entries, err := open.inventory.Catalogued(ctx); err == nil {
seat, readAt = buildSeatAmong(entries), time.Now()
} else {
fmt.Printf("the build seat is read as %s: the catalogue cannot be read: %v\n", seat, err)
}
return seat
}
link.KeepingInOrder(ctx, js, seatNow, link.Holding, func(format string, args ...any) {
fmt.Printf(format+"\n", args...)
})
}
// queueAnswer is the queue as data (novox/hq issue 455): what a caller reads to say how far back an ask
// waits — "waiting in the build queue behind 2" — which the table alone could not tell it. Text is the
// table, written from the same read: the verb answers it as its output and the rest as its answer, so the
@@ -100,7 +123,8 @@ type queueAnswer struct {
type queuedAnswer struct {
ID string `json:"id"`
State string `json:"state"`
// Kind is "check" for a pull request's merge check, "build" for anything else.
// Kind is "delivery" for a build a walk waits for, "check" for a pull request's check and any ask
// whose kind cannot be told (link.KindOf, novox/hq issue 558).
Kind string `json:"kind"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
@@ -129,10 +153,7 @@ func queueAnswerOf(q link.Queue, now time.Time) queueAnswer {
out := queueAnswer{Seat: q.Seat, Waiting: len(waiting), InFlight: len(running), Dead: len(dead),
MaxDeliver: q.MaxDeliver, Asks: []queuedAnswer{}, Text: queueText(q, now)}
one := func(a link.QueuedAsk, position int) queuedAnswer {
kind := "build"
if a.Request.Check != nil {
kind = "check"
}
kind := link.KindOf(a.Request)
since := a.AskedAt
if a.State == link.AskInFlight && a.On != "" && !a.Started.IsZero() {
since = a.Started
@@ -189,19 +210,20 @@ func queueText(q link.Queue, now time.Time) string {
case a.Was != "":
where = fmt.Sprintf("handed back by %s, to be handed out again", a.Was)
}
fmt.Fprintf(&b, " %-26s %s — %s, %s (seq %d)\n", a.ID, what(a), where, ago(a.AskedAt), a.Seq)
fmt.Fprintf(&b, " %-26s %-8s %s — %s, %s (seq %d)\n", a.ID, link.KindOf(a.Request), what(a), where,
ago(a.AskedAt), a.Seq)
}
}
if len(waiting) > 0 {
fmt.Fprintln(&b, "\nwaiting:")
fmt.Fprintln(&b, "\nwaiting, in the order they are handed out:")
for _, a := range waiting {
fmt.Fprintf(&b, " %-26s %s — %s (seq %d)\n", a.ID, what(a), ago(a.AskedAt), a.Seq)
fmt.Fprintf(&b, " %-26s %-8s %s — %s (seq %d)\n", a.ID, link.KindOf(a.Request), what(a), ago(a.AskedAt), a.Seq)
}
}
if len(dead) > 0 {
fmt.Fprintf(&b, "\ndead — handed out as often as the worker allows (%d) and never settled, still held:\n", q.MaxDeliver)
for _, a := range dead {
fmt.Fprintf(&b, " %-26s %s — %s (seq %d)\n", a.ID, what(a), ago(a.AskedAt), a.Seq)
fmt.Fprintf(&b, " %-26s %-8s %s — %s (seq %d)\n", a.ID, link.KindOf(a.Request), what(a), ago(a.AskedAt), a.Seq)
}
}
switch {
@@ -275,6 +297,15 @@ func cancelAsk(ctx context.Context, js *broker.JetStream, open *stores, seat str
}
}
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: seat, Accepts: []string{"build"}})
// **Where it waits now** (novox/hq issue 558): moved into order since it was read, a waiting ask has a
// later sequence, and the worker handing out what was behind its old one is not it being taken.
if ask.State == link.AskWaiting {
if q, err := link.ReadQueue(ctx, js, seat); err == nil {
if now, ok := q.Find(ask.ID); ok && now.State == link.AskWaiting {
ask.Seq = now.Seq
}
}
}
switch ask.State {
case link.AskWaiting:
if taken, err := takenSince(js, worker, ask.Seq); err != nil {
@@ -296,14 +327,25 @@ func cancelAsk(ctx context.Context, js *broker.JetStream, open *stores, seat str
"ends it there", ask.ID, started, ask.ID)
}
}
if err := js.Context().DeleteMsg(worker.Stream, ask.Seq); err != nil && !errors.Is(err, nats.ErrMsgNotFound) &&
!errors.Is(err, jetstream.ErrMsgNotFound) && !strings.Contains(err.Error(), "no message found") {
// Found by its id when it was moved into order since it was read (novox/hq issue 558).
removed, err := link.DeleteAsk(ctx, js, seat, ask)
if err != nil {
return "", fmt.Errorf("%s is marked cancelled and could not be deleted from the queue (seq %d): %w — a "+
"holder taking it ends it as cancelled", ask.ID, ask.Seq, err)
}
recordCancelled(ctx, open, ask)
return fmt.Sprintf("cancelled %s (%s, %s): deleted from the %s queue and recorded failed, %s — a plan "+
"that asked for it fails with that", ask.ID, ask.Repository, ask.State, seat, link.CancelledByHand), nil
return cancelSaid(ask, seat, removed), nil
}
// cancelSaid is what a cancel says: deleted from the queue only when it was.
func cancelSaid(ask link.QueuedAsk, seat string, removed uint64) string {
where := fmt.Sprintf("deleted from the %s queue", seat)
if removed == 0 {
where = fmt.Sprintf("no longer in the %s queue to delete, and marked cancelled, so a holder taking it "+
"ends it", seat)
}
return fmt.Sprintf("cancelled %s (%s, %s): %s and recorded failed, %s — a plan "+
"that asked for it fails with that", ask.ID, ask.Repository, ask.State, where, link.CancelledByHand)
}
// holderLooks is how long a cancel waits for a holder that took the ask before the mark to say it
+22 -2
View File
@@ -24,7 +24,7 @@ func aQueueOfEachKind(now time.Time) link.Queue {
State: link.AskInFlight, On: "ace", Started: started,
Request: link.BuildRequest{ID: "build-running", Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host"}}},
{Seq: 3, ID: "build-first", Repository: "ssh://forge/novox/mesh-catalog", Ref: "main", AskedAt: asked,
State: link.AskWaiting},
State: link.AskWaiting, Request: link.BuildRequest{ID: "build-first", Repository: "ssh://forge/novox/mesh-catalog"}},
{Seq: 4, ID: "build-second", Repository: "ssh://forge/novox/mesh-catalog", Path: "modules/gitea", Ref: aCommit,
AskedAt: now.Add(-time.Minute), State: link.AskWaiting,
Request: link.BuildRequest{ID: "build-second", Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog"}}},
@@ -51,7 +51,7 @@ func TestTheQueueVerbAnswersEachAskAsData(t *testing.T) {
t.Errorf("in flight: %+v", running)
}
first, second := byID["build-first"], byID["build-second"]
if first.Position != 1 || first.Kind != "build" || first.Commit != "" || first.Ref != "main" || first.Machine != "" ||
if first.Position != 1 || first.Kind != "delivery" || first.Commit != "" || first.Ref != "main" || first.Machine != "" ||
!first.Since.Equal(now.Add(-5*time.Minute)) {
t.Errorf("first waiting: %+v", first)
}
@@ -62,6 +62,13 @@ func TestTheQueueVerbAnswersEachAskAsData(t *testing.T) {
if dead := byID["build-dead"]; dead.Position != 0 || dead.State != link.AskDead {
t.Errorf("dead: %+v", dead)
}
// And the table says each ask's kind too (novox/hq issue 558).
for _, line := range []string{"build-running check ", "build-first delivery ",
"build-second check ", "build-dead check "} {
if !strings.Contains(got.Text, line) {
t.Errorf("the table does not say %q:\n%s", line, got.Text)
}
}
// In the table's order: in flight, then the waiting by position, then the dead.
var order []string
for _, a := range got.Asks {
@@ -135,3 +142,16 @@ func TestTheQueueNeverShowsARepositorysPassword(t *testing.T) {
}
}
}
// A cancel says it deleted the ask only when it did: moved into order and gone since, it says so (novox/hq
// issue 558).
func TestACancelSaysDeletedOnlyWhenItDeleted(t *testing.T) {
ask := link.QueuedAsk{ID: "build-1", Repository: "novox/x", State: link.AskWaiting, Seq: 4}
if said := cancelSaid(ask, link.TheBuildMachine, 9); !strings.Contains(said, "deleted from the") {
t.Errorf("removed, it says: %s", said)
}
if said := cancelSaid(ask, link.TheBuildMachine, 0); strings.Contains(said, "deleted from") ||
!strings.Contains(said, "no longer in the") {
t.Errorf("gone already, it says: %s", said)
}
}
+1 -16
View File
@@ -253,24 +253,9 @@ func withoutKept(ctx context.Context, open *stores, node string, moves []invento
keeping = keepingRecorded(ctx)
}
keeps, err := recordedKept(keeping, open, node)
if err != nil {
if err != nil || len(keeps) == 0 {
return moves, err
}
// And a joining module held at the build it runs (novox/hq ADR 0328 §8): its new build is no move this send makes,
// and judging it would read the gate as superseded by its own send (the re-review of 96e15f86, item 1).
held, err := holdJoining(ctx, open.inventory, node, nil)
if err != nil {
return moves, err
}
if len(held) > 0 && keeps == nil {
keeps = map[string]string{}
}
for m, h := range held {
keeps[m] = h.commit
}
if len(keeps) == 0 {
return moves, nil
}
var out []inventory.CarriedMove
for _, mv := range moves {
if was, kept := keeps[mv.Module]; kept && sameCommit(was, mv.From) {
+4 -26
View File
@@ -690,7 +690,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
// being behind. Now every module of the tier whose first machine is the same is sent there in one
// send, judged there by one gate — kept on the first of them, the others gated by it — and the rest
// of the tier's machines are sent once, together, when the gates have passed.
var pending, waitsForAPerson []string
var pending []string
firstOn := map[string][]string{}
runningOf := map[string][]string{}
var rest []string
@@ -706,20 +706,6 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
return false, err
}
runningOf[m] = running
// **A joining build held for a person is not sent** (novox/hq ADR 0328 §8, the re-review of 96e15f86): its
// machines keep the build they run until its library is taken (or their engine keeps own groups), so sending
// it would send nothing new and judge the old build. It waits for a person, said; the plan stays open, the other
// modules are sent and judged, and it is sent on the walk after nothing holds it — no new merge.
if state.FirstAt == nil {
why, err := heldIn(ctx, inv, m, running)
if err != nil {
return false, err
}
if why != "" {
waitsForAPerson = append(waitsForAPerson, why)
continue
}
}
// **A build whose source is unchanged is never a move** (novox/hq issue 280): made from the source
// of the build every machine running it was sent, it is registered with that build's artifacts —
// nothing to send, and nothing for a gate to judge.
@@ -882,17 +868,9 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
strings.Join(sent, ", "))
return true, nil
}
if len(pending) > 0 || len(waitsForAPerson) > 0 {
note := "tier " + fmt.Sprint(p.Tier) + " built; "
if len(pending) > 0 {
note += "waiting for " + strings.Join(pending, "; ") + " to report it applied before the rest are sent"
}
if len(waitsForAPerson) > 0 {
if len(pending) > 0 {
note += "; and "
}
note += "waiting for a person: " + strings.Join(waitsForAPerson, "; ")
}
if len(pending) > 0 {
note := "tier " + fmt.Sprint(p.Tier) + " built; waiting for " + strings.Join(pending, "; ") +
" to report it applied before the rest are sent"
changed := p.State != inventory.PlanRolling || p.Note != note
p.State = inventory.PlanRolling
p.Note = note
+90
View File
@@ -0,0 +1,90 @@
package main
import (
"encoding/json"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// TestReplay201ThePlansSend replays novox/hq issue 201 (2026-10-02): two merges a minute apart made two plans; the
// controller's plan rolled the newer controller onto the control node, and the node-engine's plan then sent that
// machine the controller build before it, whose binary refused the seat row the newer one had widened. Here the
// control node, anchor, was last sent the newer controller c2 while the mesh's record of the controller is the
// build before it, c1, which passed its gate. The outcome asserted: a send that is no person's push — composed as a
// plan's send composes it (keepingRecorded, what sendKeeps gives every send through sendToEach) — carries the
// controller at c2, the build the machine was last sent, and not the older record; a person's push still carries
// the record. Written with only what the controller had before its fix (mesh-controller #273) — the inventory's
// build, gate, assignment and send records, keepingRecorded, planFor and declarationFor — so it is laid over the
// commit before; the fix's own TestReplay201 reads what the fix added. Needs the store (MESH_TEST_POSTGRES), and
// is named in testdata/store-tests.
func TestReplay201ThePlansSend(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
const older, newer = "c1aaaaaaaaaa", "c2bbbbbbbbbb"
old, now := time.Now().Add(-2*time.Hour), time.Now().Add(-time.Minute)
record := func(commit string, asked time.Time) inventory.Build {
t.Helper()
manifest, _ := json.Marshal(catalogue.Manifest{Module: "mesh-controller", Version: "1"})
b := inventory.Build{ID: "build-mesh-controller-" + commit, Module: "mesh-controller", Commit: commit,
Repository: "novox/mesh-controller", Manifest: manifest, Asked: asked, At: asked,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:" + commit + "0000000000000000"}}}
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
return b
}
// The older controller rolled out and passed its gate; the newer one was asked after it and registered.
before := record(older, old)
if err := inv.RecordGate(ctx, inventory.GateVerdict{Build: before.ID, Module: "mesh-controller", Commit: older,
Machines: []string{"anchor"}, Verdict: inventory.GatePassed}); err != nil {
t.Fatal(err)
}
record(newer, now)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"}, inventory.Source{
Repository: "novox/mesh-controller", BuiltFrom: newer, Head: newer, Asked: now}); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
// The controller's plan sent anchor the newer controller.
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"mesh-controller": newer}); err != nil {
t.Fatal(err)
}
// And the mesh's record of the controller is the build before it — whatever made it so.
if err := inv.RestoreModule(ctx, before); err != nil {
t.Fatal(err)
}
// The controller each send to anchor carries: a plan's send keeps recorded builds (keepingRecorded, what
// sendKeeps gives every send but a person's push); a person's push composes the mesh's record.
controllerSent := func(sends string, keeping bool) string {
t.Helper()
c := ctx
if keeping {
c = keepingRecorded(ctx)
}
plan, settings, err := planFor(c, open, "anchor")
if err != nil {
t.Fatalf("%s to anchor did not compose: %v", sends, err)
}
declared, err := declarationFor(c, open, "anchor", plan, settings)
if err != nil {
t.Fatalf("%s to anchor did not compose: %v", sends, err)
}
return declared.Builds["mesh-controller"]
}
if got := controllerSent("a plan's send", true); got != newer {
t.Fatalf("a plan's send carried anchor the controller %s, older than the %s anchor was last sent: the "+
"older binary would read the row its successor widened", short(got), short(newer))
}
if got := controllerSent("a person's push", false); got != older {
t.Fatalf("a person's push carried the controller %s, not the mesh's record %s", short(got), short(older))
}
}
-15
View File
@@ -18,12 +18,6 @@ import (
// make a machine look out of date for ever, or send something `plan` never showed.
type sendable struct {
Resources []map[string]any
// Groups, Identities and Libraries are what the declaration says of libraries and joining modules (novox/hq ADR
// 0328): composed only for an engine that keeps libraries, and not sent at all when empty — an older engine
// refuses a key it does not know, whole.
Groups []catalogue.GroupEntry
Identities []catalogue.IdentityEntry
Libraries []catalogue.LibraryEntry
// Sequence orders this send against every other to the same node: one higher each time, taken
// under the node's hold just before the body is made (novox/hq 04-ISSUES/107). Zero is not sent
// at all, which a host reads as "no order claimed" — the shape of every declaration before this.
@@ -116,15 +110,6 @@ func (s sendable) Body() ([]byte, error) {
if s.Generation > 0 {
envelope["generation"] = s.Generation
}
if len(s.Groups) > 0 {
envelope["groups"] = s.Groups
}
if len(s.Identities) > 0 {
envelope["identities"] = s.Identities
}
if len(s.Libraries) > 0 {
envelope["libraries"] = s.Libraries
}
// An empty declaration is deliberate here — the node owns nothing the mesh put there
// (novox/hq issue 127) — and the host refuses an empty body unless it is told the emptiness
// is meant, so a truncated or mis-composed body is never mistaken for "own nothing".
+21
View File
@@ -0,0 +1,21 @@
package main
import (
"fmt"
"github.com/novox/mesh-controller/internal/catalogue"
)
// unboundLines say which optional requirements of a machine's modules nothing assigned answers
// (novox/hq ADR 0330 §7): left unbound rather than refused, with nothing installed for them, so the
// module runs without — and said, because a data source that never appears is otherwise a silence.
// Each binds without anybody's act on the first composition after a provider is assigned.
func unboundLines(plan catalogue.Resolution) []string {
lines := make([]string, 0, len(plan.Absent))
for _, a := range plan.Absent {
lines = append(lines, fmt.Sprintf(
" unbound %s, optional for %s — nothing assigned provides it; it binds once a provider is "+
"assigned (novox/hq ADR 0330)", a.Requirement, a.Consumer))
}
return lines
}
+59
View File
@@ -0,0 +1,59 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A machine running grafana without influxdb says so (novox/hq ADR 0330 §7, issue 166): `plan` and
// `node` list the optional requirement nothing assigned answers, and the requirement is left
// absent rather than refused — influxdb is in the catalogue and is not installed for it.
func TestPlanAndNodeListAnOptionalRequirementLeftUnbound(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "influxdb", Version: "1",
Provides: catalogue.FromAnywhere("influxdb-api")})
register(t, open, catalogue.Manifest{Module: "grafana", Version: "1",
Requires: []string{"influxdb-api"}, OptionalRequires: map[string]bool{"influxdb-api": true},
Resources: []map[string]any{{"id": "state", "type": "directory", "path": "/var/lib/grafana", "mode": "0700"}}})
if _, err := assign(ctx, open, "laptop", "grafana"); err != nil {
t.Fatalf("grafana could not be assigned on a mesh without influxdb: %v", err)
}
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
for _, m := range plan.Modules {
if m.Module == "influxdb" {
t.Fatal("influxdb was installed for grafana's optional requirement (ADR 0330 §3)")
}
}
const said = "unbound influxdb-api, optional for grafana"
if shown := printed(t, func() error { return planCommand(ctx, []string{"laptop"}) }); !strings.Contains(shown, said) {
t.Errorf("plan laptop does not say influxdb-api is unbound:\n%s", shown)
}
if shown := printed(t, func() error { return showNode(ctx, open, "laptop") }); !strings.Contains(shown, said) {
t.Errorf("node show laptop does not say influxdb-api is unbound:\n%s", shown)
}
// And once influxdb is assigned somewhere, grafana binds to it with no act on grafana.
if _, err := assign(ctx, open, "anchor", "influxdb"); err != nil {
t.Fatal(err)
}
plan, _, err = planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if len(plan.Absent) != 0 {
t.Fatalf("still absent after influxdb was assigned: %+v", plan.Absent)
}
var bound bool
for _, n := range plan.Needs {
bound = bound || n.For == "grafana" && n.Name == "influxdb-api" && n.From == "anchor"
}
if !bound {
t.Fatalf("grafana was not bound to influxdb on anchor: %+v", plan.Needs)
}
}
+3
View File
@@ -739,6 +739,9 @@ func watchTheMesh(ctx context.Context, open *stores, server *link.Server, bus li
// and the answer chosen is performed on its warrant.
startAsking(watching, open, server, bus.Conn, keeper)
go forgettingOldHeals(watching, open.inventory)
// And the build queue's order (novox/hq issue 558): a check that has waited its while goes ahead of the
// deliveries asked after it even when nothing new is asked.
go keepingTheQueueInOrder(watching, open, server.JetStream())
fmt.Printf("watching the mesh: %d signal(s) every %s, %d probe(s) every %s; what is wrong is kept in %s "+
"and said as %s events\n", len(watchedRows()), watchEvery, len(runnableProbes()), doctorEvery,
broker.ConditionsBucket, conditions.Seat)
+1 -1
View File
@@ -36,4 +36,4 @@ require (
// committed. Every build (the build agent's `go build`, the Dockerfile) compiles from vendor/ and
// fetches nothing; go refuses to build when vendor/ and this file disagree, so a pin moved without
// `go mod vendor` fails loudly, at once, everywhere.
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b h1:PGdZJHUU9rCDucdrZIqSs4DGyxlo4lokvwmofpGk9RI=
git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b/go.mod h1:tcTK4LMs1d6JpZUwy3hSHMFG/MIuDr/XFs7/nbeaW/c=
git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f h1:8N5OW2mdTNIck2pe4EciTYX5NsrPsHrTLENGNIWYNTU=
git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f/go.mod h1:tcTK4LMs1d6JpZUwy3hSHMFG/MIuDr/XFs7/nbeaW/c=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c h1:l5onJwoIeH8yE/PjVlEeoPyXBsHp2NQiWLllz2fwX7s=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
+1 -1
View File
@@ -82,7 +82,7 @@ func theCarriedAccounts(t *testing.T) string {
// the commit the mesh runs, or as captured in testdata/beside (internal/beside, novox/hq issue 432).
func theTemplate(t *testing.T) []map[string]any {
t.Helper()
path := filepath.Join(beside.Dir(t, "mesh-host"), "examples", "foundation-first-node.lock")
path := filepath.Join(beside.Dir(t, "mesh-host"), "examples", "foundation-first-node-nats.lock")
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
+1 -8
View File
@@ -343,11 +343,6 @@ func AskJudgeDirectoriesSubject(node string) string {
return "mesh.node." + node + ".ask.judge-directories"
}
// AskAccessActSubject is where the controller asks one machine's node-engine to take a shared folder on the
// operator's warrant (novox/hq ADR 0325 §8; mesh-host internal/link AskAccessActSubject): a request answered once,
// signed as a hand-over is (link.AccessActContext), for the same reason.
func AskAccessActSubject(node string) string { return "mesh.node." + node + ".ask.access-act" }
// inbox is a principal's own reply space. No user is ever granted a bare `_INBOX.>` (design 25
// §4): with one account, inbox privacy is the permission list or it is nothing, so each user's
// inbox is derived from its own identity and its permissions name that prefix and no other.
@@ -628,9 +623,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
// And asking it for a fresh search for setuid programs (novox/hq issue 361), answered the same way.
AskSetuidSearchSubject(p.Node),
// And asking it to judge its declared directories, read-only (novox/hq issue 358), answered the same way.
AskJudgeDirectoriesSubject(p.Node),
// And asking it to take a shared folder on the operator's warrant (novox/hq ADR 0325 §8), answered the same way.
AskAccessActSubject(p.Node)}
AskJudgeDirectoriesSubject(p.Node)}
// The node-engine witnesses the core builds it places (novox/hq to-be 45 §8, ADR 0236; the
// contract is lease/witness.go): it asks its own machine's node tools PING, and, where the
// machine runs the controller, reads the lease's one key — read, never written.
+1 -1
View File
@@ -34,7 +34,7 @@ accounts {
} }
{ user: "node.one", password: "$2a$11$nnnnnnnnnnnnnnnnnnnnnn", permissions: {
publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.INFO.NODES.one", "$SRV.PING.node-tools.one", "mesh.control.one.>"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.access-act", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.judge-directories", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.judge-directories", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "one.nats", password: "$2a$11$bbbbbbbbbbbbbbbbbbbbbb", permissions: {
-5
View File
@@ -101,11 +101,6 @@ var WritersTable = []WriterRow{
{State: "a judgement of a machine's directories asked of it", Writer: "controller",
KeptIn: "nowhere: the engine answers it and keeps nothing", Others: "the engine verifies the mesh's signature and answers, or refuses",
Subjects: []string{"mesh.node.*.ask.judge-directories"}, Writes: isController},
// The take of a shared folder asked of a machine's engine on the operator's warrant (novox/hq ADR 0325 §8):
// signed as a hand-over is, under a signing context of its own.
{State: "the take of a shared folder asked of a machine", Writer: "controller, on the operator's warrant",
KeptIn: "the machine, which records the act", Others: "the engine verifies the mesh's signature and takes it, or refuses",
Subjects: []string{"mesh.node.*.ask.access-act"}, Writes: isController},
{State: "a machine's applied state and its report", Writer: "the node-engine's apply queue",
KeptIn: "the machine; the report on the bus", Others: "the reconcile and a delivery enqueue, never apply",
// And its health statement between reports (novox/hq ADR 0240): the same writer stating the same
-1
View File
@@ -18,7 +18,6 @@ var designRows = []string{
"a hand-over asked of a machine",
"a fresh setuid search asked of a machine",
"a judgement of a machine's directories asked of it",
"the take of a shared folder asked of a machine",
"a machine's applied state and its report",
"the controller lease",
"plans and their tiers",
+2 -4
View File
@@ -939,16 +939,14 @@ func replaceGoFiles(from, into string) error {
}
// problemLines are the lines of a module check's output that name a problem: not a manifest found ok, not
// the count, not the closing note, not a warning.
// the count, not the closing note.
func problemLines(s string) []string {
var out []string
for _, line := range strings.Split(s, "\n") {
line = strings.TrimSpace(line)
switch {
case line == "", strings.Contains(line, ": ok"), strings.Contains(line, "problem(s) in"),
strings.Contains(line, "manifest(s) checked"),
// A warning is said and fails nothing (novox/hq ADR 0325: what is refused once the catalogues moved).
strings.HasPrefix(line, "warning: "):
strings.Contains(line, "manifest(s) checked"):
continue
}
out = append(out, line)
+35 -23
View File
@@ -26,21 +26,10 @@ import (
// serves — whose keys are agreed by the requirement's name, not by this file.
// bound is where a module says a value from one of its bindings belongs:
// ${bound:<provision>.<key>}.
var bound = regexp.MustCompile(`\$\{bound:([a-z0-9][a-z0-9.-]*[a-z0-9]):([a-z0-9][a-z0-9_-]*)\}`)
// boundUsed are the (provision, key) pairs a file's content asks for, first appearance first.
func boundUsed(content string) [][2]string {
var used [][2]string
seen := map[string]bool{}
for _, m := range bound.FindAllStringSubmatch(content, -1) {
if key := m[1] + ":" + m[2]; !seen[key] {
seen[key] = true
used = append(used, [2]string{m[1], m[2]})
}
}
return used
}
// ${bound:<provision>:<key>}, and for an optional requirement the value it takes while absent,
// ${bound:<provision>:<key>|default:<value>} (novox/hq ADR 0330 §4). A default is never empty: an
// absent requirement is never rendered as an empty string.
var bound = regexp.MustCompile(`\$\{bound:([a-z0-9][a-z0-9.-]*[a-z0-9]):([a-z0-9][a-z0-9_-]*)(?:\|default:([^}\n]+))?\}`)
// knownFor is everything a module may name from one of its bindings.
//
@@ -125,7 +114,11 @@ func plainly(value any) string {
// serve, is refused. Left as it was, the literal `${bound:x:y}` would be written into a
// configuration file and read as a value — a connection to a host called `${bound:x:y}`, failing
// somewhere that names neither the module nor the mesh.
func boundInto(resource map[string]any, known map[string]map[string]string, module string) error {
//
// **An optional requirement left absent takes the default its placeholder names** (novox/hq ADR
// 0330 §4), and one that names none is refused rather than rendered empty. The module check refuses
// that already; this is the same rule where the file is made, for a manifest registered before it.
func boundInto(resource map[string]any, known map[string]map[string]string, module string, absent map[string]bool) error {
if fmt.Sprint(resource["type"]) != "file" {
return nil
}
@@ -133,24 +126,43 @@ func boundInto(resource map[string]any, known map[string]map[string]string, modu
if !ok {
return nil
}
for _, pair := range boundUsed(content) {
provision, key := pair[0], pair[1]
var refused error
filled := bound.ReplaceAllStringFunc(content, func(token string) string {
if refused != nil {
return token
}
m := bound.FindStringSubmatch(token)
provision, key, fallback := m[1], m[2], m[3]
values, has := known[provision]
if !has {
return fmt.Errorf(
if absent[provision] {
if fallback != "" {
return fallback
}
refused = fmt.Errorf(
"%s has a file that says ${bound:%s:%s}, and %q is optional and absent here: it names no "+
"value for then (${bound:%s:%s|default:<value>}), so it would be written empty "+
"(novox/hq ADR 0330 §4)", module, provision, key, provision, provision, key)
return token
}
refused = fmt.Errorf(
"%s has a file that says ${bound:%s:%s}, and %s does not require %q. It may name %s",
module, provision, key, module, provision, orNothing(namesOfBindings(known)))
return token
}
value, said := values[key]
if !said {
return fmt.Errorf(
refused = fmt.Errorf(
"%s asks its %s binding for %q, and what answers it says %s",
module, provision, key, orNothing(namesOfKeys(values)))
return token
}
resource["content"] = strings.ReplaceAll(
content, fmt.Sprintf("${bound:%s:%s}", provision, key), value)
content = resource["content"].(string)
return value
})
if refused != nil {
return refused
}
resource["content"] = filled
return nil
}
+11 -69
View File
@@ -246,25 +246,6 @@ type Rendering struct {
// statement's contract is link.RootContract or later). To an older, strict engine the field is not
// sent, and the account it names is not judged — which the self-check says, as not judged.
JudgesRoot bool
// KeepsLibraries says this machine's node-engine makes the mesh's users and library groups and keeps libraries
// (novox/hq ADR 0328: its statement's contract is link.LibraryContract or later). To an older, strict engine none
// of it is sent, and a module that joins is left out of its declaration, naming why.
KeepsLibraries bool
// Identities is each joining module's allocated number, by module (ADR 0328 §4): the same on every machine. A
// joining module with none here is left out, as at a question asked before its first send.
Identities map[string]ModuleIdentity
// Libraries is every library the operator defined, by name (§1); LibraryGroups each one's allocated group number
// (§3); TakenLibraries those whose data the operator let the mesh take on their home, on the filesystem it is on
// now (§7); LibraryPaths every sub-path any module in the mesh asks of each, kept on its home (§6).
Libraries map[string]Library
LibraryGroups map[string]int
TakenLibraries map[string]bool
LibraryPaths map[string][]string
// RunnerJoinsOwnGroups says this machine's engine puts the tool runner's account in a joining module's own group
// and takes a directory's setgid mode (novox/hq ADR 0334 §1, §2: its statement's contract is link.OwnGroupContract
// or later); to an older one neither the identity's runner nor the setgid bit is sent.
RunnerJoinsOwnGroups bool
}
// RootField is a user resource's field saying the account must never become root without a person
@@ -339,11 +320,6 @@ type Composed struct {
// naming its module and why (novox/hq ADR 0255): that piece is left out and the machine is told
// everything else. Push and plan say it; the merge gate refuses a change that adds one.
Unplaced []string
// Groups, Identities and Libraries are what the declaration says of libraries and joining modules (novox/hq ADR
// 0328): composed only for an engine that keeps libraries, and sent only when not empty.
Groups []GroupEntry
Identities []IdentityEntry
Libraries []LibraryEntry
}
// ForeseenSealed is what stands for an own secret a send will make, in a composition asked ahead of
@@ -382,9 +358,7 @@ func (r Resolution) LeftOutBecause(settings SettingsBy, adopted bool) map[string
out[m.Module] = &UnreadManifestError{Module: m.Module, said: why}
continue
}
// A placement stored ahead of the build that declares its access is passed by, not a reason to leave the
// module out (novox/hq issue 575): it was judged against that build where it was stored.
if err := ComposesWith(m, settings[m.Module], adopted); err != nil {
if err := JudgeSettings(m, settings[m.Module], adopted); err != nil {
out[m.Module] = err
}
}
@@ -406,8 +380,7 @@ func (r Resolution) Compose(with Rendering) (Composed, error) {
received := map[string]map[string][]Contribution{}
leftOut := map[string]string{}
var foreseen, unplaced []string
var joined []Joined
resources, err := r.compose(with, owner, received, leftOut, &foreseen, &unplaced, &joined)
resources, err := r.compose(with, owner, received, leftOut, &foreseen, &unplaced)
if err != nil {
return Composed{}, err
}
@@ -421,10 +394,8 @@ func (r Resolution) Compose(with Rendering) (Composed, error) {
})
}
sort.Strings(unplaced)
composed := Composed{Resources: resources, Owner: owner, Received: received, LeftOut: leftOut,
Foreseen: foreseen, Unplaced: unplaced}
composed.Groups, composed.Identities, composed.Libraries = r.libraryEnvelope(with, joined)
return composed, nil
return Composed{Resources: resources, Owner: owner, Received: received, LeftOut: leftOut,
Foreseen: foreseen, Unplaced: unplaced}, nil
}
// BusMembershipID names the resource carrying a machine's membership for the new bus, and
@@ -435,26 +406,13 @@ const BusMembershipPath = "/var/lib/mesh/membership-next.json"
func (r Resolution) compose(with Rendering, owner map[string]string,
received map[string]map[string][]Contribution, leftOut map[string]string,
foreseen, unplaced *[]string, joinedOut *[]Joined) ([]map[string]any, error) {
foreseen, unplaced *[]string) ([]map[string]any, error) {
// **A setting is judged where it is stored, and an impossible one costs a module, not a
// machine** (novox/hq ADR 0163, rule 6). A definition that moved under a stored setting makes
// this module uncomposable; it is left out of the declaration — its held things kept, its
// containers untouched, the machine told so by name — and the machine is told everything else.
// Before placing, because a placement is a setting too.
left := r.LeftOut(with.Settings, with.Adopted)
// And a module whose libraries cannot be composed here, or that waits for one to be taken (novox/hq ADR 0328 §2,
// §8): left out, naming why, its containers untouched — so a container keeps the user it ran as.
for _, m := range r.Modules {
if _, already := left[m.Module]; already {
continue
}
if why := libraryLeftOut(m, r.Node, with); why != "" {
if left == nil {
left = map[string]string{}
}
left[m.Module] = why
}
}
kept := make([]Manifest, 0, len(r.Modules))
var stillBackedUp []Manifest
for _, m := range r.Modules {
@@ -595,20 +553,6 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
}
resources := m.Resources
// Its identity, where it joins (novox/hq ADR 0328 §4): its user and groups, and where its libraries are here.
var joined Joined
joins := Joins(m)
var libPaths map[string]string
if joins {
if joined, err = joinedOf(m, with); err != nil {
return nil, err
}
libPaths = libraryPaths(m, with)
if joinedOut != nil {
*joinedOut = append(*joinedOut, joined)
}
}
// What the mesh computes for this module goes FIRST, before the module's own resources.
//
// **Order is stated, not derived — the host does not sort** (novox/hq ADR 0005), so
@@ -906,6 +850,11 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// An account depends on nothing the mesh computes.
accounts, rest := accountsFirst(resources)
resources = append(append(accounts, first...), rest...)
// **A file delivered only while its optional requirement is bound is left out while it is
// absent** (novox/hq ADR 0330 §5), and the field is dropped from the rest: the node-engine
// parses strictly and knows no such field.
absent := r.AbsentFor(m.Module)
resources = withoutAbsent(resources, absent)
// No container is given the mesh's names (novox/hq ADR 0148). It used to be: every
// container got the whole roster as `--add-host` entries at creation, and a name that
@@ -1059,13 +1008,6 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
return nil, err
}
ownerInto(copied, placed)
// And its identity and libraries, where it joins (novox/hq ADR 0328 §4, §5, §10): after the placement's
// owner, which a joining module may not set.
if joins {
if err := joinedInto(copied, m, joined, libPaths, with.RunnerJoinsOwnGroups); err != nil {
return nil, err
}
}
// **After settings, and that is the whole reason it is here.** A module's file
// content is where a setting lands, so a placeholder may only exist once the setting
// has been put in — filling secrets first would look at content that is not yet what
@@ -1076,7 +1018,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// **Substituted here, not on the machine.** A bound value is not secret — the mesh
// holds it in the clear — so there is nothing for the host to be the only witness of,
// and sending it already filled in means the host learns no new field.
if err := boundInto(copied, known, m.Module); err != nil {
if err := boundInto(copied, known, m.Module, absent); err != nil {
return nil, err
}
// And what the module could not have written: the machine it turned out to be
-907
View File
@@ -1,907 +0,0 @@
package catalogue
import (
"fmt"
"path/filepath"
"regexp"
"sort"
"strings"
)
// Libraries, and the modules that join them (novox/hq ADR 0328).
//
// **A library is a named tree of shared data with one home** (§1). The operator defines it — its name, the machine it
// lives on, its root there and the machines it reaches — as the mesh-wide `libraries` setting of the controller's own
// module, a trusted setting set at the controller's terminal or on the operator's Approve of a proposal (ADR 0277),
// never through a verb alone. Libraries never overlap on a machine (§13), and a library never shares a name with a
// module (§4), so `mesh-<library>` and `mesh-<module>` never collide.
//
// **A module asks for a library by name, sub-path and use** (§2): a manifest's `libraries` entries {id, library,
// path, use}. On each machine an entry resolves to the library's root joined with the path; a module assigned to a
// machine the library does not reach is left out there, naming the library. `${library:<id>}` is that host path, and
// is filled only where a host path belongs (§5): a container's volume source, and a word of the module's tools, which
// run on the machine. A `read` entry is mounted read-only (§10).
//
// **A module joins by using the model** (§14): it declares `libraries` or uses `${identity:…}`. It is given one number
// from the mesh's range, as its user's and its own group's (§4); its user's primary group is the group of the first
// library it uses, the others are added groups, and with no library its own group is primary. `${identity:user}`,
// `${identity:uid}`, `${identity:group}` and `${identity:gid}` are filled wherever its resources take values (§5), and
// its directories are owned by its user and primary group (§4). A module that does neither is composed exactly as
// before.
//
// **The range is 50000–59999** for every number the mesh allocates — a joining module's and a library's group — from
// one sequence, so a module's own group and a library's never share a number. It lies above the numbers distributions
// give people and below the 60000s Debian keeps for its daemons, and nowhere near root's 0 or a root group's (0, adm's 4,
// wheel's 10): no allocated number is root's (§15). The store refuses a row outside it.
//
// **Only to an engine that reads it** (Rendering.KeepsLibraries: its statement's contract is link.LibraryContract or
// later), the declaration carries `groups`, `identities` and `libraries` (libraryEnvelope). To an older engine a module
// that joins is left out of the declaration, naming why — its placeholders could not be filled — and everything else is
// composed as before.
// The range the mesh allocates user and group numbers from (novox/hq ADR 0328 §3, §4, §15). The store holds the same
// bounds.
const (
IdentityLow = 50000
IdentityHigh = 59999
)
// AllocatableID says whether n is a number the mesh may give: inside the range, so never 0 nor a root group's.
func AllocatableID(n int) bool { return n >= IdentityLow && n <= IdentityHigh }
// ADR0328 is how the checks name the decision their rules come from.
const ADR0328 = "novox/hq ADR 0328"
// MeshName is the user or group a module or a library is on every machine: mesh-<name>.
func MeshName(name string) string { return "mesh-" + name }
// IdentityName is the user a joining module runs as, and its own group.
func IdentityName(module string) string { return MeshName(module) }
// ---- libraries, as the operator defines them -----------------------------------------------------------------
// LibrariesSetting is the key of the controller's mesh-wide settings layer that defines the libraries (§1).
const LibrariesSetting = "libraries"
// LibrariesModule is the module whose mesh-wide layer holds LibrariesSetting: the controller's own.
const LibrariesModule = "mesh-controller"
// Library is one library as the operator defined it.
type Library struct {
Name string `json:"-"`
// Home is the machine the data lives on; Root its path there, the same on every machine it reaches.
Home string `json:"home"`
Root string `json:"root"`
// Reach is every machine the library reaches, its home among them.
Reach []string `json:"reach,omitempty"`
}
// Reaches says whether the library reaches a machine: its home, or one it names.
func (l Library) Reaches(node string) bool { return node == l.Home || hasName(l.Reach, node) }
// Machines is every machine the library reaches, its home first.
func (l Library) Machines() []string {
out := []string{l.Home}
for _, n := range l.Reach {
if n != l.Home && !hasName(out, n) {
out = append(out, n)
}
}
return out
}
// Group is the library's group: mesh-<library>.
func (l Library) Group() string { return MeshName(l.Name) }
var libraryName = regexp.MustCompile(`^[a-z][a-z0-9-]*$`)
// ParseLibraries reads the `libraries` setting: {<name>: {home, root, reach}}, or refuses it in words.
func ParseLibraries(v any) (map[string]Library, error) {
if v == nil {
return nil, nil
}
blocks, ok := v.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s is a { <library>: {home, root, reach} } map (%s §1)", LibrariesSetting, ADR0328)
}
out := map[string]Library{}
for name, raw := range blocks {
if !libraryName.MatchString(name) {
return nil, fmt.Errorf("the library %q: a library's name is lowercase letters, digits and dashes, starting "+
"with a letter", name)
}
fields, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("the library %s is {home, root, reach}, not %v", name, raw)
}
l := Library{Name: name}
for k, v := range fields {
switch k {
case "home":
l.Home, _ = v.(string)
case "root":
l.Root, _ = v.(string)
case "reach":
list, ok := v.([]any)
if !ok {
return nil, fmt.Errorf("the library %s's reach is a list of machines", name)
}
for _, n := range list {
s, _ := n.(string)
if s == "" {
return nil, fmt.Errorf("the library %s's reach names a machine that is not a name: %v", name, n)
}
l.Reach = append(l.Reach, s)
}
default:
return nil, fmt.Errorf("the library %s says %q; a library is its home, its root and its reach, and "+
"nothing about an owner, a group, a mode or a number (%s §1)", name, k, ADR0328)
}
}
if strings.TrimSpace(l.Home) == "" {
return nil, fmt.Errorf("the library %s names no home machine", name)
}
if !strings.HasPrefix(l.Root, "/") {
return nil, fmt.Errorf("the library %s's root %q is not an absolute path", name, l.Root)
}
l.Root = filepath.Clean(l.Root)
if why := systemPath(l.Root); why != "" {
return nil, fmt.Errorf("the library %s's root is %s: %s, and the node-engine keeps a library's root in its "+
"group — no library is ever there", name, l.Root, why)
}
sort.Strings(l.Reach)
out[name] = l
}
return out, nil
}
// LibrariesProblems refuses what no library may be together (§4, §13): a library named as a registered module, and two
// libraries on one machine whose roots lie inside each other — each named.
func LibrariesProblems(libs map[string]Library, modules []string) []string {
var problems []string
names := make([]string, 0, len(libs))
for n := range libs {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
if hasName(modules, n) {
problems = append(problems, fmt.Sprintf("the library %s has the name of the module %s: %s would be both "+
"(%s §4)", n, n, MeshName(n), ADR0328))
}
}
for i, a := range names {
for _, b := range names[i+1:] {
la, lb := libs[a], libs[b]
shared := ""
for _, m := range la.Machines() {
if lb.Reaches(m) {
shared = m
break
}
}
if shared == "" {
continue
}
if within(la.Root, lb.Root) || within(lb.Root, la.Root) {
problems = append(problems, fmt.Sprintf("the libraries %s (%s) and %s (%s) overlap on %s: a library's "+
"root lies neither inside another's nor around it — divide one library by the paths its modules "+
"ask for (%s §13)", a, la.Root, b, lb.Root, shared, ADR0328))
}
}
}
return problems
}
// within says path is root or lies below it.
func within(path, root string) bool {
return path == root || strings.HasPrefix(path, strings.TrimSuffix(root, "/")+"/")
}
func hasName(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
// ---- a module's use of them ---------------------------------------------------------------------------------
// LibraryUse is one `libraries` entry of a manifest: the id the module's resources name it by, the library, the
// sub-path within it, and the use.
type LibraryUse struct {
ID string `json:"id"`
Library string `json:"library"`
Path string `json:"path"`
Use string `json:"use"`
}
// At is the entry's use, read by default.
func (u LibraryUse) At() string {
if u.Use == "" {
return AccessRead
}
return u.Use
}
// libraryRef is how a module names one of its library entries: ${library:<id>}.
var libraryRef = regexp.MustCompile(`\$\{library:([a-z0-9][a-z0-9-]*)\}`)
// identityRef is a module's identity by name: ${identity:user|uid|group|gid}.
var identityRef = regexp.MustCompile(`\$\{identity:(user|uid|group|gid)\}`)
// Joins says a module joins the model (§14): it declares libraries or uses its identity anywhere.
func Joins(m Manifest) bool {
if len(m.Libraries) > 0 {
return true
}
found := false
for _, r := range m.Resources {
walkStrings(r, func(s string) {
found = found || identityRef.MatchString(s) || strings.Contains(s, "${identity:")
})
}
for _, b := range m.Bundles {
for _, v := range b.Env {
found = found || strings.Contains(v, "${identity:")
}
}
return found
}
// walkStrings calls see with every string a value holds, however deep.
func walkStrings(v any, see func(string)) {
switch v := v.(type) {
case string:
see(v)
case []any:
for _, x := range v {
walkStrings(x, see)
}
case map[string]any:
for _, x := range v {
walkStrings(x, see)
}
}
}
// mapStrings is v with every string it holds replaced by fill's answer, however deep; the first error stops it.
func mapStrings(v any, fill func(string) (string, error)) (any, error) {
switch v := v.(type) {
case string:
return fill(v)
case []any:
out := make([]any, len(v))
for i, x := range v {
y, err := mapStrings(x, fill)
if err != nil {
return nil, err
}
out[i] = y
}
return out, nil
case map[string]any:
out := make(map[string]any, len(v))
for k, x := range v {
y, err := mapStrings(x, fill)
if err != nil {
return nil, err
}
out[k] = y
}
return out, nil
}
return v, nil
}
var libraryEntryID = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// numberWords are the setting names that carry a user or group number or an owner: none for a module that joins (§5).
// "user" and "group" are left out: an application's own login or group name is a setting like any other.
var numberWords = map[string]bool{"uid": true, "gid": true, "puid": true, "pgid": true, "user-id": true,
"group-id": true, "owner": true}
// libraryProblems is what the module check refuses of a manifest's libraries and identity (§2, §4, §5, §14):
//
// - an entry that is not {id, library, path, use} with a relative sub-path inside its library, or names a host
// path, an owner, a group, a mode or a number (the entry's shape refuses any other key);
// - two entries with one id, or a `${library:<id>}` naming none;
// - `${library:<id>}` anywhere but a container's volume source (and a word of its tools, a bundle's env);
// - `${identity:…}` other than user, uid, group or gid;
// - in a module that joins: a directory naming its owner. A setting of it naming a number or an owner is refused
// where it is set (JoiningSettingProblem).
func libraryProblems(m Manifest) []string {
var problems []string
ids := map[string]bool{}
for _, u := range m.Libraries {
where := fmt.Sprintf("%s's library entry %q", m.Module, u.ID)
switch {
case !libraryEntryID.MatchString(u.ID):
problems = append(problems, where+": an id is lowercase letters, digits and dashes")
case ids[u.ID]:
problems = append(problems, where+": two entries share the id")
}
ids[u.ID] = true
if !libraryName.MatchString(u.Library) {
problems = append(problems, fmt.Sprintf("%s names the library %q, which is not a library's name", where,
u.Library))
}
clean := filepath.Clean(u.Path)
if u.Path == "" || strings.HasPrefix(u.Path, "/") || clean == ".." || strings.HasPrefix(clean, "../") ||
strings.ContainsAny(u.Path, "${}") {
problems = append(problems, fmt.Sprintf("%s's path %q is a sub-path inside the library, relative and "+
"staying inside it — never a host path (%s §2)", where, u.Path, ADR0328))
}
if u.At() != AccessRead && u.At() != AccessReadWrite {
problems = append(problems, fmt.Sprintf("%s is used %q; a use is %q or %q", where, u.Use, AccessRead,
AccessReadWrite))
}
}
for _, r := range m.Resources {
kind := fmt.Sprint(r["type"])
for field, v := range r {
walkStrings(v, func(s string) {
for _, tok := range regexp.MustCompile(`\$\{identity:[^}]*\}`).FindAllString(s, -1) {
if !identityRef.MatchString(tok) {
problems = append(problems, fmt.Sprintf("%s's resource %v says %s: the identity is "+
"${identity:user}, ${identity:uid}, ${identity:group} or ${identity:gid} (%s §5)", m.Module,
r["id"], tok, ADR0328))
}
}
for _, match := range libraryRef.FindAllStringSubmatch(s, -1) {
if !ids[match[1]] {
problems = append(problems, fmt.Sprintf("%s's resource %v says ${library:%s}, and declares no "+
"library entry %q", m.Module, r["id"], match[1], match[1]))
}
source := ""
if kind == "container" && field == "volumes" {
source = volumeSource(s)
}
if !strings.Contains(source, match[0]) {
problems = append(problems, fmt.Sprintf("%s's resource %v says %s in its %s: a library's path "+
"is a host path, filled only in a container's volume source — the process reads the "+
"container side its volume gives (%s §5)", m.Module, r["id"], match[0], field, ADR0328))
}
}
})
}
}
if !Joins(m) {
return problems
}
problems = append(problems, toolsDirectoryProblems(m)...)
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
if owner, said := r["owner"]; said {
problems = append(problems, fmt.Sprintf("%s: the directory %v names its owner %v; a module that joins "+
"owns its directories by its user and primary group, and names no owner (%s §4)", m.Module,
r["id"], owner, ADR0328))
}
}
}
return problems
}
// toolsDirectoryProblems refuses a joining module whose tools name one of its directories whose mode closes it to its
// group (novox/hq ADR 0334 §3): the tool runner reads it through the module's own group, so it needs group read and
// enter.
func toolsDirectoryProblems(m Manifest) []string {
modes := map[string]string{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
mode, _ := r["mode"].(string)
modes[fmt.Sprint(r["id"])] = mode
}
}
var words []string
for _, b := range m.Bundles {
for _, v := range b.Env {
words = append(words, v)
}
}
if m.Build != nil {
for _, a := range m.Build.Artifacts {
for _, v := range a.Env {
words = append(words, v)
}
}
}
seen := map[string]bool{}
var problems []string
for _, w := range words {
for _, match := range dirRef.FindAllStringSubmatch(w, -1) {
id := match[1]
mode, declared := modes[id]
if !declared || seen[id] {
continue
}
seen[id] = true
var n uint64
if _, err := fmt.Sscanf(mode, "%o", &n); err != nil || mode == "" {
continue // its mode is judged where the directory is
}
if n&0o050 != 0o050 {
problems = append(problems, fmt.Sprintf("%s's tools name its directory %s, whose mode %s closes it to its "+
"own group: the tool runner reads it through that group, so it needs group read and enter (0750; "+
"the mesh adds setgid, 2750) (novox/hq ADR 0334 §3)", m.Module, id, mode))
}
}
}
sort.Strings(problems)
return problems
}
// LibraryWarnings is what the module check warns of a module's libraries (§4): an image that drops added groups
// reaches only the first library the module uses.
func LibraryWarnings(m Manifest) []string {
seen := map[string]bool{}
var libs []string
for _, u := range m.Libraries {
if !seen[u.Library] {
seen[u.Library] = true
libs = append(libs, u.Library)
}
}
if len(libs) < 2 {
return nil
}
return []string{fmt.Sprintf("%s uses the libraries %s: its user's primary group is %s, and %s are added groups — an "+
"image that drops added groups when it changes user reaches only %s (%s §4)", m.Module, strings.Join(libs, ", "),
MeshName(libs[0]), strings.Join(meshNames(libs[1:]), ", "), libs[0], ADR0328)}
}
func meshNames(names []string) []string {
out := make([]string, len(names))
for i, n := range names {
out[i] = MeshName(n)
}
return out
}
// JoiningSettingProblem refuses, where a setting is set, a layer of a module that joins that carries a user or group
// number or an owner (§5): a key whose name says so, or an owner a `places` setting names — its directories are owned
// by its user and primary group, and it reads its own as ${identity:…}.
func JoiningSettingProblem(m Manifest, layers []Layer) error {
if !Joins(m) {
return nil
}
for _, l := range layers {
for k := range l.Values {
if numberWords[k] {
return fmt.Errorf("%s joins, and the layer %q sets %q: no setting of a module that joins carries a user "+
"or group number or an owner — it reads its own as ${identity:…} (%s §5)", m.Module, l.From, k, ADR0328)
}
}
}
return PlacesOwnerProblem(m, layers)
}
// PlacesOwnerProblem refuses an owner a `places` setting names for a module that joins (§4, §5): its directories are
// owned by its user and primary group.
func PlacesOwnerProblem(m Manifest, layers []Layer) error {
if !Joins(m) {
return nil
}
placed, err := Places(m, layers)
if err != nil {
return nil // refused by name where it is judged
}
var owned []string
for id, p := range placed {
if p.Owner != "" {
owned = append(owned, id)
}
}
if len(owned) == 0 {
return nil
}
sort.Strings(owned)
return fmt.Errorf("%s places %s with an owner, and it joins: its directories are owned by its user and primary "+
"group, and no setting of it carries an owner (%s §4, §5)", m.Module, strings.Join(owned, ", "), ADR0328)
}
// ---- composition ------------------------------------------------------------------------------------------------
// ModuleIdentity is a joining module's allocated number: its user's and its own group's.
type ModuleIdentity struct {
Module string
UID int
}
// Joined is a joining module's identity on a machine, as its placeholders are filled and its directories owned.
type Joined struct {
Module string
UID int
// Primary is its user's primary group, by name and number; Added its other groups, by name, in the order used.
Primary string
PrimaryGID int
Added []string
AddedGIDs []int
}
// librariesOf is the libraries a module uses, by name, in the order its entries first name them.
func librariesOf(m Manifest) []string {
var out []string
for _, u := range m.Libraries {
if !hasName(out, u.Library) {
out = append(out, u.Library)
}
}
return out
}
// joinedOf is module m's identity here: its number, and its groups from the libraries it uses. Refused, naming
// what, where a number is missing or outside the range (§15).
func joinedOf(m Manifest, with Rendering) (Joined, error) {
id, has := with.Identities[m.Module]
if !has {
return Joined{}, fmt.Errorf("%s joins, and has no number yet: it is allocated at its first send", m.Module)
}
if !AllocatableID(id.UID) {
return Joined{}, fmt.Errorf("%s's number is %d, outside %d–%d: no number the mesh gives is root's or a root "+
"group's, and none is sent (%s §15)", m.Module, id.UID, IdentityLow, IdentityHigh, ADR0328)
}
j := Joined{Module: m.Module, UID: id.UID, Primary: IdentityName(m.Module), PrimaryGID: id.UID}
for i, lib := range librariesOf(m) {
gid, has := with.LibraryGroups[lib]
if !has || !AllocatableID(gid) {
return Joined{}, fmt.Errorf("%s uses the library %s, whose group has no number the mesh may give (%s §3, "+
"§15)", m.Module, lib, ADR0328)
}
if i == 0 {
j.Primary, j.PrimaryGID = MeshName(lib), gid
continue
}
j.Added = append(j.Added, MeshName(lib))
j.AddedGIDs = append(j.AddedGIDs, gid)
}
return j, nil
}
// libraryLeftOut is why module m is left out of this machine's declaration for its libraries, or "": a library it
// uses not defined or not reaching the machine (§2), or an engine that cannot be told it joins. A module held at the
// user it ran as until its libraries are taken (§8) is not left out: it is composed at the build it ran before it
// joined (cmd/mesh-controller holdJoining).
func libraryLeftOut(m Manifest, node string, with Rendering) string {
if !Joins(m) {
return ""
}
if !with.KeepsLibraries {
return fmt.Sprintf("%s joins the mesh's users and libraries (%s), and the node-engine on %s is older than "+
"them: it is left out until that engine is updated", m.Module, ADR0328, node)
}
for _, lib := range librariesOf(m) {
l, defined := with.Libraries[lib]
switch {
case !defined:
return fmt.Sprintf("%s uses the library %s, which is not defined: the operator defines it (%s §1)",
m.Module, lib, ADR0328)
case !l.Reaches(node):
return fmt.Sprintf("%s uses the library %s, which does not reach %s (%s §2)", m.Module, lib, node, ADR0328)
}
}
if _, has := with.Identities[m.Module]; !has {
return fmt.Sprintf("%s joins, and its number is allocated at its first send", m.Module)
}
return ""
}
// libraryPaths is where each of module m's library entries is on this machine: its library's root joined with the
// entry's path, by entry id.
func libraryPaths(m Manifest, with Rendering) map[string]string {
out := map[string]string{}
for _, u := range m.Libraries {
if l, ok := with.Libraries[u.Library]; ok {
out[u.ID] = filepath.Join(l.Root, filepath.Clean(u.Path))
}
}
return out
}
// identityFill fills ${identity:…} in one string.
func identityFill(s string, j Joined) string {
return identityRef.ReplaceAllStringFunc(s, func(tok string) string {
switch identityRef.FindStringSubmatch(tok)[1] {
case "user":
return IdentityName(j.Module)
case "uid":
return fmt.Sprint(j.UID)
case "group":
return j.Primary
default:
return fmt.Sprint(j.PrimaryGID)
}
})
}
// joinedInto composes one resource of a joining module (§4, §5, §10): ${identity:…} filled everywhere, a container's
// ${library:<id>} volume sources filled with the host path and a read entry mounted read-only, and a directory owned by
// the module's user and primary group.
func joinedInto(resource map[string]any, m Manifest, j Joined, paths map[string]string, setgid bool) error {
for k, v := range resource {
if k == "id" || k == "type" {
continue
}
filled, err := mapStrings(v, func(s string) (string, error) { return identityFill(s, j), nil })
if err != nil {
return err
}
resource[k] = filled
}
// Its own directories belong to its user and its own group, with setgid (ADR 0334 §1, v4): the library's group is
// its processes' primary, so what they make in a library keeps that group — and what they make in their own
// directories keeps mesh-<module> by the setgid bit. The bit only to an engine that takes a mode with it
// (Rendering.RunnerJoinsOwnGroups, link.OwnGroupContract): the one pinned refuses a mode not starting with 0.
if fmt.Sprint(resource["type"]) == "directory" {
resource["owner"] = fmt.Sprintf("%d:%d", j.UID, j.UID)
if mode, ok := resource["mode"].(string); ok && setgid {
var n uint64
if _, err := fmt.Sscanf(mode, "%o", &n); err == nil && mode != "" {
resource["mode"] = fmt.Sprintf("%04o", n|0o2000)
}
}
}
if fmt.Sprint(resource["type"]) != "container" {
return nil
}
volumes, _ := resource["volumes"].([]any)
out := make([]any, len(volumes))
for i, v := range volumes {
out[i] = v
s, ok := v.(string)
if !ok {
continue
}
source := volumeSource(s)
match := libraryRef.FindStringSubmatch(source)
if match == nil {
continue
}
path, has := paths[match[1]]
if !has {
return fmt.Errorf("%s says ${library:%s}, and has no library entry %q", m.Module, match[1], match[1])
}
filled := strings.Replace(source, match[0], path, 1) + strings.TrimPrefix(s, source)
for _, u := range m.Libraries {
if u.ID == match[1] && u.At() == AccessRead {
filled = readOnly(filled)
}
}
out[i] = filled
}
if len(volumes) > 0 {
resource["volumes"] = out
}
return nil
}
// readOnly is a volume mounted read-only: "src:dst" becomes "src:dst:ro", and options gain ro in place of rw.
func readOnly(v string) string {
parts := strings.Split(v, ":")
if len(parts) < 2 {
return v
}
if len(parts) == 2 {
return v + ":ro"
}
opts := strings.Split(parts[len(parts)-1], ",")
kept := []string{"ro"}
for _, o := range opts {
if o != "ro" && o != "rw" && o != "" {
kept = append(kept, o)
}
}
parts[len(parts)-1] = strings.Join(kept, ",")
return strings.Join(parts, ":")
}
// volumeSource is a volume's source as written: up to its first colon outside a ${…} placeholder, so
// "${dir:config}:/config" is "${dir:config}".
func volumeSource(v string) string {
depth := 0
for i, r := range v {
switch {
case r == '{' && i > 0 && v[i-1] == '$':
depth++
case r == '}' && depth > 0:
depth--
case r == ':' && depth == 0:
return v[:i]
}
}
return v
}
// bundleLibraryFill fills a joining module's tool words (a bundle's env): ${identity:…}, and ${library:<id>} with the
// host path, since its tools run on the machine (§5).
func bundleLibraryFill(value string, m Manifest, with Rendering) (string, error) {
if !strings.Contains(value, "${identity:") && !strings.Contains(value, "${library:") {
return value, nil
}
j, err := joinedOf(m, with)
if err != nil {
return "", err
}
paths := libraryPaths(m, with)
var missing error
value = libraryRef.ReplaceAllStringFunc(identityFill(value, j), func(tok string) string {
id := libraryRef.FindStringSubmatch(tok)[1]
p, has := paths[id]
if !has {
missing = fmt.Errorf("%s's tools say ${library:%s}, and it has no library entry %q", m.Module, id, id)
return tok
}
return p
})
return value, missing
}
// toolsAccount is the account module m's tools run as on this machine, or empty where they run nowhere as one: no
// runtime here, no bundle it loads, or a module left out of an adopted machine's declaration.
func (r Resolution) toolsAccount(m Manifest, with Rendering) string {
if !r.runtimeHere() || (with.Adopted && m.Filtering != nil) {
return ""
}
loads := false
for _, b := range m.Bundles {
if len(b.Loads) > 0 {
loads = true
}
}
if !loads {
return ""
}
if m.RunsAs != "" {
return m.RunsAs
}
return r.Account
}
// LibraryEntry is one library as a machine's declaration names it (§3, §6, §7, §11).
type LibraryEntry struct {
Library string `json:"library"`
Root string `json:"root"`
Group string `json:"group"`
GID int `json:"gid"`
// Home says this machine is the library's home, where the engine keeps it; Taken that the operator let the mesh
// take its existing data there (§7).
Home bool `json:"home"`
Taken bool `json:"taken"`
// Paths are the sub-paths its modules ask for, relative to its root.
Paths []string `json:"paths"`
// Runner is the tool runner's account where a module whose tools run there uses the library: a member of its
// group (§11). Empty otherwise.
Runner string `json:"runner,omitempty"`
}
// GroupEntry is one group a machine's declaration names: a library's, made on every machine the library reaches (§3).
type GroupEntry struct {
Name string `json:"name"`
GID int `json:"gid"`
}
// IdentityEntry is one joining module's user as a machine's declaration names it (§4; mesh-host internal/declaration
// Identity): the engine always makes its own group with its uid; GID is its primary group's number — its first
// library's, or its own — and Groups the numbers of its added groups.
type IdentityEntry struct {
Module string `json:"module"`
Name string `json:"name"`
UID int `json:"uid"`
GID int `json:"gid"`
Groups []int `json:"groups,omitempty"`
// Runner is the tool runner's account, put in this module's own group, where its tools run there (ADR 0334 §2).
// Only to an engine that reads it (Rendering.RunnerJoinsOwnGroups).
Runner string `json:"runner,omitempty"`
}
// libraryEnvelope is what a machine's declaration says of libraries and joining modules: the group of every library
// that reaches it, each joining module composed here, and each library a module here uses or that has its home here.
func (r Resolution) libraryEnvelope(with Rendering, joined []Joined) ([]GroupEntry, []IdentityEntry, []LibraryEntry) {
if !with.KeepsLibraries {
return nil, nil, nil
}
var groups []GroupEntry
for name, l := range with.Libraries {
if gid, has := with.LibraryGroups[name]; has && l.Reaches(r.Node) && AllocatableID(gid) {
groups = append(groups, GroupEntry{Name: l.Group(), GID: gid})
}
}
sort.Slice(groups, func(i, j int) bool { return groups[i].Name < groups[j].Name })
var ids []IdentityEntry
for _, j := range joined {
e := IdentityEntry{Module: j.Module, Name: IdentityName(j.Module), UID: j.UID, GID: j.PrimaryGID,
Groups: append([]int(nil), j.AddedGIDs...)}
if with.RunnerJoinsOwnGroups {
for _, m := range r.Modules {
if m.Module == j.Module {
e.Runner = r.toolsAccount(m, with)
}
}
}
ids = append(ids, e)
}
sort.Slice(ids, func(a, b int) bool { return ids[a].Module < ids[b].Module })
used := map[string]map[string]bool{}
runner := map[string]string{}
for _, m := range r.Modules {
if libraryLeftOut(m, r.Node, with) != "" {
continue
}
for _, u := range m.Libraries {
if used[u.Library] == nil {
used[u.Library] = map[string]bool{}
}
used[u.Library][filepath.Clean(u.Path)] = true
if a := r.toolsAccount(m, with); a != "" && runner[u.Library] == "" {
runner[u.Library] = a
}
}
}
var libs []LibraryEntry
for name, l := range with.Libraries {
home := l.Home == r.Node
if used[name] == nil && !home {
continue
}
gid, has := with.LibraryGroups[name]
if !has || !l.Reaches(r.Node) {
continue
}
paths := map[string]bool{}
for p := range used[name] {
paths[p] = true
}
if home {
// On its home the engine keeps every sub-path any module asks for, wherever it runs (§6).
for _, p := range with.LibraryPaths[name] {
paths[filepath.Clean(p)] = true
}
}
e := LibraryEntry{Library: name, Root: l.Root, Group: l.Group(), GID: gid, Home: home,
Taken: with.TakenLibraries[name], Paths: []string{}, Runner: runner[name]}
for p := range paths {
e.Paths = append(e.Paths, p)
}
sort.Strings(e.Paths)
libs = append(libs, e)
}
sort.Slice(libs, func(i, j int) bool { return libs[i].Library < libs[j].Library })
return groups, ids, libs
}
// LibraryUseShape is a module's libraries and whether it joins, in a form that compares: a move that changes either
// changes who uses its libraries (the gate's reading of an unknown or a held library).
func LibraryUseShape(m Manifest) string {
var out []string
for _, u := range m.Libraries {
out = append(out, u.ID+"\x00"+u.Library+"\x00"+filepath.Clean(u.Path)+"\x00"+u.At())
}
sort.Strings(out)
return fmt.Sprint(Joins(m), out)
}
// LibraryOfEntry is the library a module's entry id names, or "".
func LibraryOfEntry(m Manifest, id string) string {
for _, u := range m.Libraries {
if u.ID == id {
return u.Library
}
}
return ""
}
// AccessPathOn is where access id of module m is on a machine with these settings: its placement, else the
// definition's default; false where neither says, or the module declares no such access.
func AccessPathOn(m Manifest, layers []Layer, id string) (string, bool) {
_, byID, err := accessesFor(m, layers)
if err != nil {
return "", false
}
path, has := byID[id]
if !has || path == "" {
return "", false
}
return filepath.Clean(path), true
}
-429
View File
@@ -1,429 +0,0 @@
package catalogue
import (
"encoding/json"
"fmt"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-host/validate"
)
// Defends novox/hq ADR 0328 on the controller's side, row by row of its "How it is checked" table where the row is the
// controller's: libraries defined (1, 13), entries resolved (2), groups and numbers (3, 4, 15), the identity and library
// placeholders (5), a read use mounted read-only (10), the tool runner in a library's group (11), a joining module held
// until its libraries are taken (8), and a module that does not join composed exactly as before (14).
const libraryImage = "x@sha256:1111111111111111111111111111111111111111111111111111111111111111"
var mediaLibrary = Library{Name: "media", Home: "ace", Root: "/storage/media", Reach: []string{"novox"}}
// aRadarr writes movies and reads books, tells its image its identity by PUID and PGID, and has tools on the runner.
func aRadarr(t *testing.T) Manifest {
t.Helper()
m := aToolsModule(t, "radarr", "tools/index.js")
m.Libraries = []LibraryUse{
{ID: "movies", Library: "media", Path: "movies", Use: AccessReadWrite},
{ID: "books", Library: "books", Path: "shelf", Use: AccessRead},
}
m.Resources = []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "env", "type": "file", "path": "${dir:config}/identity.env", "mode": "0644",
"content": "PUID=${identity:uid}\nPGID=${identity:gid}\nUSER=${identity:user}\nGROUP=${identity:group}\nUMASK=002\n"},
{"id": "app", "type": "container", "name": "radarr", "image": libraryImage,
"env": map[string]any{"PUID": "${identity:uid}", "PGID": "${identity:gid}"},
"volumes": []any{"${dir:config}:/config", "${library:movies}:/movies", "${library:books}:/books:z"}},
}
return m
}
// aPlain is a module that neither declares libraries nor uses its identity.
func aPlain() Manifest {
return Manifest{Module: "plain", Version: "1", Resources: []map[string]any{
{"id": "data", "type": "directory", "mode": "0700", "owner": "999:999"},
{"id": "app", "type": "container", "name": "plain", "image": libraryImage, "volumes": []any{"${dir:data}:/data"}},
}}
}
func librariesRendering() Rendering {
books := Library{Name: "books", Home: "ace", Root: "/storage/books"}
return Rendering{KeepsLibraries: true,
Libraries: map[string]Library{"media": mediaLibrary, "books": books},
LibraryGroups: map[string]int{"media": 50010, "books": 50011},
Identities: map[string]ModuleIdentity{"radarr": {Module: "radarr", UID: 50020}},
TakenLibraries: map[string]bool{"media": true, "books": true},
LibraryPaths: map[string][]string{"media": {"movies", "series"}},
}
}
func composeLibraries(t *testing.T, r Resolution, with Rendering) Composed {
t.Helper()
with.Needed = map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}
with.ArtifactStore = "anchor.internal:5101"
composed, err := r.Compose(with)
if err != nil {
t.Fatal(err)
}
return composed
}
// Rows 2, 4, 5, 10: a joining module's identity is filled everywhere, its library entries resolve to root + path in its
// volume sources, a read entry mounts read-only, its directories are owned by its user and primary group — the first
// library it uses — and its other libraries are added groups.
func TestAJoiningModuleIsComposedWithItsIdentityAndLibraries(t *testing.T) {
r := Resolution{Node: "ace", Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
composed := composeLibraries(t, r, librariesRendering())
got := byID(composed.Resources)
app := got["radarr.app"]
if fmt.Sprint(app["env"]) != fmt.Sprint(map[string]any{"PUID": "50020", "PGID": "50010"}) {
t.Errorf("the container's environment is %v", app["env"])
}
want := []any{"/var/lib/radarr/config:/config", "/storage/media/movies:/movies", "/storage/books/shelf:/books:ro,z"}
if !reflect.DeepEqual(app["volumes"], want) {
t.Errorf("the volumes are %v; want %v", app["volumes"], want)
}
if content := fmt.Sprint(got["radarr.env"]["content"]); content !=
"PUID=50020\nPGID=50010\nUSER=mesh-radarr\nGROUP=mesh-media\nUMASK=002\n" {
t.Errorf("the identity file reads %q", content)
}
if owner := got["radarr.config"]["owner"]; owner != "50020:50020" {
t.Errorf("its directory is owned by %v; want its user and own group (ADR 0334 §1)", owner)
}
if len(composed.Identities) != 1 || !reflect.DeepEqual(composed.Identities[0], IdentityEntry{Module: "radarr",
Name: "mesh-radarr", UID: 50020, GID: 50010, Groups: []int{50011}}) {
t.Errorf("the identities are %+v", composed.Identities)
}
if w := LibraryWarnings(aRadarr(t)); len(w) != 1 || !strings.Contains(w[0], "reaches only media") {
t.Errorf("a module of two libraries is warned %q", w)
}
}
// Rows 3, 7, 11: every library reaching the machine has its group, with its number; each library used here or homed
// here is named with its root, whether this is its home, whether it is taken, the sub-paths kept on its home, and the
// tool runner's account where a module whose tools run there uses it.
func TestTheDeclarationNamesTheLibrariesAndTheirGroups(t *testing.T) {
with := librariesRendering()
with.TakenLibraries = map[string]bool{"media": false, "books": true}
r := Resolution{Node: "ace", Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
composed := composeLibraries(t, r, with)
raw, _ := json.Marshal(map[string]any{"groups": composed.Groups, "libraries": composed.Libraries})
want := `{"groups":[{"name":"mesh-books","gid":50011},{"name":"mesh-media","gid":50010}],` +
`"libraries":[{"library":"books","root":"/storage/books","group":"mesh-books","gid":50011,"home":true,` +
`"taken":true,"paths":["shelf"],"runner":"jochen"},` +
`{"library":"media","root":"/storage/media","group":"mesh-media","gid":50010,"home":true,"taken":false,` +
`"paths":["movies","series"],"runner":"jochen"}]}`
if string(raw) != want {
t.Errorf("the declaration says\n %s\nwant\n %s", raw, want)
}
// On a machine it reaches but is not home: the group, the library as used there, and no sub-path from elsewhere.
elsewhere := composeLibraries(t, Resolution{Node: "novox", Account: "jochen",
Modules: []Manifest{aPlain(), theRuntime(t)}}, with)
raw, _ = json.Marshal(map[string]any{"groups": elsewhere.Groups, "libraries": elsewhere.Libraries})
if string(raw) != `{"groups":[{"name":"mesh-media","gid":50010}],"libraries":null}` {
t.Errorf("on a machine it only reaches: %s", raw)
}
}
// Row 2: a module assigned to a machine its library does not reach is left out there, naming the library; one whose
// library is not defined likewise.
func TestALibraryNotReachingTheMachineLeavesItsModuleOut(t *testing.T) {
with := librariesRendering()
with.Identities["radarr"] = ModuleIdentity{Module: "radarr", UID: 50020}
composed := composeLibraries(t, Resolution{Node: "far", Account: "jochen",
Modules: []Manifest{aRadarr(t), theRuntime(t)}}, with)
if why := composed.LeftOut["radarr"]; !strings.Contains(why, "does not reach far") {
t.Errorf("radarr on a machine media does not reach: %q", why)
}
delete(with.Libraries, "books")
composed = composeLibraries(t, Resolution{Node: "ace", Account: "jochen",
Modules: []Manifest{aRadarr(t), theRuntime(t)}}, with)
if why := composed.LeftOut["radarr"]; !strings.Contains(why, "books, which is not defined") {
t.Errorf("radarr with an undefined library: %q", why)
}
}
// Row 14: a module that neither declares libraries nor uses its identity is composed exactly as before — the whole
// declaration: its resources, the filter its port reaches, the contributions it makes — on an engine that keeps
// libraries or not; on a machine a library reaches, only that library's group is added. A joining module on an older
// engine is left out, naming why.
func TestAModuleThatDoesNotJoinIsComposedAsBefore(t *testing.T) {
plain := aPlain()
plain.Listens = []Listening{{Port: 8080, From: FromEverywhere, Name: "web", Why: "its page"}}
plain.Contributes = map[string]map[string]any{"reverse-proxy": {"host": "plain", "port": 8080}}
plain.Requires = []string{"reverse-proxy"}
modules := []Manifest{plain, proxy(), {Module: "nftables", Filtering: &Filtering{Into: "/etc/mesh/filter.nft"}},
theRuntime(t)}
base := Rendering{Mesh: []string{"198.51.100.2"}, OutwardLinks: []string{"eth0"}, TunnelInterface: "mesh0"}
keeping := func(node string) Rendering {
with := librariesRendering()
with.Mesh, with.OutwardLinks, with.TunnelInterface = base.Mesh, base.OutwardLinks, base.TunnelInterface
return with
}
for _, node := range []string{"far", "ace"} {
r := Resolution{Node: node, Account: "jochen", Modules: modules}
before := composeLibraries(t, r, base)
after := composeLibraries(t, r, keeping(node))
if node == "far" {
if !reflect.DeepEqual(before, after) {
t.Fatalf("on a machine no library reaches, a module that does not join composed otherwise:\n %+v\n %+v",
before, after)
}
continue
}
groups := after.Groups
after.Groups, after.Libraries = nil, nil
if !reflect.DeepEqual(before, after) {
t.Fatalf("on %s, a module that does not join composed otherwise:\n %+v\n %+v", node, before, after)
}
if len(groups) == 0 || len(after.Identities) > 0 {
t.Fatalf("on the library's home: groups %v, identities %v", groups, after.Identities)
}
filter := fmt.Sprint(byID(before.Resources)["nftables."+FilteringID()]["content"])
if !strings.Contains(filter, "8080") || received(t, before.Resources)[0].From != "plain" {
t.Fatalf("the comparison held no filter or contribution of the module: %s", filter)
}
}
if Joins(aPlain()) || !Joins(aRadarr(t)) {
t.Fatal("joining is not read from the libraries and the identity")
}
older := librariesRendering()
older.KeepsLibraries = false
composed := composeLibraries(t, Resolution{Node: "ace", Account: "jochen",
Modules: []Manifest{aRadarr(t), aPlain(), theRuntime(t)}}, older)
if why := composed.LeftOut["radarr"]; !strings.Contains(why, "older than them") {
t.Fatalf("a joining module on an older engine: %q", why)
}
if len(composed.Groups)+len(composed.Identities)+len(composed.Libraries) > 0 {
t.Fatal("an older engine was sent groups, identities or libraries")
}
if byID(composed.Resources)["plain.app"] == nil {
t.Fatal("a module that does not join was left out beside one that does")
}
}
// Row 15: no number outside the range is sent, root's above all; the composition refuses, naming why.
func TestNoNumberTheMeshGivesIsRoot(t *testing.T) {
for _, n := range []int{0, 4, 10, IdentityLow - 1, IdentityHigh + 1} {
if AllocatableID(n) {
t.Errorf("%d is allocatable", n)
}
with := librariesRendering()
with.Identities["radarr"] = ModuleIdentity{Module: "radarr", UID: n}
with.Needed = map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}
with.ArtifactStore = "anchor.internal:5101"
r := Resolution{Node: "ace", Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
if _, err := r.Compose(with); err == nil || !strings.Contains(err.Error(), "§15") {
t.Errorf("a module numbered %d composed: %v", n, err)
}
with = librariesRendering()
with.LibraryGroups["media"] = n
with.Needed = map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}
with.ArtifactStore = "anchor.internal:5101"
if _, err := r.Compose(with); err == nil || !strings.Contains(err.Error(), "§15") {
t.Errorf("a library group numbered %d composed: %v", n, err)
}
}
}
func libraryManifest(t *testing.T, extra map[string]any) []byte {
t.Helper()
m := map[string]any{"module": "app", "version": "1", "capabilities": []any{"container-runtime"},
"libraries": []any{map[string]any{"id": "movies", "library": "media", "path": "movies", "use": "read-write"}},
"resources": []any{map[string]any{"id": "app", "type": "container", "name": "app", "image": libraryImage,
"volumes": []any{"${library:movies}:/movies"}}}}
for k, v := range extra {
m[k] = v
}
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
return raw
}
// Rows 2 and 5 at the module check: library entries are accepted by name, sub-path and use, and refused with a host
// path, an owner, a group, a mode or a number; ${library:<id>} only in a volume source; ${identity:…} only by its four
// names; a joining module names no directory owner and declares no number setting.
func TestTheModuleCheckHoldsLibrariesAndIdentityToTheDecision(t *testing.T) {
if _, err := ParseManifest(libraryManifest(t, nil)); err != nil {
t.Fatalf("a library entry was refused: %v", err)
}
entry := func(extra map[string]any) map[string]any {
e := map[string]any{"id": "movies", "library": "media", "path": "movies", "use": "read-write"}
for k, v := range extra {
e[k] = v
}
return map[string]any{"libraries": []any{e}}
}
for name, c := range map[string]struct {
extra map[string]any
want string
}{
"a host path": {entry(map[string]any{"path": "/storage/media/movies"}), "never a host path"},
"out of it": {entry(map[string]any{"path": "../other"}), "never a host path"},
"an owner": {entry(map[string]any{"owner": "1000"}), "owner"},
"a group": {entry(map[string]any{"group": "media"}), "group"},
"a mode": {entry(map[string]any{"mode": "0775"}), "mode"},
"a number": {entry(map[string]any{"gid": 1000}), "gid"},
"a use": {entry(map[string]any{"use": "write"}), "is used \"write\""},
"in the env": {map[string]any{"resources": []any{map[string]any{"id": "app", "type": "container",
"name": "app", "image": libraryImage, "env": map[string]any{"MOVIES": "${library:movies}"}}}}, "volume source"},
"in a file": {map[string]any{"resources": []any{map[string]any{"id": "f", "type": "file", "path": "/etc/app.conf",
"mode": "0644", "content": "movies=${library:movies}"}}}, "volume source"},
"on the container side": {map[string]any{"resources": []any{map[string]any{"id": "app", "type": "container",
"name": "app", "image": libraryImage, "volumes": []any{"/srv:${library:movies}"}}}}, "volume source"},
"an unknown identity": {map[string]any{"resources": []any{map[string]any{"id": "app", "type": "container",
"name": "app", "image": libraryImage, "env": map[string]any{"HOME": "${identity:home}"}}}}, "${identity:home}"},
"a directory owner": {map[string]any{"resources": []any{map[string]any{"id": "d", "type": "directory",
"mode": "0750", "owner": "1000:1000"}}}, "names its owner"},
} {
_, err := ParseManifest(libraryManifest(t, c.extra))
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: %v; want refused saying %q", name, err, c.want)
}
}
// A module that does not join keeps its directory owner and its settings as they are (§14).
plain := map[string]any{"module": "plain", "version": "1",
"resources": []any{map[string]any{"id": "d", "type": "directory", "mode": "0750", "owner": "1000:1000"}}}
raw, _ := json.Marshal(plain)
if _, err := ParseManifest(raw); err != nil {
t.Errorf("a module that does not join was refused: %v", err)
}
// A setting carrying a number, or a placement's owner, is refused for a joining module, and kept for one that does
// not.
if err := JoiningSettingProblem(aRadarr(t), []Layer{{From: "ace", Values: map[string]any{"puid": "1000"}}}); err == nil ||
!strings.Contains(err.Error(), "carries a user or group number") {
t.Errorf("a joining module's puid setting: %v", err)
}
if err := JoiningSettingProblem(aPlain(), []Layer{{From: "ace", Values: map[string]any{"puid": "1000"}}}); err != nil {
t.Errorf("a plain module's puid setting: %v", err)
}
placed := []Layer{{From: "ace", Values: map[string]any{PlacesSetting: map[string]any{
"config": map[string]any{"path": "/srv/x", "owner": "1000:1000"}}}}}
if err := PlacesOwnerProblem(aRadarr(t), placed); err == nil || !strings.Contains(err.Error(), "ADR 0328") {
t.Errorf("a joining module's placed owner: %v", err)
}
if err := PlacesOwnerProblem(aPlain(), placed); err != nil {
t.Errorf("a plain module's placed owner: %v", err)
}
}
// Row 1: the libraries are the controller's own mesh-wide setting, the terminal's or the operator's Approve alone.
func TestTheLibrariesAreTheOperatorsSetting(t *testing.T) {
controller := Manifest{Module: LibrariesModule, Version: "1"}
keys := TerminalKeys(controller)
if !hasName(keys, LibrariesSetting) {
t.Fatalf("the libraries are not among the terminal's keys: %v", keys)
}
if hasName(TerminalKeys(Manifest{Module: "radarr"}), LibrariesSetting) {
t.Fatal("another module's libraries key is the terminal's")
}
if err := ComposesWith(controller, []Layer{{From: "ace", Values: map[string]any{LibrariesSetting: map[string]any{}}}},
false); err == nil {
t.Fatal("libraries set on one machine were accepted")
}
if _, err := ParseLibraries(map[string]any{"media": map[string]any{"home": "ace", "root": "/etc/media"}}); err == nil {
t.Fatal("a library in the machine's own tree was accepted")
}
}
// What the node-engine this controller pins (mesh-host #95, ADR 0328's half) makes of a composed declaration with
// libraries and a joining module: nothing refused.
func TestTheEngineTakesWhatIsComposedForLibraries(t *testing.T) {
for _, node := range []string{"ace", "novox"} {
r := Resolution{Node: node, Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
with := librariesRendering()
with.RunnerJoinsOwnGroups = true // contract 7 (mesh-host #97): the runner on identities, setgid on own directories
with.Libraries["books"] = Library{Name: "books", Home: "ace", Root: "/storage/books", Reach: []string{"novox"}}
composed := composeLibraries(t, r, with)
body, err := json.Marshal(map[string]any{"declaration": validate.Version, "resources": composed.Resources,
"groups": composed.Groups, "identities": composed.Identities, "libraries": composed.Libraries})
if err != nil {
t.Fatal(err)
}
if problems := validate.Declaration(body); len(problems) > 0 {
t.Fatalf("%s: the pinned node-engine refuses it: %v", node, problems)
}
}
}
// novox/hq ADR 0334 §1, §2 (the review of 7ea6d36e, item 2): a joining module's own directories are owned by its user
// and its own group, not the library's; and the tool runner's account joins the own group of every joining module whose
// tools run on the machine — sent only to an engine that reads it (link.OwnGroupContract).
func TestAJoiningModulesOwnDirectoriesAreItsOwnGroups(t *testing.T) {
r := Resolution{Node: "ace", Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
got := byID(composeLibraries(t, r, librariesRendering()).Resources)
if owner := got["radarr.config"]["owner"]; owner != "50020:50020" {
t.Errorf("its directory is owned by %v; want its user and its own group", owner)
}
// ADR 0334 v4: with setgid, so what is made in it later keeps mesh-radarr though the process's primary group is the
// library's.
if mode := got["radarr.config"]["mode"]; mode != "0750" {
t.Errorf("an engine that refuses setgid modes was sent %v", mode)
}
if env := got["radarr.app"]["env"].(map[string]any); env["PGID"] != "50010" {
t.Errorf("its process's primary group is %v; want the library's, so what it makes there keeps the group", env["PGID"])
}
older := composeLibraries(t, r, librariesRendering())
if older.Identities[0].Runner != "" {
t.Errorf("an engine that cannot put the runner in a module's own group was told %q", older.Identities[0].Runner)
}
with := librariesRendering()
with.RunnerJoinsOwnGroups = true
newer := composeLibraries(t, r, with)
if got := newer.Identities[0].Runner; got != "jochen" {
t.Errorf("the runner joins %q's own group; want jochen, whose tools it runs", got)
}
if mode := byID(newer.Resources)["radarr.config"]["mode"]; mode != "2750" {
t.Errorf("its directory's mode is %v; want 2750, setgid on its own 0750", mode)
}
}
// novox/hq ADR 0334 §3: a joining module whose tools name a directory whose mode closes it to its group is refused at
// the module check, naming the directory.
func TestAToolsDirectoryIsOpenToItsGroup(t *testing.T) {
closed := map[string]any{
"libraries": []any{map[string]any{"id": "movies", "library": "media", "path": "movies", "use": "read"}},
"resources": []any{
map[string]any{"id": "config", "type": "directory", "mode": "0700"},
map[string]any{"id": "app", "type": "container", "name": "app", "image": libraryImage,
"volumes": []any{"${dir:config}:/config", "${library:movies}:/movies"}}},
"build": map[string]any{"artifacts": []any{map[string]any{"name": "tools", "kind": "bundle",
"language": "go", "system": "arch", "from": "cmd/app", "binary": "app", "loads": []any{"app"},
"env": map[string]any{"APP_CONFIG": "${dir:config}/config.xml"}}}}}
_, err := ParseManifest(libraryManifest(t, closed))
if err == nil || !strings.Contains(err.Error(), "config") || !strings.Contains(err.Error(), "ADR 0334") {
t.Fatalf("a tools' directory closed to its group: %v", err)
}
open := closed
open["resources"] = []any{
map[string]any{"id": "config", "type": "directory", "mode": "0750"},
map[string]any{"id": "app", "type": "container", "name": "app", "image": libraryImage,
"volumes": []any{"${dir:config}:/config", "${library:movies}:/movies"}}}
if _, err := ParseManifest(libraryManifest(t, open)); err != nil {
t.Fatalf("a tools' directory open to its group was refused: %v", err)
}
}
// The review of 7ea6d36e, item 6: a service's unit takes the identity too — `User=${identity:user}` — at the module
// check and in what is composed.
func TestAServicesUnitTakesTheIdentity(t *testing.T) {
unit := "[Service]\nUser=${identity:user}\nGroup=${identity:group}\nUMask=0002\nExecStart=/usr/bin/importer\n"
raw := libraryManifest(t, map[string]any{"resources": []any{
map[string]any{"id": "app", "type": "container", "name": "app", "image": libraryImage,
"volumes": []any{"${library:movies}:/movies"}},
map[string]any{"id": "watch", "type": "service", "name": "app-watch", "unit": unit}}})
m, err := ParseManifest(raw)
if err != nil {
t.Fatalf("a unit naming the identity was refused: %v", err)
}
with := librariesRendering()
with.Identities["app"] = ModuleIdentity{Module: "app", UID: 50030}
composed := composeLibraries(t, Resolution{Node: "ace", Modules: []Manifest{m}}, with)
got := fmt.Sprint(byID(composed.Resources)["app.watch"]["unit"])
if !strings.Contains(got, "User=mesh-app\nGroup=mesh-media\n") {
t.Fatalf("the unit reads %q", got)
}
}
+48 -6
View File
@@ -385,6 +385,10 @@ type Manifest struct {
// Requires are names that must be provided by something assigned to the same node.
Requires []string `json:"requires,omitempty"`
// OptionalRequires is which of Requires the module can run without (novox/hq ADR 0330): filled
// by UnmarshalJSON from the long form of a `requires` entry, {name, optional}, or from the stored
// form's `optional-requires` list; never written by hand.
OptionalRequires map[string]bool `json:"-"`
// Emits are the events this module publishes, named **locally**: `order.placed`, not a subject
// and not a routing key. The mesh derives where it lands (design 29 §1), so reorganising the
@@ -536,9 +540,6 @@ type Manifest struct {
// `directory` resource, which the mesh creates and owns: an access is mounted and nothing
// about it is reconciled, and several modules may name the same one without conflict.
Accesses []Access `json:"accesses,omitempty"`
// Libraries are the libraries this module uses, each by name, sub-path and use (novox/hq ADR 0328 §2): where a
// library is, and who may write it, is the operator's definition and the mesh's, never the module's.
Libraries []LibraryUse `json:"libraries,omitempty"`
// Computed names something in the control plane that works this module's resources out per
// node, instead of them being fixed here.
@@ -1357,6 +1358,35 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
}
delete(keys, "contributes")
}
var optionalRequires map[string]bool
if requires, ok := keys["requires"]; ok && string(requires) != "null" {
names, optional, err := readRequires(requires)
if err != nil {
return err
}
plainNames, err := json.Marshal(names)
if err != nil {
return err
}
keys["requires"] = plainNames
optionalRequires = optional
}
// The stored form: `requires` plain, and which are optional beside it (OptionalRequiresKey).
if stored, ok := keys[OptionalRequiresKey]; ok {
if string(stored) != "null" {
optional, err := readOptionalRequires(stored)
if err != nil {
return err
}
for name := range optional {
if optionalRequires == nil {
optionalRequires = map[string]bool{}
}
optionalRequires[name] = true
}
}
delete(keys, OptionalRequiresKey)
}
rest, err := json.Marshal(keys)
if err != nil {
return err
@@ -1388,6 +1418,7 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
}
*m = Manifest(fields)
m.unknown = unknown
m.OptionalRequires = optionalRequires
if len(plain) > 0 {
m.Secrets = plain
}
@@ -1410,13 +1441,22 @@ func (m Manifest) MarshalJSON() ([]byte, error) {
if err != nil {
return nil, err
}
if len(m.SecretsMany) == 0 && len(m.ContributesMany) == 0 {
if len(m.SecretsMany) == 0 && len(m.ContributesMany) == 0 && len(m.OptionalRequires) == 0 {
return raw, nil
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
return nil, err
}
if optional := optionalRequiresList(m); len(optional) > 0 {
// Beside `requires`, never inside it: an older controller reads `requires` as plain names, and one
// manifest it cannot read stops the whole catalogue (OptionalRequiresKey).
listed, err := json.Marshal(optional)
if err != nil {
return nil, err
}
keys[OptionalRequiresKey] = listed
}
if len(m.ContributesMany) > 0 {
mergedContributes := map[string]any{}
for to, values := range m.Contributes {
@@ -1696,6 +1736,10 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, LogProblems(m)...)
// And the defaults it gives its settings (setting_defaults.go, novox/hq ADR 0262).
problems = append(problems, SettingProblems(m)...)
// And what it requires optionally, and the files it delivers only while bound (novox/hq ADR 0330).
problems = append(problems, OptionalProblems(m)...)
// And which setting keys each mergeable file takes (setting_takes.go, novox/hq ADR 0326).
problems = append(problems, TakenSettingsProblems(m)...)
wellFormed := true
for _, c := range m.Claims {
if !name.MatchString(c.Name) {
@@ -1863,8 +1907,6 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, replacesProblems(m)...)
problems = append(problems, UnitSettingProblems(m)...)
problems = append(problems, OwnerSettingProblems(m)...)
// Its libraries and its identity, and what a module that joins is held to (novox/hq ADR 0328).
problems = append(problems, libraryProblems(m)...)
problems = append(problems, endpointNameProblems(m)...)
problems = append(problems, RouteProblems(m)...)
for _, port := range m.Guards {
+282
View File
@@ -0,0 +1,282 @@
package catalogue
import (
"bytes"
"encoding/json"
"fmt"
"sort"
"strings"
)
// A requirement a module can run without (novox/hq ADR 0330, issue 166).
//
// **Hard stays the default.** A requirement written as a bare name is hard, exactly as before: it
// resolves, or the assignment is refused, and a single catalogue candidate is installed for it. One
// written `{"name": "influxdb-api", "optional": true}` resolves the same way when a provider is
// already assigned in the mesh or reachable from the consumer (§2), and is otherwise **absent**: not
// refused, and nothing is installed for it, on this machine or elsewhere (§3).
//
// **An absent requirement is never rendered as an empty value** (§4–§6). Its binding, its pair
// credential and its grant are not composed, so a file that names one of them is delivered only while
// the requirement is bound (`only-when-bound`), and a `${bound:…}` placeholder of it outside such a
// file names the value it takes while absent: `${bound:<provision>:<key>|default:<value>}`. The
// module check refuses anything else, so the gap is found when the manifest is written, not on a
// machine where it reads as a blank.
// OnlyWhenBound is a resource's field naming the optional requirement it is delivered only while
// bound (ADR 0330 §5). Said in the catalogue and dropped before the declaration is sent: the
// node-engine parses strictly and knows no such field.
const OnlyWhenBound = "only-when-bound"
// requirementForm is one entry of `requires` in its long form.
type requirementForm struct {
Name string `json:"name"`
Optional bool `json:"optional,omitempty"`
}
// readRequires reads `requires` in both of its forms — a bare name, or {name, optional} — into the
// names and which of them are optional.
func readRequires(raw json.RawMessage) ([]string, map[string]bool, error) {
var entries []json.RawMessage
if err := json.Unmarshal(raw, &entries); err != nil {
return nil, nil, fmt.Errorf("requires: a list of names, or of {name, optional}: %w", err)
}
var names []string
optional := map[string]bool{}
for i, e := range entries {
var plain string
if err := json.Unmarshal(e, &plain); err == nil {
names = append(names, plain)
continue
}
var full requirementForm
dec := json.NewDecoder(bytes.NewReader(e))
dec.DisallowUnknownFields()
if err := dec.Decode(&full); err != nil {
return nil, nil, fmt.Errorf("requires[%d]: a name, or {name, optional} (novox/hq ADR 0330): %w", i, err)
}
if strings.TrimSpace(full.Name) == "" {
return nil, nil, fmt.Errorf("requires[%d] names no requirement: {name, optional}", i)
}
names = append(names, full.Name)
if full.Optional {
optional[full.Name] = true
}
}
if len(optional) == 0 {
optional = nil
}
return names, optional, nil
}
// OptionalRequiresKey is where a stored manifest keeps which of its requirements are optional: beside
// `requires`, which stays a plain list of names (mesh-controller #281's review). A controller older than
// optional requirements reads `requires` as names; the long form inside it failed that read, and with it
// the whole catalogue, so a rollback below this controller would have composed no machine. An older
// controller sets this unknown key aside instead and leaves only the module that carries it out. A
// manifest may be written either way; it is stored this way.
const OptionalRequiresKey = "optional-requires"
// readOptionalRequires reads the stored form's list of optional requirements.
func readOptionalRequires(raw json.RawMessage) (map[string]bool, error) {
var names []string
if err := json.Unmarshal(raw, &names); err != nil {
return nil, fmt.Errorf("%s: a list of the names in `requires` that are optional: %w", OptionalRequiresKey, err)
}
out := map[string]bool{}
for _, n := range names {
out[n] = true
}
return out, nil
}
// optionalRequiresList is the stored form's list: the optional requirements, sorted.
func optionalRequiresList(m Manifest) []string {
var out []string
for name, optional := range m.OptionalRequires {
if optional {
out = append(out, name)
}
}
sort.Strings(out)
return out
}
// IsOptional is whether this module requires the name optionally (ADR 0330 §1). A name it contributes
// to without requiring, or requires as a bare name, is hard.
func (m Manifest) IsOptional(requirement string) bool { return m.OptionalRequires[requirement] }
// wantsHard is whether this module wants the name as a hard requirement: required without
// `optional`, or contributed to.
func (m Manifest) wantsHard(name string) bool {
for _, w := range m.Wants() {
if w == name {
return !m.IsOptional(name)
}
}
return false
}
// Absent is one optional requirement of one consumer that nothing assigned answers (ADR 0330 §3):
// left unbound rather than refused, and listed by `plan` and `node` (§7).
type Absent struct {
// Consumer is the module that requires it optionally.
Consumer string `json:"consumer"`
// Requirement is what it requires.
Requirement string `json:"requirement"`
}
// AbsentFor is the optional requirements of one module this resolution leaves unbound.
func (r Resolution) AbsentFor(module string) map[string]bool {
out := map[string]bool{}
for _, a := range r.Absent {
if a.Consumer == module {
out[a.Requirement] = true
}
}
return out
}
// absentOf is, for each requirement left absent, every module of the set that requires it
// optionally, sorted by consumer then requirement.
func absentOf(names map[string]bool, order []string, catalogue map[string]Manifest) []Absent {
var out []Absent
for _, module := range order {
m := catalogue[module]
for _, r := range m.Requires {
if names[r] && m.IsOptional(r) {
out = append(out, Absent{Consumer: module, Requirement: r})
}
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Consumer != out[j].Consumer {
return out[i].Consumer < out[j].Consumer
}
return out[i].Requirement < out[j].Requirement
})
return out
}
// withoutAbsent drops the resources delivered only while a requirement is bound whose requirement is
// absent here (ADR 0330 §5), and the field itself from every resource that keeps it.
func withoutAbsent(resources []map[string]any, absent map[string]bool) []map[string]any {
out := make([]map[string]any, 0, len(resources))
for _, r := range resources {
when, has := r[OnlyWhenBound]
if !has {
out = append(out, r)
continue
}
if absent[fmt.Sprint(when)] {
continue
}
kept := make(map[string]any, len(r))
for k, v := range r {
if k != OnlyWhenBound {
kept[k] = v
}
}
out = append(out, kept)
}
return out
}
// secretNamesOf is every `${secret:…}` name that carries a credential of this requirement: its own
// name, and each local name it keeps several under (ADR 0094).
func secretNamesOf(m Manifest, requirement string) []string {
names := []string{requirement}
for local := range m.SecretsMany[requirement] {
names = append(names, local)
}
sort.Strings(names)
return names
}
// OptionalProblems is what the module check refuses of a manifest's optional requirements and its
// only-when-bound resources (ADR 0330 §4–§6).
func OptionalProblems(m Manifest) []string {
var problems []string
required := map[string]bool{}
for _, r := range m.Requires {
if required[r] && m.IsOptional(r) {
problems = append(problems, fmt.Sprintf("%s requires %q twice; say it once, optional or not", m.Module, r))
}
required[r] = true
}
for _, r := range optionalRequiresList(m) {
if !required[r] {
problems = append(problems, fmt.Sprintf(
"%s says %q is optional in %s and does not require it", m.Module, r, OptionalRequiresKey))
}
}
// A contribution is to whoever answers the requirement, so it goes with it: contributing to one
// that is optional is allowed, and composed only while it is bound.
for _, r := range m.Resources {
id := fmt.Sprint(r["id"])
when, marked := r[OnlyWhenBound]
on := ""
if marked {
name, ok := when.(string)
switch {
case !ok || name == "":
problems = append(problems, fmt.Sprintf(
"%s's resource %s says %s %v; it names one requirement the module requires optionally",
m.Module, id, OnlyWhenBound, when))
continue
case fmt.Sprint(r["type"]) != "file":
problems = append(problems, fmt.Sprintf(
"%s's resource %s is a %v and says %s: only a file is left out while a requirement is "+
"absent (novox/hq ADR 0330 §5)", m.Module, id, r["type"], OnlyWhenBound))
continue
case !required[name]:
problems = append(problems, fmt.Sprintf(
"%s's resource %s is delivered only when %q is bound, and %s does not require it",
m.Module, id, name, m.Module))
continue
case !m.IsOptional(name):
problems = append(problems, fmt.Sprintf(
"%s's resource %s is delivered only when %q is bound, which is a hard requirement and "+
"always bound: mark the requirement optional, or drop %s", m.Module, id, name, OnlyWhenBound))
continue
}
on = name
}
if fmt.Sprint(r["type"]) != "file" {
continue
}
content, _ := r["content"].(string)
for _, use := range bound.FindAllStringSubmatch(content, -1) {
provision, key, hasDefault := use[1], use[2], use[3] != ""
if !m.IsOptional(provision) {
if hasDefault {
problems = append(problems, fmt.Sprintf(
"%s's resource %s gives ${bound:%s:%s} a default, and %q is a hard requirement, always "+
"bound: a default it never renders only misleads a reader", m.Module, id, provision, key, provision))
}
continue
}
if !hasDefault && on != provision {
problems = append(problems, fmt.Sprintf(
"%s's resource %s says ${bound:%s:%s}, and %q is optional: while it is absent this would "+
"be empty. Name its value then — ${bound:%s:%s|default:<value>} — or deliver the file "+
"only while it is bound, with \"%s\": %q (novox/hq ADR 0330 §4)",
m.Module, id, provision, key, provision, provision, key, OnlyWhenBound, provision))
}
}
for _, provision := range m.Requires {
if !m.IsOptional(provision) || on == provision {
continue
}
for _, name := range secretNamesOf(m, provision) {
if strings.Contains(content, "${secret:"+name+"}") {
problems = append(problems, fmt.Sprintf(
"%s's resource %s says ${secret:%s}, the credential of %q, which is optional and not "+
"delivered while absent: deliver the file only while it is bound, with \"%s\": %q "+
"(novox/hq ADR 0330 §6)", m.Module, id, name, provision, OnlyWhenBound, provision))
}
}
}
}
return problems
}
@@ -0,0 +1,301 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// A requirement a module can run without (novox/hq ADR 0330, issue 166). The shape of the issue:
// grafana runs without a data source and requires `influxdb-api` softly; a mesh without influxdb
// could not assign grafana at all, and one with influxdb in its catalogue would have had it installed
// for grafana.
func optionalShelf() map[string]Manifest {
grafana := Manifest{Module: "grafana", Version: "1", Requires: []string{"influxdb-api", "shell"},
OptionalRequires: map[string]bool{"influxdb-api": true, "shell": true}}
return shelf(
Manifest{Module: "influxdb", Version: "1", Provides: FromAnywhere("influxdb-api")},
Manifest{Module: "zsh", Version: "1", Provides: Offers("shell")},
grafana,
// What wants the same names hard, for the cases where one consumer's optional is another's hard.
Manifest{Module: "telegraf", Version: "1", Requires: []string{"influxdb-api"}},
Manifest{Module: "editor", Version: "1", Requires: []string{"shell"}},
Manifest{Module: "suite", Version: "1", Requires: []string{"editor"}},
)
}
func TestARequirementIsHardUnlessMarkedOptional(t *testing.T) {
m := mustParse(t, `{"module":"grafana","version":"1",
"requires":["route",{"name":"influxdb-api","optional":true},{"name":"oidc-client"}]}`)
if strings.Join(m.Requires, " ") != "route influxdb-api oidc-client" {
t.Fatalf("the requirements read as %v", m.Requires)
}
if m.IsOptional("route") || m.IsOptional("oidc-client") {
t.Error("a requirement without `optional` read as optional; hard is the default (ADR 0330 §1)")
}
if !m.IsOptional("influxdb-api") {
t.Error("a requirement marked optional read as hard")
}
// And written back as it was read, so a manifest through the store keeps its form.
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
var again Manifest
if err := json.Unmarshal(raw, &again); err != nil {
t.Fatal(err)
}
if !again.IsOptional("influxdb-api") || again.IsOptional("route") || len(again.Requires) != 3 {
t.Fatalf("written and read back, it says %v optional %v (%s)", again.Requires, again.OptionalRequires, raw)
}
if strings.Contains(string(raw), `"name":`) {
t.Errorf("a requirement was stored in the long form: %s", raw)
}
}
// The stored form reads on a controller older than optional requirements (review of mesh-controller
// #281). Inventory.Catalogue stops at the first manifest it cannot read, so a long-form entry inside
// `requires` would have stopped every plan after a rollback. Stored, `requires` is plain names, as main's
// manifest type reads it, and which are optional sits beside it in a key such a controller sets aside.
func TestTheStoredFormReadsWhereRequiresIsAListOfNames(t *testing.T) {
m := mustParse(t, `{"module":"grafana","version":"1",
"requires":["route",{"name":"influxdb-api","optional":true}]}`)
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
// main's field, exactly: `Requires []string`.
var older struct {
Requires []string `json:"requires"`
}
if err := json.Unmarshal(raw, &older); err != nil {
t.Fatalf("a controller reading requires as names cannot read the stored form: %v\n%s", err, raw)
}
if strings.Join(older.Requires, " ") != "route influxdb-api" {
t.Fatalf("it reads the requirements as %v", older.Requires)
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
t.Fatal(err)
}
if string(keys[OptionalRequiresKey]) != `["influxdb-api"]` {
t.Fatalf("which are optional is stored as %s", keys[OptionalRequiresKey])
}
// And this controller reads the stored form back the same, leniently and strictly.
var back Manifest
if err := json.Unmarshal(raw, &back); err != nil {
t.Fatal(err)
}
strict, err := ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
for _, got := range []Manifest{back, strict} {
if !got.IsOptional("influxdb-api") || got.IsOptional("route") || len(got.Requires) != 2 {
t.Fatalf("read back as %v optional %v", got.Requires, got.OptionalRequires)
}
}
// A stored list naming something the module does not require is refused.
if _, err := ParseManifest([]byte(`{"module":"x","version":"1","requires":["route"],"optional-requires":["mqtt-topic"]}`)); err == nil {
t.Error("an optional requirement the module does not require passed the check")
}
}
func TestARequirementsLongFormRefusesWhatItDoesNotKnow(t *testing.T) {
for _, raw := range []string{
`{"module":"x","version":"1","requires":[{"name":"influxdb-api","optinal":true}]}`,
`{"module":"x","version":"1","requires":[{"optional":true}]}`,
} {
if _, err := ParseManifest([]byte(raw)); err == nil {
t.Errorf("accepted %s", raw)
}
}
}
func TestAnOptionalRequirementWithNoProviderIsAbsentAndNothingIsInstalled(t *testing.T) {
// Exactly one catalogue module could answer each — the case where a hard requirement installs it.
// Brokered (influxdb-api, from anywhere) and on this machine (shell); both passes.
for _, world := range []World{{}, {Unchecked: true}} {
got, err := Resolve(optionalShelf(), []string{"grafana"}, reachable(), world)
if err != nil {
t.Fatalf("an optional requirement with no provider refused the assignment: %v", err)
}
if have := strings.Join(names(got), " "); have != "grafana" {
t.Fatalf("assigning grafana put %s on the machine; nothing is installed for an optional requirement (ADR 0330 §3)", have)
}
if world.Unchecked {
continue
}
want := []Absent{{Consumer: "grafana", Requirement: "influxdb-api"}, {Consumer: "grafana", Requirement: "shell"}}
if len(got.Absent) != 2 || got.Absent[0] != want[0] || got.Absent[1] != want[1] {
t.Fatalf("absent: %+v, want %+v", got.Absent, want)
}
if len(got.Needs) != 0 {
t.Fatalf("an absent requirement was given a binding: %+v", got.Needs)
}
}
}
func TestAnOptionalRequirementWithAProviderBindsAsAHardOne(t *testing.T) {
world := World{Offered: map[string][]Provider{"influxdb-api": {
{Node: "anchor", At: "anchor.internal", Module: "influxdb", Serves: map[string]any{"port": 8086}}}}}
got, err := Resolve(optionalShelf(), []string{"grafana", "zsh"}, reachable(), world)
if err != nil {
t.Fatal(err)
}
if n := boundFrom(t, got, "grafana", "influxdb-api"); n.From != "anchor" || n.Module != "influxdb" {
t.Fatalf("bound to %s/%s", n.From, n.Module)
}
if len(got.Absent) != 0 {
t.Fatalf("a requirement with a provider was left absent: %+v", got.Absent)
}
// And the same assignment without the provider is absent: what binds it later is only the
// provider being assigned, with no act on grafana (ADR 0330 §7).
before, err := Resolve(optionalShelf(), []string{"grafana", "zsh"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
if len(before.Absent) != 1 || before.Absent[0].Requirement != "influxdb-api" {
t.Fatalf("absent before the provider was assigned: %+v", before.Absent)
}
}
func TestOneConsumersOptionalRequirementIsAnothersHardOne(t *testing.T) {
// telegraf wants influxdb-api hard: with nothing offering it, the assignment is refused as
// before, naming what to assign — grafana's optional does not soften telegraf's.
_, err := Resolve(optionalShelf(), []string{"grafana", "telegraf"}, reachable(), World{})
if err == nil || !strings.Contains(err.Error(), "assign influxdb") {
t.Fatalf("a hard requirement with no provider was not refused naming the candidate: %v", err)
}
// And the refusal names telegraf, whose remedy it is, not grafana, which met the name first.
if !strings.Contains(err.Error(), "wanted by telegraf") {
t.Errorf("the refusal does not name the module that wants it hard: %v", err)
}
// And a module pulled in later (suite → editor) that wants the shell hard installs it, even
// though grafana, met first, wanted it only optionally — and grafana is then bound to it.
got, err := Resolve(optionalShelf(), []string{"grafana", "suite"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
if have := strings.Join(names(got), " "); !strings.Contains(have, "zsh") {
t.Fatalf("a hard requirement met after an optional one was not installed: %s", have)
}
for _, a := range got.Absent {
if a.Requirement == "shell" {
t.Fatalf("the shell is installed and still listed absent: %+v", got.Absent)
}
}
}
func TestAHardRequirementWithNoProviderIsStillRefused(t *testing.T) {
_, err := Resolve(optionalShelf(), []string{"telegraf"}, reachable(), World{})
if err == nil || !strings.Contains(err.Error(), "influxdb") {
t.Fatalf("got %v", err)
}
}
func optionalModule(resources string) string {
return `{"module":"grafana","version":"1",
"requires":["route",{"name":"influxdb-api","optional":true}],
"secrets":{"influxdb-api":"/var/lib/grafana/influxdb-api"},
"resources":[` + resources + `]}`
}
func TestTheModuleCheckRefusesAnOptionalPlaceholderThatCouldRenderEmpty(t *testing.T) {
refused := map[string]string{
"a placeholder without a default": `{"id":"ds","type":"file","path":"/a","mode":"0644",
"content":"url: ${bound:influxdb-api:at}"}`,
"a secret outside an only-when-bound file": `{"id":"s","type":"file","path":"/a","mode":"0400",
"content":"${secret:influxdb-api}"}`,
"only-when-bound on a hard requirement": `{"id":"r","type":"file","path":"/a","mode":"0644",
"content":"x","only-when-bound":"route"}`,
"only-when-bound on something not required": `{"id":"r","type":"file","path":"/a","mode":"0644",
"content":"x","only-when-bound":"mqtt-topic"}`,
"only-when-bound on a directory": `{"id":"d","type":"directory","mode":"0700",
"only-when-bound":"influxdb-api"}`,
"a default on a hard requirement": `{"id":"r","type":"file","path":"/a","mode":"0644",
"content":"${bound:route:name|default:x}"}`,
}
for what, resource := range refused {
if _, err := ParseManifest([]byte(optionalModule(resource))); err == nil {
t.Errorf("%s passed the module check", what)
} else if what == "a placeholder without a default" && !strings.Contains(err.Error(), "|default:") {
t.Errorf("the refusal does not say how to write it: %v", err)
}
}
accepted := map[string]string{
"a placeholder with a default": `{"id":"ds","type":"file","path":"/a","mode":"0644",
"content":"url: ${bound:influxdb-api:at|default:none}"}`,
"a placeholder and a secret inside an only-when-bound file": `{"id":"ds","type":"file","path":"/a",
"mode":"0400","content":"url: ${bound:influxdb-api:at}\npassword: ${secret:influxdb-api}",
"only-when-bound":"influxdb-api"}`,
}
for what, resource := range accepted {
if _, err := ParseManifest([]byte(optionalModule(resource))); err != nil {
t.Errorf("%s was refused: %v", what, err)
}
}
}
func absentGrafana(t *testing.T, absent bool) Resolution {
t.Helper()
m := mustParse(t, optionalModule(`
{"id":"env","type":"file","path":"/var/lib/grafana/env","mode":"0644",
"content":"INFLUX=${bound:influxdb-api:at|default:none}:${bound:influxdb-api:port|default:0}\n"},
{"id":"ds","type":"file","path":"/var/lib/grafana/ds.yaml","mode":"0400",
"content":"url: ${bound:influxdb-api:at}\npassword: ${secret:influxdb-api}\n","only-when-bound":"influxdb-api"}`))
m.Requires = []string{"influxdb-api"}
m.Binds = map[string]string{"influxdb-api": "/var/lib/grafana/influxdb.json"}
r := Resolution{Node: "workstation", Modules: []Manifest{m}}
if absent {
r.Absent = []Absent{{Consumer: "grafana", Requirement: "influxdb-api"}}
} else {
r.Needs = []Needed{{Name: "influxdb-api", From: "anchor", At: "anchor.internal", For: "grafana",
Module: "influxdb", Sealed: "sealed-influx", Serves: map[string]any{"port": float64(8086)}}}
}
return r
}
func TestAnAbsentRequirementIsComposedWithItsDefaultsAndWithoutItsFiles(t *testing.T) {
out, err := absentGrafana(t, true).Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
env := fileNamed(out, "grafana.env")
if env == nil || env["content"] != "INFLUX=none:0\n" {
t.Fatalf("the defaults were not rendered: %v", env)
}
for _, id := range []string{"grafana.ds", "grafana." + SecretID("influxdb-api"), "grafana." + BoundID("influxdb-api")} {
if fileNamed(out, id) != nil {
t.Errorf("%s was delivered while influxdb-api is absent (ADR 0330 §5, §6)", id)
}
}
for _, r := range out {
if _, has := r[OnlyWhenBound]; has {
t.Errorf("%v carries %s to the node-engine, which knows no such field", r["id"], OnlyWhenBound)
}
}
}
func TestABoundOptionalRequirementIsComposedAsAHardOne(t *testing.T) {
out, err := absentGrafana(t, false).Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
if env := fileNamed(out, "grafana.env"); env == nil || env["content"] != "INFLUX=anchor.internal:8086\n" {
t.Fatalf("the bound values were not rendered: %v", env)
}
ds := fileNamed(out, "grafana.ds")
if ds == nil {
t.Fatalf("the only-when-bound file was left out while bound: %v", out)
}
if _, has := ds[OnlyWhenBound]; has {
t.Errorf("the file carries %s to the node-engine", OnlyWhenBound)
}
for _, id := range []string{"grafana." + SecretID("influxdb-api"), "grafana." + BoundID("influxdb-api")} {
if fileNamed(out, id) == nil {
t.Errorf("%s was not delivered while bound", id)
}
}
}
+3 -32
View File
@@ -1,7 +1,6 @@
package catalogue
import (
"errors"
"fmt"
"path/filepath"
"regexp"
@@ -271,31 +270,6 @@ func Places(m Manifest, layers []Layer) (map[string]Placement, error) {
// It refuses an id the module declares no access under, and a path that is not absolute. An
// access declared by path alone cannot be placed — it has no name to place it by.
func AccessPlaces(m Manifest, layers []Layer) (map[string]string, error) {
return accessPlaces(m, layers, true)
}
// UndeclaredAccessError is a layer placing an access the definition does not declare (novox/hq issue 575): refused
// where a setting is stored, unless a newer build of the module declares it; passed by where a machine is composed.
type UndeclaredAccessError struct {
Module, ID string
said string
}
func (e *UndeclaredAccessError) Error() string { return e.said }
// UndeclaredAccessPlaces refuses a layer that places an access the definition does not declare.
func UndeclaredAccessPlaces(m Manifest, layers []Layer) error {
_, err := accessPlaces(m, layers, true)
var undeclared *UndeclaredAccessError
if errors.As(err, &undeclared) {
return err
}
return nil
}
// accessPlaces is AccessPlaces; where strict is false an access the definition does not declare is passed by
// (novox/hq issue 575), as a placement stored ahead of the build that declares it.
func accessPlaces(m Manifest, layers []Layer, strict bool) (map[string]string, error) {
declared := map[string]bool{}
for _, a := range m.Accesses {
if a.ID != "" {
@@ -315,12 +289,9 @@ func accessPlaces(m Manifest, layers []Layer, strict bool) (map[string]string, e
}
for id, body := range blocks {
if !declared[id] {
if !strict {
continue
}
return nil, &UndeclaredAccessError{Module: m.Module, ID: id, said: fmt.Sprintf(
return nil, fmt.Errorf(
"%s places the access %q, which it does not declare — the setting reaches "+
"nothing. It declares %s", m.Module, id, orNothing(namesOfAccesses(m)))}
"nothing. It declares %s", m.Module, id, orNothing(namesOfAccesses(m)))
}
path, _ := body.(string)
path = strings.TrimSpace(path)
@@ -353,7 +324,7 @@ type placedAccess struct {
// placed one, the definition's where it carries a default, and refused where neither says — an
// access that resolves to nowhere would reach the machine as a mount of nothing.
func accessesFor(m Manifest, layers []Layer) ([]placedAccess, map[string]string, error) {
placed, err := accessPlaces(m, layers, false)
placed, err := AccessPlaces(m, layers)
if err != nil {
return nil, nil, err
}
+92 -2
View File
@@ -183,6 +183,10 @@ type Resolution struct {
// that keeps its data, and did not (novox/hq ADR 0232): it stays bound where its data is, and the
// controller raises an urgent condition naming the move until a person pins one or the other.
Kept []KeptBinding
// Absent is every optional requirement of this node's modules that nothing assigned answers
// (novox/hq ADR 0330 §3): left unbound rather than refused, with nothing installed for it, and
// listed by `plan` and `node` (§7). It binds on the first composition after a provider is assigned.
Absent []Absent
}
// Unhostable is one directly-assigned module the machine cannot run.
@@ -374,12 +378,80 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
// identical lines before realising there is none.
reported := map[string]bool{}
for len(queue) > 0 {
// **An optional requirement is settled last** (novox/hq ADR 0330). Whether a name is optional is
// a question about every module that wants it, and a module pulled in later may want it hard; so
// a name only optional modules want so far is put aside until the walk has nothing else, and only
// then, if still nobody wants it hard, settled by the optional rule: bound to a provider that is
// already assigned or reachable, exactly as a hard one, and otherwise absent — never refused,
// and never answered by installing anything.
var deferred []string
settling := map[string]bool{}
absent := map[string]bool{}
wantedHard := func(want string) bool {
for _, name := range order {
if catalogue[name].wantsHard(want) {
return true
}
}
return false
}
wantedOptionally := func(want string) bool {
for _, name := range order {
if catalogue[name].IsOptional(want) {
return true
}
}
return false
}
leaveAbsent := func(want string) {
absent[want] = true
reported[want] = true
}
for {
if len(queue) == 0 {
if len(deferred) == 0 {
break
}
// What a module chosen since wants hard goes back to the walk, which may install for
// it; only when none does is the rest settled, since settling installs nothing and so
// can change no other name's answer.
var hard, optional []string
for _, d := range deferred {
if wantedHard(d) {
hard = append(hard, d)
} else {
optional = append(optional, d)
}
}
if len(hard) > 0 {
queue, deferred = hard, optional
continue
}
for _, d := range optional {
settling[d] = true
}
queue, deferred = optional, nil
}
want := queue[0]
queue = queue[1:]
if chosen[want] || reported[want] {
continue
}
if !settling[want] && because[want] != "assigned" && !wantedHard(want) && wantedOptionally(want) {
deferred = append(deferred, want)
continue
}
// Walked as a hard requirement: a refusal names a module that wants it hard, never one that
// wanted it only optionally and met it first — whose remedy it is not.
if first, known := catalogue[because[want]]; !settling[want] && known && !first.wantsHard(want) {
for _, name := range order {
if catalogue[name].wantsHard(want) {
because[want] = name
break
}
}
}
// Already answered by something in the set. This is the case that makes assigning zsh do
// what a person meant by it.
//
@@ -483,6 +555,12 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
if brokered[want] {
reported[want] = true
where := world.Offered[want]
if settling[want] && len(where) == 0 {
// Nothing assigned anywhere answers it: absent, and nothing is put on a machine
// for it (ADR 0330 §3), however few modules in the catalogue could.
leaveAbsent(want)
continue
}
names := providerNames(where)
take := func(p Provider) {
if node.At != "" && p.At == "" || node.At == "" && p.At != "" || node.At == "" && p.At == "" {
@@ -579,6 +657,10 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
// research 026 found. Another node's provider never counts — node scope is never brokered.
if machineReach[want] && !isModule(catalogue, want) {
reported[want] = true
if settling[want] {
leaveAbsent(want)
continue
}
if world.Unchecked {
// The first pass's refusals take a machine off the network; the second says it.
continue
@@ -591,6 +673,13 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
}
candidates := offers[want]
if settling[want] && !(len(candidates) == 0 && len(world.Licences[want]) > 0) {
// Not answered by anything in this node's set, and only installing could answer it
// here — which is never done for an optional requirement (ADR 0330 §3), not even
// for the only candidate. A licence record still answers it, in the post-pass.
leaveAbsent(want)
continue
}
switch len(candidates) {
case 0:
if len(world.Licences[want]) > 0 {
@@ -724,7 +813,8 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
resolution := Resolution{Node: node.Name, At: node.At, PublicDomain: node.PublicDomain,
Account: node.Account, AccountHome: node.AccountHome, AgentAccount: node.AgentAccount,
AgentAccountHome: node.AgentAccountHome, Capabilities: node.Capabilities,
Because: because, Needs: needs, Unhostable: unhostable, Kept: kept}
Because: because, Needs: needs, Unhostable: unhostable, Kept: kept,
Absent: absentOf(absent, order, catalogue)}
for _, n := range providersFirst(order, catalogue) {
resolution.Modules = append(resolution.Modules, catalogue[n])
}
-4
View File
@@ -410,10 +410,6 @@ func bundleWords(m Manifest, with Rendering) (map[string]string, error) {
if value, err = portsFilledInto(value, m.Module+"'s bundle "+b.Name+" ("+word+")", m.Module, m.Listens, with); err != nil {
return nil, err
}
// A joining module's identity and its libraries' host paths: its tools run on the machine (ADR 0328 §5).
if value, err = bundleLibraryFill(value, m, with); err != nil {
return nil, err
}
if out == nil {
out = map[string]string{}
}
+176
View File
@@ -0,0 +1,176 @@
package catalogue
import (
"fmt"
"strings"
"unicode"
)
// A mergeable file names the setting keys it takes (novox/hq ADR 0326, issue 168).
//
// Before this, every key of a module's layers merged into every one of its mergeable files: the
// mesh's own words, a key meant for a `${setting:<key>}` in another file, a key for a second
// configuration file. So a module could not own two configurable files, and every receiver had to
// ignore what it did not know. ADR 0164 §11 decided that a setting names the files that read it; this
// is that relation, stated from the file's side, so the operator's own values stay declared nowhere
// (ADR 0262 §2).
// TakesSettingsField is the key a mergeable file carries to name the top-level setting keys it takes.
const TakesSettingsField = "takes-settings"
// takenKeys is the keys a resource names in takes-settings, and whether it names any list at all. An
// empty list is a list: that file takes no setting.
func takenKeys(resource map[string]any) (map[string]bool, bool) {
raw, said := resource[TakesSettingsField]
if !said {
return nil, false
}
keys := map[string]bool{}
for _, k := range stringsOf(raw) {
keys[k] = true
}
return keys, true
}
// onlyTaken is the layers with every key the file does not name left out. The file's own defaults and
// what the mesh reads itself are not touched here: settle already keeps both out of a file.
func onlyTaken(layers []Layer, taken map[string]bool) []Layer {
kept := make([]Layer, 0, len(layers))
for _, layer := range layers {
values := map[string]any{}
for key, value := range layer.Values {
if taken[key] {
values[key] = value
}
}
kept = append(kept, Layer{From: layer.From, Values: values, Default: layer.Default})
}
return kept
}
// namesItsKeys says whether a module's mergeable files name the keys they take (ADR 0326 §2): true
// only when it has one and every one carries takes-settings. A module with a silent mergeable file
// keeps today's form (§3), where every key reaches every mergeable file.
func namesItsKeys(m Manifest) bool {
merging := 0
for _, r := range m.Resources {
if how, _ := r["merge"].(string); how == "" {
continue
}
merging++
if _, said := r[TakesSettingsField]; !said {
return false
}
}
return merging > 0
}
// TakenSettingsProblems is every way a definition's takes-settings is wrong (ADR 0326 §2 and §4), in
// its own words: on a file that does not merge, not a list of keys, a key named twice, one the mesh
// reads itself, one the file keeps for itself, or one mergeable file naming its keys beside another
// that names none.
func TakenSettingsProblems(m Manifest) []string {
// Not yet here: refusing a module that declares a setting and owns a mergeable file naming no keys
// (ADR 0326 §3). betterbird is one, and the catalogue cannot name its keys before this controller
// runs, so that refusal follows once the catalogue does (novox/hq issue 574).
return takenSettingsProblems(m)
}
// takenSettingsProblems is every rule of ADR 0326 §2 and §4, judged where a definition is checked and
// where a stored one is composed.
func takenSettingsProblems(m Manifest) []string {
var problems []string
var naming, silent []string
for _, r := range m.Resources {
id := fmt.Sprint(r["id"])
how, _ := r["merge"].(string)
raw, said := r[TakesSettingsField]
if how == "" {
if said {
problems = append(problems, fmt.Sprintf("%s's %s says %s and does not merge: only a mergeable "+
"file takes settings by key (novox/hq ADR 0326)", m.Module, id, TakesSettingsField))
}
continue
}
if !said {
silent = append(silent, id)
continue
}
naming = append(naming, id)
list, isList := raw.([]any)
if !isList {
problems = append(problems, fmt.Sprintf("%s's %s: %s is a list of setting keys, and this is %T",
m.Module, id, TakesSettingsField, raw))
continue
}
protected := map[string]bool{}
for _, k := range stringsOf(r["protected"]) {
protected[k] = true
}
seen := map[string]bool{}
for _, item := range list {
key, isString := item.(string)
switch {
case !isString || !usableKey(key):
problems = append(problems, fmt.Sprintf("%s's %s: %v in %s is not a usable setting key",
m.Module, id, item, TakesSettingsField))
case seen[key]:
problems = append(problems, fmt.Sprintf("%s's %s names %q twice in %s",
m.Module, id, key, TakesSettingsField))
case meshWords[key]:
problems = append(problems, fmt.Sprintf("%s's %s names %q in %s, and the mesh reads %q itself "+
"(novox/hq ADR 0164 §14)", m.Module, id, key, TakesSettingsField, key))
case protected[key]:
problems = append(problems, fmt.Sprintf("%s's %s names %q in %s and keeps it as protected: a "+
"key is the file's or the setting's, never both", m.Module, id, key, TakesSettingsField))
}
if isString {
seen[key] = true
}
}
}
if len(naming) > 0 {
for _, id := range silent {
problems = append(problems, fmt.Sprintf("%s's %s merges and names no %s while %s does: once one "+
"mergeable file of a module names the keys it takes, every one does, an empty list for none "+
"(novox/hq ADR 0326)", m.Module, id, TakesSettingsField, strings.Join(naming, ", ")))
}
}
return problems
}
// usableKey is a key a file can be said to take: not empty, and no space, line end or control
// character. Not the lower-case rule of a `${setting:<key>}`: a file's own keys are its software's
// spelling, such as `timeZone`.
func usableKey(key string) bool {
if key == "" {
return false
}
for _, r := range key {
if unicode.IsSpace(r) || unicode.IsControl(r) || !unicode.IsPrint(r) {
return false
}
}
return true
}
// takenSaid is what a stray key's message adds for a module whose files name their keys: which keys
// each file takes, so the person sees where the key would have had to be named.
func takenSaid(m Manifest) string {
if !namesItsKeys(m) {
return ""
}
var files []string
for _, r := range m.Resources {
if how, _ := r["merge"].(string); how == "" {
continue
}
keys := stringsOf(r[TakesSettingsField])
if len(keys) == 0 {
files = append(files, fmt.Sprintf("%v takes none", r["id"]))
continue
}
files = append(files, fmt.Sprintf("%v takes %s", r["id"], strings.Join(keys, ", ")))
}
return " (" + strings.Join(files, "; ") + ")"
}
+196
View File
@@ -0,0 +1,196 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// twoFiles is a module that owns two configurable files, which issue 168 said no module could:
// every key of every layer reached both.
func twoFiles(app, sidecar []any) Manifest {
appFile := map[string]any{"id": "app", "type": "file", "path": "/etc/app.json", "merge": MergeJSON,
"content": `{"server":{"port":8080},"search":{"safe":1}}`}
sideFile := map[string]any{"id": "sidecar", "type": "file", "path": "/etc/sidecar.json", "merge": MergeJSON,
"content": `{"url":"http://127.0.0.1:8080"}`}
if app != nil {
appFile[TakesSettingsField] = app
}
if sidecar != nil {
sideFile[TakesSettingsField] = sidecar
}
return Manifest{Module: "search", Resources: []map[string]any{
appFile, sideFile,
{"id": "env", "type": "file", "path": "/etc/search.env", "content": "SITE=${setting:site-name}\n"},
}}
}
func settledContent(t *testing.T, resource map[string]any, layers []Layer) map[string]any {
t.Helper()
out, err := ApplySettings(resource, layers)
if err != nil {
t.Fatal(err)
}
if _, kept := out[TakesSettingsField]; kept {
t.Errorf("%v still carries %s; the host knows no such field", out["id"], TakesSettingsField)
}
var parsed map[string]any
if err := json.Unmarshal([]byte(out["content"].(string)), &parsed); err != nil {
t.Fatalf("%v is not JSON after settling: %v", out["id"], err)
}
return parsed
}
func TestEachMergeableFileTakesOnlyTheKeysItNames(t *testing.T) {
// novox/hq ADR 0326 §1, issue 168: two configurable files, each taking its own keys. A key for
// the other file, for a ${setting:…} elsewhere or for the mesh itself lands in neither.
m := twoFiles([]any{"server", "search"}, []any{"url"})
layers := []Layer{
{From: "the mesh", Values: map[string]any{
"server": map[string]any{"base_url": "https://search.example.tld/"},
"url": "http://127.0.0.1:9090", "site-name": "Search",
"endpoints": map[string]any{"web": map[string]any{"reach": "public"}}}},
{From: "node-a", Values: map[string]any{"search": map[string]any{"safe": 0.0}}},
}
app := settledContent(t, m.Resources[0], layers)
for _, stray := range []string{"url", "site-name", "endpoints"} {
if _, in := app[stray]; in {
t.Errorf("app took %q, which it does not name: %v", stray, app)
}
}
server, _ := app["server"].(map[string]any)
if server["base_url"] != "https://search.example.tld/" || server["port"] != 8080.0 {
t.Errorf("app's server block is %v; the setting should merge into the file's own", server)
}
if search, _ := app["search"].(map[string]any); search["safe"] != 0.0 {
t.Errorf("app's search block is %v; the node's layer should win", search)
}
side := settledContent(t, m.Resources[1], layers)
if side["url"] != "http://127.0.0.1:9090" {
t.Errorf("sidecar's url is %v; it names url", side["url"])
}
if len(side) != 1 {
t.Errorf("sidecar took keys it does not name: %v", side)
}
}
func TestAnEmptyListTakesNoSetting(t *testing.T) {
// ADR 0164 §12: a file that names no setting takes none.
m := twoFiles([]any{"server"}, []any{})
side := settledContent(t, m.Resources[1], []Layer{{From: "the mesh", Values: map[string]any{
"server": map[string]any{"port": 1.0}, "url": "x"}}})
if len(side) != 1 || side["url"] != "http://127.0.0.1:8080" {
t.Errorf("a file taking none was changed: %v", side)
}
}
func TestAFileThatNamesNothingKeepsTodaysForm(t *testing.T) {
// ADR 0326 §3, ADR 0164 §15: a module none of whose files names keys is left as it was.
m := twoFiles(nil, nil)
app := settledContent(t, m.Resources[0], []Layer{{From: "the mesh", Values: map[string]any{"url": "x"}}})
if app["url"] != "x" {
t.Errorf("a silent file should still take every key until its module names them: %v", app)
}
if unused := UnusedSettings(m, []Layer{{From: "the mesh", Values: map[string]any{"anything": 1}}}); len(unused) != 0 {
t.Errorf("today's form has nothing stray, and said: %v", unused)
}
}
func TestAKeyNoFileNamesIsStray(t *testing.T) {
// ADR 0326 §5: once the files name their keys, a key none of them names — and nothing else
// takes — is stray, and is said with what each file takes.
m := twoFiles([]any{"server", "search"}, []any{"url"})
unused := UnusedSettings(m, []Layer{{From: "the mesh", Values: map[string]any{
"server": 1, "url": "x", "site-name": "Search", "ui": map[string]any{"theme": "dark"}}}})
if len(unused) != 1 || !strings.Contains(unused[0], `"ui"`) {
t.Fatalf("only ui reaches nothing; named: %v", unused)
}
if !strings.Contains(unused[0], "app takes server, search") || !strings.Contains(unused[0], "sidecar takes url") {
t.Errorf("the stray key's message does not say what each file takes: %s", unused[0])
}
}
func TestOneNamingFileMakesEveryMergeableFileName(t *testing.T) {
// ADR 0326 §2: a module with one naming and one silent mergeable file is refused, naming the
// silent one — the silent one would otherwise still take every key.
m := twoFiles([]any{"server"}, nil)
problems := TakenSettingsProblems(m)
if len(problems) != 1 || !strings.Contains(problems[0], "sidecar merges and names no takes-settings") {
t.Fatalf("problems: %v", problems)
}
if err := JudgeSettings(m, nil, false); err == nil || !strings.Contains(err.Error(), "sidecar") {
t.Errorf("composing it should refuse it too, and said: %v", err)
}
raw, err := json.Marshal(map[string]any{"module": m.Module, "version": "1", "resources": m.Resources})
if err != nil {
t.Fatal(err)
}
if _, err := ParseManifest(raw); err == nil || !strings.Contains(err.Error(), "sidecar merges and names no") {
t.Errorf("the module check does not say it: %v", err)
}
}
func TestTakesSettingsIsRefusedWhereItCannotMeanAnything(t *testing.T) {
// ADR 0326 §4.
cases := map[string]struct {
resource map[string]any
says string
}{
"on a file that does not merge": {
map[string]any{"id": "plain", "type": "file", "path": "/x", "content": "x",
TakesSettingsField: []any{"a"}},
"does not merge"},
"not a list": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
TakesSettingsField: "server"},
"is a list of setting keys"},
"a mesh word": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
TakesSettingsField: []any{"endpoints"}},
"the mesh reads"},
"a protected key": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
"protected": []any{"secret"}, TakesSettingsField: []any{"secret"}},
"keeps it as protected"},
"named twice": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
TakesSettingsField: []any{"a", "a"}},
"twice"},
"not a key": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
TakesSettingsField: []any{"two words"}},
"not a usable setting key"},
}
for name, c := range cases {
problems := TakenSettingsProblems(Manifest{Module: "m", Resources: []map[string]any{c.resource}})
if !containsText(problems, c.says) {
t.Errorf("%s: want a problem saying %q, got %v", name, c.says, problems)
}
}
// And a file that does not merge is refused where it is settled, not handed to the host.
plain := map[string]any{"id": "plain", "type": "file", "path": "/x", "content": "x",
TakesSettingsField: []any{"a"}}
if _, err := ApplySettings(plain, nil); err == nil {
t.Error("a file that does not merge and names keys was settled")
}
}
func TestASoftwareSpelledKeyIsUsable(t *testing.T) {
// A file's keys are its software's: nodered's timeZone is not lower case and is still a key.
m := Manifest{Module: "flows", Resources: []map[string]any{{"id": "settings", "type": "file",
"path": "/x", "merge": MergeJSON, "content": "{}", TakesSettingsField: []any{"timeZone"}}}}
if problems := TakenSettingsProblems(m); len(problems) != 0 {
t.Errorf("timeZone was refused: %v", problems)
}
}
func containsText(list []string, text string) bool {
for _, s := range list {
if strings.Contains(s, text) {
return true
}
}
return false
}
+31 -34
View File
@@ -46,11 +46,24 @@ type Layer struct {
// rather than by repeating everything the rest already say.
func ApplySettings(resource map[string]any, layers []Layer) (map[string]any, error) {
how, _ := resource["merge"].(string)
taken, names := takenKeys(resource)
if how == "" {
if names {
// Refused by the module check too (TakenSettingsProblems); said here as well, for a
// definition stored by an older binary, rather than handed to the host, which knows no
// such field.
return nil, fmt.Errorf("%v says %s and does not merge: only a mergeable file takes "+
"settings by key (novox/hq ADR 0326)", resource["id"], TakesSettingsField)
}
// Not a mergeable file. Settings for it are ignored rather than silently doing nothing
// somewhere else — see UnusedSettings, which is what says so.
return resource, nil
}
if names {
// **The file takes the keys it names, and no other** (novox/hq ADR 0326, issue 168): a key
// for another file, for a `${setting:<key>}` or for the mesh itself does not land here.
layers = onlyTaken(layers, taken)
}
if how != MergeJSON {
return nil, fmt.Errorf(
"%v says it merges as %q, and this control plane knows how to merge %q",
@@ -91,6 +104,7 @@ func ApplySettings(resource map[string]any, layers []Layer) (map[string]any, err
delete(out, "merge")
delete(out, "protected")
delete(out, TrustedField)
delete(out, TakesSettingsField)
return out, nil
}
@@ -324,8 +338,11 @@ func sortedKeysAny(m map[string]any) []string {
// (novox/hq 04-ISSUES/173); and the mesh's own words — `expose`, `ports`, `reach`, `endpoints` —
// are read by the mesh. A key none of those takes is stray, and is said so rather than dropped.
func UnusedSettings(m Manifest, layers []Layer) []string {
named := namesItsKeys(m)
for _, r := range m.Resources {
if how, _ := r["merge"].(string); how != "" {
if how, _ := r["merge"].(string); how != "" && !named {
// Today's form (novox/hq ADR 0326 §3, ADR 0164 §15): a mergeable file that names no
// keys takes every key, so nothing is stray.
return nil
}
}
@@ -337,9 +354,12 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
}
lands := settingKeysUsedBy(m)
// The libraries are the controller's own mesh-wide setting (novox/hq ADR 0328 §1).
if m.Module == LibrariesModule {
lands[LibrariesSetting] = true
// A mergeable file takes the keys it names (novox/hq ADR 0326 §5).
for _, r := range m.Resources {
taken, _ := takenKeys(r)
for key := range taken {
lands[key] = true
}
}
// A key the module gives a default reaches what asks for it (novox/hq ADR 0262): setting it
// overrides the default, which is how a running module gains a setting without a gap between.
@@ -405,8 +425,8 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
}
unused = append(unused, fmt.Sprintf(
"%s sets %q, and %s has no file that merges it, asks for no ${setting:%s}, and "+
"declares no %q in what it contributes or serves",
layer.From, key, m.Module, key, key))
"declares no %q in what it contributes or serves%s",
layer.From, key, m.Module, key, key, takenSaid(m)))
}
}
sort.Strings(unused)
@@ -513,37 +533,14 @@ var networkName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
// refused where it is stored (SetSettings, with UnusedSettings) and said where a plan is read,
// and never costs a module its place.
func JudgeSettings(m Manifest, layers []Layer, adopted bool) error {
if err := UndeclaredAccessPlaces(m, layers); err != nil {
return err
}
return ComposesWith(m, layers, adopted)
}
// ComposesWith is JudgeSettings for a layer stored ahead of the definition that reads it (novox/hq issue 575): an
// access placed that this definition does not declare yet is not refused, because composition passes it by — it
// reaches nothing until the build that declares it registers, and then it places that access. Where a machine is
// declared (LeftOutBecause), and where a setting judged against a newer build of the module is held to still compose
// with the one registered.
func ComposesWith(m Manifest, layers []Layer, adopted bool) error {
// The libraries, on the controller's own layer (novox/hq ADR 0328 §1): each well formed, and only mesh-wide.
if m.Module == LibrariesModule {
for _, l := range layers {
v, has := l.Values[LibrariesSetting]
if !has {
continue
}
if l.From != MeshWideLayer && !l.Default {
return fmt.Errorf("%s is the mesh's, set on the whole mesh and never on one machine (%s §1)",
LibrariesSetting, ADR0328)
}
if _, err := ParseLibraries(v); err != nil {
return err
}
}
}
if err := settingsHoldOneLine(m.Module, layers); err != nil {
return err
}
// Which keys each mergeable file takes is the definition's to say right (novox/hq ADR 0326);
// judged here too for a definition stored by an older binary.
if problems := takenSettingsProblems(m); len(problems) > 0 {
return fmt.Errorf("%s", problems[0])
}
// With no layers too: a definition may ask for a setting nobody made — an access placed by
// nobody, a file's ${setting:…} nothing sets — and that is the same statement, missing.
if _, err := GivenPorts(m, layers); err != nil {
-4
View File
@@ -79,10 +79,6 @@ func TerminalKeys(m Manifest) []string {
}
delete(keys, PlacesSetting)
delete(keys, AccessesSetting)
// The libraries say where shared data is and who may write it (novox/hq ADR 0328 §1): the operator's alone.
if m.Module == LibrariesModule {
keys[LibrariesSetting] = true
}
rest := make([]string, 0, len(keys))
for k := range keys {
rest = append(rest, k)
@@ -59,12 +59,6 @@ var fillers = []filler{
{namespace: "setting", patterns: []*regexp.Regexp{settingRef}, fields: map[string][]string{"file": {"content"}, "service": {"unit"}, "directory": {"owner"}}},
{namespace: "dir", patterns: []*regexp.Regexp{dirRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
{namespace: "access", patterns: []*regexp.Regexp{accessRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
// A joining module's identity, wherever its resources take values; a library's host path, only in a container's
// volumes, whose source libraryProblems holds it to (novox/hq ADR 0328 §5).
{namespace: "identity", patterns: []*regexp.Regexp{identityRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file", "run", "command", "user", "name", "unit"}}},
{namespace: "library", patterns: []*regexp.Regexp{libraryRef}, fields: map[string][]string{"container": {"volumes"}},
why: "a library's path is a host path, filled only in a container's volume source; the process reads the " +
"container side its volume gives (novox/hq ADR 0328 §5)"},
{namespace: "secret", patterns: []*regexp.Regexp{placeholder}, fields: map[string][]string{"file": {"content"}},
why: "a secret is never filled into an environment variable or any other field: put it in a file the " +
"module declares and mount that (novox/hq ADR 0086)"},
+1 -74
View File
@@ -117,10 +117,6 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
}
}
}
// A module never takes a defined library's name: mesh-<module> and mesh-<library> never collide (ADR 0328 §4).
if err := i.LibraryNamed(ctx, m.Module); err != nil {
return fmt.Errorf("%s is not registered: %w", m.Module, err)
}
raw, err := json.Marshal(m)
if err != nil {
return err
@@ -933,10 +929,6 @@ func (i *Inventory) setSettings(ctx context.Context, nodeName, module string, va
// the mesh-wide layer under it when the layer is one node's, and refuses the first thing that
// cannot work (ADR 0163, rule 6). The same judgement composition makes; what passes here composes.
func (i *Inventory) judgeSettings(ctx context.Context, nodeName, module string, values map[string]any) error {
// The libraries the controller's own layer defines (novox/hq ADR 0328 §1, §4, §13).
if err := i.judgeLibraries(ctx, nodeName, module, values); err != nil {
return err
}
m, err := i.declared(ctx, module)
if err != nil {
return fmt.Errorf("%w: %s", ErrNoSuchModule, module)
@@ -966,38 +958,7 @@ func (i *Inventory) judgeSettings(ctx context.Context, nodeName, module string,
}
layers = append(layers, catalogue.Layer{From: from, Values: values})
if err := catalogue.JudgeSettings(m, layers, adopted); err != nil {
// **A placement for an access the module's newer build brings** (novox/hq issue 575): judged against the
// manifest of the module's newest build on the branch it follows, where the registered one does not declare
// the access yet — so an access can be placed before or with the delivery that adds it. It must still compose
// with the registered manifest, which passes the placement by until the build registers.
var undeclared *catalogue.UndeclaredAccessError
if !errors.As(err, &undeclared) {
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w", module, where, from, err)
}
ahead, has, aerr := i.NewestManifestOnBranch(ctx, module)
if aerr != nil {
return aerr
}
if !has {
follows, ferr := i.followsABranch(ctx, module)
if ferr != nil {
return ferr
}
if !follows {
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w; and %s follows no branch "+
"(it was registered by hand), so no build of it can be judged ahead of what is registered", module,
where, from, err, module)
}
}
if !has || catalogue.JudgeSettings(ahead, layers, adopted) != nil || catalogue.ComposesWith(m, layers, adopted) != nil {
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w; nor does the newest build of "+
"%s on the branch it follows declare it", module, where, from, err, module)
}
m = ahead
}
// A module that joins carries no user or group number or owner in a setting (novox/hq ADR 0328 §5).
if err := catalogue.JoiningSettingProblem(m, layers[len(layers)-1:]); err != nil {
return fmt.Errorf("refused: %w", err)
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w", module, where, from, err)
}
// And a key that reaches nothing, refused here where somebody can still fix the spelling:
// stored, it would be a setting somebody believes they made.
@@ -1008,40 +969,6 @@ func (i *Inventory) judgeSettings(ctx context.Context, nodeName, module string,
return nil
}
// followsABranch says a module's record names the branch it follows.
func (i *Inventory) followsABranch(ctx context.Context, module string) (bool, error) {
var ref string
err := i.store.Pool().QueryRow(ctx, `select coalesce(ref, '') from module where name = $1`, module).Scan(&ref)
if errors.Is(err, pgx.ErrNoRows) {
return false, nil
}
return ref != "", err
}
// NewestManifestOnBranch is the manifest of a module's newest successful build on the branch its record follows
// (novox/hq issue 575): what the delivery under way brings, before it registers — or the registered one's own, when
// nothing newer was built. Only a build of that very branch: never one of another branch or of none. False when no
// such build is kept, and for a module whose record follows no branch (registered by hand).
func (i *Inventory) NewestManifestOnBranch(ctx context.Context, module string) (catalogue.Manifest, bool, error) {
var raw []byte
err := i.store.Pool().QueryRow(ctx,
`select b.manifest from build b join module m on m.name = b.module
where b.module = $1 and b.failed = '' and b.manifest is not null and b.manifest::text <> 'null'
and coalesce(m.ref, '') <> '' and b.ref = m.ref
order by b.at desc limit 1`, module).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return catalogue.Manifest{}, false, nil
}
if err != nil {
return catalogue.Manifest{}, false, err
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
return catalogue.Manifest{}, false, nil // a build the mesh cannot read judges nothing
}
return m, true, nil
}
// givenIn is the machine ports a node-level settings layer gives a module, software port →
// machine port (novox/hq ADR 0100). Nothing when the layer gives none; what is not a port is left
// for composition to refuse in its own words.
+3 -12
View File
@@ -38,22 +38,13 @@ type ResourceHealth struct {
Waits []Wait `json:"waits,omitempty"`
}
// Wait is one thing a waiting resource waits for the operator to give: exactly one of Secret, Setting and Access
// (novox/hq ADR 0283, ADR 0325 §7), as link.Wait carries it.
// Wait is one thing a waiting resource waits for the operator to give: exactly one of Secret and Setting
// (novox/hq ADR 0283), as link.Wait carries it.
type Wait struct {
Part string `json:"part"`
Secret string `json:"secret,omitempty"`
Setting string `json:"setting,omitempty"`
// Access is an access of the module by its own id for it, from the engine's access-waits, and Disk that it
// waits for the disk that holds its folder rather than for the operator's word (novox/hq ADR 0325 §7, §9).
Access string `json:"access,omitempty"`
Disk bool `json:"disk,omitempty"`
// Placed is the filesystem the folder is on now, without its path (ADR 0325 §8): a warrant for another does not
// answer the wait.
Placed string `json:"placed,omitempty"`
// Reason is the engine's reason for an access wait: "keep" or "disk".
Reason string `json:"reason,omitempty"`
What string `json:"what"`
What string `json:"what"`
}
// NodeHealth is a machine's newest statement, as kept.
-331
View File
@@ -1,331 +0,0 @@
package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
"path/filepath"
"strings"
"time"
"github.com/jackc/pgx/v5"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Joining modules' numbers, library groups, taken libraries and the libraries themselves (novox/hq ADR 0328 §1, §3,
// §4, §7, §15; migration 0096).
// AllocateIdentity is a joining module's number, allocated now when it has none: one number from the mesh's range,
// mesh-wide, kept for ever, its user's and its own group's. Two controllers allocating at once agree: the second's
// insert finds the first's row.
func (i *Inventory) AllocateIdentity(ctx context.Context, module string) (catalogue.ModuleIdentity, error) {
if strings.TrimSpace(module) == "" {
return catalogue.ModuleIdentity{}, errors.New("a number is allocated for a module, and none was named")
}
if id, had, err := i.IdentityOf(ctx, module); err != nil || had {
return id, err
}
// Drawn only when the row is new; a draw lost to a concurrent insert is a gap in the range, never a number given
// twice.
if _, err := i.store.Pool().Exec(ctx,
`insert into module_identity (module, uid) select $1, nextval('mesh_number')::integer
on conflict (module) do nothing`, module); err != nil {
return catalogue.ModuleIdentity{}, fmt.Errorf("%s could not be given its number: %w", module, err)
}
id, had, err := i.IdentityOf(ctx, module)
if err == nil && !had {
err = fmt.Errorf("%s was given its number and it cannot be read back", module)
}
return id, err
}
// IdentityOf is a module's number, and false when it was never allocated one.
func (i *Inventory) IdentityOf(ctx context.Context, module string) (catalogue.ModuleIdentity, bool, error) {
id := catalogue.ModuleIdentity{Module: module}
err := i.store.Pool().QueryRow(ctx, `select uid from module_identity where module = $1`, module).Scan(&id.UID)
if errors.Is(err, pgx.ErrNoRows) {
return catalogue.ModuleIdentity{}, false, nil
}
if err != nil {
return catalogue.ModuleIdentity{}, false, err
}
return id, true, nil
}
// Identities is every joining module's number, by module.
func (i *Inventory) Identities(ctx context.Context) (map[string]catalogue.ModuleIdentity, error) {
rows, err := i.store.Pool().Query(ctx, `select module, uid from module_identity`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]catalogue.ModuleIdentity{}
for rows.Next() {
var id catalogue.ModuleIdentity
if err := rows.Scan(&id.Module, &id.UID); err != nil {
return nil, err
}
out[id.Module] = id
}
return out, rows.Err()
}
// AllocateLibraryGroup is a library's group number, allocated now when it has none, from the same sequence as the
// modules' numbers (§3).
func (i *Inventory) AllocateLibraryGroup(ctx context.Context, library string) (int, error) {
if strings.TrimSpace(library) == "" {
return 0, errors.New("a group number is allocated for a library, and none was named")
}
groups, err := i.LibraryGroups(ctx)
if err != nil {
return 0, err
}
if gid, had := groups[library]; had {
return gid, nil
}
if _, err := i.store.Pool().Exec(ctx,
`insert into library_group (library, gid) select $1, nextval('mesh_number')::integer
on conflict (library) do nothing`, library); err != nil {
return 0, fmt.Errorf("the library %s could not be given its group's number: %w", library, err)
}
if groups, err = i.LibraryGroups(ctx); err != nil {
return 0, err
}
gid, has := groups[library]
if !has {
return 0, fmt.Errorf("the library %s was given its group's number and it cannot be read back", library)
}
return gid, nil
}
// LibraryGroups is every library's group number, by library.
func (i *Inventory) LibraryGroups(ctx context.Context) (map[string]int, error) {
rows, err := i.store.Pool().Query(ctx, `select library, gid from library_group`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]int{}
for rows.Next() {
var name string
var gid int
if err := rows.Scan(&name, &gid); err != nil {
return nil, err
}
out[name] = gid
}
return out, rows.Err()
}
// Libraries is every library the operator defined (§1): the mesh-wide `libraries` layer of the controller's module.
// None when it was never set.
func (i *Inventory) Libraries(ctx context.Context) (map[string]catalogue.Library, error) {
var raw []byte
err := i.store.Pool().QueryRow(ctx,
`select values from settings where module = $1 and node is null`, catalogue.LibrariesModule).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return nil, nil
}
if err != nil {
return nil, err
}
var layer map[string]any
if err := json.Unmarshal(raw, &layer); err != nil {
return nil, err
}
return catalogue.ParseLibraries(layer[catalogue.LibrariesSetting])
}
// judgeLibraries is what a layer of the controller's module is held to when it defines libraries (§1, §4, §13): only
// mesh-wide, well formed, no library named as a registered module, none overlapping another on a machine.
func (i *Inventory) judgeLibraries(ctx context.Context, nodeName, module string, values map[string]any) error {
v, defines := values[catalogue.LibrariesSetting]
if module != catalogue.LibrariesModule || !defines {
return nil
}
if nodeName != "" {
return fmt.Errorf("refused: %s is the mesh's, set on the whole mesh and never on one machine (%s §1)",
catalogue.LibrariesSetting, catalogue.ADR0328)
}
libs, err := catalogue.ParseLibraries(v)
if err != nil {
return fmt.Errorf("refused: %w", err)
}
shelf, err := i.Catalogue(ctx)
if err != nil {
return err
}
modules := make([]string, 0, len(shelf))
for name := range shelf {
modules = append(modules, name)
}
if problems := catalogue.LibrariesProblems(libs, modules); len(problems) > 0 {
return fmt.Errorf("refused: %s", strings.Join(problems, "; "))
}
return nil
}
// LibraryNamed refuses a module whose name is a defined library's (§4): mesh-<module> and mesh-<library> never collide.
func (i *Inventory) LibraryNamed(ctx context.Context, module string) error {
libs, err := i.Libraries(ctx)
if err != nil {
return err
}
if _, clash := libs[module]; clash {
return fmt.Errorf("the module %s has the name of the library %s: %s would be both (%s §4)", module, module,
catalogue.MeshName(module), catalogue.ADR0328)
}
return nil
}
// LibraryPaths is every sub-path a module on any machine asks of each library (§6): what its home keeps. A module on a
// machine is one assigned there or one a requirement pulled in — what the machine was last sent.
func (i *Inventory) LibraryPaths(ctx context.Context) (map[string][]string, error) {
shelf, err := i.Catalogue(ctx)
if err != nil {
return nil, err
}
// Every module assigned anywhere, and every module a machine was last sent — those a requirement pulled in too.
rows, err := i.store.Pool().Query(ctx,
`select module from assignment
union select jsonb_object_keys(sent_builds) from node where sent_builds is not null and jsonb_typeof(sent_builds) = 'object'`)
if err != nil {
return nil, err
}
defer rows.Close()
seen := map[string]map[string]bool{}
for rows.Next() {
var module string
if err := rows.Scan(&module); err != nil {
return nil, err
}
for _, u := range shelf[module].Libraries {
if seen[u.Library] == nil {
seen[u.Library] = map[string]bool{}
}
seen[u.Library][filepath.Clean(u.Path)] = true
}
}
if err := rows.Err(); err != nil {
return nil, err
}
out := map[string][]string{}
for lib, paths := range seen {
for p := range paths {
out[lib] = append(out[lib], p)
}
}
return out, nil
}
// TakenLibrary is one library the operator let the mesh take, on its home (§7).
type TakenLibrary struct {
Library string
Home string
Root string
Placed string
By string
Ask string
Warrant string
Said string
// Done is whether the engine stated the take's act done: only then is the library taken.
Done bool
}
// TakenLibraries is every library whose take was started, by name; Done says it is taken.
func (i *Inventory) TakenLibraries(ctx context.Context) (map[string]TakenLibrary, error) {
rows, err := i.store.Pool().Query(ctx,
`select library, home, root, placed, taken_by, ask, warrant, said, done from kept_library`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]TakenLibrary{}
for rows.Next() {
var t TakenLibrary
if err := rows.Scan(&t.Library, &t.Home, &t.Root, &t.Placed, &t.By, &t.Ask, &t.Warrant, &t.Said, &t.Done); err != nil {
return nil, err
}
out[t.Library] = t
}
return out, rows.Err()
}
// RecordTakeStarted keeps that a take of a library was started on its home, under the operator's warrant: the
// engine said it began. The library is taken only once the engine states the act done (TakeDone). Once per home, root
// and filesystem: a second start of the same keeps the first, and one on another (it moved, and the operator was asked
// again) replaces it, not done.
func (i *Inventory) RecordTakeStarted(ctx context.Context, t TakenLibrary) error {
switch {
case t.Library == "" || t.Home == "" || !strings.HasPrefix(t.Root, "/"):
return errors.New("a taken library names itself, its home and its root")
case strings.TrimSpace(t.Warrant) == "" || strings.TrimSpace(t.Ask) == "":
return errors.New("a library is taken only on the operator's warrant, and this names none")
}
_, err := i.store.Pool().Exec(ctx,
`insert into kept_library (library, home, root, placed, taken_by, ask, warrant, said, done)
values ($1, $2, $3, $4, $5, $6, $7, $8, false)
on conflict (library) do update
set home = excluded.home, root = excluded.root, placed = excluded.placed, taken_at = now(),
taken_by = excluded.taken_by, ask = excluded.ask, warrant = excluded.warrant, said = excluded.said,
done = false
where kept_library.home <> excluded.home or kept_library.root <> excluded.root
or kept_library.placed <> excluded.placed`,
t.Library, t.Home, t.Root, t.Placed, t.By, t.Ask, t.Warrant, t.Said)
return err
}
// TakeDone keeps that the engine stated a library's take done under a warrant: the library is taken. A warrant the
// recorded take was not started under changes nothing.
func (i *Inventory) TakeDone(ctx context.Context, library, warrant string) error {
_, err := i.store.Pool().Exec(ctx,
`update kept_library set done = true where library = $1 and warrant = $2 and not done`, library, warrant)
return err
}
// TakeFailed forgets a take the engine stated failed under a warrant, never one done: the library is asked about again.
func (i *Inventory) TakeFailed(ctx context.Context, library, warrant string) error {
_, err := i.store.Pool().Exec(ctx,
`delete from kept_library where library = $1 and warrant = $2 and not done`, library, warrant)
return err
}
// SettingsAsOf is a module's settings layers on a machine as they stood at a moment: each layer — the whole mesh's,
// then the machine's — as it was set then, read from the layer standing now or from its history. What a module held at
// the build it ran is composed with: the settings it last ran with (novox/hq ADR 0328 §8).
func (i *Inventory) SettingsAsOf(ctx context.Context, nodeName, module string, at time.Time) ([]catalogue.Layer, error) {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return nil, err
}
var layers []catalogue.Layer
for _, scope := range []struct {
id *string
from string
}{{nil, catalogue.MeshWideLayer}, {&node.ID, nodeName}} {
var raw []byte
err := i.store.Pool().QueryRow(ctx,
`select values from settings
where module = $1 and node is not distinct from $2::uuid and set_at <= $3
union all
select values from (select values from settings_history
where module = $1 and node is not distinct from $2::uuid
and (set_at is null or set_at <= $3) and replaced_at > $3
order by replaced_at asc limit 1) h
limit 1`, module, scope.id, at).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
continue
}
if err != nil {
return nil, err
}
values := map[string]any{}
if err := json.Unmarshal(raw, &values); err != nil {
return nil, err
}
layers = append(layers, catalogue.Layer{From: scope.from, Values: values})
}
return layers, nil
}
-393
View File
@@ -1,393 +0,0 @@
package inventory
import (
"encoding/json"
"fmt"
"sort"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq ADR 0328 §3, §4, §15: one number per joining module (its user's and its own group's) and one per library's
// group, from one sequence and one range — never 0 or a root group's, never shared between a module and a library,
// never given twice, the same however often asked.
func TestOneNumberPerModuleAndLibraryNeverRoot(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
radarr, err := inv.AllocateIdentity(ctx, "radarr")
if err != nil {
t.Fatal(err)
}
again, err := inv.AllocateIdentity(ctx, "radarr")
if err != nil || again != radarr {
t.Fatalf("radarr was given %+v, then %+v (%v): one number per module", radarr, again, err)
}
media, err := inv.AllocateLibraryGroup(ctx, "media")
if err != nil {
t.Fatal(err)
}
if mediaAgain, err := inv.AllocateLibraryGroup(ctx, "media"); err != nil || mediaAgain != media {
t.Fatalf("the media group was given %d, then %d (%v)", media, mediaAgain, err)
}
plex, err := inv.AllocateIdentity(ctx, "plex")
if err != nil {
t.Fatal(err)
}
seen := map[int]string{}
for who, n := range map[string]int{"radarr": radarr.UID, "plex": plex.UID, "mesh-media": media} {
if n == 0 || !catalogue.AllocatableID(n) {
t.Errorf("%s was given %d: within %d–%d, never 0", who, n, catalogue.IdentityLow, catalogue.IdentityHigh)
}
if other, twice := seen[n]; twice {
t.Errorf("%s and %s share %d: a module's own group and a library's never do", who, other, n)
}
seen[n] = who
}
// Read back as every machine's composition reads them: the same numbers.
ids, err := inv.Identities(ctx)
if err != nil {
t.Fatal(err)
}
groups, err := inv.LibraryGroups(ctx)
if err != nil {
t.Fatal(err)
}
if ids["radarr"].UID != radarr.UID || groups["media"] != media {
t.Fatalf("read back %+v and %v", ids, groups)
}
// A forgotten row frees no number: the next one is new.
if _, err := inv.store.Pool().Exec(ctx, `delete from module_identity where module = 'plex'`); err != nil {
t.Fatal(err)
}
sonarr, err := inv.AllocateIdentity(ctx, "sonarr")
if err != nil {
t.Fatal(err)
}
if _, given := seen[sonarr.UID]; given {
t.Fatalf("sonarr was given %d, a number given before", sonarr.UID)
}
// The store refuses a number outside the range, root's above all.
for _, n := range []int{0, 4, 10, 1000, 49999, 60000} {
if _, err := inv.store.Pool().Exec(ctx, `insert into module_identity (module, uid) values ('byhand', $1)`, n); err == nil {
t.Errorf("the store took %d as a module's number", n)
}
if _, err := inv.store.Pool().Exec(ctx, `insert into library_group (library, gid) values ('byhand', $1)`, n); err == nil {
t.Errorf("the store took %d as a library group's number", n)
}
}
}
// libraryController registers the controller's own module, whose mesh-wide layer defines the libraries.
func libraryController(t *testing.T, inv *Inventory) {
t.Helper()
m := catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "controller", "image": "x"}}}
if err := inv.RegisterModule(t.Context(), m, Source{}); err != nil {
t.Fatal(err)
}
}
// novox/hq ADR 0328 §1, §4, §13: a library is defined only on the controller's mesh-wide layer, as a trusted setting;
// never on one machine; never overlapping another on a machine it reaches; never named as a registered module — and no
// module is registered under a defined library's name.
func TestALibraryIsDefinedOnceForTheMeshAndNeverOverlaps(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
for _, n := range []string{"ace", "novox"} {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
libraryController(t, inv)
radarr := catalogue.Manifest{Module: "radarr", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "radarr", "image": "x"}}}
if err := inv.RegisterModule(ctx, radarr, Source{}); err != nil {
t.Fatal(err)
}
media := map[string]any{"home": "ace", "root": "/storage/media", "reach": []any{"novox"}}
define := func(libs map[string]any) map[string]any {
return map[string]any{catalogue.LibrariesSetting: libs}
}
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, define(map[string]any{"media": media})); err != nil {
t.Fatalf("a library was not defined: %v", err)
}
libs, err := inv.Libraries(ctx)
if err != nil {
t.Fatal(err)
}
if l := libs["media"]; l.Home != "ace" || l.Root != "/storage/media" || !l.Reaches("novox") || !l.Reaches("ace") {
t.Fatalf("the library reads %+v", libs["media"])
}
for name, c := range map[string]struct {
node string
libs map[string]any
want string
}{
"on one machine": {"ace", map[string]any{"media": media}, "never on one machine"},
"inside another": {"", map[string]any{"media": media,
"movies": map[string]any{"home": "novox", "root": "/storage/media/movies"}}, "overlap on"},
"around another": {"", map[string]any{"movies": map[string]any{"home": "ace", "root": "/storage/media/movies"},
"media": media}, "overlap on"},
"named as a module": {"", map[string]any{"radarr": map[string]any{"home": "ace", "root": "/srv/radarr-data"}},
"name of the module radarr"},
"with an owner": {"", map[string]any{"books": map[string]any{"home": "ace", "root": "/srv/books",
"owner": "1000"}}, "nothing about an owner"},
} {
err := inv.SetSettings(ctx, c.node, catalogue.LibrariesModule, define(c.libs))
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: %v; want refused saying %q", name, err, c.want)
}
}
// Two libraries apart on different machines may share a path's prefix only where they do not meet.
apart := map[string]any{"media": media, "photos": map[string]any{"home": "elsewhere", "root": "/storage/media/photos"}}
if err := inv.JudgeSettings(ctx, "", catalogue.LibrariesModule, define(apart)); err != nil {
t.Errorf("libraries that meet on no machine were refused: %v", err)
}
// A module under a defined library's name is refused.
named := catalogue.Manifest{Module: "media", Version: "1",
Resources: []map[string]any{{"id": "s", "type": "container", "name": "media", "image": "x"}}}
if err := inv.RegisterModule(ctx, named, Source{}); err == nil || !strings.Contains(err.Error(), "name of the library media") {
t.Fatalf("a module named as a library: %v", err)
}
}
// novox/hq ADR 0328 §7: a library taken is recorded once per home and filesystem, under a warrant; one taken again on
// another filesystem replaces it.
func TestALibraryIsTakenOnceUnderAWarrant(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if err := inv.RecordTakeStarted(ctx, TakenLibrary{Library: "media", Home: "ace", By: "the operator"}); err == nil {
t.Fatal("a library was recorded taken with no warrant")
}
first := TakenLibrary{Library: "media", Home: "ace", Root: "/storage/media", Placed: "fs-000000000001",
By: "the operator", Ask: "c01", Warrant: "c01"}
if err := inv.RecordTakeStarted(ctx, first); err != nil {
t.Fatal(err)
}
again := first
again.Ask, again.Warrant, again.By = "c02", "c02", "someone else"
if err := inv.RecordTakeStarted(ctx, again); err != nil {
t.Fatal(err)
}
taken, err := inv.TakenLibraries(ctx)
if err != nil {
t.Fatal(err)
}
if got := taken["media"]; got.Ask != "c01" {
t.Fatalf("a second take of the same overwrote the first: %+v", got)
}
moved := first
moved.Placed, moved.Ask, moved.Warrant = "fs-0000000000ff", "c03", "c03"
if err := inv.RecordTakeStarted(ctx, moved); err != nil {
t.Fatal(err)
}
if taken, err = inv.TakenLibraries(ctx); err != nil || taken["media"].Ask != "c03" {
t.Fatalf("a take on another filesystem: %+v (%v)", taken["media"], err)
}
// The review of 7ea6d36e, item 3: the root is part of what was taken — a library whose root changed is taken
// again, and the new take replaces the old.
if got := taken["media"].Root; got != "/storage/media" {
t.Fatalf("the take recorded the root %q", got)
}
rooted := moved
rooted.Root, rooted.Ask, rooted.Warrant = "/storage/media2", "c04", "c04"
if err := inv.RecordTakeStarted(ctx, rooted); err != nil {
t.Fatal(err)
}
if taken, err = inv.TakenLibraries(ctx); err != nil || taken["media"].Ask != "c04" || taken["media"].Root != "/storage/media2" {
t.Fatalf("a take at another root: %+v (%v)", taken["media"], err)
}
}
// novox/hq issue 575: a module registered without the access movies, whose newest build on its branch declares it —
// the build the delivery under way brings — takes a proposal placing movies: judged against that build, not refused.
// Without such a build it is refused, as before.
func TestAPlacementIsJudgedAgainstTheBuildThatBringsTheAccess(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "ace"); err != nil {
t.Fatal(err)
}
registered := catalogue.Manifest{Module: "media-importer", Version: "1",
Accesses: []catalogue.Access{{ID: "downloads", Path: "/services/media/downloads", Mode: catalogue.AccessReadWrite}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "importer", "image": "x",
"volumes": []any{"${access:downloads}:/downloads"}}}}
if err := inv.RegisterModule(ctx, registered, Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c0ffee33"}); err != nil {
t.Fatal(err)
}
placing := map[string]any{catalogue.AccessesSetting: map[string]any{"movies": "/storage/media/movies"}}
err := inv.JudgeSettings(ctx, "ace", "media-importer", placing)
if err == nil || !strings.Contains(err.Error(), `places the access "movies", which it does not declare`) {
t.Fatalf("a placement no build declares: %v", err)
}
// main's manifest — the build of #34 — declares movies.
ahead := registered
ahead.Accesses = append([]catalogue.Access{}, registered.Accesses...)
ahead.Accesses = append(ahead.Accesses, catalogue.Access{ID: "movies", Path: "/services/media/movies",
Mode: catalogue.AccessReadWrite})
raw, err := json.Marshal(ahead)
if err != nil {
t.Fatal(err)
}
b := aBuild("b34", "media-importer", "")
b.Ref, b.Manifest = "main", raw
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
if err := inv.JudgeSettings(ctx, "ace", "media-importer", placing); err != nil {
t.Fatalf("the proposal placing movies, which main's build declares, was refused: %v", err)
}
if err := inv.SetSettings(ctx, "ace", "media-importer", placing); err != nil {
t.Fatalf("the layer placing movies was not stored: %v", err)
}
// Stored ahead, it does not cost the registered module its place: composition passes it by.
layers, err := inv.SettingsFor(ctx, "ace", "media-importer")
if err != nil {
t.Fatal(err)
}
if err := catalogue.ComposesWith(registered, layers, false); err != nil {
t.Fatalf("the registered module no longer composes with the layer stored ahead: %v", err)
}
// A typo is still a typo: an access neither build declares is refused.
typo := map[string]any{catalogue.AccessesSetting: map[string]any{"moveis": "/storage/media/movies"}}
if err := inv.JudgeSettings(ctx, "ace", "media-importer", typo); err == nil {
t.Fatal("a placement no build declares was taken")
}
}
// Review of e1c2b744: only a build of the branch the module's record follows judges a placement ahead of it — never one
// from another branch, nor one with no branch, and a module registered with no branch is refused as before, saying so.
func TestOnlyABuildOfTheFollowedBranchJudgesAPlacementAhead(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "ace"); err != nil {
t.Fatal(err)
}
registered := catalogue.Manifest{Module: "importer", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "importer", "image": "x"}}}
ahead := registered
ahead.Accesses = []catalogue.Access{{ID: "movies", Path: "/services/media/movies", Mode: catalogue.AccessReadWrite}}
raw, err := json.Marshal(ahead)
if err != nil {
t.Fatal(err)
}
placing := map[string]any{catalogue.AccessesSetting: map[string]any{"movies": "/storage/media/movies"}}
if err := inv.RegisterModule(ctx, registered, Source{Repository: "https://forge.invalid/media.git", Ref: "main",
BuiltFrom: "c0ffee"}); err != nil {
t.Fatal(err)
}
for i, ref := range []string{"feature", ""} {
b := aBuild(fmt.Sprintf("elsewhere%d", i), "importer", "")
b.Ref, b.Manifest = ref, raw
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
if err := inv.JudgeSettings(ctx, "ace", "importer", placing); err == nil {
t.Fatal("a build of another branch, or of none, judged the placement")
}
// Registered by hand, with no branch: refused, saying why.
byHand := registered
byHand.Module = "byhand"
if err := inv.RegisterModule(ctx, byHand, Source{}); err != nil {
t.Fatal(err)
}
aheadByHand := ahead
aheadByHand.Module = "byhand"
if raw, err = json.Marshal(aheadByHand); err != nil {
t.Fatal(err)
}
b := aBuild("onmain", "byhand", "")
b.Ref, b.Manifest = "main", raw
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
if err := inv.JudgeSettings(ctx, "ace", "byhand", placing); err == nil ||
!strings.Contains(err.Error(), "follows no branch") {
t.Fatalf("a module with no branch recorded: %v", err)
}
}
// The review of 7ea6d36e, item 4: every sub-path a module asks of a library is kept on its home, whether the module
// was assigned or pulled in by a requirement (it is in what its machine was last sent).
func TestTheSubPathsOfModulesPulledInByARequirementAreKept(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
record, err := inv.AddNode(ctx, "ace")
if err != nil {
t.Fatal(err)
}
for _, m := range []catalogue.Manifest{
{Module: "radarr", Version: "1", Libraries: []catalogue.LibraryUse{{ID: "m", Library: "media", Path: "movies", Use: "read-write"}}},
{Module: "importer", Version: "1", Libraries: []catalogue.LibraryUse{{ID: "i", Library: "media", Path: "incoming", Use: "read-write"}}},
} {
if err := inv.RegisterModule(ctx, m, Source{}); err != nil {
t.Fatal(err)
}
}
if _, err := inv.Assign(ctx, "ace", "radarr"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": "c1", "importer": "c2"}, 1, 1,
Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
paths, err := inv.LibraryPaths(ctx)
if err != nil {
t.Fatal(err)
}
got := append([]string{}, paths["media"]...)
sort.Strings(got)
if strings.Join(got, ",") != "incoming,movies" {
t.Fatalf("the paths kept of media are %v; want the importer's, pulled in by a requirement, too", got)
}
}
// The review of 7ea6d36e, item 5: a take the engine says it started is recorded as started, not taken; it is taken when
// the engine states the act done under that warrant, and forgotten when it states it failed, so the operator is asked
// again.
func TestALibraryIsTakenWhenTheActIsDone(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
started := TakenLibrary{Library: "media", Home: "ace", Root: "/storage/media", Placed: "fs-000000000001",
By: "the operator", Ask: "c01", Warrant: "c01"}
if err := inv.RecordTakeStarted(ctx, started); err != nil {
t.Fatal(err)
}
taken, err := inv.TakenLibraries(ctx)
if err != nil {
t.Fatal(err)
}
if got := taken["media"]; got.Done || got.Ask != "c01" {
t.Fatalf("a take only started reads %+v; want started, not done", got)
}
if err := inv.TakeDone(ctx, "media", "c99"); err != nil {
t.Fatal(err)
}
if taken, _ = inv.TakenLibraries(ctx); taken["media"].Done {
t.Fatal("another warrant's act made the take done")
}
if err := inv.TakeDone(ctx, "media", "c01"); err != nil {
t.Fatal(err)
}
if taken, _ = inv.TakenLibraries(ctx); !taken["media"].Done {
t.Fatal("the act done under the take's warrant did not make it taken")
}
// A failed act forgets a take not done, so it is asked again; never one done.
again := started
again.Root, again.Ask, again.Warrant = "/storage/media2", "c02", "c02"
if err := inv.RecordTakeStarted(ctx, again); err != nil {
t.Fatal(err)
}
if err := inv.TakeFailed(ctx, "media", "c02"); err != nil {
t.Fatal(err)
}
if taken, _ = inv.TakenLibraries(ctx); taken["media"].Ask == "c02" {
t.Fatalf("a failed take is still recorded: %+v", taken["media"])
}
}
+74
View File
@@ -0,0 +1,74 @@
package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
)
// MachineAccount is what a machine said it is made of (novox/hq issue 160), from the profile it last
// reported: the architecture and kernel family the profile always carried, and the facts a node-engine that
// knows how says beside them. Read by `node`, so what a machine says of itself is visible in one place —
// collected and shown in the same breath, not collected and read by nothing (issue 160's pattern).
type MachineAccount struct {
Architecture string `json:"architecture"`
Kernel string `json:"kernel"`
// Machine is nil when the node-engine that reported is older than the one that says it.
Machine *MadeOf `json:"machine"`
}
// MadeOf is the node-engine's account of what it is made of, as it put it on the wire. A fact it could
// not determine is absent here and named in Unknown with why — never a zero.
type MadeOf struct {
ObservedAt time.Time `json:"observed_at"`
KernelRelease string `json:"kernel_release"`
Distribution string `json:"distribution"`
Version string `json:"version"`
Processor string `json:"processor"`
Cores int `json:"cores"`
Threads int `json:"threads"`
MemoryBytes int64 `json:"memory_bytes"`
Virtual string `json:"virtual"`
BootedAt *time.Time `json:"booted_at"`
Timezone string `json:"timezone"`
Filesystems []MountedFilesystem `json:"filesystems"`
Unknown map[string]string `json:"unknown"`
}
// MountedFilesystem is one local filesystem a machine has mounted.
type MountedFilesystem struct {
Mount string `json:"mount"`
Device string `json:"device"`
Type string `json:"type"`
SizeBytes int64 `json:"size_bytes"`
FreeBytes int64 `json:"free_bytes"`
}
// MachineOf is what a node last said it is made of. had is false for a node that has never reported a
// profile at all, which is a different answer from one whose node-engine does not say the facts yet.
func (i *Inventory) MachineOf(ctx context.Context, nodeName string) (account MachineAccount, had bool, err error) {
var raw []byte
err = i.store.Pool().QueryRow(ctx, `select profile from node where name = $1`, nodeName).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return MachineAccount{}, false, fmt.Errorf("%w: %s", ErrNoSuchNode, nodeName)
}
if err != nil {
return MachineAccount{}, false, err
}
return machineFrom(raw)
}
func machineFrom(raw []byte) (MachineAccount, bool, error) {
if len(raw) == 0 {
return MachineAccount{}, false, nil
}
var account MachineAccount
if err := json.Unmarshal(raw, &account); err != nil {
return MachineAccount{}, false, err
}
return account, true, nil
}
+42
View File
@@ -0,0 +1,42 @@
package inventory
import (
"testing"
)
// The node-engine's words for a machine's facts (novox/hq issue 160), as mesh-host's profile.Machine puts
// them on the wire. A rename on either side fails here rather than as a blank on a real machine.
func TestAMachinesAccountIsReadInTheNodeEnginesWords(t *testing.T) {
raw := []byte(`{"architecture":"amd64","kernel":"linux","capabilities":[],
"machine":{"observed_at":"2026-10-12T15:00:00Z","kernel_release":"7.2.8-arch1-2",
"distribution":"Arch Linux","version":"rolling","processor":"AMD Ryzen 9","cores":8,"threads":16,
"memory_bytes":32891494400,"virtual":"physical","booted_at":"2026-10-09T08:50:34Z",
"timezone":"Europe/Brussels",
"filesystems":[{"mount":"/","device":"/dev/nvme0n1p3","type":"ext4","size_bytes":100,"free_bytes":40}],
"unknown":{"public address":"not decided"}}}`)
a, had, err := machineFrom(raw)
if err != nil || !had {
t.Fatalf("had %v, err %v", had, err)
}
m := a.Machine
if a.Architecture != "amd64" || a.Kernel != "linux" || m == nil {
t.Fatalf("%+v", a)
}
if m.KernelRelease != "7.2.8-arch1-2" || m.Distribution != "Arch Linux" || m.Version != "rolling" ||
m.Processor != "AMD Ryzen 9" || m.Cores != 8 || m.Threads != 16 || m.MemoryBytes != 32891494400 ||
m.Virtual != "physical" || m.BootedAt == nil || m.Timezone != "Europe/Brussels" ||
len(m.Filesystems) != 1 || m.Filesystems[0].FreeBytes != 40 || m.Unknown["public address"] == "" ||
m.ObservedAt.IsZero() {
t.Errorf("%+v", m)
}
}
func TestAProfileWithoutTheFactsIsAnOlderNodeEngineNotNothing(t *testing.T) {
a, had, err := machineFrom([]byte(`{"architecture":"amd64","kernel":"linux","capabilities":[]}`))
if err != nil || !had || a.Machine != nil || a.Architecture != "amd64" {
t.Errorf("%+v had %v err %v", a, had, err)
}
if _, had, _ := machineFrom(nil); had {
t.Error("a node that never reported read as having reported")
}
}
@@ -1,50 +0,0 @@
-- A module that joins has a user and a group of its own, each library has a group of its own, and a library's
-- existing data is taken once, on the operator's word (novox/hq ADR 0328 §3, §4, §7, §15).
--
-- **One sequence for every number the mesh gives.** A joining module is given one number, its user's and its own
-- group's (mesh-<module>); a library is given one, its group's (mesh-<library>). Both are drawn from this sequence, so
-- a module's own group and a library's group never share a number. The range is 50000–59999 (catalogue.IdentityLow and
-- IdentityHigh): above the numbers distributions give people, below the 60000s Debian keeps for its daemons, and
-- nowhere near root's 0 or a root group's (0, 4, 10). The checks refuse a row outside it, so no write, by hand or by a
-- later bug, can make one root's (§15). A sequence never hands a number back, and a row is kept after its module or
-- library is gone, so a number once given is never given to another: a file left owned by a removed module's user is
-- never inherited by a new one.
create sequence mesh_number as integer minvalue 50000 maxvalue 59999 start 50000 no cycle;
create table module_identity (
module text primary key,
uid integer not null unique check (uid between 50000 and 59999),
allocated timestamptz not null default now()
);
create table library_group (
library text primary key,
gid integer not null unique check (gid between 50000 and 59999),
allocated timestamptz not null default now()
);
-- **A library taken** (§7). The first time a library's root on its home holds data not yet in its group, the operator
-- is asked once for that library; on the warrant the node-engine sets the group, group write and setgid under the root,
-- and the library is taken: kept from then on without asking. The row is the record of that act — on which machine
-- root and filesystem, who approved it, under which warrant. A library whose home, root or filesystem changed since
-- (it moved to another machine, path or disk) is not taken by that warrant, and the operator is asked again; a new take
-- replaces the row.
create table kept_library (
library text primary key,
home text not null,
-- The root it was taken at: a library whose root changed since is taken again.
root text not null check (root like '/%'),
-- The filesystem the root was on when taken, as the engine names it without a path ("fs-<12 hex>"); empty when
-- the engine named none.
placed text not null default '',
taken_at timestamptz not null default now(),
taken_by text not null,
ask text not null,
warrant text not null,
-- What the node-engine answered when it started the take.
said text not null default '',
-- Whether the engine has stated the take's act done under this warrant: only then is the library taken. Until
-- then the take is under way — the library still waits, and the operator is not asked again — and an act stated
-- failed forgets the row, so the operator is asked again.
done boolean not null default false
);
+12 -3
View File
@@ -1064,9 +1064,14 @@ type Reported struct {
// SentDeclaration is what one send carried to a machine, as a gate keeps it: the declaration's digest and
// its sequence (novox/hq issue 352). A report about this declaration, or about one sequenced after it, is a
// report on what the gate sent — whatever the machine was sent since.
//
// At is when the controller sent it to that machine: the node's own send time, which is where that machine's
// `apply` duration starts. A walk measures its rest from it, not from its own record of the send, made only after
// every machine of the rest was sent (novox/hq issue 582). Zero for a send kept before it was recorded.
type SentDeclaration struct {
Digest string `json:"digest"`
Sequence int64 `json:"sequence,omitempty"`
Digest string `json:"digest"`
Sequence int64 `json:"sequence,omitempty"`
At time.Time `json:"at,omitzero"`
}
// ReportsOn says a report is about this send: the declaration itself; one the same machine was sequenced
@@ -1085,7 +1090,8 @@ func (s SentDeclaration) ReportsOn(r Reported) bool {
func (i *Inventory) SentTo(ctx context.Context, name string) (SentDeclaration, bool, error) {
var digest *string
var seq *int64
err := i.store.Pool().QueryRow(ctx, `select sent, sequence from node where name = $1`, name).Scan(&digest, &seq)
var at *time.Time
err := i.store.Pool().QueryRow(ctx, `select sent, sequence, sent_at from node where name = $1`, name).Scan(&digest, &seq, &at)
if errors.Is(err, pgx.ErrNoRows) {
return SentDeclaration{}, false, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
@@ -1096,6 +1102,9 @@ func (i *Inventory) SentTo(ctx context.Context, name string) (SentDeclaration, b
if seq != nil {
s.Sequence = *seq
}
if at != nil {
s.At = at.UTC()
}
return s, true, nil
}
+173 -44
View File
@@ -2,13 +2,10 @@ package inventory
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/jackc/pgx/v5"
)
// Where a walk's time went (novox/hq ADR 0282 decision 6, issue 382): from its merge to every machine of its
@@ -59,11 +56,14 @@ type WalkPhases struct {
TotalMS int64 `json:"total_ms,omitempty"`
Total string `json:"total,omitempty"`
// UnknownMS is the time within the walk no phase could be measured over.
UnknownMS int64 `json:"unknown_ms,omitempty"`
Phases []WalkPhase `json:"phases"`
Silent []string `json:"silent,omitempty"`
Said string `json:"said"`
Merges []MergeStart `json:"merges,omitempty"`
UnknownMS int64 `json:"unknown_ms,omitempty"`
Phases []WalkPhase `json:"phases"`
Silent []string `json:"silent,omitempty"`
// Rest is each machine of the rest's own report of the send that carried the walk's build there (novox/hq ADR
// 0282 decision 6, issue 582): when it was sent, when it said, and what.
Rest []RestReport `json:"rest,omitempty"`
Said string `json:"said"`
Merges []MergeStart `json:"merges,omitempty"`
}
// MergeStart is one merge the walk answers and when it was made: where that merge's delivery time starts.
@@ -92,8 +92,34 @@ type AppliedReport struct {
Outcome string
}
// AppliedLookup answers a machine's first report after a send to it; false when it has not reported.
type AppliedLookup func(node string, sent time.Time) (AppliedReport, bool)
// AppliedLookup answers a machine's report of the send the walk made to it: send is what the walk kept of it, and
// walkSent the walk's own record of its send to the rest; false when it has not reported.
type AppliedLookup func(node string, send SentDeclaration, walkSent time.Time) (AppliedReport, bool)
// RestReport is one machine of the rest and its own report of a send of the walk (novox/hq issue 582).
type RestReport struct {
Node string `json:"node"`
Modules []string `json:"modules"`
// Sent is when that machine was sent, by its own send time where the walk kept it; else the walk's record of
// its send to the rest, and SentSaid says so.
Sent time.Time `json:"sent"`
SentSaid string `json:"sent_said,omitempty"`
Sequence int64 `json:"sequence,omitempty"`
Reported *time.Time `json:"reported,omitempty"`
Took string `json:"took,omitempty"`
Outcome string `json:"outcome,omitempty"`
// State is applied, failed, waiting (not yet heard, within ApplySilentAfter) or silent (not heard within it:
// listed, never moving the walk's end).
State string `json:"state"`
}
// The states of a machine of the rest.
const (
RestApplied = "applied"
RestFailed = "failed"
RestWaiting = "waiting"
RestSilent = "silent"
)
// point is one moment of the walk, ending the phase named.
type point struct {
@@ -307,8 +333,8 @@ func (p Plan) points(now time.Time, applied AppliedLookup, w *WalkPhases) []poin
}
} else {
rest = latest(rest, s.SentAt)
for node := range s.Rest {
lastSends = append(lastSends, restSend{module: m, node: node, at: *s.SentAt})
for node, what := range s.Rest {
lastSends = append(lastSends, restSend{module: m, node: node, walkSent: *s.SentAt, what: what})
}
}
}
@@ -347,22 +373,73 @@ func (p Plan) points(now time.Time, applied AppliedLookup, w *WalkPhases) []poin
if applied == nil {
return append(out, point{phase: PhaseApply, tier: -1, said: "the rest's reports were not read"})
}
var end *time.Time
var waiting, failed []string
// Each machine of the rest, by its own report of the send that carried the build there (novox/hq issue 582):
// never a report of a send made after it, which another walk's send may be.
reports := map[string]*RestReport{}
var order []string
for _, s := range lastSends {
r, ok := applied(s.node, s.at)
switch {
case !ok && now.Sub(s.at) > ApplySilentAfter, ok && r.At.Sub(s.at) > ApplySilentAfter:
w.Silent = appendOnce(w.Silent, s.node)
case !ok:
waiting = appendOnce(waiting, s.node)
case r.Outcome != OutcomeApplied:
failed = appendOnce(failed, s.node+" ("+r.Outcome+")")
default:
at := r.At
end = latest(end, &at)
at, said := s.walkSent, ""
if !s.what.At.IsZero() {
at = s.what.At
} else {
said = "the walk's record of its send to the rest: this machine's own send time was not kept"
}
key := s.node + "\x00" + s.what.Digest + "\x00" + at.Format(time.RFC3339Nano)
if r := reports[key]; r != nil {
r.Modules = appendOnce(r.Modules, s.module)
continue
}
rr := &RestReport{Node: s.node, Modules: []string{s.module}, Sent: at, SentSaid: said, Sequence: s.what.Sequence}
if r, ok := applied(s.node, s.what, s.walkSent); ok {
got := r.At
rr.Reported, rr.Outcome, rr.Took = &got, r.Outcome, words(r.At.Sub(at))
}
reports[key] = rr
order = append(order, key)
}
// A machine sent twice by the walk (one tier's rest, then the next's) that said nothing of the first send has
// said it of the second, which carries the first's builds too: both are this walk's.
for _, k := range order {
rr := reports[k]
if rr.Reported != nil {
continue
}
for _, l := range order {
later := reports[l]
if later.Node == rr.Node && later.Sent.After(rr.Sent) && later.Reported != nil &&
(rr.Reported == nil || later.Reported.Before(*rr.Reported)) {
got := *later.Reported
rr.Reported, rr.Outcome, rr.Took = &got, later.Outcome, words(got.Sub(rr.Sent))
}
}
}
var end *time.Time
var waiting, failed []string
for _, k := range order {
rr := reports[k]
switch {
case rr.Reported == nil && now.Sub(rr.Sent) > ApplySilentAfter, rr.Reported != nil && rr.Reported.Sub(rr.Sent) > ApplySilentAfter:
rr.State = RestSilent
w.Silent = appendOnce(w.Silent, rr.Node)
case rr.Reported == nil:
rr.State = RestWaiting
waiting = appendOnce(waiting, rr.Node)
case rr.Outcome != OutcomeApplied:
rr.State = RestFailed
failed = appendOnce(failed, rr.Node+" ("+rr.Outcome+")")
default:
rr.State = RestApplied
end = latest(end, rr.Reported)
}
sort.Strings(rr.Modules)
w.Rest = append(w.Rest, *rr)
}
sort.SliceStable(w.Rest, func(a, b int) bool {
if w.Rest[a].Node != w.Rest[b].Node {
return w.Rest[a].Node < w.Rest[b].Node
}
return w.Rest[a].Sent.Before(w.Rest[b].Sent)
})
switch {
case len(failed) > 0:
return append(out, point{phase: PhaseApply, tier: -1, said: "not applied on " + strings.Join(failed, ", ")})
@@ -393,7 +470,8 @@ func (p Plan) points(now time.Time, applied AppliedLookup, w *WalkPhases) []poin
type restSend struct {
module, node string
at time.Time
walkSent time.Time
what SentDeclaration
}
func askedAnyOf(p Plan, tier []string) bool {
@@ -451,28 +529,79 @@ func words(d time.Duration) string {
return d.Round(100 * time.Millisecond).String()
}
// FirstAppliedAfter is a machine's first report of a send made at or after a walk's send to it, within
// ApplySilentAfter of it — the controller's `apply` durations, which measure every send to its first report (to-be
// 45 Phase 0). A declaration sent later carries the build too, so its report counts; one sent past the bound is a
// machine that slept, listed as silent and never moving the walk's end (ADR 0282 decision 1).
func (i *Inventory) FirstAppliedAfter(ctx context.Context, node string, sent time.Time) (AppliedReport, bool, error) {
var started time.Time
var ms int64
var outcome string
err := i.store.Pool().QueryRow(ctx,
`select started, took_ms, detail from duration
// FirstAppliedAfter is a machine's report of the send a walk made to it: the controller's `apply` durations, which
// measure every send to its first report (to-be 45 Phase 0), each row starting at the machine's own send time and
// naming the declaration it was about. send is what the walk kept of that send; walkSent its own record of the
// send to the rest, made after every machine of the rest was sent (novox/hq issue 582). FirstReportOf chooses.
func (i *Inventory) FirstAppliedAfter(ctx context.Context, node string, send SentDeclaration, walkSent time.Time) (AppliedReport, bool, error) {
from, to := applyRowsBetween(send, walkSent)
rows, err := i.store.Pool().Query(ctx,
`select ref, started, took_ms, detail from duration
where kind = $1 and subject = $2 and started >= $3 and started < $4
order by started limit 1`,
DurationApply, node, sent.Add(-appliedSlack), sent.Add(ApplySilentAfter)).Scan(&started, &ms, &outcome)
if errors.Is(err, pgx.ErrNoRows) {
return AppliedReport{}, false, nil
}
order by started`,
DurationApply, node, from, to)
if err != nil {
return AppliedReport{}, false, err
}
return AppliedReport{At: started.Add(time.Duration(ms) * time.Millisecond).UTC(), Outcome: outcome}, true, nil
defer rows.Close()
var ds []Duration
for rows.Next() {
d := Duration{Kind: DurationApply, Subject: node, Node: node}
var ms int64
if err := rows.Scan(&d.Ref, &d.Started, &ms, &d.Detail); err != nil {
return AppliedReport{}, false, err
}
d.Took = time.Duration(ms) * time.Millisecond
ds = append(ds, d)
}
if err := rows.Err(); err != nil {
return AppliedReport{}, false, err
}
r, ok := FirstReportOf(ds, send, walkSent)
return r, ok, nil
}
// appliedSlack is how much earlier than a walk's record of its send the machine's own record of it may be:
// the send is recorded on the machine first, then on the walk.
// applyRowsBetween is the span of a machine's `apply` durations that may hold its report of a walk's send.
func applyRowsBetween(send SentDeclaration, walkSent time.Time) (time.Time, time.Time) {
switch {
case !send.At.IsZero():
return send.At.Add(-time.Millisecond), send.At.Add(time.Millisecond)
case send.Digest != "":
return walkSent.Add(-restSendBound), walkSent.Add(ApplySilentAfter)
}
return walkSent.Add(-appliedSlack), walkSent.Add(ApplySilentAfter)
}
// FirstReportOf picks, from a machine's `apply` durations ordered by their start, its report of the send a walk made
// to it (novox/hq issue 582):
// - with the machine's own send time kept (every walk since issue 582): the duration of that very send, and of the
// declaration sent where the walk kept its digest. A report of a later send is never this walk's: a send made
// later, by another walk, would end this walk at that walk's send.
// - with only the declaration's digest kept (a walk kept before): the first duration about that declaration
// within restSendBound before the walk's record of its send, the machine sent first in the rest included.
// - with neither (a walk kept before ADR 0282's rest): the first duration started no earlier than appliedSlack
// before the walk's record of its send, as it was read before.
//
// A send past ApplySilentAfter is never read: its machine slept, and is listed silent (ADR 0282 decision 1).
func FirstReportOf(ds []Duration, send SentDeclaration, walkSent time.Time) (AppliedReport, bool) {
from, to := applyRowsBetween(send, walkSent)
for _, d := range ds {
if d.Started.Before(from) || !d.Started.Before(to) {
continue
}
if send.Digest != "" && !strings.HasPrefix(d.Ref, send.Digest+"@") {
continue
}
return AppliedReport{At: d.Started.Add(d.Took).UTC(), Outcome: d.Detail}, true
}
return AppliedReport{}, false
}
// appliedSlack is how much earlier than a walk's record of its send the machine's own record of it may be, for a
// walk that kept neither the machine's send time nor what it sent.
const appliedSlack = 2 * time.Second
// restSendBound is how long before a walk's record of its send to the rest a machine of the rest may have been
// sent, for a walk that kept what it sent and not when: the send to the rest goes machine by machine, and took
// 19 to 28 seconds on 2026-10-12 (novox/hq issue 582).
const restSendBound = 5 * time.Minute
+188
View File
@@ -0,0 +1,188 @@
package inventory
import (
"strings"
"testing"
"time"
)
// novox/hq issue 582: the walk's own record of its send to the rest is made after every machine of the rest was
// sent, and that send took 19 to 28 s on 2026-10-12. The machine sent first reported 5 s after its own send, long
// before the walk's record; read from the walk's record, its report was missed and the walk ended at another walk's
// send to it, minutes later. A walk of one tier and three machines of the rest, the send to the rest taking 20 s.
type sendRest582 struct {
judged, walkSent time.Time
walk Plan
rows map[string][]Duration
}
func walk582(t *testing.T) sendRest582 {
t.Helper()
judged := time.Date(2026, 10, 12, 17, 45, 35, 137000000, time.UTC)
ownSend := map[string]time.Duration{"g14": 500 * time.Millisecond, "novox": 10 * time.Second, "shanks": 20 * time.Second}
walkSent := judged.Add(20200 * time.Millisecond)
merged := judged.Add(-10 * time.Minute)
built, first := judged.Add(-3*time.Minute), judged.Add(-2*time.Minute)
rest := map[string]SentDeclaration{}
rows := map[string][]Duration{}
took := map[string]time.Duration{"g14": 5 * time.Second, "novox": 13 * time.Second, "shanks": 15 * time.Second}
for i, n := range []string{"g14", "novox", "shanks"} {
at := judged.Add(ownSend[n])
digest := "walk-" + n
rest[n] = SentDeclaration{Digest: digest, Sequence: int64(647 + i), At: at}
rows[n] = append(rows[n],
// This walk's send, and its report.
Duration{Kind: DurationApply, Subject: n, Node: n, Ref: digest + "@" + at.Format("2006-01-02T15:04:05.000000"),
Started: at, Took: took[n], Detail: OutcomeApplied},
// The next walk's send to the same machine, nine minutes later, and its report.
Duration{Kind: DurationApply, Subject: n, Node: n, Ref: "next-" + n + "@x",
Started: judged.Add(9*time.Minute + ownSend[n]), Took: 4 * time.Second, Detail: OutcomeApplied})
}
p := Plan{ID: "plan-582", State: PlanDone, Tier: 1, Tiers: [][]string{{"build-agent"}}, Created: merged,
Delivery: &PlanDelivery{Merges: []PlanMerge{{Repository: "novox/mesh-controller", Commit: "c", Merged: merged}}},
Times: &PlanTimes{Cut: &merged, Class: ClassCore},
Modules: map[string]*PlanModule{"build-agent": {State: "built", AskedAt: &merged, BuiltAt: &built,
FirstAt: &first, SentAt: &walkSent, Gate: &PlanGate{JudgedAt: &judged}, Rest: rest}}}
return sendRest582{judged: judged, walkSent: walkSent, walk: p, rows: rows}
}
// lookup answers from the rows as the controller's durations would.
func (w sendRest582) lookup(node string, send SentDeclaration, walkSent time.Time) (AppliedReport, bool) {
return FirstReportOf(w.rows[node], send, walkSent)
}
// The apply phase ends at the last machine's own report of this walk's send — shanks, sent at the end of the 20 s
// and reporting 15 s later — and never at the next walk's send; every machine's report is shown.
func TestTheApplyPhaseEndsAtEachMachinesOwnReport(t *testing.T) {
c := walk582(t)
w := c.walk.WalkPhases(c.judged.Add(time.Hour), c.lookup)
sumsUp(t, w)
want := c.judged.Add(35 * time.Second) // shanks: sent 20 s after the judgement, reported 15 s later
if !w.End.Equal(want) {
t.Fatalf("the walk ends at %s, want %s: its rest %+v", w.End, want, w.Rest)
}
if got := phase(w, PhaseApply, -1); got.State != PhaseMeasured || got.TookMS != want.Sub(c.walkSent).Milliseconds() {
t.Fatalf("apply: %+v", got)
}
if len(w.Rest) != 3 || len(w.Silent) != 0 {
t.Fatalf("every machine of the rest is shown with its report: %+v, silent %v", w.Rest, w.Silent)
}
g14 := w.Rest[0]
if g14.Node != "g14" || g14.State != RestApplied || g14.Reported == nil ||
!g14.Reported.Equal(c.judged.Add(5500*time.Millisecond)) || g14.Took != "5s" || g14.Sequence != 647 ||
!g14.Sent.Equal(c.judged.Add(500*time.Millisecond)) || g14.SentSaid != "" {
t.Fatalf("g14, sent first and reporting 5 s after its own send: %+v", g14)
}
}
// A walk kept before its machines' own send times were: matched by the declaration it sent, within restSendBound
// of its record of the send, so the machine sent first is still found; a walk that kept neither is read as before.
func TestAnOlderWalkIsReadByWhatItSent(t *testing.T) {
c := walk582(t)
for n, s := range c.walk.Modules["build-agent"].Rest {
s.At = time.Time{}
c.walk.Modules["build-agent"].Rest[n] = s
}
w := c.walk.WalkPhases(c.judged.Add(time.Hour), c.lookup)
sumsUp(t, w)
if !w.End.Equal(c.judged.Add(35 * time.Second)) {
t.Fatalf("by the declaration sent: end %s, rest %+v", w.End, w.Rest)
}
if !strings.Contains(w.Rest[0].SentSaid, "walk's record") || !w.Rest[0].Sent.Equal(c.walkSent) {
t.Fatalf("a send time not kept is said: %+v", w.Rest[0])
}
// Nothing of the send kept: the first report no earlier than appliedSlack before the walk's record, as before.
if r, ok := FirstReportOf(c.rows["shanks"], SentDeclaration{}, c.walkSent); !ok || !r.At.Equal(c.judged.Add(35*time.Second)) {
t.Fatalf("a walk that kept nothing of its send: %+v %v", r, ok)
}
}
// A machine that said nothing of this walk's send is waiting, then silent past ApplySilentAfter: the report of a
// later send to it is never taken as this walk's, and never moves the walk's end.
func TestALaterSendsReportIsNeverThisWalks(t *testing.T) {
c := walk582(t)
c.rows["g14"] = c.rows["g14"][1:] // g14 slept through this walk's send; the next walk's send it reported
w := c.walk.WalkPhases(c.judged.Add(5*time.Minute), c.lookup)
if w.End != nil || !strings.Contains(phase(w, PhaseApply, -1).Said, "waiting for g14") {
t.Fatalf("g14 not heard from yet: end %v, apply %+v", w.End, phase(w, PhaseApply, -1))
}
w = c.walk.WalkPhases(c.judged.Add(time.Hour), c.lookup)
sumsUp(t, w)
if len(w.Silent) != 1 || w.Silent[0] != "g14" || !w.End.Equal(c.judged.Add(35*time.Second)) {
t.Fatalf("g14 silent, the end at shanks' report: silent %v, end %s", w.Silent, w.End)
}
if w.Rest[0].State != RestSilent || w.Rest[0].Reported != nil {
t.Fatalf("g14's later report taken as this walk's: %+v", w.Rest[0])
}
}
// A machine sent by two tiers of one walk that said nothing of the first send has said it of the second, which
// carries the first tier's build too.
func TestAMachineSentTwiceByOneWalkIsHeardByItsLaterSend(t *testing.T) {
c := walk582(t)
second := c.judged.Add(4 * time.Minute)
secondSent := second.Add(time.Second)
c.walk.Tiers = append(c.walk.Tiers, []string{"route-proxy"})
c.walk.Tier = 2
asked, built := c.judged.Add(time.Minute), c.judged.Add(2*time.Minute)
first := built.Add(time.Second)
c.walk.Modules["route-proxy"] = &PlanModule{State: "built", AskedAt: &asked, BuiltAt: &built, FirstAt: &first,
SentAt: &secondSent, Gate: &PlanGate{JudgedAt: &second},
Rest: map[string]SentDeclaration{"novox": {Digest: "walk-novox-2", Sequence: 1554, At: second.Add(500 * time.Millisecond)}}}
c.rows["novox"] = []Duration{{Kind: DurationApply, Subject: "novox", Ref: "walk-novox-2@x",
Started: second.Add(500 * time.Millisecond), Took: 14 * time.Second, Detail: OutcomeApplied}}
w := c.walk.WalkPhases(c.judged.Add(time.Hour), c.lookup)
sumsUp(t, w)
want := second.Add(14500 * time.Millisecond)
if !w.End.Equal(want) || len(w.Silent) != 0 {
t.Fatalf("end %s (want %s), silent %v, rest %+v", w.End, want, w.Silent, w.Rest)
}
for _, r := range w.Rest {
if r.Node == "novox" && (r.State != RestApplied || !r.Reported.Equal(want)) {
t.Fatalf("novox, heard by its later send of this walk: %+v", r)
}
}
}
// The store: a machine's own send time is read with what it was sent, and its `apply` duration is found by it —
// the send and the duration start at one moment, the node's sent_at.
func TestAWalksRestIsReadFromEachMachinesOwnSend(t *testing.T) {
inv := ForTest(t)
ctx := t.Context()
record, err := inv.AddNode(ctx, "g14")
if err != nil {
t.Fatal(err)
}
if _, err := inv.NextSequence(ctx, record.ID); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, record.ID, "d-582", map[string]string{"build-agent": "c1"}); err != nil {
t.Fatal(err)
}
sent, found, err := inv.SentTo(ctx, "g14")
if err != nil || !found || sent.At.IsZero() || sent.Digest != "d-582" {
t.Fatalf("the machine's own send time: %+v %v %v", sent, found, err)
}
if _, ok, err := inv.FirstAppliedAfter(ctx, "g14", sent, sent.At.Add(20*time.Second)); err != nil || ok {
t.Fatalf("no report yet: %v %v", ok, err)
}
if err := inv.RecordApplyDuration(ctx, "g14", "d-582", OutcomeApplied); err != nil {
t.Fatal(err)
}
// The walk's record of its send, 20 s after the machine's own: still found, by the machine's own send.
r, ok, err := inv.FirstAppliedAfter(ctx, "g14", sent, sent.At.Add(20*time.Second))
if err != nil || !ok || r.Outcome != OutcomeApplied || r.At.Before(sent.At) {
t.Fatalf("the machine's own report of its send: %+v %v %v", r, ok, err)
}
// A later send, reported: never this walk's.
if err := inv.RecordSent(ctx, record.ID, "d-next", map[string]string{"build-agent": "c2"}); err != nil {
t.Fatal(err)
}
if err := inv.RecordApplyDuration(ctx, "g14", "d-next", OutcomeApplied); err != nil {
t.Fatal(err)
}
again, _, _ := inv.FirstAppliedAfter(ctx, "g14", sent, sent.At.Add(20*time.Second))
if !again.At.Equal(r.At) {
t.Fatalf("a later send's report read as this walk's: %+v, was %+v", again, r)
}
}
+8 -8
View File
@@ -116,7 +116,7 @@ func TestAWalkOfTwoTiersEndsAtItsRestsLastReport(t *testing.T) {
}}
reports := map[string]AppliedReport{"ace": {At: *at("05:12"), Outcome: OutcomeApplied},
"shanks": {At: *at("08:15.25"), Outcome: OutcomeApplied}}
w := p.WalkPhases(*at("30:00"), func(node string, _ time.Time) (AppliedReport, bool) {
w := p.WalkPhases(*at("30:00"), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
@@ -135,7 +135,7 @@ func TestAWalkOfTwoTiersEndsAtItsRestsLastReport(t *testing.T) {
}
// A report not yet in: the walk has no end, and its apply is unknown — never zero.
delete(reports, "shanks")
w = p.WalkPhases(*at("10:00"), func(node string, _ time.Time) (AppliedReport, bool) {
w = p.WalkPhases(*at("10:00"), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
@@ -146,7 +146,7 @@ func TestAWalkOfTwoTiersEndsAtItsRestsLastReport(t *testing.T) {
t.Errorf("apply while shanks has not reported: %+v", got)
}
// Silent past the bound: listed, not waited for.
w = p.WalkPhases(at("08:00.5").Add(ApplySilentAfter+time.Second), func(node string, _ time.Time) (AppliedReport, bool) {
w = p.WalkPhases(at("08:00.5").Add(ApplySilentAfter+time.Second), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
@@ -156,7 +156,7 @@ func TestAWalkOfTwoTiersEndsAtItsRestsLastReport(t *testing.T) {
}
// A failed report: the walk has no end, said.
reports["shanks"] = AppliedReport{At: *at("08:10"), Outcome: OutcomeFailed}
w = p.WalkPhases(*at("30:00"), func(node string, _ time.Time) (AppliedReport, bool) {
w = p.WalkPhases(*at("30:00"), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
@@ -247,7 +247,7 @@ func TestAWalksMomentsAreKeptAndItsRestsReportRead(t *testing.T) {
t.Fatal("a merge's time was not kept")
}
sent := time.Now().UTC().Add(-time.Minute).Truncate(time.Millisecond)
if _, ok, err := inv.FirstAppliedAfter(ctx, "ace", sent); err != nil || ok {
if _, ok, err := inv.FirstAppliedAfter(ctx, "ace", SentDeclaration{}, sent); err != nil || ok {
t.Fatalf("no report yet: %v %v", ok, err)
}
for _, d := range []Duration{
@@ -258,7 +258,7 @@ func TestAWalksMomentsAreKeptAndItsRestsReportRead(t *testing.T) {
t.Fatal(err)
}
}
r, ok, err := inv.FirstAppliedAfter(ctx, "ace", sent)
r, ok, err := inv.FirstAppliedAfter(ctx, "ace", SentDeclaration{}, sent)
if err != nil || !ok || r.Outcome != OutcomeApplied || !r.At.Equal(sent.Add(11*time.Second)) {
t.Fatalf("the report after the send: %+v %v %v", r, ok, err)
}
@@ -281,7 +281,7 @@ func TestALateReportIsSilentNotTheEnd(t *testing.T) {
p := walkOf(t, recordedWalk)
p.Modules["mesh-delivery"].SentAt = &sent
p.Modules["mesh-delivery"].Rest = map[string]SentDeclaration{"laptop": {Digest: "d"}, "server": {Digest: "e"}}
w := p.WalkPhases(sent.Add(3*time.Hour), func(node string, _ time.Time) (AppliedReport, bool) {
w := p.WalkPhases(sent.Add(3*time.Hour), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
if node == "server" {
return AppliedReport{At: sent.Add(20 * time.Second), Outcome: OutcomeApplied}, true
}
@@ -296,7 +296,7 @@ func TestALateReportIsSilentNotTheEnd(t *testing.T) {
Started: sent.Add(time.Hour), Took: time.Second, Detail: OutcomeApplied}); err != nil {
t.Fatal(err)
}
if _, ok, err := inv.FirstAppliedAfter(ctx, "laptop", sent); err != nil || ok {
if _, ok, err := inv.FirstAppliedAfter(ctx, "laptop", SentDeclaration{}, sent); err != nil || ok {
t.Fatalf("a send past the bound read as the walk's: %v %v", ok, err)
}
}
+11
View File
@@ -79,6 +79,8 @@ func (b *natsBuilds) Ask(ctx context.Context, request BuildRequest) error {
if _, err := b.js.Context().Publish(BuildWorkOf(b.role()), body, nats.Context(publish)); err != nil {
return fmt.Errorf("cannot submit a build: %w", err)
}
// The acting controller puts it in order (novox/hq issue 558); this only says an ask joined.
signalAsked()
return nil
}
@@ -113,6 +115,8 @@ func (b *natsBuilds) Submit(ctx context.Context, request BuildRequest,
if _, err := b.js.Context().Publish(BuildWorkOf(b.role()), body, nats.Context(publish)); err != nil {
return BuildResult{}, fmt.Errorf("%w: cannot submit a build: %w", ErrNotHandedOver, err)
}
// The acting controller puts it in order (novox/hq issue 558); this only says an ask joined.
signalAsked()
waiting, cancelWait := context.WithTimeout(ctx, wait)
defer cancelWait()
@@ -167,6 +171,10 @@ type MachineOptions struct {
// Paused is asked before every fetch: while it says so, nothing new is taken, and a build
// already running finishes. Nil is never paused.
Paused func() bool
// Taking is called once, when this machine is bound to its worker and pulls work: from then on it
// can take a build, which is what its container's health check reads (novox/hq issue 544). Nil is
// told nothing.
Taking func()
}
// MachineOverNATS takes build work from the current build role.
@@ -232,6 +240,9 @@ func (m *natsMachine) Take(ctx context.Context, do func(context.Context, Build))
"the mesh's to answer, not this machine's", m.seat, err)
}
m.sub = sub
if m.opts.Taking != nil {
m.opts.Taking()
}
for {
if ctx.Err() != nil {
+33
View File
@@ -368,3 +368,36 @@ func TestNatsAMachineOnTheRetiredBuildRoleTakesThatRolesAsks(t *testing.T) {
t.Errorf("the retired role's holder did not answer: %+v", result)
}
}
// A holder says once that it takes work, and only once it is bound to its worker (novox/hq issue 544):
// its container's health check reads that, so a holder that could not bind is never said ready.
func TestNatsAHolderSaysItTakesWorkOnceItIsBound(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx, stop := context.WithCancel(context.Background())
defer stop()
said := make(chan struct{}, 4)
machine := MachineOverNATSWith(js, "anchor", TheBuildMachine, MachineOptions{Taking: func() { said <- struct{}{} }})
defer machine.Close()
go func() { _ = machine.Take(ctx, func(context.Context, Build) {}) }()
select {
case <-said:
case <-time.After(5 * time.Second):
t.Fatal("a holder bound to its worker never said it takes work")
}
time.Sleep(300 * time.Millisecond)
if len(said) != 0 {
t.Fatalf("a holder said it takes work %d more time(s)", len(said))
}
// A role the mesh raised no worker for cannot be bound, and its holder says nothing.
bounded, end := context.WithTimeout(ctx, 10*time.Second)
defer end()
other := MachineOverNATSWith(js, "anchor", "mesh-no-such-seat", MachineOptions{Taking: func() { said <- struct{}{} }})
defer other.Close()
if err := other.Take(bounded, func(context.Context, Build) {}); err == nil {
t.Fatal("a machine with no worker took work")
}
if len(said) != 0 {
t.Fatal("a machine that could not bind said it takes work")
}
}
-70
View File
@@ -8,7 +8,6 @@ package link
import (
"encoding/base64"
"encoding/json"
"github.com/novox/mesh-host/rootsearch"
"strconv"
"strings"
@@ -437,34 +436,6 @@ const ReadinessContract = 2
// parses strictly and would refuse the whole declaration for it.
const RootContract = 3
// IdentityContract is the statement of an engine that gives each module its identity and keeps shared folders
// (novox/hq ADR 0325; mesh-host internal/link IdentityContract): it reads a declaration's `identities`, a container's
// resolved `identity`, an access's `users` and `taken`, states an access it may not keep yet as waiting
// (ResourceHealth.AccessWaits), and hears the take act (AskAccessAct). Only to such an engine is any of it sent: an
// older one parses strictly and would refuse the whole declaration for it. Four, the grant's number (ADR 0312,
// superseded), is what engines stated before; the controller sends no grant to any engine (#270).
const IdentityContract = 5
// LibraryContract is the statement of an engine that makes the mesh's users and library groups and keeps libraries
// (novox/hq ADR 0328, which supersedes ADR 0325): it reads a declaration's `groups`, `identities` and `libraries`,
// states a library it may not keep yet as waiting, and hears the take of a library. Only to such an engine is any of
// it sent; five, ADR 0325's, is sent nothing new.
const LibraryContract = 6
// OwnGroupContract is the statement of an engine that also puts the tool runner's account in a joining module's own
// group, named by an identity's `runner`, and takes a directory's mode with the setgid bit, such as "2750" (novox/hq
// ADR 0334 §1, §2). Not stated by any engine yet: the engine must add `runner` (an account's name) to its declaration's
// Identity and put that account in the group mesh-<module>, and accept a directory mode whose first digit is 2.
const OwnGroupContract = 7
// KindRunner is the tool runner's membership as the engine states it (novox/hq ADR 0334 §2; mesh-host internal/link):
// `runner:<module>` of a joining module, in its own group; `runner:library:<name>` of no module, in a library's group.
const KindRunner = "runner"
// KindLibrary is a library's own verdict on its home, as the engine may state it (novox/hq ADR 0328): healthy once
// kept. Read beside an access verdict naming the library until the engine's shape is settled.
const KindLibrary = "library"
// ReasonRoot starts the reason of an account verdict that found a way to root (ADR 0266); the node-engine's
// own words (mesh-host internal/accounts ReasonRoot).
const ReasonRoot = "can become root without a person"
@@ -609,43 +580,6 @@ type Wait struct {
What string `json:"what"`
}
// WaitKindAccess is the kind of a wait on an access (novox/hq ADR 0325 §7; mesh-host internal/link).
const WaitKindAccess = "access"
// AccessWait is one access waiting for the operator (novox/hq ADR 0325 §7): the access by its module's own id for
// it, the part (its module), and what the operator gives, in words with no path — "let the mesh keep the shared
// folder movies", or "mount the disk that holds books". Beside Waits because mesh-sdk's health.Wait names a secret or
// a setting and nothing else. From an engine at IdentityContract or later.
type AccessWait struct {
Kind string `json:"kind"`
Access string `json:"access"`
Part string `json:"part"`
What string `json:"what"`
// Placed names, with no path, the filesystem the folder is on now ("fs-<12 hex>"), on a wait for the operator's
// word: a warrant given for another filesystem (the folder moved) does not answer it, and the controller asks
// again (mesh-host internal/link AccessWait.Placed).
Placed string `json:"placed,omitempty"`
// Reason is what the wait is for: WaitKeep (the operator's word to take the library) or WaitDisk (its disk
// mounted).
Reason string `json:"reason"`
}
// The reasons of an access wait (mesh-host internal/link).
const (
WaitKeep = "keep"
WaitDisk = "disk"
)
// KindAccess is an access's verdict (novox/hq ADR 0325; mesh-host internal/link KindAccess): healthy while each user
// has its right, waiting while the mesh may not keep its folder yet or its disk is not there, unhealthy otherwise —
// and unknown when its filesystem could not be named this look (a hung mount), carrying no wait.
const KindAccess = "access"
// KindKeptEntries is the mesh's entries on a shared folder that nobody wants and that could not be taken out yet
// because its disk was away (novox/hq ADR 0325 §8; mesh-host internal/link): unhealthy, said under the access that
// last named it (`<access id>~entries`), which may no longer be declared. The node-engine's own.
const KindKeptEntries = "kept-entries"
// ResourceHealth is one long-running resource's state.
type ResourceHealth struct {
Module string `json:"module"`
@@ -672,10 +606,6 @@ type ResourceHealth struct {
// Waits is what a resource in StateWaiting waits for (novox/hq ADR 0283). Empty otherwise, and from an
// engine older than that.
Waits []Wait `json:"waits,omitempty"`
// AccessWaits is what an access stated waiting waits for (novox/hq ADR 0325 §7); Act the last walk under its
// folder, kept as the engine said it. From an engine at IdentityContract or later.
AccessWaits []AccessWait `json:"access-waits,omitempty"`
Act json.RawMessage `json:"act,omitempty"`
}
// HealthSaid is the health event's body: the machine and its statement. The machine is read from the
+26 -10
View File
@@ -62,7 +62,11 @@ type QueuedAsk struct {
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
AskedAt time.Time `json:"asked-at,omitempty"`
State string `json:"state"`
// Stored is when the stream first took it, a move into order kept (novox/hq issue 558).
Stored time.Time `json:"stored,omitempty"`
State string `json:"state"`
// movedFrom is the MovedFromHeader it carries: the sequence of the message a move copied it from.
movedFrom string
// On and Started are the holder building an in-flight ask and when it started, from its started
// event. Was is the holder that started an in-flight ask and is no longer building it — handed
// back, to be handed out again — with Started its start there.
@@ -236,15 +240,7 @@ func ReadQueue(ctx context.Context, js *broker.JetStream, seat string) (Queue, e
if err != nil {
return q, fmt.Errorf("reading the %s work queue: %w", seat, err)
}
ask := QueuedAsk{Seq: msg.Sequence}
if json.Unmarshal(msg.Data, &ask.Request) == nil {
r := ask.Request
ask.ID, ask.Repository, ask.Path, ask.Ref = r.ID, r.Repository, r.Path, r.Ref
if at, ok := BuildAskedAt(r.ID); ok {
ask.AskedAt = at
}
}
asks = append(asks, ask)
asks = append(asks, askOf(msg))
seq = msg.Sequence + 1
}
@@ -269,6 +265,26 @@ func ReadQueue(ctx context.Context, js *broker.JetStream, seat string) (Queue, e
return q, nil
}
// askOf is one ask as the stream holds it: what its request says, and when the stream took it.
func askOf(msg *jetstream.RawStreamMsg) QueuedAsk {
ask := QueuedAsk{Seq: msg.Sequence, Stored: msg.Time}
// Moved into order, it keeps when the stream first took it (novox/hq issue 558).
if at, err := time.Parse(time.RFC3339Nano, msg.Header.Get(QueuedAtHeader)); err == nil {
ask.Stored = at
}
if json.Unmarshal(msg.Data, &ask.Request) == nil {
r := ask.Request
ask.ID, ask.Repository, ask.Path, ask.Ref = r.ID, r.Repository, r.Path, r.Ref
if at, ok := BuildAskedAt(r.ID); ok {
ask.AskedAt = at
}
} else {
// Nothing read is nothing kept: a half-read request would pass for a delivery (novox/hq issue 558).
ask.Request = BuildRequest{}
}
return ask
}
// WithoutPassword is a repository as a listing may say it: a URL's password, should one ever be written
// into it, left out (novox/hq issue 455). Only for what is shown: the ask itself keeps what it was given,
// which a cancel's outcome and a rebuild's match are read from.
+426
View File
@@ -0,0 +1,426 @@
package link
import (
"context"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
)
// The order the build queue hands asks out in (novox/hq issue 558).
//
// **A build a walk waits for goes before a pull request's check.** The queue was first come, first served,
// and a merged change's delivery build waited minutes behind checks while its walk, and every walk after
// it, waited on the build. So:
//
// - a delivery is handed out before any check;
// - a check that has waited CheckWaitsAtMost goes before every delivery asked after it, so no check
// waits for ever behind a stream of deliveries;
// - within a kind, first come, first served, by when it was asked;
// - an ask whose kind cannot be told counts as a check: a delivery taken late breaks a budget, a check
// taken late breaks nothing.
//
// **Kept in the stream, never beside it.** The queue is the seat's work queue, which the server hands out
// in stream order to whichever holder pulls; nothing else holds the order, so a restarted controller
// finds it as it was left. An ask that must go behind another is moved to the tail — deleted from where
// it was and published again, its body unchanged, under the same id — while the worker is paused, so no
// holder takes it in the moment it moves. Only waiting asks move: what a holder took is never touched.
// The kinds of ask, as the queue says them.
const (
KindDelivery = "delivery"
KindCheck = "check"
)
// CheckWaitsAtMost is how long a check waits before it goes ahead of the deliveries asked after it.
var CheckWaitsAtMost = 10 * time.Minute
// KindOf is an ask's kind: a check for a pull request's check — a recheck the merge tool asks included —
// a dry run, or an ask nothing could be read from; a delivery for any other build.
func KindOf(r BuildRequest) string {
if r.ID == "" || r.Check != nil || r.DryRun {
return KindCheck
}
return KindDelivery
}
// askedOrStored is when an ask was asked, as its id says, or else when the queue took it.
func (a QueuedAsk) askedOrStored() time.Time {
if !a.AskedAt.IsZero() {
return a.AskedAt
}
return a.Stored
}
// QueueOrder is the order the waiting asks are to be handed out in, now: the deliveries and every check
// that has waited CheckWaitsAtMost, by when each was asked, then the other checks, by when each was asked.
func QueueOrder(waiting []QueuedAsk, now time.Time) []QueuedAsk {
var first, then []QueuedAsk
for _, a := range waiting {
if KindOf(a.Request) == KindCheck && now.Sub(a.askedOrStored()) < CheckWaitsAtMost {
then = append(then, a)
continue
}
first = append(first, a)
}
byAsking := func(s []QueuedAsk) {
sort.SliceStable(s, func(i, j int) bool {
if ti, tj := s[i].askedOrStored(), s[j].askedOrStored(); !ti.Equal(tj) {
return ti.Before(tj)
}
return s[i].Seq < s[j].Seq
})
}
byAsking(first)
byAsking(then)
return append(first, then...)
}
// toMove is the asks to move to the tail, in this order, for the queue to stand as order says: everything
// after the longest start of order that the queue already holds in that order.
func toMove(queue, order []QueuedAsk) []QueuedAsk {
at := make(map[uint64]int, len(queue))
for i, a := range queue {
at[a.Seq] = i
}
kept, last := 0, -1
for kept < len(order) {
i, ok := at[order[kept].Seq]
if !ok || i < last {
break
}
last = i
kept++
}
return order[kept:]
}
// orderPasses keeps one pass at a time. Only the acting controller's loop runs one (KeepingInOrder), so
// there is one; the lock holds a test's two at once to that too.
var orderPasses sync.Mutex
// pausedFor bounds a pause the pass could not end itself: a controller that stops mid-pass leaves the
// worker paused this long at most. moveMargin is the part of it no move starts in: a pass stops moving
// with at least this left, and the next pass moves the rest, so no move runs past the pause.
const (
pausedFor = 20 * time.Second
moveMargin = 5 * time.Second
publishWaits = 3 * time.Second
)
// QueuedAtHeader carries, on a moved ask, when the stream first took it: a move keeps an ask's place in
// time, its id unread or not.
const QueuedAtHeader = "Mesh-Queued-At"
// MovedFromHeader names, on a move's copy, the sequence of the message it is a copy of: with QueuedAtHeader,
// what proves a second message of one id is a move's copy and not an ask delivered again (ADR 0293), whose
// headers are its original's — and so never name that original's own sequence.
const MovedFromHeader = "Mesh-Moved-From"
// Pass is what one pass did: the asks it moved, and each second copy it removed, in words.
type Pass struct {
Moved []string
Removed []string
}
// asked is the signal an ask sends the acting controller's pass loop, in this process. An ask asked in
// another process — a verb run as its own command — is seen by the loop as the queue's last sequence moving.
var asked = make(chan struct{}, 1)
// signalAsked tells the pass loop an ask joined the queue. Never blocks, never runs a pass.
func signalAsked() {
select {
case asked <- struct{}{}:
default:
}
}
// PutInOrder moves the seat's waiting asks into the order QueueOrder says, and says each it moved. A queue
// already in order is read and left alone; the worker is paused only while asks move. Run by the acting
// controller alone (KeepingInOrder): two passes in two processes would each resume what the other paused.
func PutInOrder(ctx context.Context, js *broker.JetStream, seat string) (Pass, error) {
var pass Pass
orderPasses.Lock()
defer orderPasses.Unlock()
api, err := jetstream.New(js.Conn())
if err != nil {
return pass, err
}
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: seat, Accepts: []string{"build"}})
waiting, extra, delivered, err := readWaiting(ctx, api, worker, seat)
if err != nil || (len(extra) == 0 && len(toMove(waiting, QueueOrder(waiting, time.Now()))) == 0) {
return pass, err
}
// Paused, then read again: what the worker handed out up to the pause is not waiting any more.
pauseEnds := time.Now().Add(pausedFor)
if delivered >= 0 {
if _, err := api.PauseConsumer(ctx, worker.Stream, worker.Name, pauseEnds); err != nil {
return pass, fmt.Errorf("the %s worker could not be paused to put its queue in order, so it is left "+
"first come, first served: %w", seat, err)
}
defer func() {
resuming, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
defer cancel()
if _, err := api.ResumeConsumer(resuming, worker.Stream, worker.Name); err != nil {
fmt.Printf("the %s worker could not be resumed after its queue was put in order; it resumes by "+
"itself by %s: %v\n", seat, pauseEnds.Format(time.RFC3339), err)
}
}()
if waiting, extra, _, err = readWaiting(ctx, api, worker, seat); err != nil {
return pass, err
}
}
stream, err := api.Stream(ctx, worker.Stream)
if err != nil {
return pass, err
}
// A second copy of one ask — a move that published and stopped before it deleted — goes first.
for _, a := range extra {
if err := stream.DeleteMsg(ctx, a.Seq); err != nil && !errors.Is(err, jetstream.ErrMsgNotFound) {
return pass, fmt.Errorf("a second copy of %s (seq %d) could not be removed: %w", a.ID, a.Seq, err)
}
pass.Removed = append(pass.Removed, fmt.Sprintf("%s at seq %d, a move's second copy of the message at seq %s",
a.ID, a.Seq, a.movedFrom))
}
for _, a := range toMove(waiting, QueueOrder(waiting, time.Now())) {
if delivered >= 0 && time.Until(pauseEnds) < moveMargin {
// The rest waits for the next pass, under a pause of its own.
break
}
ok, err := moveToTheTail(ctx, js, stream, worker.Name, seat, a)
if err != nil {
return pass, err
}
if ok {
pass.Moved = append(pass.Moved, a.ID)
}
}
return pass, nil
}
// readWaiting is every ask the worker has not yet handed out, in queue order, but each second copy of one
// ask; the second copies, to remove; and the last sequence the worker handed out, -1 when there is no
// worker, so nothing is handed out to pause.
//
// **A second copy is proven, never guessed from an id**: a waiting message whose MovedFromHeader names an
// earlier message of the same request id still in the queue, and whose QueuedAtHeader is that message's
// first-queued time — a move that published its copy and stopped before it deleted the original. The
// original stays, waiting or handed out (a holder's then, and the copy would build it twice). Any other
// second message of one id — an ask a dead letter delivered again beside its original (ADR 0293) — is a
// real ask and stays.
func readWaiting(ctx context.Context, api jetstream.JetStream, worker broker.Consumer, seat string) (
waiting, extra []QueuedAsk, delivered int64, err error) {
stream, err := api.Stream(ctx, worker.Stream)
if err != nil {
return nil, nil, 0, fmt.Errorf("the %s work queue (%s) cannot be read: %w", seat, worker.Stream, err)
}
delivered = -1
if consumer, err := stream.Consumer(ctx, worker.Name); err == nil {
info, err := consumer.Info(ctx)
if err != nil {
return nil, nil, 0, fmt.Errorf("the worker of %s cannot be read: %w", seat, err)
}
delivered = int64(info.Delivered.Stream)
} else if !errors.Is(err, jetstream.ErrConsumerNotFound) {
return nil, nil, 0, fmt.Errorf("the worker of %s cannot be read: %w", seat, err)
}
// From the first: the stream holds only what is unsettled, so this is a handful of reads.
// Every message by sequence, with its id and first-queued time, to prove a copy against.
type held struct {
id string
stored time.Time
}
bySeq := map[string]held{}
var seq uint64 = 1
for n := 0; n < 10000; n++ {
msg, err := stream.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(BuildWorkOf(seat)))
if errors.Is(err, jetstream.ErrMsgNotFound) {
break
}
if err != nil {
return nil, nil, 0, fmt.Errorf("reading the %s work queue: %w", seat, err)
}
seq = msg.Sequence + 1
a := askOf(msg)
a.movedFrom = msg.Header.Get(MovedFromHeader)
from, known := bySeq[a.movedFrom]
again := a.ID != "" && a.movedFrom != "" && known && from.id == a.ID && from.stored.Equal(a.Stored)
bySeq[strconv.FormatUint(a.Seq, 10)] = held{id: a.ID, stored: a.Stored}
switch {
case int64(a.Seq) <= delivered:
case again:
extra = append(extra, a)
default:
waiting = append(waiting, a)
}
}
return waiting, extra, delivered, nil
}
// moveToTheTail publishes the ask's body again at the queue's tail and then deletes it where it was. **The
// copy first**: a publish that fails deletes nothing, and a pass that stops between the two leaves two
// copies, which the next pass sees by their one id and makes one (readWaiting). A move never loses an ask.
// Looked at once more just before: one the worker has handed out since is a holder's now and stays. False
// when it was not moved.
func moveToTheTail(ctx context.Context, js *broker.JetStream, stream jetstream.Stream, workerName, seat string,
a QueuedAsk) (bool, error) {
if consumer, err := stream.Consumer(ctx, workerName); err == nil {
info, err := consumer.Info(ctx)
if err != nil {
return false, err
}
if info.Delivered.Stream >= a.Seq {
return false, nil
}
}
msg, err := stream.GetMsg(ctx, a.Seq)
if errors.Is(err, jetstream.ErrMsgNotFound) {
return false, nil
}
if err != nil {
return false, err
}
again := &nats.Msg{Subject: msg.Subject, Data: msg.Data, Header: nats.Header{}}
for k, v := range msg.Header {
// The server's own headers stay behind: a message id would be taken for a duplicate and dropped.
if !strings.HasPrefix(k, "Nats-") {
again.Header[k] = v
}
}
// Its place in time goes with it: when the stream first took it, as askOf read it; and the message it
// is a copy of, which proves it one.
again.Header.Set(QueuedAtHeader, askOf(msg).Stored.UTC().Format(time.RFC3339Nano))
again.Header.Set(MovedFromHeader, strconv.FormatUint(a.Seq, 10))
// Inside the margin, so the publish ends while the worker is still paused.
publishing, cancel := context.WithTimeout(ctx, publishWaits)
ack, err := js.Context().PublishMsg(again, nats.Context(publishing))
cancel()
if err != nil {
return false, fmt.Errorf("%s (seq %d) could not be put behind, and stays where it is: %w", a.ID, a.Seq, err)
}
// **Handed out meanwhile** — a bus slow enough that the pause ended: the original is a holder's, and the
// copy goes instead, so it is built once.
if consumer, err := stream.Consumer(ctx, workerName); err == nil {
info, err := consumer.Info(ctx)
if err != nil || info.Delivered.Stream >= a.Seq {
if derr := stream.DeleteMsg(ctx, ack.Sequence); derr != nil && !errors.Is(derr, jetstream.ErrMsgNotFound) {
return false, fmt.Errorf("%s was handed out while it moved and its copy (seq %d) could not be "+
"removed; the next pass removes it: %w", a.ID, ack.Sequence, derr)
}
if err != nil {
return false, fmt.Errorf("whether %s was handed out while it moved cannot be read, so its copy was "+
"removed and it stays where it was: %w", a.ID, err)
}
if info.Delivered.Stream >= ack.Sequence {
return false, fmt.Errorf("%s was handed out while it moved, and its copy too: it may be built twice", a.ID)
}
return false, nil
}
}
if err := stream.DeleteMsg(ctx, a.Seq); err != nil && !errors.Is(err, jetstream.ErrMsgNotFound) {
return false, fmt.Errorf("%s was copied behind and its original (seq %d) not removed; the next pass "+
"removes the second copy: %w", a.ID, a.Seq, err)
}
return true, nil
}
// KeepingInOrder runs the passes, on the acting controller alone, until ctx ends: when an ask in this
// process says so, when the queue's last sequence moved — an ask from another process — and every
// passEvery, for a check that comes of age with nothing asked. seat says the build seat to order; acting
// whether this controller acts (Holding); said what a pass did.
func KeepingInOrder(ctx context.Context, js *broker.JetStream, seat func() string, acting func() bool,
said func(format string, args ...any)) {
tick := time.NewTicker(orderLooks)
defer tick.Stop()
var lastSeq uint64
var lastPass time.Time
for {
signalled := false
select {
case <-ctx.Done():
return
case <-asked:
signalled = true
case <-tick.C:
}
if !acting() {
continue
}
name := seat()
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: name, Accepts: []string{"build"}})
info, err := js.Context().StreamInfo(worker.Stream)
if err != nil {
continue
}
if !signalled && info.State.LastSeq == lastSeq && time.Since(lastPass) < passEvery {
continue
}
lastSeq, lastPass = info.State.LastSeq, time.Now()
passing, cancel := context.WithTimeout(ctx, pausedFor+10*time.Second)
pass, err := PutInOrder(passing, js, name)
cancel()
for _, r := range pass.Removed {
said("the %s queue: removed %s", name, r)
}
if len(pass.Moved) > 0 {
said("the %s queue put in order: %s moved behind what goes first", name, strings.Join(pass.Moved, ", "))
}
if err != nil {
said("the %s queue could not be put in order: %v", name, err)
}
if info, err := js.Context().StreamInfo(worker.Stream); err == nil {
// Its own moves are not an ask.
lastSeq = info.State.LastSeq
}
}
}
// orderLooks is how often the loop looks whether an ask joined from another process; passEvery how often
// it passes with nothing asked.
var (
orderLooks = 2 * time.Second
passEvery = 30 * time.Second
)
// DeleteAsk removes a waiting or dead ask from the queue: at its sequence, or — moved into order since it
// was read — the copy carrying its id. Says the sequence it removed, 0 when neither was there any more.
func DeleteAsk(ctx context.Context, js *broker.JetStream, seat string, ask QueuedAsk) (uint64, error) {
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: seat, Accepts: []string{"build"}})
gone := func(err error) bool {
return errors.Is(err, nats.ErrMsgNotFound) || errors.Is(err, jetstream.ErrMsgNotFound) ||
(err != nil && strings.Contains(err.Error(), "no message found"))
}
err := js.Context().DeleteMsg(worker.Stream, ask.Seq)
if err == nil {
return ask.Seq, nil
}
if !gone(err) {
return 0, err
}
if ask.ID == "" {
return 0, nil
}
q, err := ReadQueue(ctx, js, seat)
if err != nil {
return 0, err
}
for _, a := range q.Asks {
if a.ID == ask.ID && a.Seq != ask.Seq && a.State != AskInFlight {
if err := js.Context().DeleteMsg(worker.Stream, a.Seq); err != nil && !gone(err) {
return 0, err
} else if err == nil {
return a.Seq, nil
}
}
}
return 0, nil
}
+441
View File
@@ -0,0 +1,441 @@
package link
import (
"context"
"encoding/json"
"fmt"
"reflect"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
)
// The order the build queue hands asks out in (novox/hq issue 558): a build a walk waits for before any
// pull request's check, a check that has waited ten minutes before every delivery asked after it, and
// first come, first served within a kind.
func anAsk(seq uint64, id string, asked time.Time, check bool) QueuedAsk {
r := BuildRequest{ID: id, Repository: "/r"}
if check {
r.Check = &CheckRequest{Owner: "novox", Repo: "r"}
}
return QueuedAsk{Seq: seq, ID: id, AskedAt: asked, Request: r, State: AskWaiting}
}
func idsOf(asks []QueuedAsk) []string {
out := []string{}
for _, a := range asks {
out = append(out, a.ID)
}
return out
}
func TestADeliveryIsHandedOutBeforeTheChecksAskedBeforeIt(t *testing.T) {
now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
waiting := []QueuedAsk{
anAsk(1, "check-a", now.Add(-3*time.Minute), true),
anAsk(2, "check-b", now.Add(-2*time.Minute), true),
anAsk(3, "delivery-d", now.Add(-time.Minute), false),
}
want := []string{"delivery-d", "check-a", "check-b"}
if got := idsOf(QueueOrder(waiting, now)); !reflect.DeepEqual(got, want) {
t.Fatalf("handed out as %v, want %v", got, want)
}
// Only the checks ahead of it move behind it; the delivery stays where it is.
if got := idsOf(toMove(waiting, QueueOrder(waiting, now))); !reflect.DeepEqual(got, []string{"check-a", "check-b"}) {
t.Fatalf("moved %v", got)
}
}
func TestACheckThatWaitedTenMinutesGoesBeforeALaterDelivery(t *testing.T) {
now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
// As the queue stands after a delivery put the young checks behind it, ten minutes ago and since.
waiting := []QueuedAsk{
anAsk(5, "delivery-early", now.Add(-12*time.Minute), false),
anAsk(6, "delivery-later", now.Add(-time.Minute), false),
anAsk(7, "check-old", now.Add(-11*time.Minute), true),
anAsk(8, "check-young", now.Add(-9*time.Minute), true),
}
want := []string{"delivery-early", "check-old", "delivery-later", "check-young"}
if got := idsOf(QueueOrder(waiting, now)); !reflect.DeepEqual(got, want) {
t.Fatalf("handed out as %v, want %v", got, want)
}
if got := idsOf(toMove(waiting, QueueOrder(waiting, now))); !reflect.DeepEqual(got, []string{"delivery-later", "check-young"}) {
t.Fatalf("moved %v", got)
}
}
func TestAnAskWhoseKindCannotBeToldIsACheck(t *testing.T) {
for _, r := range []BuildRequest{
{}, // unreadable: nothing was read from it
{ID: "build-1", Check: &CheckRequest{}}, // a pull request's check, or a recheck the merge tool asked
{ID: "build-2", DryRun: true}, // looked at, never delivered
} {
if got := KindOf(r); got != KindCheck {
t.Errorf("%+v is a %s", r, got)
}
}
if got := KindOf(BuildRequest{ID: "build-3", Repository: "/r"}); got != KindDelivery {
t.Errorf("a build is a %s", got)
}
}
// Against a real server: two checks wait, a delivery is asked, and the next free holder takes the delivery.
func TestNatsADeliveryIsTakenBeforeTheChecksQueuedAheadOfIt(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx, stop := context.WithCancel(context.Background())
defer stop()
asks := BuildsOn(js, TheBuildMachine)
now := time.Now()
for i, r := range []BuildRequest{
{ID: NewBuildID(now.Add(-3 * time.Minute)), Repository: "/a", Check: &CheckRequest{Owner: "novox", Repo: "a"}},
{ID: NewBuildID(now.Add(-2 * time.Minute)), Repository: "/b", Check: &CheckRequest{Owner: "novox", Repo: "b"}},
{ID: NewBuildID(now), Repository: "/d"},
} {
if i == 2 {
// And one nobody can read, whose kind cannot be told: it waits with the checks.
if _, err := js.Context().Publish(BuildWork(), []byte("not an ask")); err != nil {
t.Fatal(err)
}
}
if err := asks.Ask(ctx, r); err != nil {
t.Fatal(err)
}
}
// The acting controller's loop puts it in order, told by the asks.
go KeepingInOrder(ctx, js, func() string { return TheBuildMachine }, func() bool { return true },
func(string, ...any) {})
time.Sleep(500 * time.Millisecond)
took := make(chan string, 4)
machine := MachineOverNATS(js, "anchor")
defer machine.Close()
go func() {
_ = machine.Take(ctx, func(ctx context.Context, work Build) {
took <- work.Request().Repository
_ = work.Announce(ctx, BuildResult{ID: work.Request().ID, On: "anchor", Failed: "no"})
_ = work.Done()
})
}()
var order []string
for len(order) < 3 {
select {
case r := <-took:
order = append(order, r)
case <-time.After(10 * time.Second):
t.Fatalf("taken so far: %v", order)
}
}
if want := []string{"/d", "/a", "/b"}; !reflect.DeepEqual(order, want) {
t.Fatalf("taken as %v, want %v", order, want)
}
}
// --- the moves, against a real server ------------------------------------------------------------
func publishAsk(t *testing.T, js *broker.JetStream, r BuildRequest) {
t.Helper()
body, _ := json.Marshal(r)
if _, err := js.Context().Publish(BuildWork(), body); err != nil {
t.Fatal(err)
}
}
func aCheck(asked time.Time, repository string) BuildRequest {
return BuildRequest{ID: NewBuildID(asked), Repository: repository, Check: &CheckRequest{Owner: "novox", Repo: "r"}}
}
func aDelivery(asked time.Time, repository string) BuildRequest {
return BuildRequest{ID: NewBuildID(asked), Repository: repository}
}
// queued is the stream's asks, in queue order.
func queued(t *testing.T, js *broker.JetStream) []QueuedAsk {
t.Helper()
q, err := ReadQueue(context.Background(), js, TheBuildMachine)
if err != nil {
t.Fatal(err)
}
return q.Asks
}
// takeAll runs a holder that records how often it was handed each repository.
func takeAll(t *testing.T, ctx context.Context, js *broker.JetStream) (chan string, func() map[string]int) {
t.Helper()
took := make(chan string, 100)
var mu sync.Mutex
counts := map[string]int{}
machine := MachineOverNATS(js, "anchor")
t.Cleanup(machine.Close)
go func() {
_ = machine.Take(ctx, func(ctx context.Context, work Build) {
mu.Lock()
counts[work.Request().Repository]++
mu.Unlock()
took <- work.Request().Repository
_ = work.Announce(ctx, BuildResult{ID: work.Request().ID, On: "anchor", Failed: "no"})
_ = work.Done()
})
}()
return took, func() map[string]int {
mu.Lock()
defer mu.Unlock()
out := map[string]int{}
for k, v := range counts {
out[k] = v
}
return out
}
}
// A move that published its copy and stopped before deleting the original leaves two copies of one ask;
// the next pass makes them one, and a copy of one a holder already took is removed, never built twice.
func TestNatsAMoveCutShortLeavesOneCopyAfterTheNextPass(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx := t.Context()
now := time.Now()
check := aCheck(now.Add(-2*time.Minute), "/c")
publishAsk(t, js, check)
publishAsk(t, js, aDelivery(now.Add(-time.Minute), "/d"))
moveCopyOf(t, js, check.ID) // the copy, published; the original never deleted
if pass, err := PutInOrder(ctx, js, TheBuildMachine); err != nil || len(pass.Removed) != 1 {
t.Fatalf("removed %v: %v", pass.Removed, err)
}
var got []string
for _, a := range queued(t, js) {
got = append(got, a.Repository)
}
if want := []string{"/d", "/c"}; !reflect.DeepEqual(got, want) {
t.Fatalf("after the pass the queue holds %v, want %v", got, want)
}
// And one a holder took before the copy was published: the copy goes, the holder's stays.
taken := aCheck(now, "/taken")
publishAsk(t, js, taken)
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: TheBuildMachine, Accepts: []string{"build"}})
sub, err := js.Context().PullSubscribe(worker.Filters[0], worker.Name, nats.Bind(worker.Stream, worker.Name), nats.ManualAck())
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
// Take everything waiting up to and including /taken, and keep /taken unsettled.
for {
msgs, err := sub.Fetch(1, nats.MaxWait(2*time.Second))
if err != nil || len(msgs) == 0 {
t.Fatalf("nothing to take: %v", err)
}
var r BuildRequest
_ = json.Unmarshal(msgs[0].Data, &r)
if r.Repository == "/taken" {
break
}
_ = msgs[0].Ack()
}
moveCopyOf(t, js, taken.ID)
if _, err := PutInOrder(ctx, js, TheBuildMachine); err != nil {
t.Fatal(err)
}
n := 0
for _, a := range queued(t, js) {
if a.ID == taken.ID {
n++
}
}
if n != 1 {
t.Fatalf("%d copies of the ask a holder took are in the queue, want 1", n)
}
}
// moveCopyOf publishes what a move publishes for the ask with this id, as a pass stopped before its delete
// would leave it.
func moveCopyOf(t *testing.T, js *broker.JetStream, id string) {
t.Helper()
for _, a := range queued(t, js) {
if a.ID != id {
continue
}
body, _ := json.Marshal(a.Request)
msg := &nats.Msg{Subject: BuildWork(), Data: body, Header: nats.Header{}}
msg.Header.Set(QueuedAtHeader, a.Stored.UTC().Format(time.RFC3339Nano))
msg.Header.Set(MovedFromHeader, fmt.Sprint(a.Seq))
if _, err := js.Context().PublishMsg(msg); err != nil {
t.Fatal(err)
}
return
}
t.Fatalf("%s is not queued", id)
}
// A dead letter delivered again whose original stayed beside it (ADR 0293: its removal failed) is a real
// ask with the same id, and is never removed as a second copy — even when the original was itself a move's
// copy, whose headers the delivery carries.
func TestNatsAnAskDeliveredAgainBesideItsOriginalStays(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx := t.Context()
ask := aCheck(time.Now(), "/again")
publishAsk(t, js, aCheck(time.Now().Add(-time.Minute), "/before"))
publishAsk(t, js, ask)
publishAsk(t, js, aDelivery(time.Now(), "/d"))
if _, err := PutInOrder(ctx, js, TheBuildMachine); err != nil { // /again becomes a move's copy
t.Fatal(err)
}
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: TheBuildMachine, Accepts: []string{"build"}})
sub, err := js.Context().PullSubscribe(worker.Filters[0], worker.Name, nats.Bind(worker.Stream, worker.Name), nats.ManualAck())
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
var original *nats.Msg
for original == nil {
msgs, err := sub.Fetch(1, nats.MaxWait(2*time.Second))
if err != nil || len(msgs) == 0 {
t.Fatalf("nothing to take: %v", err)
}
var r BuildRequest
_ = json.Unmarshal(msgs[0].Data, &r)
if r.ID == ask.ID {
original = msgs[0] // given up on, never settled: dead, and kept
continue
}
_ = msgs[0].Ack()
}
// Delivered again as DeliverAgain does: its headers, and its own.
again := &nats.Msg{Subject: BuildWork(), Data: original.Data, Header: nats.Header{}}
for k, v := range original.Header {
again.Header[k] = v
}
again.Header.Set(AgainHeader, "7")
if _, err := js.Context().PublishMsg(again); err != nil {
t.Fatal(err)
}
pass, err := PutInOrder(ctx, js, TheBuildMachine)
if err != nil || len(pass.Removed) != 0 {
t.Fatalf("removed %v: %v", pass.Removed, err)
}
n := 0
for _, a := range queued(t, js) {
if a.ID == ask.ID {
n++
}
}
if n != 2 {
t.Fatalf("%d messages of the ask delivered again are queued, want its original and the delivery", n)
}
}
// Two passes at once: nothing is built twice, and nothing is lost.
func TestNatsTwoPassesAtOnceBuildNothingTwice(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx, stop := context.WithCancel(t.Context())
defer stop()
took, counts := takeAll(t, ctx, js)
now := time.Now()
const asks = 24
var passes sync.WaitGroup
for p := 0; p < 2; p++ {
passes.Add(1)
go func() {
defer passes.Done()
for i := 0; i < 15; i++ {
_, _ = PutInOrder(ctx, js, TheBuildMachine)
}
}()
}
for i := 0; i < asks; i++ {
repository := fmt.Sprintf("/%02d", i)
if i%3 == 2 {
publishAsk(t, js, aDelivery(now.Add(time.Duration(i)*time.Millisecond), repository))
} else {
publishAsk(t, js, aCheck(now.Add(time.Duration(i)*time.Millisecond), repository))
}
}
passes.Wait()
for got := 0; got < asks; got++ {
select {
case <-took:
case <-time.After(15 * time.Second):
t.Fatalf("taken %d of %d: %v", got, asks, counts())
}
}
time.Sleep(time.Second)
c := counts()
for i := 0; i < asks; i++ {
if n := c[fmt.Sprintf("/%02d", i)]; n != 1 {
t.Errorf("/%02d was built %d times", i, n)
}
}
}
// A check that has waited ten minutes goes before a delivery asked after it, even one queued ahead of it.
func TestNatsACheckThatWaitedTenMinutesIsTakenBeforeALaterDelivery(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx, stop := context.WithCancel(t.Context())
defer stop()
now := time.Now()
publishAsk(t, js, aDelivery(now.Add(-time.Minute), "/delivery"))
publishAsk(t, js, aCheck(now.Add(-11*time.Minute), "/old-check"))
if _, err := PutInOrder(ctx, js, TheBuildMachine); err != nil {
t.Fatal(err)
}
took, _ := takeAll(t, ctx, js)
var order []string
for len(order) < 2 {
select {
case r := <-took:
order = append(order, r)
case <-time.After(10 * time.Second):
t.Fatalf("taken so far: %v", order)
}
}
if want := []string{"/old-check", "/delivery"}; !reflect.DeepEqual(order, want) {
t.Fatalf("taken as %v, want %v", order, want)
}
}
// A cancel that read an ask before it moved deletes the moved copy, found by its id, and says when
// there was nothing left to delete.
func TestNatsACancelRacingAMoveDeletesTheMovedCopy(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx := t.Context()
now := time.Now()
publishAsk(t, js, aCheck(now.Add(-time.Minute), "/c"))
read := queued(t, js)[0]
publishAsk(t, js, aDelivery(now, "/d"))
if pass, err := PutInOrder(ctx, js, TheBuildMachine); err != nil || len(pass.Moved) != 1 {
t.Fatalf("moved %v: %v", pass.Moved, err)
}
removed, err := DeleteAsk(ctx, js, TheBuildMachine, read)
if err != nil || removed == 0 || removed == read.Seq {
t.Fatalf("removed seq %d (read at %d): %v", removed, read.Seq, err)
}
for _, a := range queued(t, js) {
if a.ID == read.ID {
t.Fatalf("the cancelled ask is still queued at seq %d", a.Seq)
}
}
if removed, err := DeleteAsk(ctx, js, TheBuildMachine, read); err != nil || removed != 0 {
t.Fatalf("an ask gone already is said removed at %d: %v", removed, err)
}
}
// An ask nothing can be read from keeps its place in time when it moves.
func TestNatsAnUnreadableAskKeepsItsAgeWhenItMoves(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx := t.Context()
if _, err := js.Context().Publish(BuildWork(), []byte("not an ask")); err != nil {
t.Fatal(err)
}
first := queued(t, js)[0].Stored
time.Sleep(50 * time.Millisecond)
publishAsk(t, js, aDelivery(time.Now(), "/d"))
if pass, err := PutInOrder(ctx, js, TheBuildMachine); err != nil || len(pass.Moved) != 1 {
t.Fatalf("moved %v: %v", pass.Moved, err)
}
asks := queued(t, js)
if len(asks) != 2 || asks[1].ID != "" || !asks[1].Stored.Equal(first) {
t.Fatalf("after the move the unreadable ask was taken at %v, first at %v: %+v", asks[1].Stored, first, asks)
}
}
-114
View File
@@ -1,114 +0,0 @@
package link
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
)
// The take act: a shared folder that already holds data is taken by the mesh once, on the operator's word
// (novox/hq ADR 0325 §8).
//
// The controller asks the operator once for that folder on that machine (ADR 0259, at the level approve). On the
// warrant it asks the machine's node-engine, on the engine's own subject (broker.AskAccessActSubject), signed with
// the mesh's key as a hand-over is (SignedHandOver) under a signing context of its own (AccessActContext), to take
// the folder: the engine adds the entries of the users it was declared for that folder to everything under it —
// following no link, entering no other filesystem — answers at once that it started or why it refused, and states
// the walk on the access's verdict. **Nothing is taken without a warrant**: the ask names the warrant the operator
// gave, and the controller sends it only from the warrant (cmd/mesh-controller decideTake).
//
// The engine's own shape (mesh-host internal/link AccessActAsk, AnswerAccessAct): a hand-over's fields and
// {act: "take", access: <the access's resource id in the declaration>, warrant}.
// AccessActContext is prefixed to an act ask's bytes before signing: never another ask's signature, nor a
// declaration's. The engine holds the same words.
const AccessActContext = "novox-mesh access-act v2\n"
// ActTake is the one act the controller asks.
const ActTake = "take"
// AccessActAsk is what the controller asks: a hand-over's fields — the node it is for, the folder's absolute path as
// the declaration places it, who asked, when the ask stops being good and a nonce the engine takes once — and the
// act's own: which act, the access by its resource id, and the warrant the operator gave.
type AccessActAsk struct {
Node string `json:"node"`
Path string `json:"path"`
By string `json:"by"`
Expires time.Time `json:"expires"`
Nonce string `json:"nonce"`
Act string `json:"act"`
Access string `json:"access"`
Warrant string `json:"warrant"`
}
// MaxActField bounds an access id and a warrant, as the engine does.
const MaxActField = 200
// TakeWithin is how long the controller waits for the engine's answer: it starts the walk and answers, and the
// walk's end is stated on the access's verdict.
const TakeWithin = 30 * time.Second
// SignTake fills a take ask's act, expiry and nonce and signs it with the mesh's key over AccessActContext. It refuses
// an ask with no warrant, no path or no access — and one the engine would refuse for its shape: nothing is ever asked
// of an engine that it would have to refuse.
func SignTake(ctx context.Context, signer Signer, ask AccessActAsk) ([]byte, error) {
ask.Access, ask.Warrant = strings.TrimSpace(ask.Access), strings.TrimSpace(ask.Warrant)
switch {
case signer == nil:
return nil, errors.New("no signing key, so nothing can be asked of an engine")
case ask.Warrant == "":
return nil, errors.New("a take is asked only on the operator's warrant, and this one names none")
case ask.Node == "" || !strings.HasPrefix(ask.Path, "/"):
return nil, errors.New("a take names its machine and the folder's absolute path")
case ask.Access == "":
return nil, errors.New("a take names the access whose folder it takes")
}
// The warrant is one short name; the access is the resource id the declaration gave it, which may hold a path's
// slashes (catalogue.AccessID) — the engine judges it against what it was declared.
if len(ask.Warrant) > MaxActField || strings.ContainsAny(ask.Warrant, "\r\n\x00/") {
return nil, fmt.Errorf("the take's warrant %q is not one short name", ask.Warrant)
}
if len(ask.Access) > MaxActField || strings.ContainsAny(ask.Access, "\r\n\x00") {
return nil, fmt.Errorf("the take's access %q is not one line", ask.Access)
}
nonce := make([]byte, 16)
if _, err := rand.Read(nonce); err != nil {
return nil, err
}
ask.Act, ask.Nonce, ask.Expires = ActTake, hex.EncodeToString(nonce), time.Now().UTC().Add(HandOverGood)
raw, err := json.Marshal(ask)
if err != nil {
return nil, err
}
signature, err := signer.Sign(ctx, append([]byte(AccessActContext), raw...))
if err != nil {
return nil, fmt.Errorf("cannot sign the ask: %w", err)
}
return json.Marshal(SignedHandOver{Ask: raw, Signature: signature})
}
// AskTake asks one machine's node-engine to take a shared folder on the operator's warrant, and reads its answer. An
// error is the ask not reaching an engine, or an answer that is not one; a refusal is the engine's and comes back in
// the answer.
func AskTake(ctx context.Context, conn *nats.Conn, signer Signer, ask AccessActAsk, timeout time.Duration) (HandOverAnswer, error) {
if conn == nil {
return HandOverAnswer{}, errors.New("this controller is not on the bus")
}
body, err := SignTake(ctx, signer, ask)
if err != nil {
return HandOverAnswer{}, err
}
return askSigned(ctx, conn, broker.AskAccessActSubject(ask.Node), body, ask.Node, timeout, askWords{
what: "the take of a shared folder", nothing: "nothing was taken", issue: "ADR 0325",
unknown: "whether it started is not known — the access's verdict on the machine says whether it is kept",
record: "a start"})
}
+2 -2
View File
@@ -8,7 +8,7 @@ private addresses. That is acceptable while mesh-controller is private; if it ev
a second place to clean besides the catalogue itself.
mesh-catalog b9de001833b1b61b297182b8e3bcdae1cbdeace9 modules/*/module.json, modules/nats/Dockerfile
mesh-host 7a41e120635b3f7b6cb94cca8b3fc4093e4e6110 examples/foundation-first-node.lock
mesh-host a76baa4b47f07a8b2bab065603414530bd02b53a examples/foundation-first-node-nats.lock
mesh-sdk 33917f91ac3cc213b879b32f3c70e2aa70395f98 conformance/events/module-event.json
To move them, from this repository's root, with the two repositories checked out beside it:
@@ -18,7 +18,7 @@ To move them, from this repository's root, with the two repositories checked out
git -C ../mesh-catalog archive <commit> modules | tar -x -C testdata/beside/mesh-catalog --wildcards \
'modules/*/module.json' 'modules/nats/Dockerfile'
find testdata/beside -name module.json -exec mv {} {}.captured \;
git -C ../mesh-host archive <commit> examples/foundation-first-node.lock | tar -x -C testdata/beside/mesh-host
git -C ../mesh-host archive <commit> examples/foundation-first-node-nats.lock | tar -x -C testdata/beside/mesh-host
and write the commits here. The SDK is captured at the commit go.mod pins for git.novox.be/novox/mesh-sdk/go
(the last part of its pseudo-version; novox/hq issue 449), which internal/link's conformance test holds it to:
@@ -1,17 +1,11 @@
// foundation-first-node.lock — what a machine must be before a mesh exists: the one genesis bundle
// (novox/hq ADR 0331 §2), on the bus the mesh runs on (ADR 0106, ADR 0131, design 25).
// foundation-first-node-nats.lock — what a machine must be before a mesh exists, on the bus being
// built (novox/hq ADR 0106, design 25).
//
// The whole bootstrap (novox/hq 03-DESIGN/01-to-be/07-the-foundation.md): a container runtime, a
// filter, a store, a database per context, those contexts' schemas, the bus with its certificate
// and its first user list, and the control plane running on top of them.
//
// **The mesh writes its own user list, and at genesis there is no mesh yet to write it.** So this
// carries the first one — the controller's own account — and the control plane's address names the
// same account. Both say `bootstrap`, and both are placeholders: the installer makes the account's
// password, puts it in the address and its hash in the list (mesh-host internal/bootstrap
// rootsecrets.go), and refuses a bundle whose bus connection it does not recognise. Until the first
// node has enrolled the installer places the list the controller composes; from the bus's adoption
// as the `nats` module on, the module receives it like any module (ADR 0331 §1, §5).
// The same twelve steps as foundation-first-node.lock, with one difference that matters: **the mesh
// writes its own user list, and at genesis there is no mesh yet to write it.** So this carries the
// first one — the controller's own account, at a well-known bootstrap password, exactly as the store
// is reached at `postgres:bootstrap` and the old bus at `guest:guest`. It is rotated with those, and
// from the controller's first composition onward the file is the controller's to write.
//
// The accounts file is its own file beside the server's configuration, because the server's own
// settings belong to whoever raises it and the users belong to the mesh (design 25 §4). Both live in
@@ -21,18 +15,7 @@
//
// No `verify` on the TLS block, deliberately — that setting makes the server demand a *client*
// certificate, and nothing in the mesh presents one: a host pins this server's exact certificate and
// authenticates with a password (ADR 0004, design 25 §4). The control plane pins it too, through
// MESH_BROKER_CERTIFICATE, and dials the bus on the machine's own loopback.
//
// PINNED BY DIGEST, and the digest is not decoration: a tag can be made to point at a different
// image, and this file is applied on a machine with no mesh to ask about anything (ADR 0006). The
// registry prefix is the lab's; the digests are upstream's.
//
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
// creates is owned by root, and the database runs as somebody else inside the container — so it
// could not write, and the container crash-looped. A named volume lets the image set up its own
// ownership, and outlives the container, which is what you want for the thing holding the mesh's
// state.
// authenticates with a password (ADR 0004, design 25 §4).
{
"declaration": 1,
@@ -523,20 +523,6 @@ func refusedPlaces(resources []declaration.Resource) map[string]error {
switch res := r.(type) {
case *declaration.Directory:
err = refusePath(res.Path, placingDirectory, moduleOfID(res.ID), res.Owner)
// And outside every home, the walk's judgement before anything is applied (novox/hq issue 358): a
// link another account could make in its path refuses it, and with it what is below it, whatever
// order they are declared in. The apply walks again at the moment of use. A directory handed over
// keeps its own, stricter walk (issue 356), which refuses any link at all.
if err == nil && meshDirsByDescriptor && homeAbove(res.Path) == "" &&
!HandedOver()[handOverKey(res.ID, res.Path)] {
if judged := judgeMeshDir(res.Path); judged != nil {
err = judged
var link *LinkInPathError
if errors.As(judged, &link) {
err = &PlacementRefusedError{Path: res.Path, Why: "it " + link.why()}
}
}
}
case *declaration.Access:
err = refusePath(res.Path, placingAccess, moduleOfID(res.ID), "")
default:
+3 -16
View File
@@ -35,19 +35,6 @@
./cmd/mesh-controller TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend
./cmd/mesh-controller TestAGateJudgesTheOtherMovesBesideAKeptCore
./cmd/mesh-controller TestReplay401
# novox/hq ADR 0328 §1, §3, §4, §7, §13, §15 and issue 575: numbers allocated once and never root, libraries defined
# once and never overlapping, a library taken under a warrant, a placement judged against the build that brings it.
./internal/inventory TestOneNumberPerModuleAndLibraryNeverRoot
./internal/inventory TestALibraryIsDefinedOnceForTheMeshAndNeverOverlaps
./internal/inventory TestALibraryIsTakenOnceUnderAWarrant
./internal/inventory TestAPlacementIsJudgedAgainstTheBuildThatBringsTheAccess
./internal/inventory TestOnlyABuildOfTheFollowedBranchJudgesAPlacementAhead
./cmd/mesh-controller TestALibrarysVerdictIsSaidForTheModulesThatUseIt
./cmd/mesh-controller TestAHeldModuleIsComposedAsBeforeItJoined
./cmd/mesh-controller TestALibraryIsTakenOnlyWhenTheEngineSaysTheActIsDone
./internal/inventory TestTheSubPathsOfModulesPulledInByARequirementAreKept
./internal/inventory TestALibraryIsTakenWhenTheActIsDone
./cmd/mesh-controller TestADeliveryWaitsForALibraryAndJudgesTheRest
./cmd/mesh-controller TestAJoiningModuleIsHeldOnAnEngineOlderThanOwnGroups
./cmd/mesh-controller TestAHeldModuleKeepsTheSettingsItRanWith
./cmd/mesh-controller TestARunnersVerdictIsSaidAndJudgedByWhetherTheMoveChangesIt
./cmd/mesh-controller TestReplay201ThePlansSend
./internal/inventory TestAWalksRestIsReadFromEachMachinesOwnSend
./cmd/mesh-controller TestPlanAndNodeListAnOptionalRequirementLeftUnbound
+5 -348
View File
@@ -12,7 +12,6 @@ import (
"fmt"
"io"
"net"
"path/filepath"
"reflect"
"regexp"
"slices"
@@ -133,13 +132,7 @@ func (d *Directory) validate(where string, _ bool) []string {
if d.Path == "" {
problems = append(problems, where+": a directory needs a path")
}
// A directory's mode may carry the setgid bit (2xxx), so what is made beneath it keeps its group (novox/hq ADR
// 0334 §1); never setuid or sticky, which mean nothing the mesh asks of a directory it owns.
mode := d.Mode
if len(mode) == 4 && mode[0] == '2' {
mode = "0" + mode[1:]
}
return append(problems, checkMode(where, mode)...)
return append(problems, checkMode(where, d.Mode)...)
}
// File is a file with literal content. The host renders nothing.
@@ -463,51 +456,14 @@ const (
// **The distinction 04-ISSUES/036 and 026 turn on.** A `directory` resource is the mesh's own —
// it creates it, sets its owner and mode, and removes it when empty ([ADR 0030](novox/hq)). A
// media library, a download spool is the operator's: it existed before the mesh, several modules
// read and write it at once, and the mesh must not create, chown, chmod or remove it. The host
// confirms it is there and mounts it; nothing else. Library data is a library's (novox/hq ADR 0328),
// not an access.
// read and write it at once, and the mesh must not create, chown, reconcile or remove it. The
// host confirms it is there and mounts it; nothing else.
type Access struct {
ID string `json:"id"`
Type Type `json:"type"`
Path string `json:"path"`
// Mode is how this module reaches the path: read or read-write. Absent narrows to read.
Mode string `json:"mode,omitempty"`
// Users and Taken are what a controller speaking ADR 0325 sends (the identities that use the folder, and
// that the operator's warrant was carried out): accepted, so its declaration is not refused whole, and not
// acted on since ADR 0328 superseded ADR 0325 — no ACL entry is kept.
Users []AccessUser `json:"users,omitempty"`
Taken bool `json:"taken,omitempty"`
// Grant is what a controller older than ADR 0325 still sends (ADR 0312's group grant, superseded): accepted
// so its declaration is not refused whole, and never acted on — no account is put in a group for data (§5).
Grant json.RawMessage `json:"grant,omitempty"`
}
// The uses an access's user has (ADR 0051 §3), and where it runs (ADR 0325 §4).
const (
FromContainer = "container"
FromToolRunner = "tool-runner"
)
// AccessUser is one identity that uses an accessed folder: its name and uid as the controller allocated them, its
// use, and where it runs. A tool runner's account may come with no uid (0 or absent): the engine reads it by name.
type AccessUser struct {
Identity string `json:"identity"`
UID int `json:"uid,omitempty"`
Use string `json:"use"`
From string `json:"from"`
}
// UnmarshalJSON reads a user as strictly as a resource is read.
func (u *AccessUser) UnmarshalJSON(raw []byte) error {
type plain AccessUser
var p plain
if err := strictDecode(raw, &p); err != nil {
return fmt.Errorf("an access's user: %w", err)
}
*u = AccessUser(p)
return nil
}
func (a *Access) Identity() string { return a.ID }
@@ -526,43 +482,9 @@ func (a *Access) validate(where string, _ bool) []string {
problems = append(problems, fmt.Sprintf(
"%s: an access is %q or %q, not %q", where, AccessRead, AccessReadWrite, a.Mode))
}
for _, u := range a.Users {
switch {
case u.Identity == "" || !accountName.MatchString(u.Identity):
problems = append(problems, fmt.Sprintf("%s: a user of an access is an account's name, and %q is not one",
where, u.Identity))
case u.UID < 0 || (u.UID == 0 && u.From != FromToolRunner):
// Never root (novox/hq ADR 0325 §11): an entry for uid 0 would be a right root has anyway, and a
// number below it is no account. A tool runner's account alone may come without its number: the
// controller cannot know the operator's account's, and the engine reads it by name
// (apply.ResolveRunners), refusing one that is root's.
problems = append(problems, fmt.Sprintf("%s: the user %s has uid %d, and a module's identity is never "+
"root or below it (novox/hq ADR 0325 §11)", where, u.Identity, u.UID))
}
if u.Use != AccessRead && u.Use != AccessReadWrite {
problems = append(problems, fmt.Sprintf("%s: the user %s uses it %q, and a use is %q or %q", where,
u.Identity, u.Use, AccessRead, AccessReadWrite))
}
if u.Use == AccessReadWrite && a.Mode != AccessReadWrite {
problems = append(problems, fmt.Sprintf("%s: the user %s writes an access declared %s", where,
u.Identity, orRead(a.Mode)))
}
if u.From != FromContainer && u.From != FromToolRunner {
problems = append(problems, fmt.Sprintf("%s: the user %s runs from %q, and that is %q or %q", where,
u.Identity, u.From, FromContainer, FromToolRunner))
}
}
return problems
}
// orRead is an access's mode as said: absent narrows to read.
func orRead(mode string) string {
if mode == "" {
return AccessRead
}
return mode
}
func (u *User) Identity() string { return u.ID }
func (u *User) Kind() Type { return TypeUser }
func (u *User) Target() string { return u.Name }
@@ -1238,78 +1160,6 @@ type Container struct {
// Health is how this resource is ready (novox/hq ADR 0240 rule 2, Phase B): one kind and its
// timing, judged by the node-engine beside liveness. Absent: judged alive or not, and nothing more.
Health *Health `json:"health,omitempty"`
// RunAs is who the container runs as, resolved by the controller from the module's identity and the
// image's declared interface (novox/hq ADR 0325 §2, §3): told by environment (the image reads PUID and
// PGID, say) or by the runtime's own user option. Part of the spec, so a change recreates it. Absent: the
// container runs as its image says. Its key is `identity`, the manifest's word; the field is not, because
// a resource's Identity is its id.
RunAs *ContainerIdentity `json:"identity,omitempty"`
}
// The ways an image is told who to run as (novox/hq ADR 0325 §2).
const (
IdentityByEnv = "env"
IdentityByRun = "run"
)
// ContainerIdentity is who a container runs as, and how its image is told.
type ContainerIdentity struct {
// Mode is IdentityByEnv or IdentityByRun.
Mode string `json:"mode"`
// UserEnv and GroupEnv are the variables the image reads the user's and the group's number from, for
// IdentityByEnv: "PUID", "PGID".
UserEnv string `json:"user_env,omitempty"`
GroupEnv string `json:"group_env,omitempty"`
UID int `json:"uid"`
GID int `json:"gid"`
}
// UnmarshalJSON reads an identity as strictly as a resource is read.
func (i *ContainerIdentity) UnmarshalJSON(raw []byte) error {
type plain ContainerIdentity
var p plain
if err := strictDecode(raw, &p); err != nil {
return fmt.Errorf("a container's identity: %w", err)
}
*i = ContainerIdentity(p)
return nil
}
// envName is a variable's name as a runtime takes it.
var envName = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
func (i *ContainerIdentity) problems(where string, env map[string]string) []string {
if i == nil {
return nil
}
var problems []string
switch i.Mode {
case IdentityByEnv:
if !envName.MatchString(i.UserEnv) || (i.GroupEnv != "" && !envName.MatchString(i.GroupEnv)) {
problems = append(problems, fmt.Sprintf("%s: an identity told by environment names the variables "+
"it is told in, and %q and %q are not both names", where, i.UserEnv, i.GroupEnv))
}
for _, name := range []string{i.UserEnv, i.GroupEnv} {
if _, set := env[name]; set && name != "" {
problems = append(problems, fmt.Sprintf("%s: env sets %s, which the identity sets; one value "+
"for one variable", where, name))
}
}
case IdentityByRun:
if i.UserEnv != "" || i.GroupEnv != "" {
problems = append(problems, where+": an identity told by the runtime's user option names no variable")
}
default:
problems = append(problems, fmt.Sprintf("%s: an identity is told by %q or %q, not %q", where,
IdentityByEnv, IdentityByRun, i.Mode))
}
// Never root (novox/hq ADR 0325 §11): refused here, so nothing of the declaration is made or started.
if i.UID <= 0 || i.GID <= 0 {
problems = append(problems, fmt.Sprintf("%s: the identity is uid %d, gid %d, and a module's identity is "+
"never root, nor a number below it (novox/hq ADR 0325 §11)", where, i.UID, i.GID))
}
return problems
}
func (c *Container) Identity() string { return c.ID }
@@ -1392,7 +1242,6 @@ func (c *Container) validate(where string, _ bool) []string {
problems = append(problems, where+": networks is for a container that keeps running; a step "+
"runs and exits, and joins nothing afterwards")
}
problems = append(problems, c.RunAs.problems(where+": container "+c.Name, c.Env)...)
return append(problems, checkImage(where, c.Image)...)
}
@@ -1543,185 +1392,6 @@ type Declaration struct {
// the generation of what it puts back, which may be older than what the machine applied, and is
// not refused for it.
PutBack bool
// Identities is the identity of every module assigned to this machine (novox/hq ADR 0328 §4): an account
// and a group `mesh-<module>` with the numbers the controller allocated once for the module, the same on
// every machine. The engine makes each before any resource is applied, and removes what it made of one no
// longer declared. Absent is a controller older than ADR 0325.
Identities []Identity
// Groups is the group of every library this machine is reached by (novox/hq ADR 0328 §3): `mesh-<library>`
// with the number the controller allocated once, mesh-wide. The engine makes each, and removes what it made of
// one no longer declared. Absent is a controller older than ADR 0328.
Groups []Group
// Libraries is every library this machine is reached by (ADR 0328 §1, §6, §9): its root here, its group, whether
// this is its home, whether the operator's warrant to take it was carried out, the sub-paths its modules ask for,
// and the tool runner's account to put in its group (§11). Absent is a controller older than ADR 0328.
Libraries []Library
}
// Identity is one module's identity, as the controller allocated it (ADR 0328 §4): its user `mesh-<module>` with
// uid, its own group of the same name and number, its primary group gid (its own, or its first library's group),
// and the groups of its other libraries as added groups.
type Identity struct {
Module string `json:"module"`
Name string `json:"name"`
UID int `json:"uid"`
GID int `json:"gid"`
Groups []int `json:"groups,omitempty"`
// Runner is the tool runner's account, put in the module's own group (novox/hq ADR 0334 §2) when its tools run
// on this machine. Absent for a module that declares no tools here, and from a controller older than ADR 0334.
Runner string `json:"runner,omitempty"`
}
// Group is one library's group, as the controller allocated it (ADR 0328 §3).
type Group struct {
Name string `json:"name"`
GID int `json:"gid"`
}
// GroupPrefix starts every library group's name, as IdentityPrefix starts every module's.
const GroupPrefix = "mesh-"
// LibraryAccessPrefix starts the access a library's wait and its take act name: `library:<name>`.
const LibraryAccessPrefix = "library:"
// Library is one library as this machine sees it (ADR 0328).
type Library struct {
Library string `json:"library"`
// Root is the library's root on this machine: on its home the data itself, elsewhere the mount point the mounts
// module mounts it at.
Root string `json:"root"`
Group string `json:"group"`
GID int `json:"gid"`
// Home is that the library's data lives on this machine: only there does the engine keep it.
Home bool `json:"home"`
// Taken is the controller saying the operator's warrant to take it was carried out (§7).
Taken bool `json:"taken,omitempty"`
// Paths are the sub-paths its modules ask for, relative to the root.
Paths []string `json:"paths,omitempty"`
// Runner is the tool runner's account, put in the library's group (§11). Absent when no module's tools use it.
Runner string `json:"runner,omitempty"`
}
// checkLibraries judges the groups and libraries on arrival: every library's group declared, named for it, never
// root's number; paths that stay below the root; a runner that is an account's name; and no two roots that nest
// (§13).
func checkLibraries(groups []Group, libs []Library) []string {
var problems []string
byName := map[string]int{}
numbers := map[int]string{}
for _, g := range groups {
where := fmt.Sprintf("group %q", g.Name)
if !strings.HasPrefix(g.Name, GroupPrefix) || !accountName.MatchString(g.Name) {
problems = append(problems, fmt.Sprintf("%s: a library's group is named %s<library>", where, GroupPrefix))
}
if g.GID <= 0 {
problems = append(problems, fmt.Sprintf("%s: gid %d, and a library's group is never root, nor a number "+
"below it (novox/hq ADR 0328 §15); nothing of this declaration was made", where, g.GID))
}
if _, twice := byName[g.Name]; twice {
problems = append(problems, where+": declared twice")
}
if other, used := numbers[g.GID]; used {
problems = append(problems, fmt.Sprintf("%s: gid %d is %s's too", where, g.GID, other))
}
byName[g.Name], numbers[g.GID] = g.GID, g.Name
}
seen := map[string]bool{}
for _, l := range libs {
where := fmt.Sprintf("library %q", l.Library)
switch {
case l.Library == "" || strings.ContainsAny(l.Library, "/ \t\n"):
problems = append(problems, where+": a library is named by one word")
case seen[l.Library]:
problems = append(problems, where+": declared twice")
}
seen[l.Library] = true
if !filepath.IsAbs(l.Root) || filepath.Clean(l.Root) != l.Root || l.Root == "/" {
problems = append(problems, fmt.Sprintf("%s: its root %q is not an absolute, clean path below /", where,
l.Root))
}
if gid, ok := byName[l.Group]; !ok || gid != l.GID {
problems = append(problems, fmt.Sprintf("%s: its group %s (gid %d) is not one this declaration makes",
where, l.Group, l.GID))
}
if l.Group != GroupPrefix+l.Library {
problems = append(problems, fmt.Sprintf("%s: its group is %s%s, and %s is not", where, GroupPrefix,
l.Library, l.Group))
}
for _, p := range l.Paths {
if p == "" || filepath.IsAbs(p) || filepath.Clean(p) != p || p == ".." || strings.HasPrefix(p, "../") {
problems = append(problems, fmt.Sprintf("%s: the path %q is not a clean path below its root", where, p))
}
}
if l.Runner != "" && !accountName.MatchString(l.Runner) {
problems = append(problems, fmt.Sprintf("%s: its runner %q is not an account's name", where, l.Runner))
}
}
for i, a := range libs {
for _, b := range libs[i+1:] {
if a.Root != "" && b.Root != "" && (a.Root == b.Root || strings.HasPrefix(a.Root, b.Root+"/") ||
strings.HasPrefix(b.Root, a.Root+"/")) {
problems = append(problems, fmt.Sprintf("library %q: its root and library %q's nest, and libraries "+
"never overlap (novox/hq ADR 0328 §13)", a.Library, b.Library))
}
}
}
return problems
}
// IdentityPrefix starts every module identity's name.
const IdentityPrefix = "mesh-"
// checkIdentities judges the identities on arrival: each named for its module, never root, and no name or number
// twice — two modules with one account would write as each other.
func checkIdentities(ids []Identity, groups []Group) []string {
var problems []string
names, uids := map[string]bool{}, map[int]string{}
libraryGroups := map[int]bool{}
for _, g := range groups {
libraryGroups[g.GID] = true
}
for _, id := range ids {
where := fmt.Sprintf("identity %q", id.Name)
if id.Module == "" || id.Name != IdentityPrefix+id.Module || !accountName.MatchString(id.Name) {
problems = append(problems, fmt.Sprintf("%s: a module's identity is named %s<module>, and this one is "+
"for the module %q", where, IdentityPrefix, id.Module))
}
if id.UID <= 0 || id.GID <= 0 {
problems = append(problems, fmt.Sprintf("%s: uid %d, gid %d, and a module's identity is never root, "+
"nor a number below it (novox/hq ADR 0328 §15); nothing of this declaration was made", where,
id.UID, id.GID))
}
if names[id.Name] {
problems = append(problems, where+": declared twice")
}
if other, used := uids[id.UID]; used {
problems = append(problems, fmt.Sprintf("%s: uid %d is %s's too", where, id.UID, other))
}
// Its primary group is its own (gid == uid) or a library's this declaration makes (ADR 0328 §4); so is
// every added group. A module's own number is never a library group's.
if id.GID != id.UID && !libraryGroups[id.GID] {
problems = append(problems, fmt.Sprintf("%s: its primary group gid %d is neither its own (%d) nor a "+
"library's this declaration makes", where, id.GID, id.UID))
}
if libraryGroups[id.UID] {
problems = append(problems, fmt.Sprintf("%s: uid %d is a library group's number too", where, id.UID))
}
for _, g := range id.Groups {
if !libraryGroups[g] {
problems = append(problems, fmt.Sprintf("%s: its added group gid %d is not a library's this "+
"declaration makes", where, g))
}
}
if id.Runner != "" && !accountName.MatchString(id.Runner) {
problems = append(problems, fmt.Sprintf("%s: its runner %q is not an account's name", where, id.Runner))
}
names[id.Name], uids[id.UID] = true, id.Name
}
return problems
}
// LeftOutModuleOf says which left-out module a recorded resource belongs to, if any: its id is the
@@ -1891,13 +1561,6 @@ type envelope struct {
// sends them only to a host whose reports say `reads_generation`.
Generation int64 `json:"generation,omitempty"`
PutBack bool `json:"put_back,omitempty"`
// Identities is optional on the wire too (novox/hq ADR 0325 §3, ADR 0328 §4). **An older host refuses this key**; a
// controller sends it only to a host whose health says link.IdentityContract.
Identities []Identity `json:"identities,omitempty"`
// Groups and Libraries are optional too (novox/hq ADR 0328). **An older host refuses these keys**; a controller
// sends them only to a host whose health says link.LibraryContract.
Groups []Group `json:"groups,omitempty"`
Libraries []Library `json:"libraries,omitempty"`
}
func parse(raw []byte, allowActions bool) (*Declaration, error) {
@@ -1915,10 +1578,8 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
}
d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption, Sequence: env.Sequence,
Epoch: env.Epoch, LeftOut: env.LeftOut, Generation: env.Generation, PutBack: env.PutBack,
Identities: env.Identities, Groups: env.Groups, Libraries: env.Libraries}
problems := checkIdentities(env.Identities, env.Groups)
problems = append(problems, checkLibraries(env.Groups, env.Libraries)...)
Epoch: env.Epoch, LeftOut: env.LeftOut, Generation: env.Generation, PutBack: env.PutBack}
var problems []string
if env.Generation < 0 {
// Below zero would read as "none claimed" and pass every refusal of an older generation.
problems = append(problems, fmt.Sprintf("an assignment generation below zero (%d) is not one the "+
@@ -2071,10 +1732,6 @@ func checkWhileStopped(resources []Resource) []string {
return problems
}
// accountName is what an identity's or an access user's name may be: the portable account name the user tools
// take, optionally ending in `$`, never starting with a dash — it is handed to them as an argument.
var accountName = regexp.MustCompile(`^[A-Za-z0-9_.][A-Za-z0-9_.-]*\$?$`)
func strictDecode(raw []byte, into any) error {
// DisallowUnknownFields is the whole point rather than strictness for its own sake: a
// field the host does not know is a thing the control plane believes it asked for.
+2 -2
View File
@@ -78,7 +78,7 @@ github.com/nats-io/nkeys
# github.com/nats-io/nuid v1.0.1
## explicit
github.com/nats-io/nuid
# github.com/novox/mesh-host v0.0.0 => git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b
# github.com/novox/mesh-host v0.0.0 => git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f
## explicit; go 1.26.0
github.com/novox/mesh-host/internal/declaration
github.com/novox/mesh-host/rootsearch
@@ -138,4 +138,4 @@ golang.org/x/text/width
# golang.org/x/time v0.15.0
## explicit; go 1.25.0
golang.org/x/time/rate
# github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b
# github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f